Staged Maven build in Hudson with shared workspace

Staged Builds

Continous Integration is all about providing feedback, the faster the better. It is all being said by Martin Fowler, I'll just quote a crucial sentence from the Staged Builds section:

Probably the most crucial step is to start working on setting up a staged build.

So true, I had the opportunity to think about it each time when I had to wait for 40 long minutes before I see the build turning red or green. These exhaustive cryptographic tests taking 8 minutes or these cool integration tests taking 6 minutes - let them run each time but not as a part of the commit build. Enough said, check the article

Hudson provides all needed parts to set up a staged maven build, but I wasn't able to find a clear guidance on how to glue a staged build together. I found some useful hints and even better comments here. There is definitely more than one way of setting up a staged build, here is how I put the puzzle pieces together:

The scenario
Let's start with an imaginary scenario to illustrate the point. Let there be 3 modules: A, B and C. Module B's tests take too long an are unlikely to fail. Module C is an assembly using A and B and on which some integration tests are run.

Let's see how to make a 3-stage build:
* stage1 compiles and tests A, only compiles B
* stage2 tests (the already compiled) B
* stage3 packages and tests C (using A and B snapshots).

The key is to let the 3 stages run in the same workspace, so that each stage picks up where the previous one stopped. Behavior can be coupled to a stage with a profile, so for this example I define 3 maven profiles - stage1, stage2 and stage3.

It is important that the profiles add, not suppress tasks. Then the profiles can be used in a additive way, i.e. -Pstage1,stage2 will execute all tasks linked to both stage1 and stage 2. to take tests as example, do no -DskipTests=true in a stage, but define empty test files set in one stage and extend it in the next stage.

The parent pom:

<modules>
  <module>A</module>
  <module>B</module> 
</modules>
<profiles>
  <profile>
   <id>stage3</id>
   <modules>
    <module>C</module>
   </modules>
  </profile>
</profiles>

Nothing special in the A's pom. The B's pom:

<build>
<plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <configuration>
    <includes>
     <include>No default tests included</include>
     </includes>
    </configuration>
   </plugin>
  </plugins>
</build>

<profiles>
  <profile>
   <id>stage2</id>
   <build>
   <plugins>
    <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <configuration>
      <includes>
       <include>**/Test*.java</include>
       <include>**/*Test.java</include>
       <include>**/*TestCase.java</include>
      </includes>
     </configuration>
    </plugin>
   </plugins>
  </build>
</profile>
</profiles>

 
Needed Hudson Plug-ins

The URL SCM plug-in lets you access archives from another build as SCM URLs. The trick is to let one stage provide the workspace for the next stage. Potentially this means passing huge archives between build nodes, more on that later. The M2 Extra Steps Plugin is needed to unzip the archive from the previous stage before starting the maven build.

Stage 1 Job

Define a Maven job that checks out the workspace from SCM. In the job settings, check 'Archive artifacts' and select the whole workspace (pattern **/*.*). The goal: clean, install.

Stage 2 Job

Copy Stage1 Job to a new job Stage2. In the Source Code Management section, select URL Copy. The URL to watch is something like http://hudson-server/job/stage1/lastSuccessfulBuild/artifact/*zip*/archive.zip. Go to Configure M2 Extra Build Steps and add an Execute Shell step: unzip archive.zip The goal: -Pstage2, install. Note that there is no clean. Finally, archive the workspace for the next stage.

Stage 3 Job

The Stage3 is much like stage2. The URL to watch will be http://hudson-server/job/stage2/lastSuccessfulBuild/artifact/*zip*/archive.zip. The goal will contain -Pstage2 but no clean.

Conclusion

This setup is pretty straightforward an pretty new for me, but I already notice a few points I am not really happy with:
- Packing, transferring and unpacking the workspace could cost a lot of time.
- The separate projects are too much aware about the build they will be part of. I'd prefer to be able to tweak the build without editing each particular project. Possibly inventing some profile names more descriptive than stage1, stage2 is the way to go.