Staged Maven build in Hudson with shared workspace

Staged Builds

Continous Integration is all about providing feedback, the faster the better. It is all being said by Martin Fowler, I'll just quote a crucial sentence from the Staged Builds section:

Probably the most crucial step is to start working on setting up a staged build.

So true, I had the opportunity to think about it each time when I had to wait for 40 long minutes before I see the build turning red or green. These exhaustive cryptographic tests taking 8 minutes or these cool integration tests taking 6 minutes - let them run each time but not as a part of the commit build. Enough said, check the article

Hudson provides all needed parts to set up a staged maven build, but I wasn't able to find a clear guidance on how to glue a staged build together. I found some useful hints and even better comments here. There is definitely more than one way of setting up a staged build, here is how I put the puzzle pieces together:

The scenario
Let's start with an imaginary scenario to illustrate the point. Let there be 3 modules: A, B and C. Module B's tests take too long an are unlikely to fail. Module C is an assembly using A and B and on which some integration tests are run.

Let's see how to make a 3-stage build:
* stage1 compiles and tests A, only compiles B
* stage2 tests (the already compiled) B
* stage3 packages and tests C (using A and B snapshots).

The key is to let the 3 stages run in the same workspace, so that each stage picks up where the previous one stopped. Behavior can be coupled to a stage with a profile, so for this example I define 3 maven profiles - stage1, stage2 and stage3.

It is important that the profiles add, not suppress tasks. Then the profiles can be used in a additive way, i.e. -Pstage1,stage2 will execute all tasks linked to both stage1 and stage 2. to take tests as example, do no -DskipTests=true in a stage, but define empty test files set in one stage and extend it in the next stage.

The parent pom:

<modules>
  <module>A</module>
  <module>B</module> 
</modules>
<profiles>
  <profile>
   <id>stage3</id>
   <modules>
    <module>C</module>
   </modules>
  </profile>
</profiles>

Nothing special in the A's pom. The B's pom:

<build>
<plugins>
  <plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-surefire-plugin</artifactId>
   <configuration>
    <includes>
     <include>No default tests included</include>
     </includes>
    </configuration>
   </plugin>
  </plugins>
</build>

<profiles>
  <profile>
   <id>stage2</id>
   <build>
   <plugins>
    <plugin>
     <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-surefire-plugin</artifactId>
     <configuration>
      <includes>
       <include>**/Test*.java</include>
       <include>**/*Test.java</include>
       <include>**/*TestCase.java</include>
      </includes>
     </configuration>
    </plugin>
   </plugins>
  </build>
</profile>
</profiles>

 
Needed Hudson Plug-ins

The URL SCM plug-in lets you access archives from another build as SCM URLs. The trick is to let one stage provide the workspace for the next stage. Potentially this means passing huge archives between build nodes, more on that later. The M2 Extra Steps Plugin is needed to unzip the archive from the previous stage before starting the maven build.

Stage 1 Job

Define a Maven job that checks out the workspace from SCM. In the job settings, check 'Archive artifacts' and select the whole workspace (pattern **/*.*). The goal: clean, install.

Stage 2 Job

Copy Stage1 Job to a new job Stage2. In the Source Code Management section, select URL Copy. The URL to watch is something like http://hudson-server/job/stage1/lastSuccessfulBuild/artifact/*zip*/archive.zip. Go to Configure M2 Extra Build Steps and add an Execute Shell step: unzip archive.zip The goal: -Pstage2, install. Note that there is no clean. Finally, archive the workspace for the next stage.

Stage 3 Job

The Stage3 is much like stage2. The URL to watch will be http://hudson-server/job/stage2/lastSuccessfulBuild/artifact/*zip*/archive.zip. The goal will contain -Pstage2 but no clean.

Conclusion

This setup is pretty straightforward an pretty new for me, but I already notice a few points I am not really happy with:
- Packing, transferring and unpacking the workspace could cost a lot of time.
- The separate projects are too much aware about the build they will be part of. I'd prefer to be able to tweak the build without editing each particular project. Possibly inventing some profile names more descriptive than stage1, stage2 is the way to go.

Generating IBM RSA Websphere projects with Maven

Rational Software Architect (RSA) may not be the best development environment, but when it comes to WAS-integrated IDEs, there are not that many choices. RSA is the familiar Eclipse packaged together with some plugins, which provide integration with Websphere. Starting with a Websphere Web application with RSA is easy - just choose the right project type and you can immediately run the application in Websphere from the IDE.

If you are working with Maven, however, there are very good reasons why you will not want to create any projects drom RSA. All of them boil down to the DRY principle - Don't Repeat Yourself. The Maven POM files already describe your projects and dependencies, why would you want to do this second time in RSA? You are going to make your release builds (and continuous integration) with Maven anyway, and any RSA project changes you make have no impact on the build. You may end up bulding something different than what you develop. Maintaining dependencies at two places - in maven poms and in your RSA project is not good, and Maven is much better in the dependency stuff.

Besides, why lock to RSA? If you can generate the IDE files for RSA/Websphere, so you can for Eclipse/Tomcat, for example.

The Maven Eclipse plugin generates the needed IDE files. The point is you want to be able to use them directly in RSA. It took me some time to get the POMs right, so I thought it is worth sharing.

Typically in a Websphere app, you have an EAR project, a Web project and several JAR projects. In this example, I start with 3 projects: ERA, WAR, JAR. Define them in the parent POM:

<modules>
 <module>app-jar</module>
 <module>app-web</module>
 <module>app-ear</module>
<modules>

Further in the parent POM, declare you are using java 5.0:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-compiler-plugin</artifactId>
 <configuration>
  <source>1.5</source>
  <target>1.5</target>
 </configuration>
</plugin>

N.B. Define 1.5 and not 5.0, this was fatal for my RSA installation!

There's nothing special about the JAR project, the generated eclipse project is fine. For the Web project, few more settings are essential. Assuming you have a maven webapp archetype project, include the following under build/plugins in the project's pom:

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-eclipse-plugin</artifactId>
 <configuration>
  <wtpversion>1.5</wtpversion>
  <wtpContextName>myapp</wtpContextName>
  <additionalProjectFacets>
     <jst.java>5.0</jst.java>
     <com.ibm.websphere.extended.web>6.1</com.ibm.websphere.extended.web>
     <com.ibm.websphere.coexistence.web>6.1</com.ibm.websphere.coexistence.web>
  </additionalProjectFacets>
 </configuration>
</plugin>

The generated project will be a Dynamic Java project, and RSA wil recognise src/main/webapp as the web root folder. The web context name is also define in the plugin settings.

Similar setting are needed for the EAR POM:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-eclipse-plugin</artifactId>
    <configuration>
     <wtpversion>1.5</wtpversion>
     <additionalProjectFacets>
      <jst.ear>1.4</jst.ear>
      <com.ibm.websphere.extended.web>
       6.1
      </com.ibm.websphere.extended.web>
      <com.ibm.websphere.coexistence.web>
       6.1
      </com.ibm.websphere.coexistence.web>
     </additionalProjectFacets>
    </configuration>
   </plugin>

The generated project is recognized by RSA as an EAR project. Of course, you need a correct application.xml in src/main/application/META-INF. This project doesn't contain any code, but is still usefull for maven, as it will build the release EAR.

Once that all is done, go the the project root and run mvn eclipse:eclipse with any extra options you may need. Import the generated projects in RSA (File/Import/Existing projects int workspace). The resulting file structure is the well known maven webapp structure:



To run the application from RSA, drag the EAR project to 'Web Sphere Application Sever' in the Servers view and start te server.

There is one issue that has not been addressed here - how do you get all those WAS-specific IBM files. Update follows.

Profiles

It's a good idea to set all those settings in maven under a profile. For the next environment, add a new profile.

Hiccups

Yes, they are very present:
- Go to the app-web project, Properties, J2EE dependencies, Web libraries - all library dependencies are there but not the project dependency on app-dom, while it obviously should be. Add it. I couldn't find a solution for this, tips about it will be very appreciated.
- Make sure that the web module element in application.xml has no id attribute. The id runs out of sync each time project files are generated.
- Sadly enough, after fiddling with application and server deployment configuration, now and then I just have to restart RSA to make the web application running.