Last year, before the Christmas holidays , I described how we do middleware integration testing at XebiaLabs and I described the way we deploy test servlets by wrapping them in WAR and EAR files that get generated on the fly. There is only one thing left to explain; how do we integrate these tests into a continuous build using Maven and VMware?
Running the middleware integration tests
So let's start with the Maven configuration. As I mentioned in the first blog of this series, the integration tests are recognizable by the fact that the classnames end in Itest. That means they won't get picked up by the default configuration of the Maven Surefire plugin. And that is fortunate because we don't always want to run these tests. Firstly they require a very specific test setup (the application server configurations should be in an expected state, see below) and secondly they can take a long time to complete and that would get in the way of the quick turnaround we want from commit builds in our continuous integration system.
But when we do want to run the middleware integration tests, we can configure the Maven Surefire plugin with a Maven build profile like so:
To run the middleware integration tests in addition to the regular tests, we can tell Maven to use this profile with the -P flag:
$ mvn -Pitest test
If we want this profile to run only the middleware integration tests, we need to leave out the lines that include the standard classnames (Test*.java, *Test.java and *TestCase.java).
Defining the expected state for the target middleware
A big challenge when performing middleware integration tests is to ensure that the middleware we test against is in a correct state. For example, the test might expect an application server cluster to be running but no application to be deployed on it. In a regular test we may set up a test fixture in a method marked with @Before, but in this case that might not be so simple. Confuguring the middleware to be in the expected state would be an integration in itself!
The first approach to address this issue is to write all middleware integration tests in such a way that they revert any changes they make. For example, a test that creates a datasource also removes it. In this way we combine the test for the creation-step with the test for the destruction-step. The disadvantage here is that these two steps can no longer be tested independently, but the advantage is that the combined test runs more quickly. For this approach to be successful, all tests using the same target middleware should agree on a common expected state. We configure the target middleware in that state once and all the tests return the middleware to that state after completion. With a setup like this we can quickly run one or more integration tests, either interactively from an IDE such as Eclipse, or from Maven.
But there is one problem with this. What if the test fails? In that case the middleware might not be left in the correct state. One could argue that the steps needed to revert the middleware should be placed in an @After method but they may still fail. More importantly, those steps are also part of the test so they don't really belong there!
Reverting the target middleware to the expected state with VMware
This is where VMware comes in. By installing the target middleware in a hypervisor such as VMware, we can make a snapshot when the middleware is in the expected state. Then we can revert to that state before executing a single test or a test suite. The simplest thing is to manually revert from the hypervisor administration console before running the tests. But we can also revert the images automatically, so that we can integrate these tests into our continuous integration builds.
The images we use for our middleware integration tests run on VMware vSphere 4 and we've found the vSphere SDK for Perl, and most specifically the provided vmrun.pl sample script, to be the easiest way to interface with VMware vSphere. As an example, this is the script we use to revert our JBoss image:
IMAGE_NAME="[datastore1] jboss-42/CentOS 5.3, JBoss 4.2.3-GA.vmx"
SNAPSHOT_NAME="Ready for itest"
echo "Reverting VMWare image $IMAGE_NAME to snapshot $SNAPSHOT_NAME"
$VMRUN -host $VMWARE_HOST -username $VMWARE_USERNAME -password $VMWARE_PASSWORD \
revertToSnapshot "$IMAGE_NAME" "$SNAPSHOT_NAME"
echo "Waiting $SETTLE_TIME seconds for things to settle"
We do need to provide the proper values for the environment variables VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD when running this script. After the image has been reverted we wait for 60 seconds to allow the image to settle. The amount of seconds needed here is hard to determine exactly but this value works well for us.
To have Maven run this script before the integration tests are executed, but only when a certain profile is selected, we have added this configuration to the POM:
<!-- We really want this to execute right before the test phase. -->
<!-- This is the best we can do. -->
Putting it together
With the two profiles introduced in this blog, we tell Maven to revert the VMware images and execute all middleware integration tests by giving the following command:
$ mvn -Prevert-vmware-images -Pitest clean test
We have configured our continuous integration system to run this command once a day, as a secondary build, and configured it with the right values for the VMWARE_HOST, VMWARE_USERNAME and VMWARE_PASSWORD environment variables. In this way the middleware integration tests do not slow down the standard commit build (they take about 90 minutes to complete in our case!). As an added benefit, we know when the VMware images will be used by the tests so we can use them for manual experiments at other times.
In this blog and the first two blogs of this series I have explained how we use JUnit, Maven and VMware to write repeatable, manageable middleware integration tests. These tests have helped us gain a lot of confidence in the stability of Deployit's middleware integrations. And because this test-support code is part of Deployit's plugin API, Deployit plugin developers can use it to get the same confidence in their own code!