Strategy to Make CI Builds Self Testing
This is post 5 in a series called ‘How to get started with CI’. The last post talked about how to implement CI to your first CI project. This post will extend that implementation by making sure that a build is ‘self testing.
A ‘self-testing’ build is a build that runs all of the required tests as part of it. It is important to stress the term tests here as there are different types of tests, unit, integration, regression, UI etc.). In general tests as described run in the following way:
- unit tests – run in isolation, mock any dependencies and should be fast
- integration / regression tests – require some sort of setup of data / environment and will be slower to run
- UI / acceptance tests are slower to run and will require a replica of the application to test against. These will be a lot slower.
- It is important to understand that if a build that has triggered
because of a VCS check in and doesn’t break the build then we cannot
assume the integration of the code has worked.
We will not know that the integration has been successful until the code has been tested.
- The most efficient way of testing the code is with our test suite.
If the integration fails with obvious errors, e.g. a compilation error,
then that’s a huge clue that something has gone wrong.
- My usual strategy for CI builds and their tests runs is as follows:
1. On VCS check in run all the unit tests
This will give us fast feedback about problems at a code level and will tell us if the code still behaves as expected. For this instant feedback to happen, builds (including test runs) must be fast. IF the runs are not fast then development could potentially be impacted by developers working with code that doesn’t build or hasn’t been tested. Always think of the 10 minute rule for the time a build starts till completion.
2. Every 2 hours run the integration / regression tests
These tests are very important to us but are a step below the priority of unit tests. They will be slower to run since we require data to be setup and torn down for the tests. They can require network and database interaction so we cannot hang around waiting for them to complete.
3. Nightly run the UI / Acceptance tests
These are again still import but the lowest of the priority order for a standard CI build. There will always be a round of testing by QAs or whoever before a release so there is time to fix any errors arising. From my nightly strategy, errors in these tests will only take at maximum a day to appear to a team. These tests are slower again and may require a full compilation and installation of an application to run against so running them overnight means there should be no real problem with time.
The steps listed about are the steps I use for a project is running a normal CI environment. These would change when an environment changes to include continuous delivery of the application. This will be discussed in a further post.
This is only a suggested approach from the experiences I have had with CI. If you run your builds in a different manner then I would love to hear about it – leave a comment below.