This post summarizes what I’ve learned from various sources about making acceptance or black-box tests maintainable. This topic is of great interest to me because I believe in the benefits that acceptance tests can bring (such as living documentation) but I’m also very much aware that it is all too easy to create an unmaintainable monster whose weight eventually crushes you. So the question is how to navigate the minefield to get to the golden apple?
The key elements that contribute to the maintainability of acceptance tests are:
- Aligned business, software, and test models => small change in business requires only a similarly small change in the software and a small change in tests (Gojko Adzic explains that very well in his JavaZone 2012 talk Long-term value of acceptance tests)
- The key to gaining the alignment is to use business language in all the three models from the very start, building them around business concepts and relationships
- Testing under the surface level, if possible
- Prefer to test your application via the service layer or at worst the servlet layer; only test on the UI level if you really have to and only as little as possible for UI is much more brittle (and also difficult to test)
- The more you want to test the more you have to pay for it in the terms of maintenance effort. Usually you decide so that you cover the part(s) of the application where the most risk is – the best thing is to do cost-benefit evaluation.
- Isolating tests from implementation by layers of test abstraction
- Top layer: Acceptance tests should only describe “what” is tested and never “how” to test it. You must avoid writing scripts instead of specifications.
- Layer 2: Instrumentation – right below the acceptance test is an instrumentation layer, which extracts input/output data from the test and defines how to perform the test via a high-level API, provided by the next level (we could say a test DSL) such as “logInUser(X); openAccountPage();”
- Layer 3: High-level test DSL: This layer contains all the implementation details and exposes to the higher layer high-level primitives that it can use to compose the tests without depending on implementation details (ex.: logInUser may use HtmlUnit to load a page, fill a form, post it)
(And of course many, if not all, of the rules for creating maintainable unit tests apply as well.)
What others have to say
On the three layers of test isolation:
- Business rule or functionality level: what is this test demonstrating or exercising. Ideally illustrated with realistic key examples. For example: Free delivery is offered to customers who order two or more books, illustrated with an example of a customer who orders one book and doesn’t get free delivery and an example of a customer who orders two books and gets free delivery.
- User interface workflow level: what does a user have to do to exercise the functionality through the UI, on a higher activity level. For example, put the specified number of books in a shopping cart, enter address details, verify that delivery options include or not include free delivery as expected.
- Technical activity level: what are the technical steps required to exercise the functionality. For example, open the shop homepage, log in with testuser and testpassword go to the /book page, click on the first image with the book CSS class, wait for page to load, click on the ‘Buy now’ link and so on.
(The Secret Ninja Cucumber Scrolls, page 109 – 110)
On aligned business, software and test models:
The principle of symmetric change
The best software is the one where the technical software model is aligned with the relevant business domain model. This ensures that one small change in the business domain (new requirements or changes to existing features) results in one small change in software. The same is true for the other artefacts produced for software — tests and documentation. When the tests are aligned with the business domain model, one small change in business will result in one small change in tests, making the tests easy to maintain.
Tests described at a low level technical detail, in the language of technical UI interactions, are everything but aligned with a business model. If anything, they are aligned with the current design and layout of user interfaces. A small change in business requirements can have a huge impact on such tests, requiring hours of updates to tests after just a few minutes of changes to the code.
(The Secret Ninja Cucumber Scrolls, page 111)
This post is very much based on the following resources:
- The Secret Ninja Cucumber Scrolls (free pdf) by David de Florinier and Gojko Adzic, 2011-03-16
- http://concordion.org/ by David Peterson
- Specification by example, Gojko Adzic, 2011, Manning, ISBN 978-1617290084
- Long-term value of acceptance tests, Gojko Adzic, a talk at JavaZone 2012