The key to testing is to create the best test case for a given application. Test case creation and implementation is both costly and time consuming, meaning that typically the goal is to optimize this process by planning a moderate-sized and efficient test set which reveals most of the bugs.
However, testing itself is not enough: according to a huge amount of collected data published by Capers Jones in 2012, on average only about 85% of bugs are found in the testing stage. Defect prevention in IT is as important as disease prevention in healthcare.
There are various methods available, from formal inspection to static analysis. Applying these methods can result in significant improvement, with significantly fewer bugs going undetected.
Constraint Driven Testing (CDT)
Our method originates from category-partitioning, published by Thomas J. Ostrand and Marc J. Balcer in 1988. We have sought to establish a widely applicable test design method which:
- Optimizes the number of test cases
- Introduces a test data adequacy criterion to determine when to stop creating additional testing
- Adapts the method for agile testing
Although it is possible to use boundary value and equivalence class testing, etc., these methods do not ensure an optimal number of test cases. The solution is CDT, a systematic method which enables us to create very efficient tests.
We start from a user story/feature specification. This feature shouldn’t be too complicated, i.e. no more than six lines of text. If the feature is too complex it should then be split into smaller ones. We then analyze the feature and try to divide it into categories related to preconditions, inputs, and sometimes outputs. For example, for a login feature we can select the login name and password as categories.
The categories are then further divided into choices: for example, a login name can be either existent or non-existent.
There are constraints among the choices: for example, in the case of a non-existent login name the password is irrelevant, and will not be tested by this test case. Another constraint is that you cannot select a password which is the same as your login name.
Now, if we have all these things together, we can start creating the test specification. A test specification contains test cases where sometimes direct values may substituted by choices, i.e. a login name as “existent” instead of “Smith”. Test specification creation must start by satisfying all of your constraints.
When these constraints are processed, we then make or generate a table, where the columns are the categories and the rows contain the tests with the selected choices:
Choice2 (for Category1)
Choice1 (for Category2)
Choice4 (for Category3)
The final step is to replace the choices with actual values.
This method can be automated, i.e. you create the categories, then the constraints based on them and the tests are generated. To ease the generation, the constraints can be written similarly to the gherkin syntax:
WHEN card owner IS yes AND new books’ price IS 49.99 THEN price reduction for new books IS 10%
New Test Data Adequacy Criterion
A significant advantage of this method is that it is possible to define a suitable test data adequacy criterion. This criterion helps us to decide when to stop creating additional tests. The criterion here is very simple: each choice has to be selected at least once. However, if there is a constraint stating that choices are irrelevant at a given test case/specification, then no selected choices are considered to have fulfilled the criterion.
When making the test frames and selecting the categories, choices and constraints, any incompleteness, ambiguity or inconsistency is easily revealed. This is an excellent review method, since the specification is deeply processed and analyzed for test creation, whereas no simple or systematic reading would be able to reveal the deep problems contained in the specification.
Another source of defect prevention is the test method itself. Good and readable test cases drive developers to make better code. TDD is also considered to be a good method with 37% defect prevention efficiency, and since CDT results in optimized tests, we can assume a higher prevention rate when considering only the test first element of prevention.
When to Use CDT?
CDT can be used in every test design process to create test cases for the latter test implementation phase, and even for unit and component testing. Testers can create the test cases from the functional specification or user story and the developers can then implement them. The method can also be used for keyword-driven testing, when categories can be considered as keywords.
However, the biggest advantage is that CDT and our other method KDAT can be united into a complex model based testing method referred to as 4Test. 4Test can be automated, resulting in the generation of executable test cases.
With this method, developers can avoid test design, leaving it to professional testers who are experts in the field. The tests are fully optimized, since no missing or superfluous tests will be created, assuming that both the feature description and the application of CDT are correct. Maintenance of test specification is very easy and totally independent from both the implementation and the selected testing tools.
The method involves a test data adequacy criterion. This is much better than code coverage criteria, since it can be used prior to coding, it is unambiguous and it can always be fully (100%) satisfied.
The method also aids defect prevention at no additional cost, by simply following the necessary steps in order to produce the test specification, meaning that it is an efficient way to unite defect prevention and test design.
CDT is clearly agile, though it can be used for other methodologies as well.
You can find this article extended with an example here.