Constraint Driven Testing to Improve Agile Development

Acceptance Test Driven Development (ATDD) is a collaborative development methodology. The ATDD cycle contains three elements: discuss, develop, and deliver.

In the first phase, the whole team discusses the user story. If anything is inconsistent, superfluous, or dubious, then good questions are needed to detect the problem and improve the story. The next step of the method is to create acceptance-level tests based on the improved story. The test should be understandable for everybody.

Specification by example (SBE) is a similar method to ATDD, but there are some key differences. With SBE, the first step is to derive the right scope from specific goals, i.e. to start from the problems and claims, instead of initial solutions.  Although this can be very important, the possibility of taking this step depends on the customers.

From here the next steps are similar: discussion within the team assures common understanding and improves the story. Here, examples are used for clarifying and improving the specification, and serve as tests to be automated. Executable specification, i.e. a set of examples, is also considered to be “living documentation”, since whenever the automated and readable tests pass, the specification and the code match.

These two methods are very good for creating a good specification which contains the problem to be solved and is therefore more stable. These methods also help shared understanding among every project member and stakeholder. However, they are not perfect for designing the tests. Although it is easy to give examples or tests for a given story, it is very difficult to give tests that cover the story, so that no more test cases are required and no superfluous tests are involved. For example, the number of tests given by certified testers for a very simple story was 9-10, when the optimal number is 5. Since the tests/examples are not optimal, the living documentation or the test cases of ATDD/SBE will not be perfect either.   

Constraint Driven Testing

The solution is to extend the methods by a collaborative test design which results in much better tests. Our design method is to create a test model from which a test-specification can be generated. This is an understandable alternative of the specification. The core part of the collaborative test design is Constraint Driven Testing (CDT) and we refer to the whole process including the collective test design as CDT as well.

The CDT Agile Process

The product owner’s goal is search functionality in an open document. When the pattern has been found, the cursor has to be set at the first character of the text. It is then possible to search forwards or backwards to find the subsequent or previous pattern in the text. After the last match, the next search result is the first match in the document.

The team contained the product owner, business analyst, the developer, and the tester. After understanding the task, the team members considered the following questions:

• Is it a match if a pattern begins in a line and ends in another line in the text?
• Is it a match if the patterns are overlapped, e.g. “abab” in “ababab”?
• Is it necessary to find the first matching pattern independently of where the cursor is?

The team agreed, that for all the above questions the answer is yes. According to this completion the specification (user story) is as follows.

Specification: Search

The pattern to be searched has to be written in the navigation window. Up and down arrows help the search. By clicking down, the nearest matching text following the text cursor will be found. By clicking up, the nearest matching text preceding the text cursor will be found. By clicking on “enter”, the first match in text will always be found. When a searched pattern starts in one line and ends in another line in the same paragraph of the text, the matching holds. Overlapped matches have to be considered to be further results. After the last match and by selecting down, the first match will be selected. After the first match and selecting up, the last match will be selected. 

In the original methods, tests/examples are selected by the team. However, it can happen that we select superfluous tests without knowing when we should stop the process. In any case, the tests are selected in an ad hoc way. Our CDT method, is, however, a systematic solution.

The three key elements (i.e. our model) of our method are: to find the categories; the choices for each category; and the constraints among the choices. Categories are initial states, inputs and outputs such as login name or password. Choices are values or generalized values such as many, or integers between 1 and 20.

Constraints connecting choices such as password and login name have to be different. Constraint description contains only categories, choices and keywords to form text understandable for machines and humans as well. Here is an example:

if searching sequence of operators is down and place of cursor is behind the last matching pattern, then serial number of the pattern found is first

Where keywords are bold, categories are underlined, and choices are italics.

Our aim is to specify as many constraints as possible, since these constraints are the staring points of the test cases. Then a test specification is generated, which can be considered as an alternative specification. During this generation all the constraints and choices are covered by at least one test case. In this way our test data adequacy criterion is always satisfied. Finally, some missing output is manually added to the test specification and some choice gets real value, for example “many” is substituted by 1000. With this we get the tests from the test specification.  

In practice, this is an iterative method:

• Create the categories with obvious choices based on the specification.
• Make the constraints based on the specification, the categories and the choices.
• Insert missing categories and choices based on the constraints.
• Make test specification starting from the constraints, with new choices and constraints to be added if necessary.
• Make additional tests to cover all the choices.
• Validate the tests based on the constraints.
• Insert missing tests, if any.

The advantage of this iterative solution is that we can gradually improve our test entities as they interact with each other. This is not possible for traditional methods, where test cases are independent and there is no direct dependence to other test cases. 

Solve the Example

Categories with choices:

• start searching: (down, up, enter)
• relation of the text and the pattern: (simple, overlapped, overlapped and multi-lined)
• number of matching patterns: (0 (S), 1, 2, more than four, many (S))
• serial number of the pattern found: (no matching, first, last, [first, last, second], [last, first, last], [first, first, first], [second, third, fourth, third, second, first, last], )
• place of cursor (in the text related to the matching patterns): (before the first matching pattern, behind the last matching pattern, between the first and the second matches)
• size of the pattern: (empty (S), 1 (S), usual, same as the text (S), larger than the text (S))
• searching sequence of operators: (down, up, enter, [down, down, down, up, up, up, up], [down, down, enter, up, up, up, enter] , [down, up,  enter], [down, down, down]  [down, …., down])

We can delete the first category, since it is involved in the last one. We therefore have 6 categories, which is a manageable size. “Usual” choice means a pattern greater than 1 and less than the size of the text.  (S) means single, i.e this choice has to be selected only once.

Constraints

• If searching sequence of operators is down and place of cursor is behind the last matching pattern, then serial number of the pattern found is first
• If searching sequence of operators is up and place of cursor is before the first matching pattern, then serial number of the pattern found is last
• If place of cursor is between the first and the second matches and searching sequence of operators is enter, then serial number of the pattern found is first
• If size of the pattern is empty and searching sequence of operators is [down, up,  enter], then serial number of the pattern found is no matching
• If number of matching patterns is two, and relation of the text and the pattern is overlappedand multi-lined, and the place of cursor is between the first and the second matches, and searching sequence of operatorsis [down, down, down] then serial number of the pattern found is [last, first, last]
• If relation of the text and the pattern is overlapped, then size of the pattern is usual
• If number of matching patterns is 1, and searching sequence of operators is [down, up, enter], then serial number of the pattern found is [first, first, first]
• If place of cursor is between and size of the pattern is1and number of matching patterns is more than four and searching sequence of operators is [down, down, down, up, up, up, up], then serial number of the pattern found is [second, third, fourth, third, second, first, last]
• If size of the pattern is same as the text,and searching sequence of operators is enter then serial number of the pattern found is first
• If size of the pattern is larger than the text, and searching sequence of operators is up then serial number of the pattern found is no matching

In some cases constraints and tests are very similar. Based on the categories, choices, constraints and the test data adequacy criterion to cover each choices at least once we obtain the test specification as follows.

The Test Specification

We can see that for any constraint there is at least one related test which fulfils the constraint. In addition, every choice has been covered by at least one test case. Therefore, the test adequacy criterion is satisfied. To take this test specification with average work intensity took two hours and fifteen minutes.

The agile team, by applying traditional methodology, made 55 test cases, which could have been reduced to 20-25.

We Detected Some Issues, Probably Bugs in Word 2013

It should be noted that although Word 2013 has an unknown and slightly different specification, we can use it as an implemented version to validate our tests. All tests with overlapping patterns failed because it is a missing function in MS Word.

Similarly, all the tests validating the operation to find the first matching failed. In many cases I would like to be able to search from the beginning of the text, but there is no simple possibility in the tool. I also think that overlapped match is a type of match and Word should handle this.

I’m a Word user and for me these are unambiguously bugs.