{{announcement.body}}
{{announcement.title}}

Case Study: Complex UI Testing

DZone 's Guide to

Case Study: Complex UI Testing

This new tool is a large and complex UI with dozens of actions proposed to the user and a drastic performance requirement.

· Performance Zone ·
Free Resource

Large and Complex UI 90%+ Covered by Tests

In this previous post, I used the example of the new NDepend v2020.1 graph. This new tool is a large and complex UI with dozens of actions proposed to the user and a drastic performance requirement (it scales live on 100.000+ elements). This graph implementation is 90% covered by tests. It is not because there is a lot of UI code that it should not be well tested. 

We didn't spend a good part of our resources in writing tests just for the sake of it. We did it because we know by experience that it'll pay off: probably a few bugs will be reported as for all 1.0 implementation although beta test phases already caught some. But we are confident that it won't take a lot of resources to fix them. We can look forward to the future confidently (like supporting properly .NET 5 that will be released in November 2020). 

And indeed 10 days after its 1.0 release, no bug has been reported (nor logged) on this new graph although many users downloaded it: so far it looks rock-solid and we can focus on what's next.

The picture below shows all namespace, classes, and methods of graph implementation. Smaller rectangles are methods and the color of each rectangle indicates how well a method is covered by tests. We tolerate some gaps in UI code, while non-UI code like Undo/Redo actions implementations are 100% covered. Experience told us how to balance our resources and that everything does not have to be perfect to achieve high maintainability.

thermal image

How Did We Achieve a High Coverage Ratio on UI Code?

It is easy: we have a simple MVC (Model View Controller) design. Some controller classes contain the logic for all actions the user can do and those classes pilot the UI. Concretely in our scenario actions are load/save, change group-by, change layout direction, zoom, generate a call graph for this method, change filters...

Then we wrote a test suite that first starts the UI and then invokes all actions. Each complex peculiarity of each action gets fully tested, hence complex actions get invoked several times by tests but differently each time, to make sure all scenarios get tested.

The video below shows the UI under testing: more than 40 actions get tested in less than a minute. It would take more than an hour to do all this work manually and any change in code could potentially ruin the validity of manual tests.

In such a complex UI many classes are not directly related to UI. For example, the grape of classes that describe the underlying model is tested separately.

As usual, a side benefit of writing tests is better to design: the code gets structured in a way that makes it easy to invoke it through tests. Concretely some abstractions are introduced (that wouldn't make sense without tests), some classes and some methods get split, some logic gets refined, and as a result, developers are happy to live in a codebase where the logic is smoothly implemented.

High Coverage Ratio Is Not Enough: Assertions to the Rescue

Typically at this point comes the remark: but code coverage is not enough, results must be asserted. And indeed, if nothing gets asserted nothing gets tested even if the code is entirely covered by tests. We want tests to fail if something can go wrong.

Of course, our tests contain many assertions for example load / save actions are invoked and asserted this way:

Java
 




x
12


 
1
  // Save and undo until DefaultGraphAction
2
         InvokeOnUIThread(() => Assert.IsTrue(m_GraphController.TrySaveGraphInFile(graphFilePath, "My Desc", out failureReason)));
3
         InvokeOnUIThread(() => m_GraphController.ExecuteUndoRedoGraphAction(new OnUndoTillItemAction<IGraphAction>((uint)(2 - 1)))); // -1 coz 0-based
4
         WaitUntil(stack => stack.GetUndoItems()[0] is DefaultGraphAction);
5
         Assert.IsTrue(m_GraphController.GraphContext.NodesSelected.Count == 0);
6
 
7
         // Load and make sure we have 2 nodes selected
8
         InvokeOnUIThread(() => Assert.IsTrue(m_GraphController.TryLoadGraphFromFile(graphFilePath, out string  graphDescFromFile, out failureReason)));
9
         WaitUntil(() => m_GraphController.GraphContext.NodesSelected.Count == 2);
10
         Assert.IsTrue(m_GraphController.GraphContext.UndoRedoStack.UndoItemsCount == 3);
11
         Assert.IsTrue(m_GraphController.GraphContext.MessageForUser == @"Graph loaded from file:  File.ndgraph
12
My Desc");



But these assertions are not enough. Per definition, the UI code contains tons of visual peculiarities represented by states that can be potentially corrupted. As a consequence our UI code is stuffed with thousands of assertions: everything that can be asserted gets asserted.

  • A Rectangle with width/height in a certain range
  • The state of a node or an edge when another element gets selected (is it a caller, a callee...?).
  • The current application state when a new graph is demanded by the controller.
  • The graph UI contains much asynchronous computation to avoid UI freezing. This leads to many assertions to check that mutable states are not corrupted by concurrent accesses.

All those states asserted would be hardly reachable from test code. However, they get naturally accessed by the UI code itself so it is the right place to assert that they are not corrupted.

Btw, We still use the good old System.Diagnostics.Debug.Assert(...) for that, it has several advantages:

  • It is simple.
  • It is understood by tools like Roslyn/Resharper/CodeRush analyzers.
  • An assertion that fails cannot be missed both when running automatic tests and when running manual tests on the Debug mode version.
  • Debug assertions are removed by the compiler in Release mode: assertions are not executed in production and users get better performance. The idea is to not consider users as testers: code released in production is supposed to be rock-solid. Assertions are like scaffolding that gets removed when a building gets delivered. If there is still a bug we'll discover it from users' feedback, from production logs or our manual tests.
developers vs others

Debug.Assert(...) is enough for us and, understandably, some other teams want a more sophisticated assertion framework. The key is to take the habit to assert everything that can be asserted when writing code (UI code or not). Each assertion is a guard that helps to make the code rock-solid. Also, each assertion improves the code readability. At code-review time we've all been wondering: can this integer be zero? can this string be empty? can this reference be null?. Hopefully C#8 non-nullable discards the last question but so many questions remain open without assertions.

Design by Contracts

This idea of stuffing code with assertions is an important software correctness methodology named DbC, Design by Contract, that is worth knowing. Contracts mean much more than the usual approach with exception:

Java
 




x
9


 
1
C
2
void Method(int argument) { 
3
   if(argument == 0) { throw new ArgumentOutOfRangeException(); 
4
   // ...
5
}
6
void Method(int argument) { 
7
   if(argument == 0) { throw new ArgumentOutOfRangeException(); 
8
   // ...
9
}



  • Explicitly throwing an exception says: zero is tolerated, it is not a bug, but you won't get the result you'd like, be prepared to catch some exceptions.
  • Writing a contract says: don't even think of passing a zero value. The real type of argument is not Int32 it is [Int32 minus 0]. Ideally, such violation could be caught by compilers and analyzers (and is indeed sometimes caught as we saw in the screenshot above).

Conclusion

Any complex UI can be automatically tested as long as:

  • It is well designed with some controllers that pilot the UI and that can be invoked from tests.
  • UI code gets stuffed with assertions to make sure that no state becomes corrupted at runtime.

In short assertions embedded in code tested matter as much as assertions embedded in tests. If an assertion gets violated there is a problem, no matter the assertion location, and it must not be missed nor ignored. This powerful idea doesn't necessarily apply only to UI code and is known as DbC, Design by Contract.

Actually, in this post, I added a third principle to achieve high code maintainability and high code correctness: layered code + high coverage ratio by test + contracts.

Topics:
coverage ratio ,perforamnce ,ui ,ui testing

Published at DZone with permission of Erik Dietrich , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}