a translated version of an article Rob and I wrote for the Dutch .NET Magazine.
Visual Studio Test Edition will be significantly extended in the upcoming version . Next to support for developers and testers this version will also support user / acceptance testers. With these features the upcoming version of Visual Studio will cover the testing spectrum completely.
Everyone knows that
testing is an important part in the development of software and it is
quite unusual for the testing department to be completely separated
from the departments that develop software. This separation of worlds
is not only physical, but also present in the area of tooling.
Cooperation is hard to find even though cooperation of developers and
testers using a structured approach offers many advantages. It
increases the efficiency of the entire application life cycle and
improves the quality of the resulting product, which combined results
in reduction of cost.
In this article you will be able to see how Visual 2010 improves upon the cooperation between the software development team and testers quite a bit.
An example from reality
Cooperation does not happen automatically. Developers and testers have a natural tendency to not (want to) understand one another. From that perspective it is hard to realize that you need each other to deliver software of good quality.
... tend to not understand each other
The consequence of this virtual (and sometimes real) wall is disastrous for quality. Many bugs are caused especially by suppositions and interpretations of, generally, unclear specifications. Meetings about these bugs cost a lot of time, especially if they cannot be reproduced.
After a development cycle of 5 man-years a test is done by a testing team. The testing team vigorously starts running test cases that they prepared during the implementation phase. After 3 months their judgment is: Negative advice on release. Because of (8) fatal and (22) big errors they advise not move the system to production.
Understandably, this was quite a disappointment for the project team as well as the stakeholders. So a task force is formed to contain the delay and thus damage that has occurred. After careful analysis it turned out not to be as bad as it looked: After filtering testing errors (20%), nice-to-haves (20%), and changes (which the testers had not been made aware of; 30%) repair costs were estimated at 200 hours (the remaining 30%).
All in all the implementation team have done a good job. Still, better communication from both sides, before and during the testing phase, would probably have resulted in a positive judgment and thus CELEBRATION. Quite a bit different from a cold shower!
Cooperation in the Application Lifecycle
The typical example in the above paragraph clearly shows that cooperation between test departments and development departments is often far from obvious. If no effort is put into creating a cooperative relationship between developers and testers the proverbial ”wall” will appear all by itself. In general, this wall causes confusion, irritation, and forced limits of tasks and thus loss of quality. The so-called ‘ping pong’ effect with (non-reproducible) errors is the most clear and most often occurring example of this.
Software development and all roles involved are supposed to strive for one common goal: providing added value to the customer. It does not matter what kind of application needs to be implemented. All tasks and roles need to reach for this one single goal. The synergy between roles, processes, and tools is also commonly known as Application Lifecycle Management [ALM]. Notions like accountability, governance, and compliance are regularly used when talking about ALM. But all these notions refer to the simple notion of cooperation.
Application Lifecycle Management.
There would be a better mutual understanding if developers and testers would have more insight in each other’s work and challenges. Annoyances disappear and people work together on a common cause. Visual Studio Team System 2010 will better support this cooperation. By giving the tester aids for performing his/her tasks and making him/her a first-class-citizen within the Team Foundation Server, he/she has and gives more insight in tasks to be done, progress on these tasks and what steps still need to be performed.
Aside from this insight, which often means quite a change of culture, the tester has better insight in and access to the ALM-Artifacts from other roles. For example, use-case and activity diagrams from business and information analysts and other diagrams from designers and architects are easily accessible through Visual Studio Team Architect Edition. Cooperation not only occurs by giving insight in work activities, but also by working on common products.
To get a clear picture of how Visual Studio Team System 2010 will support testers and ensures a better cooperation, first must be looked at goals and tasks which testers try to achieve and perform.
In the age of the monoliths it was possible to delay testing until the acceptance test. Because the monolith had all of its functionality in a compact manner in one big component, it was easy to discover errors quickly and they could be repaired even quicker. That era has passed. If you depend too much on the end test, in current architectures, it is quite possible that you will not achieve the desired on-time, on-budget delivery.
High quality requirements and short time-to-market for information systems demands a 180 degree turn in regards to how one thinks about tests. Instead of delaying the mindset has to be “Finding Bugs as early as possible”.
Each phase of the development
process has its own test goals and depth of test coverage. The depth
particularly depends upon the risks: higher probability of damage
requires better test coverage.
These test goals are especially aimed at discovering errors that can be found in that phase (“as early as possible”).
- during unit testing the internal logic of the unit is tested,
- the unit integration test needs to show that units ”understand” each other.
- the system test (which is called readiness scan in the Acceptance Test Guidance of Patterns of the Practices Group, see helpful links) needs to prove/demonstrate that the system complies with the functional requirements that have been agreed to,
- finally, the acceptance test considers how the system fits into the environment in which it will operate.
Test goals and (depth of) test coverage per phase are made compatible with each other using a Master Test Plan (see TMap® www.tmap.net).
Types of tests in the Application lifecycle.
Every test must be able to rely on previous tests, only then will the complexity of errors be contained to errors that were introduced during the last step. Setting up a learning cycle makes it possible to correct shortcomings, if any, in the testing process chain (see further in this article). Also, the use of tooling plays a crucial part. Running test cases by hand, as done in the nowadays widely used Agile process, is not attainable. So automated support is a must.
However, experience shows that reckless use of tools results in more chaos rather than gains. Right from the start tests need to be set up in a structured manner to contain maintenance efforts within workable limits.
Each phase of the development process and its corresponding test goal introduces a specific role with specific abilities and tasks. These roles are supported by specific components of VSTS 2010. In short, the following kinds of testers exist:
- The development tester.
This tester knows the inner workings of the system and makes use of test-frameworks to automate the tests that he wants to perform. This testing role, which is often if not always performed by developers, has been supported for a while now by technical aids as Unit test frameworks and the current version of Visual Studio. This functionality of VSTS is not only available in the Test Edition, but also fully present in the Developer Edition, which confirms that these tasks are primarily done by developers. VSTS 2010 eases the life of the development tester by intelligent support of Test Driven Development.
- The system tester.
The system tester tests the system for completeness, whether requirements have been met. These are not only functional requirements, but also quality requirements like performance and security.
System testers use a lot of scripts to automate their activities, he/she would like to perform the same tests using various infrastructures and configurations. VSTS has had Load- and Web testing for quite a while in the Test Edition and with the introduction of VSTS 2010 the creation of test scripts is simplified by introduction of the so called ‘CodedUI’ framework.
- The acceptance tester.
The acceptance tester, also called user tester or black box tester, truly is a tester who performs a test of the application and knows nothing about the internal workings of the application. He/she constructs test cases using functional requirements, using various testing techniques.
Acceptance testers, who often still use Excel sheets to perform their tasks, are abundantly well supported in Visual Studio 2010. Test cases are registered in TFS and he/she has access to a specific tool based on optimal support of, usually, manual testing activities.
Enough about testing and cooperating. What does it look like and where exactly is cooperation supported. In the following paragraphs the abilities of Visual Studio Team System 2010 are explained more precisely using the test goals and test tasks as mentioned above.
Democratizing Application lifecycle management.
With their vision “Democratizing Application lifecycle management” Microsoft has further extended the support of Visual Studio Team System 2010 for ALM. As can be read in this publication of .NET Magazine, additional supporting aids are available to improve cooperation and various roles are better supported.
Several techniques and components in the testing area have been added to Visual Studio. The most remarkable added feature is the standalone [acceptance]test tool Camano. Camano is a test management environment where test cases can be introduced, which are work items of a new work item type ‘test case’ in Team Foundation Server. Test cases can be performed using TestRunner, which has the ability to record actions done by the tester and send these along with a findings report. The TestRunner does not only capture the screen of the tester, but also gathers information about the system that is being tested on and creates a file that can be used within Visual Studio to step through the tester’s actions in debug mode. This file is actually a complete stack trace that mimics a real-time debug experience. Microsoft calls this “Historical debugging”.
Test plan overview in Camano from which Test cases can be executed.
Apart from supporting the execution of acceptance tests Visual Studio Test Edition also has the option to automate it. Using CodedUI technology, which generates C# code from actions that have been executed in the window, UI tests can be included in the daily or continuous build process.
A lot of new things, but how do you use them? In the following paragraphs some examples are shown.
Test Impact View for the development tester.
As mentioned in the previous paragraph, the development tester has been provided with ample tools to perform his/her tasks in the current as well as previous version of Visual Studio. For instance, the test framework and the test coverage reports are there to prove quality.
Microsoft has still found new ways to support the work of development testers and those who do Test Driven Development. The Test Impact Analysis is the most eye-catching.
Impact Analysis screen
The Test Impact View provides developers [test developers] an overview of all unit tests that have been affected after a code change, and thus need to be rerun. Aside from the tests that need to be run, the Test Impact View also provides an overview of code that has been modified. This overview also enables efficient testing of code changes.
Camano, TestRunner and CodedUI for system and acceptance testers
Acceptance tests verify the system regarding compatibility with the environment and functionality, as mentioned in previous paragraphs. Those that play the role of acceptance tester can do this totally ‘disconnected’ from the development team. All that is required is a list of tests that needs to be run. If the application does not comply with what is described in the test, this is sent to the developer to fix.
There are three challenges in this simple scenario:
First: test cases are based upon business requirements. Nine out of ten times these requirements have been established by the business at the start of the project and during development of the application these have been updated due to new insights. This causes a problem for the tester. How does he know that the test case he is executing is up-to-date and can be traced back to a user requirement.
In Visual Studio 2010 this challenge can be met by linking test cases to User Stories [in the Agile Template] or Requirements [in the CMM process template]. This way reports can be generated about these User Stories. Camano, an aid to the acceptance tester separate from Visual Studio, can tell if User Stories do not have a test case yet and vice versa. It can also show test cases that are not linked to User Stories, which can mean that this test is not needed.
Test impact Camano, User Stories without tests.
Linking a test case to a User Story
Just like the test ‘Impact Analyze’ in Visual Studio, Camano has an impact view that shows which tests need to be rerun not only because of code modifications, but also because of modifications in User Stories. This helps the tester understand the work of the business analysis [and developer] better, by indicating something has changed, what has changed, and that he needs to modify something, i.e. rerun a test.
The second problem is the issues. The tester notices something that is not correct in the application and sends this issue in an email or some other more advanced bug reporting system with high priority to the development department. Email back to the tester with modification status: ‘bug solved’ or ‘not reproducible’. The tester verifies, notices that the bug is still there and sends back an email, somewhat irritated. The start of the so-called ‘bug ping pong’ (?! Is there an acceptable idiom in English for this?).
Since the first release of Team Foundation Server in 2005 this ‘bug’ has been recognized as a work item. This made it easier and clearer for developers and testers to register bugs, to manage, and to make reports. However, there is no real integration with acceptance testers. They can introduce issues by using the WebAccess tool of TFS, but one does not really solve the previous challenges with that.
In Visual Studio and the standalone testing tool Camano this integration with acceptance testers comes true. With Camano’s TestRunner it is possible to save a lot of necessary information within the issue registration system that can be useful to the developer to reproduce the issue and solve it.
Microsoft TestRunner makes a snapshot of the environment, operating system version, service pack and all other possible system information. Aside from this system data, the testers can make screenshots during testing and embed these in the bug report. Not only screen prints can be embedded, but also a video of the actions that the tester has taken can be put into the findings report, which is indexed by test steps in the test case. All this additional information helps the developer to find a solution quicker and easier and makes sure that the wall, the irritation, between developers and testers becomes smaller or disappears entirely.
Issues work item with testing steps in the details-tab.
The third challenge is the workload of the tester. For example, when a bug is solved he needs to rerun his test case. This in itself is not a big challenge, because he has recorded everything in Excel or some other tool. But what if the bug, for example, is a part of something that is used in hundreds of pages or screens? Then all these pages need to be tested again, which is not very effective for the tester indeed.
Camano makes it possible to record Test Cases and to run these later on in TestRunner, so that the tester does not to go through all the steps manually again. A big improvement to reduce repetitive activities. Another addition to Visual Studio that makes life easier for the acceptance and system testers is the CodedUi, with which it has been made possible to make scripts for the user interface that is hard to test automatically.
Various options to add UI tests to test projects. The ‘automation strip’ option can reuse Camano’s strips.
The code that is necessary to make these tests possible can be extracted from tests that have been executed in Camano. Only small adjustments need to be done to let them indeed run automatically [for example using variable data]. Using this load tests, regression tests, and user tests can be automated, through which the example where hundreds of test pages need to be retested can be prevented.
Test tools do not structure the test process, they only make the work of the tester easier. On the contrary, automation of unstructured testing increases CHAOS. In Camano it is still possible to create many test cases that do not make any sense. What does a structured test approach look like? One can think of many ways to do this.
Generally it can be said that a structured test approach is characterized by:
- Providing structure, so that it is clear what needs to be done by whom, when, and in what order.
- Containing the entire scope and descriptions of the complete range of relevant aspects.
- Providing concrete aids, so that the wheel need not be reinvented all the time.
- Managing test activities taking time, money, and quality into consideration.
An organization needs to choose with which level of ambition this structured test process is implemented in projects.
In Visual Studio Team System 2010 it is possible to record a process template for structured testing in TFS. Sogeti is currently working on two versions of the “TMap® process template”:
- The standalone version, suitable for acceptance testers.
- The merge version that is attached “behind” the process template used by the development team.
TMAP process template for structured testing in VSTS 2010
Still, only (structured) testing is not sufficient. If a developer does not know what quality needs to be delivered, he does not know when he is done, that is a tester does not know if an error should or should not have been found by the developer himself.
So it is important to agree on what level of quality needs to be delivered. In TMap® Next (the structured test approach of Sogeti) the notion basic quality has been introduced. This basic quality describes not only the needed test coverage, but also provides aids for the way in which tests need to be done, and also describes the necessary proof process and conformance.
Using this ”definition of finished” it is possible to setup a learning cycle: not only correct errors, but also learn how to prevent them!
Structured testing, making the entire test process more efficient, so testers can do their work based on risk, (basic) quality, and learning cycle is something that can be realized using Visual Studio Team System 2010. Agreements about procedures, basic quality, and other things need to find their ways into the process templates of Team Foundation Server.
In Visual Studio Team System 2010 all test roles are provided with clear and better support within the Application Lifecycle. Testers do not use their own separate technical tools anymore, but use integrated tools that are used by architects and developers. Effectively tearing down the wall between developers and testers.
But good tools are not enough. Also, a clear separation of roles, tasks, and authorizations are necessary. Finally and most importantly, a structured approach determines how successful you are with your test strategy.
- Information on VSTS2008 Test Edition: http://msdn.microsoft.com/en-us/vsts2008/products/bb933754.aspx
- Information on VSTS2010 Test Edition (Rosario): ttp://msdn.microsoft.com/en-us/vsts2008/bb725993.aspx