Service Virtualization and Manual Testing in Agile Teams. Is it Still Necessary?

This is a story about a time I consulted for a department at a large multinational enterprise.

There were around 80 people working there on delivering software products for internal use.

The management team had implemented the Extreme Programming methodology, a type of agile software development, and used it for more than ten years. They had more than one hundred middleware applications (microservices) exposing APIs to other departments. Each application was tested by anywhere between hundreds to thousands of different types of automated tests after every code commit.

Despite those high levels of automation and 83-98% automated test code coverage (depending on the application), every team was still manually conducting exploratory testing of their software, with the help of service virtualization.

Let us explore how the management justified the value of manual exploratory testing and using service virtualization in that type of environment, and how has that been implemented.

Their Software Delivery Lifecycle

Developers would pick up a new story to work on, and start with writing acceptance tests. They would discuss the acceptance tests contents with the business and the testers working in the team. They would then proceed to write unit tests and production code.

Once the user story has been implemented and was working as expected, the testers would test the application manually. Quite often they would find unexpected behavior that the developers did not anticipate. They would discuss with developers if immediate code changes were required or a new story needed to be raised to tackle the problem quickly.

They would also find serious bugs once every 2-4 user stories. The costs of the bugs found leaking to production would be so high that it easily justified having an extra person on every team, a tester performing manual exploratory testing.

Why Is Service Virtualization Useful?

A tester, while manually exploring how the application behaves, would simulate different types of hypothetical responses from third party and backend applications. 

It was often because of those hypothetical scenarios that the bugs were found. The more backend or third party systems were involved in one transaction, the more likely permutations of different non-typical responses would result in unexpected system behavior.

Often, those test scenarios also included valid responses that have not been covered by acceptance tests by developers. Developers are also people, so they sometimes missed obvious scenarios or made simple mistakes.

So, even though the developers have thought they had implemented everything well enough, and the new functionality can go to production as is, testers would make another judgment call at what else is likely to happen in production environments. That resulted in new stories or bug fixes being raised.

After testing in isolation, the tester always performed manual integration testing with backend and third-party services.

Service Virtualization Tools Used

All this would not be possible without using a service virtualization tool fitted well for manual exploratory testing in Agile environments. Since there were no appropriate tools available on the market at that time, this particular organization decided to create one in-house. They spent 14 days over a period of 9 months developing and perfecting a tool for that team. That tool was specific to the applications they had been developing.

The developers had used an open source tool called Wiremock that was used in their acceptance tests. They had built a GUI on top of Wiremock with several extensions to allow for better usability for manual exploratory testing.

Because the developers were using Wiremock in their acceptance tests, it was natural for the testers to import the same virtual services (or stubs, as they called them) to their GUI tool. Using the same base technology proved to be very efficient.

Lessons Learned on Custom-Built Tools

The technical leads noticed that they had five teams that developed very similar tools to perform similar service virtualization and stubbing tasks. That totaled up to around 80 man-days of development resources. That was an inefficient use of resources. 

An improvement on that strategy would be to use an off-the-shelf tool instead of creating new tools in-house for every new project, provided it exists. It would help them to hit the ground running, save time and costs, and reduce risks of bugs in the custom-built tooling.

Benefits of a Lightweight Tool

At the company where I currently work, we have taken all of those experiences into account while developing a new service virtualization and API mocking tool for Agile teams called Traffic Parrot. 

It is a tool that provides powerful service virtualization and API mocking capabilities but stays flexible and lightweight enough to be used in highly Agile teams by both developers and testers. Thanks to its many advanced features, it can be also used in less Agile environments while transitioning to the new delivery process. It can then be very cost effective.

Wiremock is also another good tool to explore. It is designed for developers but is sometimes used by testers as well, especially in automated tests.

Summary

We have learned that manual testing can be valuable even in an environment where there is a lot of automation. We had a look at why service virtualization is key to effective manual exploratory testing in highly Agile environments. We also explored how important it is to use off-the-shelf open source or commercial tools fitted for Agile teams, and listed two tools worth trying out.

Next Steps

If you have a lot of automated tests but once every other release experience costly production bugs go ahead and investigate if manual exploratory testing could help to address that problem. You do not have to hire people to run this small experiment. You can start by having existing team members that did not work on a given functionality put on a “testing hat.” They can test the application pretending they do not know how to code, as a tester would.

Download the free Traffic Parrot community edition and get started. It could increase the efficiency of finding new bugs and reproducing existing bugs and thus create more peace of mind for the testing team.

Download Wiremock and explore how to use it in JUnit tests.