Over a million developers have joined DZone.

Crowdtesting: Overview of a Powerful QA Testing Tool

DZone's Guide to

Crowdtesting: Overview of a Powerful QA Testing Tool

Crowdsourcing your software testing in addition to your in-house testing brings end-users into the testing process, a huge pool of available testers and devices.

· DevOps Zone ·
Free Resource

Don’t let inefficiencies in software testing lead to delayed deployments and poor quality products. Get the 90 Days to Better QA Guide by Rainforest QA for best practices to avoid these common pitfalls.

Crowdsourcing in IT

Crowdtesting (crowdsourced testing) has its roots in the Open Source movement, Mozilla being one of the most famous organizations built on crowdsourced testing and development. As organizations started to explore the potential of vast online communities contributing to various projects it was a logical step for the IT industry to also begin to draw on this new form of labor. In the last decade, crowdtesting also became part of the set of tools that companies use to develop commercial software, as it offers previously unattainable scalability and brings the unpredictability of a real environment into the testing process.

Applying crowdsourcing can make the testing process more flexible, always in accordance with the demands of the actual project. It is in no way a challenger of in-house QA experts, but rather a tool in the hands of IT managers to allocate their resources in a way that provides the highest quality with the lowest possible investment. This article will investigate the processes, merits, and challenges of crowdtesting to provide you a better understanding of this new branch of software quality assurance and its possible place in your own development lifecycle.

The Definition of Crowdtesting

Crowdtesting means utilizing individual contributors from outside the company in the testing process, generally through a specialized crowdtesting provider. In the case of traditional outsourcing, the testers are employees of the supplier, while crowdtesters are
individual contributors from a large online community, usually working in their spare time. An adequately large and diverse crowd provides IT managers with an almost infinite pool of testers with different skills to draw on for any kind of testing project.

Best Suited for Consumer Software Testing

Theoretically, crowdtesting can be used to test any kind of software, but it shows its real strength when utilized for testing user-centric products aimed at a large user base. Therefore, it is generally considered when testing online and mobile applications, websites and consumer devices. Today’s consumer applications have to work flawlessly in various environments on a dizzying number of devices, operating systems, browsers, etc. Crowdtesting is capable of mirroring this diversity during the testing process to help development teams cope with the continuously evolving consumer technology. While consumer software is suited best for crowdtesting, it can also be utilized for enterprise applications. This “Bring-Your-Own-Crowd™” approach, however, requires a different setup and management than crowdtesting for consumer applications.

Internal Testing versus Crowdtesting

Internal testing and crowdtesting are not alternatives, they complement each other. Outsourced testing is often viewed as an alternative for in-house testing teams. On the other hand, crowdtesting is a solution for QA managers to optimize the distribution of resources for every project, outsourcing the tasks more suitable for the crowd and focusing the efforts of the in-house testing experts on the areas requiring very strong domain knowledge. Testing’s need for human resources is often very volatile: it peaks before a release and can become almost non-existent after. As crowdtesting is almost always provided on an on-demand basis, it is a cost-effective way to help in peak periods. Given that a crowd is large enough, crowdtesting is almost infinitely scalable: it can provide very large numbers of testers if needed and can be totally suspended without high transition costs.

Long gone are the days when software quality assurance had to rely only on a single method: today, exceptional quality can be achieved cost-effectively by combining various methods.

The Four Core Advantages of Crowdtesting

Testing on Real Devices

The members of the crowd use their own devices to test software: there is no emulation, no virtualization. The devices are not sterile lab equipment- they are in everyday use, with their owners’ favorite applications installed on them. Testing on emulated devices is a good first step to finding the most obvious compatibility issues, but to provide exceptional software quality and find all interferences with other commonly installed applications, real devices are necessary. A large crowd provides cost-effective access to almost any real device with perceptible market presence – and every device comes with a readily available tester.

No Labs: Real Environment

Crowdtesting takes place in a real environment. The testers work from their homes, on the go, or anywhere they have time for it. This quality of crowdtesting is increasingly useful in the era of mobility and online services: various connection speeds and types can be tested anywhere around the globe without any kind of simulation.

Infinite Scalability

It is hard to scale QA processes cost-effectively. An in-house testing team is almost non-scalable. An outsourced testing provider employs a particular number of testers and aspires to utilize every one of them at all times. Crowdtesting providers, however, don’t have such constraints when undertaking projects: all of their testers are freelancers readily available for work. If the crowdtesting provider has access to a large enough pool of testers, almost any target group and amount of testers/devices can be accessible within days. A typical crowdtest comprises anything between ten and one hundred testers. This previously unimaginable flexibility can demolish constraints hindering quality assurance processes by removing the bottleneck of scalability.

User Perceived Quality

The issues discovered by testers from outside the company won’t all be functional problems: testers first encountering the software often categorize serious usability and design problems as bugs. Instead of dismissing these feedbacks as “noise,” using them can help dramatically improve the value provided by the software for its end-users. Anyone deeply involved in the project – like the developers and the in-house testing team – will be blind to these issues, so the insights of fresh pairs of eyes are invaluable.

Two Approaches: Structured or Exploratory Bug Testing

When utilizing crowdtesting for functional software testing, generally two strongly different approaches can be taken: structured testing with strict test case management or explorative testing.

Exploratory Bug Testing

Crowdsourced exploratory bug testing can be useful in various stages of software development and is usually applied in conjunction with in-house testing. The approach’s main drawback is that it requires very little preparation on the QA side: there is no need for writing test cases or careful test documentation. Testers execute realistic use cases (on their own or by following a high-level script), thus uncovering bugs and issues that would cause the most problems during real life usage. Exploratory testing is a great way to test mobile apps’ device compatibility and to regularly sanity test extensive online systems, where wall-to-wall structured testing would be too expensive to conduct on a regular basis. Exploratory testing lacks the strict thoroughness of structured testing, but in exchange provides flexibility while being cost-effective. In many cases, structured testing would be overkill as large numbers of testers working in the exploratory method can uncover enough bugs to keep developers busy for weeks. As the testers execute real life use cases, it is also ensured that the issues mostly bothering the end users will surface.

Structured Testing

Structured crowdtesting can be used to test a software’s functionality wall-to-wall. Depending on the depth of domain knowledge required for the project, testers with specific skillsets can be selected to execute the prepared test cases and document the outcomes. Crowdsourcing works especially well in the case of structured testing, as test cases can easily be grouped into test suites, which can be assigned to the selected testers. Test cases also provide an excellent basis to track the testers’ progress during the project. Structured crowdtesting demands substantial preparation: test cases must be very clear and specific, as testers outside of the company generally require more explanation. In certain cases, it is also necessary to create supporting documents to explain specific functions of the software or the testing environment. To create a test plan that is mutually exclusive and collectively exhausting – meaning there are no gaps and no overlaps between test cases – takes time and expertise, therefore it is often outsourced to the crowdtesting provider. The required preparations make structured crowdtesting less flexible than explorative crowdtesting, but if a test is repeated multiple times, preparations can be decreased or even omitted entirely. Subsequently, structured crowdtesting is an excellent tool for regression testing and can be successfully applied in agile software development to test the regular releases along the same test cases.

Crowdtesting is not the solution to every problem of software QA, but it offers great help to cope with the rapid evolution of consumer technology, the extreme device diversity, and the ever-accelerating software development processes. When used to include IT experts and professional testers from the online community, it provides a large readily available workforce to solve scalability problems. As every member of the crowd owns various devices that they can utilize for testing, crowdtesting provides a huge pool of available test devices. Last but not least, crowdtesting helps to include end-users in the testing process, thus bringing the developers much nearer to the final consumers of their creations. In a new era of IT driven by consumers, it is imperative for every company to bridge the gap between users and creators.

Discover how to optimize your DevOps workflows with our on-demand QA solution, brought to you in partnership with Rainforest QA.

crowdtesting ,quality assurance ,testing ,devops

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}