Thanks to Maciej Gryka, Head of Data Science for Rainforest QA for sharing his insights on the state of artificial intelligence (AI) today.
Q: How are you and your company involved in artificial intelligence (AI)?
A: Rainforest QA is the industry’s only AI-powered crowd testing platform. With AI, we are able to deliver fast, comprehensive testing capabilities on demand with results returned between 30 minutes to a few hours, much faster than any alternative on the market. We use AI in two primary ways to better serve our customers:
Tester reputation. We employ machine learning to determine whether the 60,000 testers on our platform are executing work optimally. For example, the platform evaluates the speed, accuracy, and consistency of test results produced by individual testers, as well as compared to others. Positive and negative outcomes increase/decrease tester reputation accordingly. This data is aggregated into a weighted quorum AI reputation algorithm that ranks the reliability of individual testers. If a tester’s reputation drops below an acceptable threshold, the platform can recommend more training, divert work away from the tester, or remove the tester from the work pool entirely. If a tester’s reputation increases, the platform will route increasing levels of work until the tester is at capacity.
In short, the better the tester’s reputation, the more work they receive, and vice versa. Testers who reach an “elite” level of reputation are additionally granted higher-impact sets of tasks that help better train our machine learning algorithm. For example, when test accuracy is harder to determine, elite testers sometimes manually review unclear test results or even the body of work of another tester. After helping classify this data with higher reliability, the input is used as training data to improve the machine learning model, which in turn improves the overall performance and accuracy of the platform.
Tester error reduction. Development teams want accurate test results they can depend on. When they receive a false positive (a test result that a tester marked as a bug but in fact is not a bug), this wastes development time. And this can be particularly damaging to a company releasing a mobile app into the App Store, where a rejected app during the review process can delay a release for several days or more. To increase test results accuracy, especially at the scale and volume at which we operate, it runs a machine learning model that aims to claim as few false positives as possible without allowing any false negatives (missed bugs) slip past. To do so, we dynamically weight a test with an accuracy probability based on the tester’s underlying reputation score. This helps the platform understand which test results are trustworthy and can be surfaced with confidence to customers. In certain cases where it is harder to discern if test results are accurate for a given test, the algorithm will trigger the test to be routed through multiple concurrent testers and then uses a quorum algorithm to help determine whether a regression test is accurately reporting the truth. Running this method at scale and in aggregate better surfaces a greater number of accurate test outcomes (signal) versus false positives (noise).
Q: What are the keys to a successful AI strategy?
A: You need two things: being able to assess whether AI is a good fit to solve your problem (and if so, which solution to choose) and having enough data to make it work.
One way around this is to combine the human touch with the speed and power of AI algorithms. Our testers have executed over 52 million test steps on our platform and we’ve been able to collect the data from those tests to develop machine learning algorithms that can identify subpar work and prevent poor testing habits from impacting our customers’ test results. Not only is it important to have a large and high-quality data set, it’s also equally important to have the right people interpret the results accurately.
This points to another successful strategy: all of our collected data is useful for learning about software testing in general. A major benefit of aggregating hundreds of organizations tests and observing how they test and what common pitfalls they encounter is that we can detect patterns. The datasets that result from this type of collection and the insights derived help surface what is most important for QA teams to focus on in order to ship higher quality product.
Q: How can companies benefit from AI?
We’re just beginning to see how AI can impact the workplace. One thing AI solves well is helping execute mundane tasks at scale, where accuracy matters, but that is a poor use of person’s time (largely because mistakes can be easily made). By offloading these kinds of tasks, AI enables people to focus on higher-value activities. For instance, a supply chain management company has numerous practical applications for AI. On the other hand, a creative ad agency likely has fewer because of the non-standard, unique, and customized creative work they do for each client’s campaign.
Q: How has AI changed in the past year?
A: People are getting comfortable both with the strengths of AI and seeing how useful and cool it can be, but they are also starting to understand limitations and risks. In addition to success stories like Alexa and Google Photos, there are some cautionary tales of chatbots learning to be prejudiced after being fed this kind of data. We are slowly becoming aware of the fact that AI systems can be gamed.
The industry at large is now in unusual agreement over how useful openness is in science. Even historically very secretive companies (such as Apple) are now sharing their research efforts publicly. This is the only way for them to attract top talent and be considered a top player pushing the field forward.
Reinforcement learning and methods that need fewer data are getting more focus. Many problems that can be solved with large amounts of labeled data (i.e. what supervised learning deals with) are now solved. Many of the remaining challenges are in domains where we don’t have enough labeled data (this is where unsupervised learning helps) and situations where AI agents need to deal with a constantly changing real world (reinforcement learning).
Q: What are the technical solutions you, or your clients, use for your AI initiatives?
A: We use a variety of technical solutions for our AI initiatives. We use Python, probably the most common programming language for data science. We also use TensorFlow, an open-source software library for machine learning that was developed by Google for its AI solutions. Another more classical machine learning algorithm we make use of is called Random Forest, which is a versatile method capable of performing both regression and classification tasks. We also leverage recent deep learning architectures to understand text and images, which is where we can use large amounts of data our testers provide. In addition to human-labeled data, we are also experimenting with unsupervised and reinforcement learning solutions, which have a potential to revolutionize many new areas of work.
Q: What are real-world problems you, or your clients, are solving, or hope to solve, with AI?
A: Testing is a critical part of software development but one few people love, much less like doing. By leveraging our platform and its underlying AI technology, we’re able to make the process much easier and quicker for them while maintaining high-quality results. What keeps us going is knowing that we are automating the boring, mundane tasks that no human wants to do. Some situations will require a human touch, while AI can do repetitive tasks or apply computational power beyond the ability of human capabilities to improve products in ways that would be difficult or undesirable for humans to manage.
Q: What are the most common issues you see preventing companies from realizing the benefits of AI?
A: There are a lot of misconceptions about AI and how we can apply it, which I believe is holding a lot of companies back from either using it or using it successfully. A lot of people don’t understand what they can and can’t use it for or how to truly harness the data they have at their disposal; you need to not only have good data but also know how to put it to work. Over time, I predict this will change as there is better education in the market and as more companies successfully implement AI.
Another issue is the lack of available talent. Machine learning and data engineers are required to manage today’s solutions but they are short supply.
Q: What are your biggest concerns regarding AI today?
A: The biggest concern I have regarding AI today is that people see AI technology as a silver bullet to the problems they have without considering what it can really accomplish. Oftentimes, people attempt to use AI to solve problems without understanding where AI falls short.
For example, there are many things that AI simply cannot replace, such as tasks or jobs that are open ended or require emotional, intellectual or intuitive IQ. AI is great at solving specific tasks that are given, such as classifying a photo or locating a specific object in a photo, but often I see people trying to apply it to scenarios that are simply ill suited to it.
Q: What skills do developers need to be proficient on AI projects?
A: The most important skill developers can focus on is to keep up with the latest research being released from academic and corporate institutions. Research in the area of AI and machine learning is constantly being released, so developers need to be reading all the time to keep up.
On top of that, proficiency in general software development is a must. There are many situations in which we need to build tools for data cleaning, visualization, interfacing with other systems, etc. that we simply have to build ourselves.