Using Machine Learning to Fix End-to-End Testing
E2E testing should deliver the most value for customers. However, they're unreliable. And that's where machine learning comes in.
Join the DZone community and get the full member experience.Join For Free
I’m a serial entrepreneur who’s spent time working at Google, VMware, Stackdriver, and now, my second startup that I've co-founded, mabl. We started mabl earlier this year to help developers with the painful challenge of testing their applications, especially as development cycles are getting shorter with the adoption of CI/CD and continued automation. As we started the company, I did a lot of research, primarily by reaching out to developers to understand their take on software testing, especially those that have embraced DevOps. We also launched a survey to get quantitative feedback.
In addition, I found lots of blogs written about the challenges of testing applications in a modern developer workflow. One of the blog posts that I was drawn to was from Mike Wacker, a software engineer at Google who was also a developer at Microsoft previously. Mike wrote a blog post in 2015 about why end-to-end tests just don’t matter anymore and developers should focus on unit tests primarily and, to a lesser extent, integration tests, as the basis for ensuring quality in their code. If you haven’t read it and are involved in testing, you should. His premise was (I’m summarizing and probably not doing enough justice to Mike):
- E2E testing should deliver the most value for customers since they’re replicating real user scenarios. Mike argues that all product stakeholders including developers, QA, and managers agree to this general premise.
- E2E tests, however, are not reliable (i.e. flaky) and take too long to complete (i.e. run overnight). And when there are bugs, the cause is hard to diagnose. Mike argues that these issues with E2E tests should force us to consider a different testing approach for ensuring customers are happy.
- Unit tests are the immediate solution because they find bugs and remedies faster. These, combined with integration tests — which help test numerous components together, identifying bugs before lengthy E2E tests are written — are the solution recommended by Mike to the broken promises of E2E tests.
Mike was totally correct with his conclusion, for the time. Why waste time on E2E tests when you already know the challenges they present? Instead, compensate by adapting your testing process.
Enter Machine Learning
As Sundar Pichai discussed at IO 2017, Google is revamping all of its products to be AI-first, given the advancements in computer vision, voice recognition, and machine learning. We’ve already seen the advancements Google has delivered from a consumer perspective for years, from self-learning thermostats to traffic navigation to speech recognition. The next step for Google is applying AI for business use cases, which it’s already doing within Google Cloud. At mabl, we agree with Google. Specifically, we believe ML could have a profound impact on improving the lives of developers, allowing them to spend more time building great products and less time on repetitive tasks. This is why we’re challenging ourselves to solve the specific problems that Mike pointed out with E2E testing.
Some of the things our team is focusing on include:
As Mike mentioned in his blog, tests often fail due to not being reliable. When this happens, developers lose confidence in their tests and just ignore them. We believe mabl should be able to automatically detect when steps in trained tests are failing because of a slight change to the UI. Mabl should recommend a fix to the user and automatically update the test script as well. Our goal is to eliminate flaky tests so developers can spend more time on coding and less time on maintaining their tests.
A developer searching for a needle in a haystack to solve a test failure isn’t the best use of that developer's time. Given that mabl has different types of machine-created tests running all the time, she’s learning much about the application and can use this data to provide evidence of what and where pieces of an application aren’t working.
Speed of Tests
Mike specifically called out waiting for tests to run overnight just to find out if a specific component is broken. We’ve also heard from several of our users that manual QA or even outsourced QA to manual testers still takes the time of a normal human to test. We believe tests should be continuous, not overnight, and should run at the speed of machine time, not human time.
One thing Mike didn’t mention is about learning from past test runs. We believe mabl should continuously learn from the tests she’s running as well as the feedback users are giving her, just like ML applied to consumer patterns.
As Mike pointed out, end-to-end testing is difficult today (as it was in 2015 when Mike wrote his blog post). However, instead of replacing E2E with unit and integration test coverage, we believe we should fix E2E testing. We believe that advancements in machine learning allow us to both fix E2E testing and even go beyond what a human is able to do with test automation frameworks. We’ve been at this for eight months and we’ve learned a lot along the way. We’ve solved some hard problems but still have many more problems to tackle.
Published at DZone with permission of Izzy Azeri, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.