Executive Insights on the State of Automated Testing

To gather insights on the state of automated testing today, we spoke with 20 executives who are familiar with automated testing. Here’s who we talked to:

• MURALI PALANISAMY, EVP, CHIEF PRODUCT OFFICER, APPVIEWX

• YANN GUERNION, DIRECTOR OF PRODUCT MARKETING, AUTOMIC

• ERIC MONTAGNE, TECHNOLOGY PM, BARCLAYCARD

• GREG LUCIANO, DIRECTOR OF SERVICES, BUILT.IO

• AMIT PAL, QA MANAGER, BUILT.IO

• DONOVAN GREEFF, HEAD OF QA, CURRENCYCLOUD

• SHAHIN PIROOZ, CTO, DATAENDURE

• LUKE GORDON, SENIOR SOLUTIONS ENGINEER, DIALEXA

• DANIEL SLATTON, QA MANAGER, DIALEXA

• ANDERS WALLGREN, CTO, ELECTRIC CLOUD

• CHARLES KENDRICK, CTO, ISOMORPHIC

• BRYAN WALSH, PRINCIPAL ENGINEER, NETAPP

• DEREK CHOY, VP OF ENGINEERING, RAINFOREST QA

• SUBU BASKARAN, SENIOR PRODUCT MANAGER, SENCHA

• RYAN LLOYD, VP PRODUCTS, TESTING AND DEVELOPMENT, SMARTBEAR

• GREG LORD, DIRECTOR OF PRODUCT MARKETING, SMARTBEAR

• CHRISTOPHER DEAN, CEO, SWRVE

• WOLFGANG PLATZ, FOUNDER AND CHIEF PRODUCT OFFICER, TRICENTIS

• PETE CHESTNA, DIRECTOR OF DEVELOPER ENGAGEMENT, VERACODE

• HARRY SMITH, TECHNOLOGY EVANGELIST, ZERTO

Key Findings

1. The key to automating testing to improve speed and quality is strategic automation and developer adoption. You must have a platform in place that enables you to use automated test development on a consistent basis and understand how to write automated test cases that automate repeatable core functionality. Automate the entire CI/CD process to reduce the amount of human interaction. This will accelerate the SDLC and lead to more reliable and secure code and applications. Increase speed and quality at every step by learning from unit and regression test results.

2. Developers need to take accountability for their work by integrating the proper unit tests at the beginning of the SDLC. Do not think of testing as someone else’s job. Inspect quality from a functional and non-functional point of view from the beginning.

There have been several changes to automated testing recently: cloud-based testing solutions, the speed with which data is available, and the rise of container services. Cloud computing has boosted the performance of automated testing. It makes it much easier to run load tests with many users at one time. Cloud/grid-based Selenium testing is becoming increasingly mature and popular. Services like Xamarin Test  Cloud and Testdroid now provide physical device hardware through
a web interface in a cloud-based Device-as-a-Service model. The digital transition, including mobile, is accelerating development testing. This impacts both the frontend and the backend of an organization. Test processes are automated more quickly. This provides more data-driven insights from the tests and improves the stability of the build. Containers are a big deal because they allow you to push around code in a completely fixed environment. The rise of container services like Docker have brought great changes to automated testing, reducing many of the environmental risks.

3. Many technical solutions are used for automated testing since there are a lot of different aspects to automated testing. Of the 33 different solutions mentioned, Selenium was mentioned most frequently: Selenium with a Capyvara layer on top making the automation more portable, Selenium WebDriver tools, and Node.js-based Selenium frameworks.

4. Real-world problems solved by automated testing are speed to market and lower costs. Automation reduces development time and reduces the time it takes to discover and fix problems. Automation enables you to complete QA and production in two days instead of two weeks. New versions are rolled out faster and enable human developers and engineers to focus on the complex tasks of software development, product strategy, test strategy, and on the more difficult areas to test within the product. Running a script automatically can be done much faster than a human can run through code or validate functionality.

All of this saved time results in reduced resource costs, since you can go to product faster with automation, and there are fewer resources used to test at scale. Companies save by hiring fewer manual testing resources. Companies who don’t automate are paying a “stupid tax.”

5. The most common issues affecting the automation of testing is the brittleness of the automation process, people, and lack of processes. If the product is dynamic or Agile it becomes a difficult effort to keep automated tests up to date. The real challenge is setting the test up in a manner that allows you to cope with maintenance challenges that come with automation.

Comprehensively solve the issue by taking care of the automation, test data management, and service virtualization to provide a stable infrastructure. People unwilling to change are the number one roadblock. They don’t believe a manual test can be automated when any process can be automated. To promote culture change, get the development and testing teams excited about the potential outcomes and let them see how quickly their code gets to production.

Define KPIs and pass/fail criteria for all performance tests. You cannot become Agile without repeatable processes.

6. The future of automated testing is integrating artificial intelligence (AI) and machine learning (ML) into the CI/CD/DevOps process so that testing is completely automated. Leverage AI/ML to automate automation. Learn from the tests to automate the process. This will result in self-healing code. Orchestrate the test process end-to-end and integrate the DevOps CD release cycle. AI/ML based solutions can identify and proactively check areas known to introduce critical issues.

7. The biggest concerns with automated testing today revolve around the fragmentation in the marketplace, the lack of an end-to-end solution, and the perspective of the people involved. The automated test space is very crowded. Several third-party solutions keep popping up and it is very difficult to have the knowledge to put together a “best of breed” solution for your company’s needs.

Nothing is managed holistically. There is no connection across the siloes. There’s need for greater automation to maintain accuracy and reliability of the process. However, there’s still a fear of failure. It’s not about the technology. It’s about understanding the required comprehensiveness of the approach. "It’s not the quality of the daily work; it’s the improvement in the quality of the daily work."

8. Skills developers need to ensure their code and apps perform well with automated testing are:
1) the ability to write tests; 2) understanding automation; 3) good coding skills; and, 4) a broader vision that includes CI/CD/DevOps and what the company is trying to accomplish. Create test code and test cases defining what data to test. Don’t write bad tests. Write tests in the smallest, most elegant way possible. Developers writing code are the best people to direct the testing on their code.

Be aware of how quality engineers write automation. As you make code changes, think about how the automation will need to change. Put effort into getting automation right. Cut out repetitive work. It may not seem like much now, but it will add years of productivity to your life. Build trust in automation by running it regularly during implementation.

Write code so the next person will be able to see what you’ve done and make changes as necessary. Code efficiently so you’re not wasting someone’s phone battery. Follow the coding standards of modular programming, unit testing, exception handling, Selenium, and Appium, and strive for CI/CD. Developers need the “DevOps” attitude – testing is an extension of the development process. Product quality doesn’t stop once the coding stops.

9. While there were several different “additional considerations,” the current and future state of automation, as well as the cost and ROI of automation were mentioned by several respondents.

There’s a stigma that automated testing is difficult to do; however, there are plenty of resources available for manual testers to get into automated testing. Everyone from developers, QA, DevOps, product owners, and stakeholders are responsible for automated testing. Everyone should be involved in automation. Do people know how to write automated tests? Testing is a problem with developers, not the testers. Shift left; test early while still in development. How often are software developers involved in the automation strategy for a given project?

Think about the cost of automation. How much do you invest in people, environments, tools, and maintenance? How do you successfully estimate the cost to reduce surprises down the road? We need to be able to quantify the value automated testing provides by optimizing time to market, reducing mean time to recovery, and ensuring applications are secure.