What Problems Does Automated Testing Solve?

To gather insights on the state of automated testing today, we spoke with twenty executives who are familiar with automated testing and asked, "What are some real-world problems being solved by automated testing?"

While there was a lot of diversity in the answers, speed to market, time savings, and cost savings were mentioned most frequently.

Speed

• Reducing mean time to discovery of problems and the cycle time is reduced as is wasted time. Find problems earlier rather than later. Doing so saves a person-day per week in rework, according to the State of DevOps report.

• Th ability to reduce the two weeks to go through QA and production to two days by automating the process. Validation is not automated. It is either human or scripted.

• Web apps in the forefront of B2B/B2C reduced time to market don’t create apps create frameworks. 1) One hour versus weeks = time-saving.  
• The ability to ship changes quickly. Granted, automated testing is not the complete solution for that, but having good automation is vital to continuous delivery. The benefits to this are quicker response time to requested changes, bug fixes, and faster time to market.
• Automated testing is what allows you to deliver fixes and features with more confidence.  As such, it accelerates development and allows new versions to be rolled out faster – this has impacts in basically every industry, from biotech to defense.
• Speed. When we started in 2006 we used Waterfall and manually tested everything. Nothing was reproducible. With Agile we added machinery to increase cycle times. Backup with tests you trust for fast, repeatable testing to run on demand. Think about the quality of code and the operating system in the container.
• Rapid software development in a highly competitive product environment is a key component of solid product quality and is truly a differentiator in the market. Automation allows us to use the human component of our engineering organization for the complex tasks of software development, product strategy, test strategy, and simply focusing on the more difficult areas to test the product – things that cannot be easily automated. Some examples of real world problems include distilling customer workflows down to their core components and automating them. This both protects the customer release over release as well as gives us unique, use-case based coverage. The market is expecting fast-twitch movement with regards to software delivery – so we enable ourselves and our customers to benefit from rapid turnaround with regards to new features.
• Automated testing reduces regression test time by 60% versus manual testing. We also have data to show clients about the reliability of our new version to help convince them of the quality and reliability. We’re able to support hundreds of customers with greater reliability.
• Saving time: Running a script automatically can be done much faster than a human can run through code or validate functionality through manual testing.

• Automated testing solves the duration issue. That can accelerate speed to market, meaning that the testing cycles are reduced significantly.

• Manual effort required for repetitive tasks is reduced. Feedback loops are shortened – problems are discovered more quickly for key areas.
• Reduction in time-consuming manual testing, faster delivery of large scale implementations and product launches, reduction in the occurrence of customer impacting scenarios introduced by bad code. 

Cost Savings

• Companies who don’t automate are paying a “stupid tax.”
• Reduced costs of hiring manual testing resources.
• Another example is that in a tight IT spending environment, we cannot rely on manual tests to get us to a shipping state in a reasonable amount of time.

• Reducing Resource Costs: Since you can produce faster with automation, the resources used to test at scale go down.

• In addition to that, if you set it up right, it will help you reduce costs significantly. What we see coming up more and more in the industry is the use of automated sequences, not for testing but for robotic process automation (RPA). Verifications are not used to verify test results, but to ensure that process automation has been sufficient. We see RPA as a trend that gains significant interest today and is being adopted by our customers enthusiastically.

Other

• The most common use case is e-commerce companies like Stub Hub and EBay tracking outflows with release cycles weekly or daily to update the site. As they update the site the automation needs to be updated. They are leveraging AI/ML algorithms to make these updates. We have clients in the education technology and healthcare industries doing the same thing.
• Able to scale as apps become increasingly complex with APIs, UIs, and modules. Very difficult to scale manually. Regression testing of complex applications. Data driven testing with different data sets enable you to have full coverage of a broad set of use cases. Parallel testing in infrastructure. The headache of testing across browsers. If you’re using a third-party API you may have to pay for every API call. We provide technology to virtualize APIs to test without incurring penalties for testing.
• Team coordination while running DevOps. Multiple teams, multiple processes. Compliance, installation of an automated process to meet the expectations of compliance.
• Scalability. Able to roll out DR testing. Migration testing. Able to create and automate exact environments without impacting production and without requiring experts. Sufficiently flexible to reduce downtime. Testing on the actual environment – the exact test platform.
• Enhance adoption and use. Personalization increases sales by 70%. Apply predictive models to automated testing to improve UX. 1:1 marketing and personalization.
• Data Security: Solving for security issues and avoiding data breaches. 
• Removing Human Error: Automated testing is programmatic, so there is less room for error. Testing manually provides a lot of opportunity for mistakes.
• Reintroduction of Old Bugs: Automation can catch old issues before they get introduced into new builds.
• Improved Reporting: Automated testing provides detailed reports of how many test cases pass or fail to quickly identify issues for the development team to resolve.
• Coverage: the more you test the more coverage you have with browsers and mobile devices.

What are some real-world problems you've seen solved by automated testing?

 Here’s who we talked to:

• Murali Palanisamy, EVP and Chief Product Officer, AppViewX
• Yann Guernion, Director of Product Marketing, Automic
• Eric Montagne, Technology PM, Barclaycard
• Greg Luciano, Director of Services and Amit Pal, QA Manager, Built.io
• Donovan Greeff, Head of QA, Currencycloud
• Shahin Pirooz, CTO, DataEndure
• Luke Gordon, Senior Solutions Engineer and Daniel Slatton, QA Manager, Dialexa
• Anders Wallgren, CTO, ElectricCloud
• Charles Kendrick, CTO, Isomorphic
• Bryan Walsh, Principal Engineer, NetApp
• Derek Choy, V.P. of Engineering, Rainforest QA
• Subu Baskaran, Senior Product Manager, Sencha
• Ryan Lloyd, V.P. Products, Testing and Development and Greg Lord, Director of Product Marketing, SmartBear
• Christopher Dean, CEO, Swrve
• Wolfgang Platz, Founder and Chief Product Officer, Tricentis
• Pete Chestna, Director of Developer Engagement, Veracode
• Harry Smith, Technology Evangelist, Zerto