To understand the current and future state of automated testing, we spoke to 14 IT professionals intimately familiar with automated testing. We asked them, "What are some real-world problems being solved by automated testing?"
Here's what the respondents told us:
- With API Discover and Inspect, a customer on all three clouds found S3 buckets open to the public as the result of a misconfiguration by a developer in an overseas office. Many data breaches are the result of changes. We are now seeing all the changes happening on a daily basis are publicly monitoring. Changes need to be monitored and detected in the event they go sideways.
- We help clients scale security with a lot of manual processes on the backend. We help them figure out how to use typical tools and improve their understanding of security requirements and identify what they really want to test for. Standardize as much as you can to make verification easier. Providing help on the development and security side to moves both groups towards the end goal.
- 1) As organizations undergo a digital transformation, they must automate more in order to keep up with the speed of change expected by the business. Today’s applications must be delivered on an ever-increasing variety of smartphones, tablets, laptops, and wearables – all running different operating systems and browsers – and all the while delivering a superior UX. Test automation is a critical factor for continuous testing in order to make fast delivery happen. 2) The ability to emulate services and other dependencies will improve automated testing by isolating the system under test from reliance on other systems that may be unavailable, unstable, or costly to use.
- When clients come to us, they have large manual regression tests (i.e., 1,500 manual UI regression tests that would take a couple of weeks). We help them reduce cycle time while increasing test coverage by replacing manual testing with automated. Agile teams want the developer and tester being interchangeable for a workload balancing point of view.
- The main problem that automated testing solves is having the ability to validate flows in a continuous and replicable way with deterministic results.
- Automation can improve time-to-market and quality-to-market. It can also address interfaces that are not easy, or even possible, to access manually. Another interesting problem that automation solves is continuous delivery. Without test automation, continuous delivery doesn’t happen.
- There are cases where customers have a robot turn on instead of booting and it just starts flashing. The maker of the PC boards used on the robot get one or two instances with multi-day cycling on/off to recreate the problem. If we reboot any unit every couple of thousand times it will fail once or twice. Automated testing enables us to scale up testing to test more cycles, more quickly to identify potential failures.
- 1) UI intensive applications with a lot of workflows involved. Automation works well with backend and data-driven applications. When you have a UI with heavy workflows, automation becomes challenging. When the UI is updated, maintaining the test becomes challenging. Automation takes too much of the engineers time. Our clients off-load to us. We handle QA so engineers can focus on what they need to do. Most people underestimate the amount of work necessary to maintain automated tests. Feature updates require test updates. This slows down product development while engineers have to update the tests. We recommend a hybrid approach with automated testing done in-house while the other is outsourced to us.
- Reduce manual regression testing time, enabling testers to focus on the more important problems, giving developers a sense of confidence that the code they produce is of high quality.
- 1) Citibank had a digital focus initiative to get its app rating from 3 to 4.5. Massively increased the speed of time to market and did in-sprint performance testing to achieve their goal. 2) We helped Spanx increase revenue by delivering a better user experience, monitored their products and automated testing to optimize their website. 3) Cerner uses us for all testing to reduce time to market and time to deploy. We give clients confidence through our customer-specific testing function.
- A large U.S. telco provider moved their on-prem apps to the cloud and consolidated applications across different business domains two years ago. We helped them leverage out-of-the-box CI/CD tooling and testing to shorten development cycles. They are now able to release every few weeks and continuously monitor the health of their applications. We helped them automate their entire launch process and accelerated product launches and enabled monitoring from tablets while they were out of the office. We helped them significantly reduce the time to deliver new features and applications.
- Automating testing is having real-world impacts across industries. For example, in the travel industry, NIIT Technologies is helping one of the largest commercial airlines in the U.S. keep up with its rapid expansion by testing its growing IT landscape and mitigating functional risk, increasing operational efficiency, and enabling 99% system reliability. For the travel sector, automated testing means faster growth, lower costs, and increased safety for all involved. The emerging technology landscape and ever-changing customer needs to be mapped to it provides three basic challenges to industries: 1) Rapid changes to the custom product /newer versions/pinpointed solutions for end-users. 2) Customer satisfaction post-production: error-free solutions; better user experience (UX). 3) Compatibility with the existing. Automated testing is the only key to provide the resolution to all these challenges.
- 1) A payments customer built out its digital wallet on every e-commerce site. If a developer for Amazon accidentally breaks digital wallet integration the customer loses business. Now, every day they are able to go to the top several hundred websites and run a live checkout process to confirm their integration is still working. We provide real-time reporting via insight to internal teams so they can reach out to merchants to advise of issues that need to be addressed. 2) We perform inventory validation of an international retail site by stoking up the customers’ entire catalog of products globally and visiting sites and product pages and verify the item is in stock, sizes, colors, prices. 3) IoT car company for connected apps. We've taken responsibility from the core operations team. We have a team to test their in-dash car system. We are able to relay information to the mobile app, talk to the data plug and simulate going on drives, and verify data is received and processed and sent to the mobile app across multiple form factors.
Here’s who shared their insights:
- Drew Horn, Senior Director of Automation, Applause
- Angie Jones, Senior Developer Advocate, Applitools
- Isa Vilacides, Director of Engineering, CloudBees
- Himanshu Dwivedi, CEO, Data Theorem
- Antony Edwards, COO, Eggplant
- Kevin Fealey, Senior Manager Application and Product Security, EY
- Hans Buwalda, CTO, LogiGear
- Malcolm Isaacs, Senior Solutions Manager, Micro Focus
- Madan Mohan, Global Head of Travel and Transportation, NIIT Technologies
- Jared Go, CEO, OhmniLabs
- Derek Choy, CIO, Rainforest QA
- Nancy Kastl, Executive Director of Testing Services, SPR
- Rishikesh Palve, Integration Product Manager, TIBCO
- Ray Wu, CEO, Wynd