Benefits of Automated Testing
Benefits of Automated Testing
Read about more than a dozen benefits to automated testing being seen in six key industries according to industry executives.
Join the DZone community and get the full member experience.Join For Free
Discover how quick and easy it is to secure secrets, so you can get back to doing what you love. Try Conjur, a free open source security service for developers.
To gather insights for the current and future state of automated testing, we asked 31 executives from 27 companies, "What are some real-world problems being solved by automated testing?" Here's what they told us:
- Simply put, automated testing is invaluable for 1) Saving time - as tests are run automatically 24/7; 2) Reporting - we get daily insights; 3) Consistency and accuracy: manual test cycles can lead to mistakes, whereas automated testing achieves accurate results every time; 4) Saving money; 5) Reducing resources, e.g. manual testers; 6) Full-coverage testing.
- 1) Generic transformation of a pipeline from releasing once a year to 17-20 times per year. Solving the testing and QA bottleneck. We work with the customer to drive these transformations through automation of several test types. 2) Next generation of digital transformation in automotive and health with unique use cases to automate, test, and cover. The connected car is their #3 vertical. It enables testing of the full user experience from app to backend servers and connecting to the backend. It benefits health, from supplying drugs to managing consumption in a digital fashion, and taking accountability for how and when you are consuming drugs. We create a virtualized patient in the cloud. Companies can track them with connected devices. We report to the mobile app and give visibility to the patient and report back to the doctor. IoT is the next generation of the digital transformation.
- The initial use case for was for automated testing – 5,000 tests of firmware, linear tests took forever. We distributed tests over an array of machines. Automated testing reduced cycle time. It helped with UI testing running Selenium tests in a cluster to speed up Selenium tests. Selenium Grid is one way to do this.
- As a testing cloud platform, we enable our clients to test across a variety of browsers and devices. We also offer debugging tools like how to pull JS console log and hard files from the browser. We help customers finding bugs and solve them quickly. Zooming out, our larger customers are running tens of thousands of tests per day and can get overwhelmed with information and data. We introduced analytics to sort through the data to find bottlenecks and the root cause of the errors. More mature companies are switching from an internal Selenium grid to move to the cloud because they don’t have the platform coverage they want – test Mac, Safari, and iOS. How to get better coverage. Hard to maintain with evergreen browsers. We do this for them. Error rates with thousands of tests per day. Selenium can be a tricky protocol. Don’t want to spend all of the time to chase down the errors. Increasing velocity. We smoke test on every pull test or commit, so we need to stand up 100 nodes to get feedback to the developer quicker. If you're without a grid, you need to get to the world of CI. Spend more time on best practices – test writing and framework — if you don’t have the expertise to take the testing framework and optimize it with a high level of parallelism.
- Traditional security teams simply cannot scale in a DevOps world. Automated security testing is the key to allowing those teams to scale. Security teams need to work closely with developers, but that method of communication must be via automated testing. Those security teams that leverage automated security testing directly within the development lifecycle have had a greater ability to keep pace with Agile and DevOps development shops.
- Automated testing is what allows you to deliver fixes and features with more confidence. As such, it accelerates development and allows new versions to be rolled out faster – this has impacts in basically every industry, from biotech to defense.
- Automated testing is essential when you must validate complex scenarios that involve a lot of data. For example, we have one airline customer who uses our software to ensure that their web booking system works properly. Our ability to support nested loops in test cases solved a unique problem for them of validating data for multiple origins and destination points. We have another customer that is using our solution to test a mobile app that controls handheld medical devices. Their app must work perfectly, so 100% test coverage is essential. Automated testing helps them deliver a quality product.
- We moved from monolithic to large-scale microservices with automated testing. You must do this to scale and communicate. We use it in our day-to-day working environment. Every piece of code that goes to testing has 90% code coverage.
- 1) Shift left – a client with a big QA team used to perform testing at the end. Once they recognized the need to test early and often, they began writing tests when they wrote application code. The QA team became part of the engineering team. Core testing took place early in the SDLC. We are able to release much faster (50%) and the quality of code has automatically improved, and costs fell due to finding bugs earlier in the cycle.
- Value-add of the company is helping the client become cloud-native developers. Focus on the business logic to open broader set of testing possibilities. Take business logic and run in a mock environment with a unit testing framework. Embedded testing will look like J2EE testing. All traditional Java testing frameworks can be used.
- 1) Cross-browser testing, 2) Cross-device testing, 3) Regression testing UI/UX, 4) Localization testing – ensure testing in all languages. Extend testing, expanding to post-production – monitoring and continuous testing in the dynamic technology world where microservices and OSs are changing. By monitoring 24/7, we help companies overcome the problem.
- DevOps and Agile practices testers are seen as valuable. We give insights to the QA team, so they can influence quality from inception to production. Look at analytics to see where bugs or people are adversely affecting quality. We give them a space because we collect metrics from automated and testing tools and provide holistic analytics to improve quality as early as possible. We provide a requirements traceability matrix – heat map of what’s working and what’s not. Look at coverage from a logical point of view. This requirement to process is complicated and has a lot of bugs. Providing an intelligent idea of coverage.
- Before the release, tests are running which usually get a lot of complex code level bugs. The user log-in was broken with two-factor authentication code. The error captured by the automated testing and fixed before the release. In another case, the public share links were broken. Changing from long to short URL was broken. Both were caught by automated testing.
- With scripted technology, you cannot manage the changes to the test scripts. At the frequency and cadence of the agile or DevOps process. May take one sprint to do but two or three sprints to execute the tests. Using a model-based approach you are able to make changes to and execute changes in real time because it's codeless. A client with test infrastructure was approaching three times the amount of code as their application. Since you’re working with lines of code you have to keep up with the changing code with the changes of scripts. We update the code with an abstracted model.
- Real world problems include the ability to make sure you're actually running your tests as part of your development process and making sure that you don't miss them or skip them. Modern automated testing has advantages like code cleanliness. It can actually evaluate syntax. It can validate comments to make sure that comments are actually inserted in the code. It can do things to make sure that you are actually implementing good development practices and good coding practices into your code base. Automated testing is more about testing code that's already built or has already been checked in, as opposed to code that's in flight. Before, you couldn't put the responsibility back on the developer. That's the other thing that's really changed with automated testing in the past few years — automated testing is now putting more of the test responsibility on the developers themselves, as opposed to this separate and distinct team of QA or QE. This is what's happening for a lot of teams that are transitioning to a continuous delivery model. Most people that have gone from waterfall to agile to continuous delivery–their tasks actually become different because the testing itself is integrated into your code check-in process. One of the things that transition implies is that you don't really need this Quality Assurance or Quality Engineering organization, or you don't leverage them in the same way.
- We had a new video player, new iOS, 15% error rate. There was a simple bug in how it initialized the player. It was reduced to less than one percent.
- E-commerce companies like retail, banking, and insurance, have a catalog of products or services. Websites are dynamic, personalized, and accessed from all over the world. Clients need to run rapid tests on their storefronts. Rapid and regular changes to the website are being made. Others are people with mobile apps, airlines, banks, apps that customers use need to make sure testing on a broad range of devices to know features work. Test features of the device. Tech companies with mobile apps work. Twitter is a big customer.
- A large telecom company is using AI to solve testing problems. The contractor at gate 2 has to go back to gate 3 to identify how many test scripts to test and how many FTEs they will need for approval. We gave him a way to auto-generate a model whereby he can specify tests relative to the model and he can do the tests himself since everything is automated. His margins went through the roof.
- Rabobank in Netherlands – 500 branches, tremendous value using agile requirements designer to automate test cases. Improved efficiency by 30%. Financial services rolled out across 70% reduction in test script creation. Auto Trader cut integration time from three days to three hours. They saved 567-man hours, or 2.5 people per release, and avoided $300,000 in test hardware and software costs. They reduced defects by 25%. Used our continuous testing platform with an e-commerce retailer contract launching a new Rhianna line two months to prepare for promotion and final design, Five days for load testing. SaaS-based platform tested at 10X which what is the site handled for more than 18 straight hours after Rhianna tweeted about the product.
- We work with the Federal Government and overlay their legacy systems to identify potential vulnerabilities.
- Nokia gets calls from customers, network service providers, all cell towers, 200,000 in North America, has hardware underneath to manage the wireless transmissions. Occasionally they need to be updated to go from 4G to 5G. We need solutions to update to meet requirements test before deploying and then monitor once they are in the field. Design, validate, build, deploy.
- Algorithmic trading firms. When building a fully automated CI/CD pipeline process, they will embed our solution with Jenkins and be part of the ecosystem so when the code is checked in, it can move to testing and decide which test they want to run. Results within 30 minutes. It is fed back into CI/CD tool to determine what the next step is. If it doesn't pass, the tool will push the code into the bug tracking system, JIRA, and it is sent back to the developer, then retest what failed. Fully integrated into CI/CD.
- Financial services brokerage applications to APIs to release behavior-driven design frameworks more quickly. There is a templated test developers could write to cover security.
- Frequently, it takes more time to write automated tests for a code change than the time it takes to make the change itself, so it could be difficult to get buy-in on writing the test in the first place. But automated tests give you: 1) More often, faster, reproducible and reliable test runs. This is due to the fact that once a test is automated, the cost of running it is minimum. In addition to that, automated tests are less prone to failures due to human errors. 2) Continuous feedback and hence confidence on your code changes. This enables a faster delivery cycle long term and enables teams for practicing continuous integration and delivery. 3) As Martin Fowler said, “if something hurts, do it more often”. Automated tests could suffer from flakiness due to several factors, one of them being that the product is not testable enough. Having a culture of writing automated tests will make your code more testable.
- Gil Sever, CEO and James Lamberti, Chief Marketing Officer, Applitools
- Shailesh Rao, COO, and Kalpesh Doshi, Senior Product Manager, BrowserStack
- Aruna Ravichandran, V.P. DevOps Products and Solutions Marketing, CA Technologies
- Pete Chestna, Director of Developer Engagement, CA Veracode
- Julian Dunn, Director of Product Marketing, Chef
- Isa Vilacides, Quality Engineering Manager, CloudBees
- Anders Wallgren, CTO, Electric Cloud
- Kevin Fealey, Senior Manager Application Security, EY Cybersecurity
- Hameetha Ahamed, Quality Assurance Manager, and Amar Kanagaraj, CMO, FileCloud
- Charles Kendrick, CTO, Isomorphic Software
- Adam Zimman, VP Product, LaunchDarkly
- Jon Dahl, CEO and Co-founder, and Matt Ward, Senior Engineer, Mux
- Tom Joyce, CEO, Pensa
- Roi Carmel, Chief Marketing & Corporate Strategy Officer, Perfecto Mobile
- Amit Bareket, CEO and Co-founder, Perimeter 81
- Jeff Keyes, Director of Product Marketing, and Bob Davis, Chief Marketing Officer, Plutora
- Christoph Preschern, Managing Director, Ranorex
- Derek Choy, CIO, Rainforest QA
- Lubos Parobek, Vice President of Product, Sauce Labs
- Walter O'Brien, CEO and Founder, Scorpion Computer Services
- Dr. Scott Clark, CEO and Co-founder, SigOpt
- Prashant Mohan, Product Manager, SmartBear
- Sarah Lahav, CEO, SysAid Technologies
- Antony Edwards, CTO, Eggplant
- Wayne Ariola, CMO, Tricentis
- Eric Sheridan, Chief Scientist, WhiteHat Security
- Roman Shaposhnik, Co-founder V.P. Product and Strategy, Zededa
Opinions expressed by DZone contributors are their own.