To understand the current and future state of automated testing, we spoke to 14 IT professionals intimately familiar with automated testing. We asked them, "What are the most common failures you see affecting the automation of testing?"
Here's what the respondents told us:
Lack of Planning
- 1) Customers know they need to do something about testing, but they do not have a plan to deploy and use technology. Identify the deployment plan and the path to an ROI upfront. 2) Too much focus on tools and processes. Too many companies spend too long messing around with internal frameworks and solutions. They wind up with analysis paralysis and waste assets in testing. A company will spend three years building a mature test practice infrastructure and then a new manager comes in and throws out the system. A lot of “not invented here” that goes on in testing. Look at the tools you have on board and figure out how to use them to accomplish your goals.
- Lack of design and the structure of tests can lead to fragmented test cases with many low-level steps, which makes automation maintenance difficult. Often applications are not automation-friendly, not offering hooks for recognizing UI elements, or white-box or API access to internal data. Another big aspect is making sure the team is properly trained from the start, often those who attempt automation need education in order to do this.
- 1) The biggest pain points are time and design. Failure to take the time for test-driven design. If the design is right with all of the stakeholders, you can do it right. 2) Save time to develop the visualization side. Build to the interface so you can test.
- Abandoned automation where someone tried it and it was non-maintainable. They didn’t spend the necessary time to build the right framework with maintainability in mind. People only want the code in one place. They have high maintenance costs. It takes too long to run long workflows. Failure to do UI automation lead to one thing changing and it breaks due to unstable automation. Lack of proper skills. Automation is done as an afterthought. Not thinking the best way to automate for the long term.
- There is a list of common issues people fail on when writing automated tests: 1) Writing tests that are not robust enough and pass or fail under certain circumstances which makes them unreliable and developers lose trust on results. 2) Writing tests that take too long to run so you limit the speed of your entire build or you incur in timeouts. 3) Write tests that are dependent on others, so if one fails then the rest will likely fail. 4) Write tests based on assumptions that can change unnoticed and could break the production code.
- A culture change is required for companies going through transformation and scaling. For scaling, it’s easy to download Selenium or Appian and write tests and run locally. More test cases need to partner with DevOps and plug into the pipeline before things fall apart. Be aware of the hidden costs of open source. Know what you are doing and how to architect to have an enterprise-grade framework, process, and test code in place so when you amplify the process, you’re not amplifying smaller problems. Culture is important. Engineering, product, DevOps, and release automation can lose confidence if test automation doesn’t work and everyone can get bitter about test automation. Do test automation properly the first time and change the culture so everyone sees test automation as enabling and super important.
- People test and scan too early. If you test and scan too early, you’re testing a bunch of code that doesn’t work. If there is no functionality, there is no attack surface. You need to find a balance between a stable, real build. Coming in early with threat modeling is good, with automation you can overlook things that come in later in development.
- A lot of issues arise because developers are not following good coding practices. We help them adopt a set of application development best practices. It’s common to see false failures because of factors other than bugs. 5-10% of failures are due to intermittent time out issues not related to a defect. Another problem area is new features being added in every cycle which causes changes in design which become difficult when you have an automated test suite in place. Also, areas where tests don’t add value or capture the right behavior when tests need to be done.
- The most common failure I see affecting the automation of testing is a lack of catering to the specific needs of a given business. The automation of testing can be looked at as a “lock and key” model. The key or automation that works for one company won’t be the same for another. Other automated testing failures are highly automated test creation and maintenance time, low speed of automated test execution and automation that’s not able to detect and prevent defects.
- A number of factors contribute to automation challenges. Examples include the speed with which applications change, resulting in brittle automated tests that break regularly; automating the wrong things such as tests that are only run rarely; difficulty staffing developer-tester roles that are in demand as organizations shift their testing left to earlier in the development phases and continuing throughout the development pipeline.
- A big problem is a reliance on third-party tooling and the lack of understanding of what value third-party tooling adds. Without the objective evaluation of alternatives, there’s no reason to switch away from them.
- It’s important to not test too much. If you are going through a lot of change in the product, too much testing will slow you down. If you have a stable code base, you’re ready for automated testing. You don’t need as much code coverage if the code base is still changing.
- 1) Going to 100% automation does not lead to the best ROI. People think they are sophisticated and successful but fail to realize the true cost and speed implications. 2) People tend to underestimate the complexity, effort, and skills required to build and maintain automated tests and they end up failing on the entire automation project.
- Many of the failures of test automation initiatives are due to the lack of proper planning and strategy. Test automation is a software development project and must be treated as such to succeed. Otherwise, teams will see themselves in maintenance hell with unreliable tests that totally defeat the purpose for which they were built.
Here’s who shared their insights:
- Drew Horn, Senior Director of Automation, Applause
- Angie Jones, Senior Developer Advocate, Applitools
- Isa Vilacides, Director of Engineering, CloudBees
- Himanshu Dwivedi, CEO, Data Theorem
- Antony Edwards, COO, Eggplant
- Kevin Fealey, Senior Manager Application and Product Security, EY
- Hans Buwalda, CTO, LogiGear
- Malcolm Isaacs, Senior Solutions Manager, Micro Focus
- Madan Mohan, Global Head of Travel and Transportation, NIIT Technologies
- Jared Go, CEO, OhmniLabs
- Derek Choy, CIO, Rainforest QA
- Nancy Kastl, Executive Director of Testing Services, SPR
- Rishikesh Palve, Integration Product Manager, TIBCO
- Ray Wu, CEO, Wynd