To understand the current and future state of automated testing, we spoke to 14 IT professionals intimately familiar with automated testing. We asked them, "What are your biggest concerns around automated testing today?"
Here's what the respondents told us:
- Lack of adoption. The goal of automated testing is to release code. There’s a concern if the security program stops releases. It will be kicked out of the automated pipeline. Change the security process if you find issues in the release process you have to include in one of the 20 releases. Security programs have to be OK with certain issues not be fixed in the current release but fixed in future releases. Taking waterfall security programs and applying to them Agile and security will ultimately be kicked out.
- 1) How far behind everyone is. We have not seen advancements in test automation that keeps pace with the evolution of IT. We’re not moving quickly enough because of “not invented here” thinking. Automated testing is still very manual. 2) Too many people are not doing performance testing yet. You cannot justify not doing performance testing.
- 1) Even though everyone wants it, only 40% of testing is automated, the rest is still manual. It comes with a cost. An automation engineer is expensive versus a low-cost manual test. We can automate but we need to make the investment in automation. Hire skilled people and get the frameworks built. The more you automate, the more scripts you have to run, and you see a quicker payback period.
- Automated testing is still treated as a side-show, not getting enough attention from teams and managers, thus preventing automated testing from reaching its full potential. Test automation needs to be treated as a product and needs serious investment from the start. Companies that overlook the benefits of testing can face risks such as finding bugs later in the SDLC that become enormously expensive to fix or facing high abandon rates due to a buggy application.
- 1) The opportunities for automated testing today are enormous. Although there are a lot of tools available, the biggest challenge is going to be finding staff who know how to use these tools effectively. 2) Organizations need to be able to take advantage of opportunities to automate, which means that they must be able to provision test environments, including emulated services and test data. On the subject of test data, regulations such as GDPR must be taken into consideration, with requirements around data masking of personally identifiable information (PII) that can be used to identify a specific individual and is a concern, particularly when using production data for testing.
- The number of third-party tools and the lack of new functionality, resources, and the skills gap. There is not a thorough understanding of security vulnerabilities. It’s easier to train a developer interested in security than train a security person to be a developer. The developer is able to assist with remediation with some guidance.
- The demand for test automation professionals is higher than the supply. There is such a need for people with test automation skills, and there are not enough people trained to fill this void. This leads to teams tasking their existing developers and testers with creating a test automation project without getting them the proper training or providing them with the time needed to treat test automation more than just a side project. To help with this problem, I’m leading an , which provides free test automation courses.
- 1) A variety of tools are available. Invest the time necessary to analyze which tools fit your requirements, and identify the skills needed to use the tools and write the automated tests. 2) Using containers to streamline setting up test environments and running test automation. 3) Maintenance and the high cost of ownership. With aggressive plans to roll out new features, the design may change with features and break automation. This results in changing the automated tests which require a lot of time and effort.
- My biggest concern about automated testing is while companies will involve and adapt to technology in testing, they may miss the business need of driving higher, faster and better automation. ROI will not remain the only factor driving higher automation, it will be driven by how the automation is showcased to and perceived by the clients.
- End-to-end tests continue to be problematic because they are difficult to maintain and slow to run. There is no great substitute to them (cypress-mocked user interface tests or contract testing do not suffice).
- Start-ups should begin with best practices around test-driven design with hardware, software, and logging to get data. More of a guiding framework for start-ups.
- Work with vendors and try their software to help you succeed. You need to be able to trust and do things on a massive scale. It’s a tough situation where you don’t know what you don’t know.
- There is an imbalance between people’s ability to do automated tests versus the amount of coverage they are trying to achieve. More mature can support a higher degree of automation. We see a number of companies trying to achieve a high level of automation without sufficient maturity. Blindly pursuing 100% automation without understanding the strategy is problematic.
Here’s who shared their insights:
- Drew Horn, Senior Director of Automation, Applause
- Angie Jones, Senior Developer Advocate, Applitools
- Isa Vilacides, Director of Engineering, CloudBees
- Himanshu Dwivedi, CEO, Data Theorem
- Antony Edwards, COO, Eggplant
- Kevin Fealey, Senior Manager Application and Product Security, EY
- Hans Buwalda, CTO, LogiGear
- Malcolm Isaacs, Senior Solutions Manager, Micro Focus
- Madan Mohan, Global Head of Travel and Transportation, NIIT Technologies
- Jared Go, CEO, OhmniLabs
- Derek Choy, CIO, Rainforest QA
- Nancy Kastl, Executive Director of Testing Services, SPR
- Rishikesh Palve, Integration Product Manager, TIBCO
- Ray Wu, CEO, Wynd