To gather insights on the state of automated testing today, we spoke with twenty executives who are familiar with automated testing and asked, "What are your biggest concerns around automated testing?"

Three themes came through in the discussion: 1) fragmentation of solutions; 2) the lack of an end-to-end solution; and, 3) people.

Fragmentation

• Fragmentation in the market. There are still a lot of growth opportunities. It is slowly evolving and improving to keep up with all the changes in technology.
• The automated test space is very crowded. People are using crowd-sourced testing but not doing justice to the product or the apps are not doing automated testing. If automated testing is not there, it should be a huge red flag. Have it up front in the SDLC.
• With the number of third-party solutions that keep popping up, I firmly believe automated testing needs to fit in with the development team and their process for delivering code. There is no template that will work for everyone; the ideas and processes are still driven by humans, which means there will always be different ways of doing the same thing.
• A few things we keep an eye on are: Are we using the right technologies for the job? Are we empowering our automation engineers to work as quickly as possible given the product and environment? How do we test at the lowest layer, automatically, with the greatest return? Can we optimize the product continuously to enable better automation? Are we able to keep up with the Platform/Product differences especially since our product can run on many different physical devices and within many different environments? Creating automation that leverages the commonalities is important from a scale perspective, but it is also a challenge and a concern because we don’t want to maintain a lot of similar scripts. We want the best bang for the buck. Examples here are new technologies like 3D XPoint vs. Virtual filers, 100gb NICs vs. VMWare logical interfaces. Things like performance and behavior differences can require automation fine-tuning.

Lack of an End-to-End Solution

• While you can cobble together a solution, there is no one end-to-end solution. It’s difficult to know all the technologies and tools to pull everything together. Pick a suite that fits your world. Find a partner that knows the different tools and technologies to give you direction on the best fit for you.
• It goes with orchestration. There are silos of automation. Nothing is managed holistically. There is no interconnection between silos. Manual testing leads to latency and delays.
• Not being exact or thorough enough. Moving to automation more for visualization and speed. We need to enable and be consistent. Account for how different real environments are and what the impact is. Understand the environment and document. Scale as you grow and add more people. There is more need for automation to maintain accuracy and reliability of the process.
• Selenium WebDriver remains very finicky and flaky.  There is a steady trip of incompatibilities between minor versions of Selenium, minor versions of browsers, and minor versions of Selenium’s long list of dependencies. This is one of the reasons cloud/grid solutions are so appealing – you can have a third-party sort all of this out for you.  But bigger picture, the Selenium organization is either underfunded or just choosing not to address this as a serious issue.
• Our concerns are not related to technology. With the solution we bring to market, the technology challenges have already been solved. My concerns are around the non-comprehensive adoption of test automation. A lot of our customers, early on, think test automation is about UI automation. You need to address test automation as a real ‘change agenda.’ One that establishes sustainable automation, with the right tooling and the right concepts.

People

• Fear of failure. "Perfect" is the enemy of "good enough." In Agile, you're getting a bunch of changes and deploying. In DevOps, it’s OK to fail quickly and fix quickly with fast cycle times.
• Again, it’s not a concern about technology, but whether you understand the required comprehensiveness of the approach.

Other

• Not having automated tests. They're poorly written. It’s painful to tell an organization you have no hope unless you tackle this problem first. There’s hope. There are a lot of stories about starting small automated test suites. Retrospective and root cause analysis write tests and keep running them to see patterns that screw you up. Improving automated tests will have a huge impact downstream. Scott Prue, working with mainframes, started with five tests and built on the benefits. It’s not the quality of the daily work, it’s the improvement in the quality of the daily work.
• Brittleness of the whole exercise of automation is costly and only partially successful.
• The time sync between automated test development and feature development. Testing is frequently playing catch-up to features.
• Apart from everyday concerns like load and whether a web or mobile app shows up properly, there are other serious issues that almost all companies face today, namely security. Big stories in the news about data breaches and hacking happen on what seems like a daily basis. To help temper this, automating testing is critical for every company, but not everyone does it yet. QA teams are often afraid of automation because they’re worried it will replace them, but they should embrace automation since it frees up time to do better, more challenging work like writing tests. It might require them to learn a little bit of code, but that’s a positive thing for a career. So, it’s a misnomer that automated testing will replace people because there is no replacement for manual testing. But you need to do both to scale and to produce bug-free code.
• From a philosophical perspective, I’d say “why” and “when” automated testing should be done. Automated testing is designed to add value and solve certain areas of the software development lifecycle. Not using automated testing can allow critical vulnerabilities to progress into production environments unnoticed. On the other hand, misusing automated testing can add unnecessary overhead and impact your business.
• Making sure that we are investing automation effort in areas that will deliver the largest ROI…basically automating where it counts the most instead of automation for automation’s sake.

What are your biggest concerns around automated testing?

 Here’s who we talked to:

• Murali Palanisamy, EVP and Chief Product Officer, AppViewX
• Yann Guernion, Director of Product Marketing, Automic
• Eric Montagne, Technology PM, Barclaycard
• Greg Luciano, Director of Services and Amit Pal, QA Manager, Built.io
• Donovan Greeff, Head of QA, Currencycloud
• Shahin Pirooz, CTO, DataEndure
• Luke Gordon, Senior Solutions Engineer and Daniel Slatton, QA Manager, Dialexa
• Anders Wallgren, CTO, ElectricCloud
• Charles Kendrick, CTO, Isomorphic
• Bryan Walsh, Principal Engineer, NetApp
• Derek Choy, V.P. of Engineering, Rainforest QA
• Subu Baskaran, Senior Product Manager, Sencha
• Ryan Lloyd, V.P. Products, Testing and Development and Greg Lord, Director of Product Marketing, SmartBear
• Christopher Dean, CEO, Swrve
• Wolfgang Platz, Founder and Chief Product Officer, Tricentis
• Pete Chestna, Director of Developer Engagement, Veracode
• Harry Smith, Technology Evangelist, Zerto