To gather insights on the state of automated testing today, we spoke with 20 executives who are familiar with automated testing and asked, "What are the most significant changes to automated testing in the past year?"
While there was a lot of diversity in the answers, cloud-based testing solutions, fast data, and containers were mentioned most frequently.
Cloud-Based Testing Platforms
- Services such as Xamarin Test Cloud and Testdroid are now providing physical device hardware through a web interface, cloud-based Device-as-a-Service model. Previously, using emulators couldn’t be considered as a real-world test since you’re removing the hardware component which plays a big role in performance and load testing. It’s hard to emulate hardware (processor and memory performance.)
- Cloud-based device labs, such as SauceLabs and Browserstack have also continued to grow in popularity and connectors are often included within many UI testing frameworks. These services have helped our team expand our coverage without needed to purchase and maintain a physical device lab.
- Time boxing them. A shift in skillset. When centered around manual testing, it is less technical. Automation has increased the technical proficiency of software testers today. The composition of the development team with more developers taking responsibility for automation. The growth in API testing has been driven by mobile and APIs and the need for a guaranteed level of service. Convergence of test automation and infrastructure you run the tests on – different browser configurations. Tap into service in the cloud.
- So much activity with regards to funding. More investments in cloud-based test automation. Variations of Selenium. More people are using frameworks more comfortably. W3C stands for Selenium – new standard for how web driver evolves with all stable versions.
- Cloud computing has really helped to boost the performance of automated testing. One of the easiest examples to highlight would be in running load tests. Cloud computing has allowed us to test with a vast number of users with ease.
- Two things: 1) Selenium 3 release. They really haven’t added much, but the whole industry now must deal with various backwards-compatibility issues, so this will be a major theme this year. 2) Cloud/grid-based Selenium testing is becoming increasingly mature and popular. It now makes sense for many organizations with lighter requirements around automated testing. We are hoping to see significant improvements in the extensibility of such offerings over the coming years.
- It's getting bigger and going to the cloud. We started in the DR and BC space. Failover test for DR. Run without impacting the production site. People are now using for development testing, for final production versions, upgrades, and patches on servers. Automated test any time of day as much as needed.
- First, with our significant reliance on small tests as the foundation of our testing strategy, we have adopted Mutation Testing in the Java spaces to ‘test the tests’. Mutation testing alters the source code by injecting defects and then executes the unit testing to see if the coverage can detect the faults. This ensures that we don’t get a false sense of security from unit test coverage that doesn’t provide actual value in detecting faults. Second, we have pushed to retire older tools and frameworks in our inventory that don’t support our CI/CD strategy. Our modernized tools ensure that automated tests can integrate into our pipelines and execute unattended with each build. Third, we have built a custom repository and dashboard system to aggregate and report test results, relate the data to our agile management and control systems, and ensure release readiness. Unlike the enterprise systems previously used, the new tools and frameworks are decentralized and unconnected, so we needed to fill the gap to integrate them into a meaningful system.
- Clients are beginning to realize the importance of monitoring and responding to real-time end-user feedback. 25% of mobile apps are dropped after the first use and 90% are dropped within 90 days. Retention and engagement are key. Optimize onboarding and customer satisfaction.
- Layers are built on top of Selenium. Putting AI/ML aspect into automation for continuous improvement. Automation requires maintenance as applications change. It can become expensive with regards to dollars, resources, and time. Very brittle. If the automation doesn’t change as the application changes, the automation breaks and stops the process. We want to leverage AI/ML to smooth out the changes – automate the tests that need to be automated. Have an algorithm learn from test data. No human involvement is required to make changes to the script.
- The digital transition is accelerating development cycles. More testing, more frequently for the ERP, BI like SAP, and digital applications. The backend is less agile than the front end. Traditional business all have ERP and legacy apps to do agile testing on the backend.
- The speed with which we get results. We’re three to five years out from I-ast instrumental JDS.
- Agile, DevOps, TDD and CI/CD. Test processes are automated and are automated more quickly. This provides more insights with data from the tests and improves the stability of the build. Over time automated testing will replace manual testing. Having automation testing at the beginning speeds delivery.
- Automated testing has evolved significantly over the last few years. Our mantra is to provide test automation in a fashion that does not require programmers anymore. This is a significant change. We offer this paradigm not only for UI automation but for API test automation and service virtualization. We also see huge potential for using artificial intelligence for test automation, specifically in UI test automation. Through AI we can drive test automation to a new level. That’s because we can use image-based recognition to detect and interact with controls that are exposed to the user interface. This is, in a sense, the final chapter to UI test automation, and will make UI test automation technology-agnostic, which is huge progress.
- Docker containers are a big deal. They enable you to push around code in a completely fixed environment. The ability to move environments and processes reduces pain. There’s a desire for DevOps and release automation. By pushing an environment around you don’t have to do configuration before deploying – you don’t have to set 67 magic environment variables.
- The rise of container services like Docker have really brought great changes to automated testing, cutting down many of the environment risks that can be associated with automated testing. These services allow for environment parity and allow automated tests to be run in lower environments with greater confidence.
- We’ve continued improvements to our Auto-Heal/Bisect layer of Testing. This falls into the continuous integration area of DevOps and is a core component of our automation strategy. We also have been optimizing our tests to run on lower cost infrastructure. Given that we ship both hardware and software products, it often makes sense to re-evaluate the best and most cost-effective device to provide us the best test coverage. In the past, creating this automation infrastructure was a challenge because we had so much product code and existing environments to deal with. It took a monumental shift in both thought processes as well as technical and environmental support to get us where we are today. The support chain needed to exist all the way from executive level to the implementing engineers.
What are the most significant changes to automated testing from your perspective?
Here’s who we talked to:
- Murali Palanisamy, EVP and Chief Product Officer, AppViewX
- Yann Guernion, Director of Product Marketing, Automic
- Eric Montagne, Technology PM, Barclaycard
- Greg Luciano, Director of Services and Amit Pal, QA Manager, Built.io
- Donovan Greeff, Head of QA, Currencycloud
- Shahin Pirooz, CTO, DataEndure
- Luke Gordon, Senior Solutions Engineer and Daniel Slatton, QA Manager, Dialexa
- Anders Wallgren, CTO, ElectricCloud
- Charles Kendrick, CTO, Isomorphic
- Bryan Walsh, Principal Engineer, NetApp
- Derek Choy, V.P. of Engineering, Rainforest QA
- Subu Baskaran, Senior Product Manager, Sencha
- Ryan Lloyd, V.P. Products, Testing and Development and Greg Lord, Director of Product Marketing, SmartBear
- Christopher Dean, CEO, Swrve
- Wolfgang Platz, Founder and Chief Product Officer, Tricentis
- Pete Chestna, Director of Developer Engagement, Veracode
- Harry Smith, Technology Evangelist, Zerto