Automated Test Changes

To understand the current and future state of automated testing, we spoke to 14 IT professionals intimately familiar with automated testing. We asked them, "What are the most significant changes to automated testing over the last year?"

Here's what the respondents told us:

DevOps/CI/CD

• 1) As we have adopted an Agile methodology of development, we have to have an Agile testing strategy. Continuously improve automated tests and integrate them into the CI/CD pipeline. 2) Another change is fast feedback by running tests in the development cycle so we can find defects during development. We started providing cloud offerings with UX feedback. UI automation is now a core part of our automation test suite. 3) As we adopt microservices, we have also broken down our monolithic automated test repository into smaller reports managed by individual teams. This has helped us scale and release features quickly. We have also built a contribution model for automation for individual teams to test their own repos.
• People are starting to realize the importance of developing custom solutions. Organizing to a more product-centric approach, break down silos, and have more shared responsibility. More than technology, it’s about culture and philosophy. We're also seeing a focus on DevOps with security integrated from the beginning. 
• Automated testing has gained a more critical role in the software development process. Before, test automation was considered a nice-to-have and was typically developed in a silo without much visibility into the results. Now, automated tests have become the gatekeeper for teams who are integrating or deploying continuously. The status of automated tests determines whether customers will receive this feature right now or not.

Embedded

• Automated tests are becoming embedded in the developer process. We are assimilating to the developers' technology – with the Jenkins plugin, automated security testing is done in preproduction. Full testing is taking place in the SDLC before production.

API-Centric

• We see a shift from UI automation to API automation. Automation used to be automating a complete workflow; however, this does not help from a DevOps perspective where small modular design automated tests are needed. We’re still seeing Selenium for browser-level automation. Also, we’re seeing a movement toward scriptless automation tools. Tool vendors are expanding on scriptless automation. 
• In the industry, with the evolution from monoliths to distributed systems it is more and more common to focus on testing the contracts between the APIs and put more efforts in post-deployment verifications in production. The reasoning behind this is reacting quickly to outages is more important in such architectures.
• Some of web testing is getting less Selenium-centric. Also testing via REST API’s is getting more attention.

Other

• Organizations are trying to do more with less and are experimenting more with artificial intelligence (AI) and machine learning (ML) to optimize testing and test strategies. They are also focusing more on how much business value they can deliver at speed, by connecting and measuring all of their end-to-end activities, including those related to automated testing, across their portfolios.
• 1) A big thing in technology right now is people taking AI, algorithms, and neural nets and using them to make a meaningful difference. 2) There is far more focus on the end user. Companies are looking forward rather than backward. More people are thinking about testing software to see that it’s good. They are determining if it will help customers do what they want to do. The stars in the app store have become more important as has knowing whether or not users accomplishing what they want to do. There has been a broadening of what people are testing – performance and usability to improve UX and CX. LOB owners want to see test results, the impact on UX and CX, more shopping, more articles read. There is an opportunity to see real ROI from automated testing. Testing should be a profit center. Be able to determine that you increased the revenue of an app by X%. Go one step further to focus on the business outcome to validate what you are doing. Monitor website or app, conversions, all technical factors. Use AI/ML to see correlations. Quantify the effect of testing on outcomes and users.
• In past years, we have seen the most significant changes in all aspects automation testing including strategy, tools, and skills. 1) Strategy: We focus on business test scenarios automation rather than test cases. Shift left strategy: In-Sprint automation to align with DevOps and sprint based development and Shift towards Service level/API automation rather than at UI only Layer. Progressive test automation with CI/CD pipelines. Service vitalization, test data, and test environment automation. Automation for business outcomes instead of focusing % test case automation. 2) Tools: Adaptation of freeware tools to reduce automation cost. Integration with build deployment, requirement management, and defect tracking tools for test lifecycle operations automation. 3) Skills: Re-Skilling of the testing workforce on emerging automation tools and frameworks.
• A lot of startups are looking for more standard ways of doing things. They do not have the time or money to explore industrial solutions. Open source choices with more universal tests are gaining favor.
• Functional UI test automation for mobile web apps. Over the past year, Selenium 4 has been a pain to build scalable reliable test automation practices across platforms. We’re playing “whack-a-mole” with reliability issues. Look at unifying protocols together to reduce frustration. This is helpful for device cloud providers since you need to support all browser combinations to commoditize the device cloud market. Headless stuff is starting to make its way into device cloud providers. Grab data in real time for automated analytics testing. Interact with UI in real-time by integrating with the device cloud. There are a lot of improvements with Appian. Test outside the pure sandbox of the application. Now simulators and emulators are able to interact with face and touch.

Here’s who shared their insights:

• Drew Horn, Senior Director of Automation, Applause
• Angie Jones, Senior Developer Advocate, Applitools
• Isa Vilacides, Director of Engineering, CloudBees
• Himanshu Dwivedi, CEO, Data Theorem
• Antony Edwards, COO, Eggplant
• Kevin Fealey, Senior Manager Application and Product Security, EY
• Hans Buwalda, CTO, LogiGear
• Malcolm Isaacs, Senior Solutions Manager, Micro Focus
• Madan Mohan, Global Head of Travel and Transportation, NIIT Technologies
• Jared Go, CEO, OhmniLabs
• Derek Choy, CIO, Rainforest QA
• Nancy Kastl, Executive Director of Testing Services, SPR
• Rishikesh Palve, Integration Product Manager, TIBCO
• Ray Wu, CEO, Wynd