DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How are you handling the data revolution? We want your take on what's real, what's hype, and what's next in the world of data engineering.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

SBOMs are essential to circumventing software supply chain attacks, and they provide visibility into various software components.

Related

  • Exploring Mobile Device Lab: Pros and Cons
  • End-To-End Testing Unveiled: Navigating Challenges With a Personal Experience
  • Automated Testing: The Missing Piece of Your CI/CD Puzzle
  • 10 Things to Remember Before You Say Yes to Automation Testing

Trending

  • Build Your Private Cloud at Home
  • Design Guards: The Missing Layer in Your Code Quality Strategy
  • How to Write for DZone Publications: Trend Reports and Refcards
  • Apache Cassandra With Java: Introduction to UDT
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Testing, Tools, and Frameworks
  4. 5 Barriers to Successful Test Automation

5 Barriers to Successful Test Automation

Time, repetition, and overburdened engineers are among the reasons enterprises have difficulty implementing test automation.

By 
George Blundell user avatar
George Blundell
·
Jul. 11, 19 · Presentation
Likes (5)
Comment
Save
Tweet
Share
11.5K Views

Join the DZone community and get the full member experience.

Join For Free

Image title


Organizations today have long understood the need to automate test execution, and 90% believe that automated testing allows testers to perform their tests quicker. Yet, QA teams are struggling to achieve sufficiently high rates of automated test execution. Slow and overly manual testing still abounds.

In 2018, 61% of organizations had automation rates lower than 50%. This article considers five reasons for these low rates of functional test automation, setting out some of the most common pitfalls to watch out for when adopting a test automation strategy.

1. Slow and Repetitious Test Creation

Slow and repetitious test creation is the primary source of test automation bottlenecks. This is because tests must exercise every single combination of user activity and data, and writing test scripts is therefore repetitious, slow, and labor-intensive. There are many overlapping test steps that need to be written to provide full coverage, and numerous new tests are required at the start of each new iteration. Automation engineers are therefore always trying to play "catch up" if they write scripts by hand, and the time spent scripting tests often outweighs the time saved during execution.

Test Automation Script
Figure 1: A scripted page object in a test automation framework. How many of these can an engineer write manually per day?


2. Low Test Coverage

Automated testing often has unacceptably low test coverage, leaving systems vulnerable to damaging bugs.

If automated tests are derived manually from imprecise and incomplete system requirements, testers cannot definitively say how "good" their tests are, nor when they have tested "enough." The tests are not measurable against requirements, making it impossilbe to know how much of a system is being covered by automated tests.

Manual test creation, therefore, leads to test coverage as low as 10-20%. The founder of the first test automation framework, Dorothy Graham, questioned automated tests’ ability to improve test coverage, saying: "It is the quality of the tests that determines whether or not bugs are found, and this has very little, if anything, to do with automation."

3. Bad Test Data

Thirdly, bad test data can undermine the speed and rigour of test automation frameworks.

For rigorous testing, test data must be available, comprehensive, and accurate. This is rarely the organizations, and 53% of the respondents to the World Quality Report in 2018 said they had a lack of appropriate test data.

Test data for most companies still means large, masked copies of production data drawn from past user activity. These copies cover only a small range of possible tests, while testers have to then search through these large "dumps" of production data to find suitable data sets. This is highly time-consuming and potentially pointless, as suitable data combinations do not always exist.

barrier to successful automation test data
Figure 2: Inaccurate and low variety test data undermines the stability and rigour of test automation frameworks.


The manual data hunt is also error-prone, leading to bad data that destabilizes automation. Test failures stemming from invalid data meanwhile flag defects that do not exist in the code, leading developers on a wild goose chase.

4. Test Maintenance

Test Maintenance is arguably the greatest barrier to successful test automation adoption.

Manually created tests are extremely brittle to system changes. Every time the system is updated there might be thousands of regression tests that might have been rendered invalid. Test engineers must then identify the impact of changes made to components across the complex system, checking which tests have been by the change. This is a vastly complex and error-prone process.

Barrier to successful test automation maintenance
Figure 3: Impossible test maintenance – arguably the greatest barrier to successful test automation.


Automation of test maintenance is, therefore, a must for rigorous testing in-sprint. However, automating maintenance is not possible when tests have been manually derived from the system requirements, as there is no formal link between the latest system designs and the tests derived from them.

In the image above, a change request is submitted and added to a bag of unconnected requirements that are not formally mapped. Engineers then must identify which tests have been impacted by each change request and update them accordingly. However, the system has more moving parts than any one person can comprehend in their heads. Test maintenance is therefore both time-consuming and error-prone.

5. Over-Burdened Engineers

Engineers with automated testing skills tend to outnumber those with only manual testing skills at organizations. This results in a core group of engineers being tasked with the whole organization’s automation, leading to a small team being severely over-worked.

Model-Based Test Automation: Automate More, Automate Faster, Automate Accurately

These five challenges present a significant barrier to achieving sufficient levels of test automation. Unless they are overcome, the potential ROI of model-based test automation will not be realized by many companies.

Here at Curiosity Software Ireland, we have developed The VIP Test Modeller, an automated test software that has conquered the challenges introduced in this blog. To find out how we have overcome these challenges, please sign up to watch our webinar on July 11th. Join Curiosity directors Huw Price and James Walker as they demonstrate how Model-Based techniques can overcome these barriers to successful test automation.

Resources

[1] Panaya (2018), The State of Functional Testing Today. Cited from https://www.devopsonline.co.uk/14159-2-ai-and-automation-vs-human-testers/. Retrieved on 1 July 2019.

[2] Panaya (2018), The State of Functional Testing Today. Cited from https://www.devopsonline.co.uk/14159-2-ai-and-automation-vs-human-testers/. Retrieved on 1 July 2019.

[3] Dortothy Graham and Mark Fewster (2009), That’s No Reason to Automate. Retrieved from http://www.dorothygraham.co.uk/downloads/generalPdfs/NoReasonAut.pdf on 1 July 2019.

[4] Capgemini, Micro Focus, Sogeti (2019), World Quality Report, 11. Retrieved from https://www.sogeti.com/globalassets/global/wqr-201819/wqr-2018-19_secured.pdfon 19 June 2019.

[Image: Pixabay]

Test data Test automation Barrier (computer science)

Opinions expressed by DZone contributors are their own.

Related

  • Exploring Mobile Device Lab: Pros and Cons
  • End-To-End Testing Unveiled: Navigating Challenges With a Personal Experience
  • Automated Testing: The Missing Piece of Your CI/CD Puzzle
  • 10 Things to Remember Before You Say Yes to Automation Testing

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: