Why You Don't Have Enough Test Automation

You probably don’t have enough end-to-end tests for your product and would love to find out what you can do about it.

But why?

Aren't the end-to-end tests the most important because they validate from an actual end-user perspective that functionality actually works?

Isn't it like using an ax instead of a fork just because it is closer?

Well, if you are using Selenium/Appium for automation then you probably noticed that it takes quite an effort to build tests. Indeed, based on our statistics with companies we talked to a human on average would be able to produce about 100 tests per year. However, if you ask how many test cases there are people usually cite thousands. So it is slow. 

Let's calculate the price of the test in this scenario. The QA Automation Engineer salary in USA is $78K on average. A person can build about 100 tests per year. Therefore, each test costs, on average $780 to build. That is shockingly expensive.

I believe, though, the number one problem with those tests is in test maintenance. What we have seen is after a year of actively building tests more than 50% of the time allocating for building tests is spent on maintaining existing tests instead. In general maintaining tests becomes a daunting task eating up a lot of time all the time. That would result in 2 things happening:

1. The price to build a test will double every year to $1,500, $3K, etc.
2. If calculated over the course of 10 years of maintenance a person would be able to build 50 tests on the first year,  25 on the second, 13 on the 3rd, 7 on the 4th, 4 on 5th, 2 on 6th, 1 on 7th and on years 8, 9, and 10 the person will be 100% of the time bogged down with test maintenance. Giving us the grand total of 100+50+25+13+7+4+2+1 = 202 tests build over the course of 11 years, making the maintenance of each test being ($78,000*11 - $780*202)/202 = $3,468 per test or 445% of the cost of building the test itself. The total cost of ownership of the test would be $3,468+$780 = $4,248

And, finally, the last argument is that they are slow to execute. Indeed if you are running these tests consecutively it might take hours (or even days) to complete. 1,000 tests each running 2 min on average would run 2,000 min or 33 hours. Which is way longer than even the duration of a nightly build.

So, therefore we should restrict the number of end-to-end tests? Nope.

I personally believe that having more end-to-end tests is very valuable from a business perspective. What if these aforementioned problems can be solved? Would it make sense to have more of those tests then? I strongly believe that yes, it would make sense.

Why Do We Have All These Issues?

Let's dig deeper into why we have all of those problems:

1. Speed and price of test creation. Well, what would you expect if you are writing basically a low-level code interacting with the browser using a low-level Selenium protocol directly?

2. Test maintenance problem comes from 2 facts. Number one - the aforementioned low-level language that would require a lot of changes in a lot of places to adapt to any meaningful change. And number two - treatment of the application under test as a white-box relying on internal concepts like XPaths, CSS Selectors, IDs that can change at any moment without having any effect on how the application functions from an end-user's perspective. Result? Tests fail even though everything is fine from an end-user's point of view.

3. Slowness. Well, as soon as tests are created the way that they can run in parallel it shouldn't be a problem - you can run them in parallel, right?

Solution

How would you solve the speed problem without sacrificing the test maintainability?

I believe the solution is to execute the specifications similar to the language you use as a user to describe how to find elements and interact with those elements on the page.

Wouldn't it be fun to use just plain English to be able to create something like this to find a product on Amazon:

xxxxxxxxxx

enter "Accelerate" into "Search"

click "Go"

click "Accelerate: The Science of Lean Software and DevOps"

Or create a lead on Salesforce:

xxxxxxxxxx

login

click "Leads"

click "New"

select "Mr" from "Salutation"

enter "Peter" into "First Name"

enter "Pan" into "Last Name"

enter "testRigor" into "Company"

click "Save"

check that page contains "Lead \"Peter Pan\" was created"

without a need to write any Selenium or any other code to actually implement it?

I think it should be obvious why the amount of test maintenance is dramatically reduced here. It is just one-to-one with end-user actions.

Pair it with the ability to record it with a Chrome Extension and you get yourself a pretty fast way of building tests too.

And, I believe, it should be obvious to anyone how to reduce the execution time. Just run them all in parallel!

Think about it, if you can increase the speed of test creation 20X and almost eliminate test maintenance bringing down the cost of ownership of a test to just $40 instead of $4K wouldn't you do it?