Over a million developers have joined DZone.

Stop Investing in Manual Regression Cycles

DZone 's Guide to

Stop Investing in Manual Regression Cycles

· DevOps Zone ·
Free Resource

Manual regressions are an old necessity in the software industry or are they? Today in a mutli-browser world and short release cycles on the rise the amount of time spent on manual regressions is rising as well. It’s not that companies are reluctant to invest in Testing & Automation (Server & UI); it seems that these investments do improve quality and quick detection of defects/regressions. However companies are having a hard time trusting these automations processes in such a way they can release a version of their software without a manual regression cycle (‘blind’ release). The problem is centered down to the visual validation of the UI, Selenium and other popular UI testing tools validate the functional part of your application, not the way it looks to an end-user. UI validation tools like Visual CI  are the solution to decreasing your manual regressions cost.

Return on investment

Let’s try and visualize the amount of effort we invest in software development, with a focus on R&D/Testing. The table below calculated off a poll of 50 companies that do invest in automation.

  • Application – All Development, new feature testing, unit tests, manual QA validations on new features
  • Server automation – Creating and maintaining server side automation (Tests that require DBs and most of the application components running)
  • UI Automation – Creating and maintaining UI side automation (Tests that require DBs and most of the application components running), yes it’s expensive.
  • Manual (Human) regression testing – all the manual tests you do on already existing features; from weekly/daily sanity tests to pre-release full regression testing.

Development & Upkeep

Server automation only

UI Automation

UI automation & Visual automation





Server automation




UI automation




Visual automation




Manual(Human) regression testing




App quality on release




If you look at the numbers in the first 2 columns of the table, you will very quickly notice that companies that start investing in UI automation do increase in quality, however the additional cost of UI automation (which is never cheap to develop & maintain) hardly decrease the time spent on manual regression testing. This is a very common pitfall; companies do not understand the high cost of UI automation and the fact that the popular UI automation frameworks on their own do not give enough confidence to ‘blindly’ release the application.

UI Functional testing – not enough

Why is UI functional testing not enough, let’s look at the following 2 images taken from the Google search page. The 1st image, the good snapshot, is the way we want to page to look. The 2nd image, the bad snapshot, has multiple parts of bad corrupted Look & Feel due to a small change in CSS (This is a real scenario that happens often in HTML development; changing a CSS in one part of the application can easily have unexpected results on another part of the application). There is a good chance the corrupted app will pass our selenium testing, the specific buttons are still visible enough to click, the icon change will not be detected. Change in font size will probably not be detected as well.

So now we have a problem, we see that we cannot trust the functional aspect of UI automation, if we had, this corrupted page would have been release to the customers, naturally it would not go unnoticed; Depending on the nature of your application it may trigger a few laughs all the way to customers losing confidence in your R&D.

Solution – Visual automation

If we take another look at the table above, we’ll notice that a small investment in visual automation (pluggable visual automation over existing UI automation framework), will actually decrease your manual regressions and let you spend and increasing amount on time on actual application development.

How is this done?
The correct way to implement visual automation is to take snapshots at certain points in your UI automation (Which should already be opening the different browsers and going through your application).
These snapshot, will usually require some basic configurations (How strict equality are we looking for? Which regions of the image to ignore due to random data?). From that point on every time the UI automation flow runs, it should compare the current snapshot to the original one, if it’s successful, continue. If it fails your developer/QA should have the following options

  • Real defect – fix (consider ignoring the snapshot for a short time until fixed)
  • New image is correct – accept changes
  • Random data changes – ignore region.

How to start

When you start, it’s always important to start simple; Use your visual automation to take snapshots of your main screens, ignore the random regions in your screens and let it run. Over time increase the coverage and consider taking more complex snapshots like: popup dialogs, different I18N.

You will see that once you start using the visual automation, your need for manual regressions will decrease and the confidence in the automation process will rise allowing you to invest more in development.

A few tips

·  Do not attempt to use cross browser image validation tools – they don’t work; browser engines update often. Take a different screenshot for every browser

·  Do not use manual visual compare solutions, Even with diff maps, humans are not good at looking at a hundreds images and looking for good vs. bad faults on a weekly basis.

·  Make sure your visual automation tool you choose can support the logic of multiple branches; Tools like Visual CI  provide such abilities.

·  Make sure the visual automation tool you choose can show you the history of a specific snapshot.

Visual automation tools that plug into your existing UI automation solution

1.  Visual CI  – A new E2E solution that provides a UI interface for seeing results as well as per snapshot configuration(Strictness, ignored regions, etc.)  - Keeps snapshots, configuration and execution history in DB

2.  PhantomCSS  – a nice but raw solution, no UI interface- you are responsible for maintain all the snapshots configuration and history

Also see my previous post 

I hope you enjoyed this post.



Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}