Software development even with UI automation testing requires manual regression testing, which costs a lot of time and resources. The simple reason is that most UI automation frameworks out there only test the functional part of your UI and not the look and feel of the UI. Your automation can run perfectly, yet some wrong CSS change messes up the entire page. This causes managers to invest a lot of time in manual regression testing before each release. With DevOps and short release cycles on the rise, this is becoming a costly problem.
A new solution?
Lately I found new software that allows to almost eliminate the need for manual regression testing before releasing a new version (assuming you have UI automation), by integrating into your existing UI testing framework. It’s called Visual CI, It will save reference images during the UI automation flow and on next runs will compare and validate the new images according to certain settings. Most importantly it will allow you to select what parts of the image should be compared (Nothing more annoying than a random ID corrupting your tests)
So if you have UI automation covering your functional requirements, and during that automation you take screenshots that are validated as well, the trust you have in automation increases dramatically.
I am very excited about this new solution, there is always some talk on the web on comparing snapshot images during automation, but it’s never a full simple solution, It’s either some concept described, or an over complicated framework that requires rewriting all your automation (Or companies telling us they do it internally, while keeping it to themselves, Google, SalesForce, etc.). Just thinking of the yearly cost in manual testing this tool can save, gives me Goosebumps.
But the question rises, does it really work? Yes, it does.
The application comes with a simple intuitive way to mark regions of unstable or random data, for example clocks, generated Ids, or random data.
Taking it out for a spin
I tried using it and the sample project runs pretty quickly and shows you the image compare abilities.
In the code, it looks like this:
//Use web driver to get the browser image (during a UI automation test) byte image = webDriver.getScreenshotAs(OutputType.BYTES); //Use the Visual CI library to compare and validate the image VisualCI.validateImage("MyImageId", "default", image);
When a mismatch occurs, it will show the original reference image, and validate image, a pixel diff image. It will then allow you to
- Fix your bug (if it’s a real defect).
- Mark region/s as ignored to avoid random data interfering with the test
- Make the new image into the reference image (Intended change)
- Mark the test as ‘ignored’
|Reference Image||Validation Image||Pixel diff image|
What's next?The tool is new and is still missing some important capabilities; it doesn't support LDAP integration and some other enterprise requirements; however it is a serious move in the right direction
For the purpose of interesting discussions, I suggest you try it, and post your response here, with answer to these 2 questions:
- Will screenshot compare tool reduce regression testing in your app/organization?
- Does this tool give enough value to use?
Get it at: http://visualci.com