Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Just Say Yes to More End-to-End Tests

DZone's Guide to

Just Say Yes to More End-to-End Tests

End-to-end tests have flaws that make this style of testing easy to dismiss. However, if you flip your thinking, these flaws can be strengths.

· DevOps Zone
Free Resource

The DevOps Zone is brought to you in partnership with Sonatype Nexus. The Nexus Suite helps scale your DevOps delivery with continuous component intelligence integrated into development tools, including Eclipse, IntelliJ, Jenkins, Bamboo, SonarQube and more. Schedule a demo today

You have probably seen this testing pyramid:

This particular image came from an article from titled Just Say No to More End-to-End Tests, which goes into some detail into the various failings of end-to-end tests and why unit tests are a far better alternative. These reasons were neatly summarized in the following table.


Unit

End-toEnd

Fast

Reliable


Isolates failures

Simulates a real user

Looking at this table and reading the comments to the article, it became clear that the whole “say no to end-to-end tests” mindset came from the point of view of a developer who prioritized quick and unambiguous feedback over the slow, flaky, and often vague results generated by end-to-end tests.

Lut let’s take a look at end-to-end tests from another point of view.

Fast

There is no reason for end-to-end tests to be so slow that developers can’t run them. There is no denying that a unit test will always be faster than an end to end test, but no one is saying that you have to stop development and watch the end to end test run.

For my own development, I have configured a personal CI server. It maintains a development environment complete with the suite of applications that I may need to test against and runs end-to-end tests upon detecting a commit. I don’t have to sit there and wait for the test to complete and I don’t even have to concern myself with the results until such time as I feel that I have reached a point where I consider my code to be working, at which point I jump into my personal CI server and inspect the report.

I suspect the stigma around the speed of end-to-end tests highlights the fact that unit tests are very well supported by IDEs and developers can quickly execute them to get meaningful results, while end-to-end tests are typically run by some central server on an inconvenient schedule. However, it doesn’t take that much to configure a personal CI server, and once that is done, end-to-end tests are actually quite easy to introduce into your development workflow.

It should also be pointed out that only developers do anything with the results of unit tests. Developers may well be able to quickly run, interpreted, code, and rerun unit tests, but unit tests mean nothing to the rest of business. It is much faster for a business analysis to read the results of an end-to-end to identify flaws and act accordingly.

Reliable

The Google article describes end-to-end tests as unreliable and flaky. In my experience, this is true, but you have to ask why end-to-end tests are flaky. Usually, it is because these tests are subject to all the uncertainties of the underlying infrastructure. Network performance, server load, browser incompatibilities, and the availability of every piece of infrastructure that hosts the final application come into play, and writing tests that can work under these circumstances is difficult.

However, if it is hard to write reliable end-to-end tests, then you have just admitted that it is hard to deliver a reliable experience to your customers because they experience the same uncertainties and instabilities that your end to end tests struggle to account for.

The technology that is used to power end-to-end tests is not inherently unreliable. Libraries like Selenium have been around for quite some time, and while new browser versions and drivers bring their own headaches, it is not hard to build a testing environment with fixed versions that can be relied upon to emulate a user's interactions in predictable ways. If end-to-end tests are flaky and unreliable, it is a sign that the environment they are testing is flaky and unreliable, and this shouldn’t be dismissed with a simple hand wave by labeling end-to-end tests themselves as flaky, unreliable, and not worth implementing.

Isolates Failures

I don’t think this feature of unit tests can be disputed. Unit tests focus on very small pieces of functionality and take every measure to isolate themselves from outside influences by mocking services and using dummy data.

The flip side of isolating failures is that you are isolated from failures. If you have mocked your database access as part of a unit test, the unit test will correctly pass even if the database is down. (Not that this means anything to your customers, who see error screens instead of a payment page.)

Screen Shot 2016-12-21 at 10.58.31 am.png

A single failing isolated unit test is a very fast and reliable way to identify a very specific problem. However, a single passing isolated unit test only provides a very small guarantee that the end product will work as expected.

Because they are not isolated from failure, end-to-end tests will typically fail if any of the underlying infrastructure or assumptions fails. It might be hard to understand why it failed, but it is very easy to detect that something has failed. This means end-to-end tests are far better at detecting unanticipated errors even if they aren’t so great at informing you of the underlying reasons.

Monitoring

Unit tests don’t tell you if your website is working. Integration tests might call out the fact that some services are failing, but do you run integration tests often enough to catch a significant outage?

By contrast, running end-to-end tests regularly in production gives you a very robust way to continuously verify the end-user experience. You are forced to write reliable tests and in the process will identify those aspects of your infrastructure that make writing reliable end-to-end tests so difficult. The end result is that the tests can be relied on by developers, that you'll have a more reliable end-user experience, and that the tests can also monitor production systems.

This comment to the original article has a great insight into the relationship between testing and monitoring:

I would argue that, at least in continuous deployment where MTTR (Mean Time to Recovery) is far more important than MTBF (Mean Time Between Failures), monitoring take precedence over tests. I would draw yet another pyramid - a monitoring pyramid - on top of the testing pyramid such that 70% is application level monitoring, 20% host monitoring and 10% KPI.

A New Summary Table

I would propose that most of the aspects of end-to-end tests that were identified as limitations by the original article are actually strengths when seen from a different point of view. So, the summary table should look more like this:


Unit

End to End

Instantly accessible by developer (was “fast”)

Accessible by non-developers

Reliable

Reflects unreliability experienced by end user

Isolates failures

Detects unanticipated failures

Tests development builds and monitors production builds

The DevOps Zone is brought to you in partnership with Sonatype Nexus. Use the Nexus Suite to automate your software supply chain and ensure you're using the highest quality open source components at every step of the development lifecycle. Get Nexus today

Topics:
software testing ,end to end testing ,devops ,software development

Published at DZone with permission of Matthew Casperson, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}