{{announcement.body}}
{{announcement.title}}
Refcard #326

End-to-End Testing Automation Essentials

E2E testing exercises an application in the same manner a user interacts with the system, and involves all components — such as hardware, networks, external services, and configuration that would be seen in a production environment. This is a long manual process that requires significant effort. Then came E2E automated testing, providing the same level of functional validation performed by a computer, but in a dramatically shorter amount of time. In this Refcard, we evaluate E2E testing automation, processes, and key benefits upon adoption.

Published: Mar. 26, 2020    |    Modified: Mar. 31, 2020
2,572
Free PDF for easy Reference

Brought to you by

Worksoft
refcard cover

Written by

author avatar Justin Albano Software Engineer, Catalogic Software, Inc.
asset cover
Refcard #326

End-to-End Testing Automation Essentials

E2E testing exercises an application in the same manner a user interacts with the system, and involves all components — such as hardware, networks, external services, and configuration that would be seen in a production environment. This is a long manual process that requires significant effort. Then came E2E automated testing, providing the same level of functional validation performed by a computer, but in a dramatically shorter amount of time. In this Refcard, we evaluate E2E testing automation, processes, and key benefits upon adoption.

Published: Mar. 26, 2020    |    Modified: Mar. 31, 2020
2,572
Free PDF for easy Reference

Written by

author avatar Justin Albano Software Engineer, Catalogic Software, Inc.

Brought to you by

Worksoft
Table of Contents

What Is End-to-End Testing?

Automation Process

Key Metrics

Key Benefits

Conclusion

Section 1

What Is End-to-End Testing?

E2E testing exercises an application in the same manner a user interacts with the system, and involves all components — such as hardware, networks, external services, and configuration that would be seen in a production environment. Since E2E testing exercises the entire system, it is often synonymized with system testing; however, it’s a very different. According to Quora, system testing is “conducted on the entire system that consists of numerous coordinated components. It helps you ensure that the system meets functional requirement specifications and operates as designed to satisfy user’s expectations.”

E2E testing is typically manual user acceptance testing. This is a long manual process that requires significant effort whereas E2E automated testing provides the same level of functional validation performed by a computer, but in a dramatically shorter amount of time. Let’s evaluate some very important differences between these two levels of testing.


This is a preview of the End-to-End Testing Automation Essentials Refcard. To read the entire Refcard, please download the PDF from the link above.

Section 2

Automation Process

E2E tests are often manual processes executed by a testing staff. This is a cumbersome task, requiring the system to be deployed by hand and for test engineers to use the UI as if they were an end-user. 

There are multiple problems with this approach: 

  1. Inefficiency: Testers tediously click and type through a UI or manually execute requests to an API. 
  2. Repeatability: Testers are likely to make a mistake or miss a step over time. 
  3. Integration: It is difficult to integrate the results of a manual test with other automated tools, such as Continuous Integration (CI) and Continuous Delivery (CD) pipelines, given that the point of CI and CD is to be unattended without human intervention. It is hard to include manual testing in an automated CI/CD pipeline given they are automated and happen on a continuous basis. 
Section 3

Key Metrics

To measure the effectiveness of the E2E tests, as well as gain insights through the execution of these tests, metrics should be collected each time E2E tests are run. These metrics can include: 

  • Execution Time: Time taken to complete the test. This time will resemble the time it will take the user to complete the workflow in production, but with some major differences. Since the test environment has less resources, but under less load, it is best to devise some mathematical estimate of production performance as a function of test performance: ETP = cET. EP is the execution time in a production environment, c is some constant factor, and ET is the execution time in the test environment. For example, there may only be one database in testing which increases the execution time by a factor of 2, but there may be 1,000 times less requests per second, which may reduce execution time by a factor of ¼. Additionally, execution time should be measured statistically, using a confidence interval with some acceptable α, since execution time will likely fluctuate between trials.
Section 4

Key Benefits

E2E testing may require a substantial up-front cost, but the benefits will likely outweigh this price for most significant applications. As we have previously seen, these benefits include: 

  • Assurance of Correctness: Since E2E tests mimic the behavior of users in a production-like environment, we can gain confidence that so long as the E2E tests pass, the user is unlikely to encounter a problem in the product. 
  • Test Coverage: E2E tests exercise an entire workflow or use case and involve many of the components of the system — including external components; thus, the coverage provided by these tests can be much larger than unit and integration tests, and may even be noticeably higher than system tests.
Section 5

Conclusion

E2E tests are an essential part of any application test plan, but these large-scale tests are often overlooked due to a mistaken conflation with system tests or their tedious nature when manually executed. By introducing automated E2E tests, developers, testers, and business personnel gain the confidence that the product can be exercised in the same manner as a customer would in an environment that closely resembles the production environment. 

This confidence alone, supplemented with the repeatability, speed, ease of understanding, and integration with existing testing automation, makes automated E2E testing well worth the price of admission. 


This is a preview of the End-to-End Testing Automation Essentials Refcard. To read the entire Refcard, please download the PDF from the link above.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}