Automation Testing Pyramid Today

The testing pyramid is a familiar concept in software engineering. The model gained greater attention in Mike Cohn's 2009 book, "Succeeding with Agile."

The bottom level of the pyramid consists of unit tests. Unit tests are written by developers and cover method and function behaviors, using test doubles to mock inputs, outputs, and external dependencies.

By applying clean code concepts, these tests are easy to create and maintain. This allows all developers to run these tests locally and check the impact of code changes.

The middle level consists of service tests for APIs and general integrations. These tests cover the composition of code units and their communication with the external world (database, web services, etc.).

The top of the pyramid consists of automated UI testing, which includes end-to-end flows throughout the entire system. Typically, these tests are slow and challenging to maintain.

In general, the core idea is:

• Separate tests with different granularity
• The more high-level you get, the fewer tests you should have

The concept has always seemed simplistic, but the main idea is still relevant, especially in a continuous delivery world. Even with this widespread concept, it is common to see automated tests written incorrectly.

• Tests written in the wrong layer (UI tests checking business logic) can result in infrequent execution due to long durations. They often fail due to issues unrelated to critical functionalities for end users.
• Unit tests are sometimes written solely for coverage and not to test business logic.
• Integration tests that solely check API results (input and output) often neglect to cover the composition between components.

Testing Pyramid Today

The discipline of software development and software architecting has matured and evolved, as well as software testing.

In modern software architectures (such as microservices, micro frontends, and SOA), tests must handle multiple independently deployable components managed by different teams. Valuable features that require thorough testing are often the result of the composition of these different components.

These architectural styles have revolutionized software testing, presenting new challenges and benefits. Today, a testing pyramid needs to include the following:

Unit Tests

The foundation of the test suite should still be unit tests. The unit tests guarantee that units of your code work as intended. and also can help teams to improve their software design because often difficulty writing a unit test can show that a module/class should be broken down into independent pieces.

Measuring the quality of unit testing is important to ensure that the tests effectively cover critical code segments. This is where the concept of mutation testing comes into play.

Mutation Testing tools basically run your unit tests against a modified version of your application code. The expected behavior when code conditionals are changed is that unit tests fail. If a unit test does not fail, it may indicate an issue with the test suite.

Contract Tests

When a system is split into numerous components, communication between these components often occurs through specific interfaces, such as REST, gRPC, or Events/Messaging.

Contracts define expectations for input and output data structures, side effects, performance, and error mapping.

Each consumer forms a different contract based on their requirements. The sum of all consumer contract tests defines the overall service contract. This concept is known as Consumer-Driven Contract.

Tools for writing contract tests:

• Pact
• Pacto

End-to-End Tests

The goal is to test the behavior of the integrated system. End-to-end tests can directly interact with microservices through their public API using an HTTP client. This allows for testing features independently of the UI or channel that utilizes them. This approach proves effective and productive, particularly for companies with omnichannel platforms based on microservices architecture.

Writing and maintaining end-to-end tests can be challenging due to the involvement of multiple components and their various integration methods. Additionally, tests must account for system asynchrony or backend processes between services. Following the testing pyramid concept is crucial. Writing as few end-to-end tests as possible is important, focusing on user journeys and making tests data-independent.

UI Tests

UI tests ensure that the user interface functions correctly. These tests can run independently of the rest of the system and be implemented using tools like Cypress or Selenium.

Exploratory Tests

Exploratory tests should be at the top of the pyramid.

Manually exploring the system with a destructive approach is extremely important. It can uncover issues and scenarios that scripted tests may not cover.

Exploratory tests also allow teams to gain a deeper understanding of the entire system and improve their automated tests.

Conclusion

Regardless of the new layers added to the pyramid, Mike's concepts remain relevant in organizing test suites and making them easy to run within the software development cycle. Following the principles of different test granularities and minimizing the number of end-to-end tests helps maintain effective and efficient testing practices.