Test Plan and Test Strategy: Best Practices That Will Make Your Product Development a Success

Without a crystal clear understanding of the processes when a team works on a software product, it can be tempting to think that all the problems stem from under-qualified QA engineers who click around randomly and ruin the hard work of the whole team. However, the value and purpose of the quality assurance process are not transparent without documentation. That's where a test plan and test strategy can help.

Even for those who are well aware of the processes, there's still the problem of measuring the quality QAs provide. If you don't measure quality, you don't have any control over the testing process or any ability to anticipate the results. So how should you know what exactly you paid for?

A test plan is an estimate subcontract, which allows you to plan expenses and resources over a long period, plan budgets, analyze performance, and increase product value. Having QA documents contributes to a structured process, which is widely accepted as a best practice. Still, many teams skip this step in their products.

In this article, I’d like to discuss the benefits of maintaining a test plan and test strategy and the most useful elements of each document for the team.

Test Plan and Test Strategy: Why You May Need Them

I have seen dozens of test plans and strategies and can confidently declare that there is no single correct, universal document that can be taken as a standard and applied to all types of projects. The content of these documents may differ from project to project, and the documents themselves can either exist separately and refer to each other, or the test strategy can be part of the test plan. Since the test plan is updated often, and the test strategy usually remains during the project, I prefer to divide them into two documents.

Below is a list of the sections to include in these two documents so that the whole team can get the most out of them.

Test Plan

• Scope of work
• Quality & acceptance criteria
• Resources (team, tools, environments)
• Test documentation and deliverables
• Risk assessment
• Process description

Test Strategy

• Testing approach
• Testing levels
• Testing types
• Compatibility testing priorities
• Impediment mitigation
• Testing phases
• Release verification
• Reporting
• Hotfix
• CI/CD testing pipeline

The value of implementing these documents affects the whole team in the broadest sense of the word, meaning everyone involved in product development. With the help of a test plan and strategy, it is easier to understand what a QA team does and coordinate work with other teams.

You Need to Grow Your Team and Don't Want to Tie All Product Knowledge to Individual Team Members.

With a well-written test plan and test strategy, the team has a unified understanding of all testing processes on the project. As a result, it can perform work efficiently even without management. Additionally, high-level documentation helps quickly get newbies up to date and synchronize the distributed team.

For example, onboarding typically includes the number of non-production hours (rate multiplied by hours), the manager's time for introductory meetings, and the time of other specialists who bring the newcomer up to date. Test documents allow you to reduce the number of hours that other departments' specialists spend on onboarding by prescribing parts of the processes in an industry-accepted format, which also saves time reading such documentation.

You Want to Design Project Timing

QA documents allow PMs to get a complete picture of what  QA engineers are testing, how the testing team will do the expected quality, and how much work at each product development phase without having in-depth knowledge of testing.

You Want to Better Control Risks

The test plan unveils what the team is responsible for and what is not under its control. For example, it lists 3rd-party services and products, boundary cases that cannot be captured in the test environment, etc. This allows for management not only of risks but of expectations as well.

You Want to Predict Product Quality

Several times in my professional life, I’ve encountered situations where the product I was managing became a partner with other major financial or medical products. Such organizations often request documentation that entirely regulates product development (including risk management, business continuity plans, product development roadmaps, etc.) They also request to be shown what set of measures the team takes to obtain the predicted quality of the product. A well-written test plan and test strategy fully cover this request in most cases.

You Want to Access New Markets and Customers

The data safety and security requirements of the entire process are very high in such industries as finance and healthcare. That's why they often require audits and certifications, which involve formalizing all processes of both the partner's team and the product development team.

Audits and certifications directly affect business expansion as they provide access to more clients. Investors can also request such audits in preparation for an IPO and conduct a Know-Your-Customer (KYC) procedure that ensures the company was never involved in crime or is not subject to any restrictions.

A test plan or test documentation may be required to cover the following gaps: development process, employees (resources), data integrity, business continuity, incident response, third-party reliance, operational resilience, infrastructure network, etc. (according to the Financial Planning Alliance).

You Want to Set up a CI/CD Process and Sync up With the Development Team

For setting up a CI/CD process, the test plan has a dedicated section that regulates the quality stages each feature accomplishes before deploying it to production. This process is invisible to stakeholders and runs in an automatic mode, but it's crucial to ensure that everything goes smoothly. Using this test plan section, developers can find answers to questions about which cases the developed functionality will be tested in, return to development, specify the stage and environment, what level of unit test coverage is expected, and what quality gates will be configured when code flows from one environment to another.

Testing Documentation: Frequent Mistakes

What (should be tested), when, who (will test), and with the help of what — if you cannot answer any of these questions, there are certainly gaps and a high risk of overlooking bugs.

Here I will consider examples of how incorrectly selected items or the absence of correctly selected items can harm the project and what to look for in the test plan and test strategy.

Example #1

Your team has grown, and it has become difficult to transfer information about processes. As a result, the number of issues has grown, and all the development processes have slowed down. Despite the lack of project time, you will save a lot more time by introducing test documentation: newly joined QA engineers can read about your processes autonomously and return to their description when necessary.

Example #2

Imagine the following situation: the project had vague exit criteria. Unfortunately, each team member interpreted them in their way, which resulted in significant bugs crashing into the release. As a result, the product release had to be postponed for a few weeks because the team logged the wrong severity level.

This was only an example of how incomplete quality criteria can affect product development. If you have already encountered such an issue, consider formalizing quality and acceptance criteria via the test plan section.

Good release criteria examples:

• “All tickets in the release backlog are implemented, deployed, and tested in accordance with the acceptance criteria and ticket description.”
• “Complete set of manual and automated tests passed after code freeze.”
• “All E2E scenarios are completed.”
• “All failed scripts have been analyzed with bug reports added to the bug tracker.”
• “Compatibility tests passed in accordance with the first priority list of browsers and OS described in the test strategy.”
• “No open bugs with Critical or Major priority.”

Example #3

During testing, the team may encounter various problems that can affect timing or quality: dropped testing environments, a team member taking sick leave, unexpected code freeze, poor requirement details, changes in requirements when they are half-finished, etc. Therefore, the team should have an emergency plan to minimize the damage from the triggered risk and a plan of countermeasures to prevent such risks. Consequently, I think risk assessment is one of the most essential sections of the test plan. It consists of a list of risks, their likelihood, impact on the testing process, priority, and a response plan.

How to Define Whether Your QA Team Needs a Documented Test Plan and Strategy?

A well-assembled test plan and a test strategy smooth out the testing process, which benefits every stakeholder of the product — users get stable software, managers get a predictable growth pipeline, engineers get a clear vision of their work areas and proceed with it faster.

But what if you have a rapidly-evolving startup with an enormous load of things to implement, never mind writing any documents? Leaving the documentation for the later stages of the product is a trade-off that frees your hands for a while. On the other hand, the same exact trade-off can increase the number of bugs, rough estimates that rarely hit the target, frustrated users, and failed commitments in the future. To keep quality engineering on top and move fast, you should know when it’s the right time to introduce testing documents for the product team. Here are five metrics that may inform you about problems in processes — they make it clear when a test plan and test strategy should be written in black and white.