The Business Value of Continuous Testing
An in-depth discussion about Continuous Testing, including the benefits and drawbacks this DevOps process can have on your code and business value.
Join the DZone community and get the full member experience.Join For Free
Why Does Software Fail?
Team members in software development make mistakes and insert defects into the software development artifacts. Defects may result in failures, or they may not, In some cases, a defect can exist that will never cause a failure in actual use, because the condition that causes the failure can never arise. In other cases, a defect will not cause a failure during testing, but could result in a failure in production. Where there are defects there is a risk of failure. Failures lower quality and business value of the IT system.
What Is Testing?
“Testing is an empirical investigation conducted to provide feedback to stakeholders and team members on quality information about the IT system.”
As Elisabeth Hendrickson Vand said:
“When I headed up quality engineering, I described my job as ‘creating feedback cycles.’ Feedback is critical because it is what allows us to steer. We must constantly validate between customer needs, our intentions, and our implementations. Testing is merely one sort of feedback.”
Feedback from testing creates confidence in the quality of the IT system. It lets us make changes quickly and with less fear. It lets us focus on the new IT capabilities that we are adding, by having the test tell us whenever we break the old IT capabilities.
Testing provides learning opportunities. The quality information is used in retrospectives to find process and quality improvement activities.
Goals of Testing
- An IT system that fulfills stakeholders' expectations.
- An IT system that achieves business goals.
- Find defects.
- Gain confidence in the level of quality of your software.
- Provide quality information for decision making, including:
- quality level.
- risk to the code quality.
- business risk.
- Keep defects out of artifacts.
- Measure the progress of development.
Problems That Hinder the Achievement of Testing Goals
Quality is something that we all agree is important, but is not given enough focus in software development. What is usually focused on is time, cost, and features, because that is easy to measure. Software teams lack the competence in testing to provide and use measurements of quality. Poor quality measurement is a result of the misconception that testing activity lengthens development time and increases development costs.
Testing has always been the elephant in the room. Psychologically, the malleable nature of software has given organizations an excuse to defer investment in testing. However, this deferral results in technical debt. Over time, technical debt compounds, escalating the risk and complexity associated with each release.
There is a lot of misconceptions around testing:
- There will usually be failures in the software, look at what Microsoft is delivering.
- To do systematic testing is expensive, so we do some ad-hoc testing at the end with the resources and time that are left in order to reach the definition of done.
- Testing the software product properly takes a long time and is too expensive.
- The traditional approach fosters the perception that building quality software is overly complex and expensive.
- One does not need specific competence in testing to test.
- Testers are developers that did not manage to make a carrier in development.
There are very few that will say that testing has no business value. But when you ask them what they are willing to spend, what feature they can wait to develop until a later iteration in order to reduce business and quality risk, provide long-term savings, or perform testing to provide quality information, you'll find that people will be vague about their willingness to invest in testing. Few can quantify, describe, and articulate the business value of testing.
Test cases are usually designed to detect functional defects. To find the defects that cause the most severe failure during operations and increase tech debt, one also needs to focus on the other quality attributes.
Many software products are not built with enough focus on testing in the beginning. Instead, organizations start testing late in the development cycle and only with ad-hoc testing. When testing begins, the product is of low quality and contains a lot showstoppers and serious failures that lengthen the development schedule. Often times, orgs take shortcuts that lower the software quality and increase the software's tech debt.
What happens is that when the software product is put into production with a lot of failures to fix, it results in a lot of support work. So instead of developing a new feature, the team members are occupied with failure discovery, failure fixing, and support.
Code defects are introduced during coding. If these defects make their way into production, it will result in an increase of development costs. The earlier defects are found and removed, the lower the cost of defect removal.
Testing is not something that we should only start once our code is complete. Janet Gregory says:
"Throw away the ‘then’ in ‘code then test’ – replace it with ‘and.’ And maybe, also reverse the order to say, ‘test AND code.’ Put the test first."
Gregory has also noted:
"By shifting left, it feels like the development process is a linear process, and it’s not."
When code is written but not tested we have partially complete software. Software development churn is associated with partially complete software. When testing occurs long after coding, test-and-fix churn is inevitable.
Much of the risk of software development lies in partially complete work. Partially done software creates the following problems:
- it gets lost.
- it becomes obsolete.
- it hides quality problems.
- it ties up money.
What Is Continuous Testing?
Because testing is so essential, we should be doing it all the time as an integral part of the DevOps lifecycle. Rather than 'shifting left,' we should test in every phase of the SDLC.
Your goal should be to build quality code from the start, not test it in later. You should avoid creating defects in the first place. You should not allow defects to be inserted into the artifacts in the first place. If this is not possible, then you should test the artifacts after each small step, so that defects are caught immediately after they occur. When a defect is found, you stop-the-line, find its cause, and fix it immediately.
To keep defects out of code as it is being developed, tests should be written before we code and executed frequently after a small increment of code is added. New code should not be written before all the tests pass. Team members should integrate both code and tests into the software repository as often as possible—every hour or so—and run a test suite to see that no defects have been introduced. If the tests don’t pass, they do not add new code until the problem is fixed or the failing code is reverted.
When a feature is implemented, you should execute a regression test suite. Finding defects should be the exception, not the rule, during regression testing. If regression testing routinely triggers test-and-fix cycles, then the development process is defective.
- Automated unit and acceptance tests should be run against every commit to version control in order to give developers fast feedback on their changes.
- Developers should be able to run all automated tests on their workstations in order to triage and fix defects.
- Testers should be performing exploratory testing continuously against the latest builds to come out of CI.
- No one should be saying they are “done” with any work until all relevant automated tests have been written and are passing. Implementing continuous delivery means creating multiple feedback loops to ensure that high-quality software products get delivered to users more frequently.
- When implemented correctly, the process of releasing new versions to stakeholders should be a routine activity that can be performed on demand at any time.
Continuous testing uses automated tests to ensure teams receive immediate feedback to quickly mitigate as many risks as possible throughout the software development lifecycle. Moreover, team members are able to continuously learn about their IT system and what can be done to increase quality. When the automated tests pass, teams are confident that their code is releasable. Furthermore, they are confident that test failures indicate a real defect.
Automated tests should not be handed over to QA or outsourced to a third-party. When team members are involved in creating and maintaining acceptance tests, there are two important effects:
- the code becomes more testable.
- team members care more about the tests and will invest more effort into maintaining and fixing them.
Business Value of Continuous Testing
We get the focus on quality by creating a feedback loop by using systematic testing that informs us about the quality of the software product. This allows us to:
- find and remove defects early while they are smaller, cheaper, and easier to fix.
- remove defects before they cause failure in production.
- create feedback that could be used for organizational learning that improves processes and the product quality of future development. When failures occur, we treat them as opportunities for learning, as opposed to a cause for punishment and blame.
- develop software products without living in fear that a failure can occur in production at any moment. Defects will be detected long before they end up as failures in production that lower the stakeholders' satisfaction and decrease the business value of the software product.
Every team member is responsible for testing the software product. Having team members share responsibility for the quality not only improves outcomes but also accelerates learning. It’s impossible for a team member to learn anything when someone yells at them for a defect inserted in the software product six months ago. That’s why we need to provide feedback to everyone as quickly as possible, in minutes, not months.
Below are some of the benefits of Continuous Testing:
- It reduces failure costs.
- If the support and operation are tired of being surprised by unknown quality problems, we can fix that by:
- finding defects long before they occur as failures in production.
- reducing defects that are inserted into the artifacts.
- enabling confident, informed decisions about quality by delivering solid information.
- If the stakeholders want to enjoy the marketing and sales advantages of an improved reputation for quality, testing will help with that.
- If the stakeholders want to have smoother, more predictable releases, testing will help with that.
- If the stakeholders want to have increased confidence upon release, systematic testing can help to ensure that what really matters gets tested.
- If the stakeholders want protection from legal liability, testing can help by ensuring compliant, secure code.
- If the stakeholders are in the safety-critical or mission-critical business, and want to be sure of gaining business value from the software product, we can work with the team throughout the lifecycle to solve as many problems as we can prior to release.
- Ensures a higher level of quality.
- Provides quality information for decisions about the:
- software development process.
- software product.
- Aids in preventing and finding defects.
- Helps us to understand the product.
- business risks.
- process risks.
- quality risks.
- Decreases the amount of incomplete requirements, architecture, and design due to:
- Ensures as many defects as possible are found before being released to production.
- Test early and often: Your code will be tested throughout the development, delivery, testing, and deployment cycles.
- Earn stakeholders' loyalty by accomplishing continuous improvement and quality.
- Automates your test cases to decrease time spent testing.
- Increases release rate by speeding up delivery to production.
- Reduces business risks by assessing potential problems before they become an actual problem.
- Creates communication transparency by eliminating handover between the development, testing, and operations teams.
Problems With Continuous Testing
While continuous testing has myriad benefits, there are several challenges that software development teams must take into consideration:
- Adjust to DevOps: Provide professionals with the right tools and training for Continuous Testing within Agile and DevOps environments.
- Change in culture: Cultural shifts among your development and testing teams may happen if traditional processes are maintained.
- Update testing strategy: Maintaining only traditional testing methods and test data management that is not clearly defined keeps continuous testing from reaching its full potential.
- Code integration: Developers who don’t integrate their code on a regular basis (recommended several times daily) create issues with duplicated coding efforts and non-compatible code.
- Test environments: Make sure your test environments work within your code repository base for seamless testing of the newest available code.
- Production environments: Also, make sure your production environments reflect the test environment to ensure every area was properly tested.
- Visibility: Gaining visibility throughout the DevOps pipeline.
- It's difficult to design and maintain meaningful test cases that are aligned with the stakeholders' expectations.
- It's difficult to understand how to design test cases from the requirement specifications.
- There's a risk of accidentally missing or designing incorrect test cases.
- It can take too much time to maintain automation scripts.
- There can be a lack of test data.
- Your non-functional testing can be out of focus.
- Release complexity: Releases often involve multiple IT systems with dependencies, different technology, and potentially conflicting resources.
- Difficult to understand what needs to be tested.
- Difficult to identify problems by gathering data across the pipeline.
- Difficult to coordinate and share knowledge across the different tools in the DevOps pipeline.
Published at DZone with permission of Stein Korsveien. See the original article here.
Opinions expressed by DZone contributors are their own.