Release Testing is Risk Management Theatre
Release Testing is Risk Management Theatre
Join the DZone community and get the full member experience.Join For Free
Your feedback matters—tell Capital One DevExchange what you would do with their Money Movement API.
Release Testing is high cost, low value Risk Management Theatre
“You cannot inspect quality into a product” – Harold Dodge
The adoption of Continuous Delivery often leads to the discovery of suboptimal practices within an organisation, and the Release Testing antipattern is a common example. What is Release Testing, and why is it an example of Risk Management Theatre?
“I was a principal test analyst. I worked in a separate testing team to the developers. I spent most of my time talking to them to understand their changes, and had to work long hours to do my testing” - Suzy
The traditional testing strategy of many IT organisations was predicated upon a misguided belief described by Elisabeth Hendrickson as “testers test, programmers code, and the separation of the two disciplines is important“. Segregated development and testing teams worked in sequential phases of the product value stream, with each product increment handed over to the testers for a prolonged period of testing prior to sign-off.
This strategy had a detrimental impact on both product lead times and quality. The handover period between development and testing inserted delays into the value stream, creating large feedback loops that increased rework and lead times. Furthermore, the segregation of development and testing implicitly assigned authority for changes to developers and responsibility for quality to testers. This disassociated developers from defect consequences and testers from business requirements, invariably resulting in high defect counts and lower quality.
“I was a product tester. I worked in an agile team with developers and a business analyst. I contributed to acceptance tests and would then manually test changes. I don’t miss the old ways of working” – Dwayne
Since the publication of the Agile Manifesto a range of lightweight development processes have introduced a radically different testing approach to the IT industry. Agile methods advocate cross-functional teams of co-located developers and testers, in which testing is considered a continuous activity and there is a shared commitment to product quality. Developers and testers collaborate on practices such as Test Driven Development and Acceptance Test Driven Development in accordance with the Test Pyramid, which recommends a large number of automated unit and acceptance tests in proportion to a small number of automated end-to-end and manual tests.
The Test Pyramid favours automated unit and acceptance tests as they offer a greater value at a lower cost. Test cost is a function of execution time, determinism, and robustness directly proportional to System Under Test size, and as automated unit and acceptance tests have a minimal scope they provide fast, deterministic feedback. In contrast automated end-to-end and manual tests use a much larger System Under Test and produce slower, less deterministic, and more brittle feedback.
This testing strategy is a vast improvement upon its predecessor. Uniting developers and testers in a product team eliminates handover delays and recombines authority with responsibility, resulting in a continual emphasis upon product quality and lower lead times.
“I was an operational acceptance tester. I worked in a separate testing team to the developers and functional testers. I never had time to find defects or understand requirements, and always got the blame” - Jamie
The transition from siloed development and testing teams to cross-functional product teams is a textbook example of how organisational change enables Continuous Delivery – faster feedback and improved quality will unlock substantial cycle time gains and decrease opportunity costs. However, all too often these benefits are impeded by Release Testing – an additional phase of automated and/or manual end-to-end regression testing, performed on the critical path independent of the product team.
Release Testing is a risk reduction strategy that aims to reduce defect probability in a minimal execution time, and it is often seen as a guarantee of product quality. However, Release Testing is actually a disproportionately costly practice due to the following characteristics:
- Independent testing phase – a segregated testing team re-inserts handover delays into the value stream and divorces the product team from final responsibility for quality, increasing feedback loops and rework
- Extensive end-to-end tests – the large System Under Test of each end-to-end test equates to slow and less deterministic feedback, resulting in long execution times and high maintenance costs
- Critical path constraints – testing on the critical path means release testers must always work to a deadline, and they will inevitably be compelled to curtail their test coverage to meet expectations
When viewed through a Continuous Delivery frame the high cost and limited value of Release Testing become evident, and attempting to redress that imbalance is a zero-sum game. Decreasing the cost of Release Testing means fewer end-to-end tests, which will decrease test coverage even further. Increasing the value of Release Testing means more end-to-end tests, more testers, and/or more time, which will respectively increase execution time, communication overheads, and/or opportunity costs. Release Testing can therefore be considered an example of what Jez Humble describes as Risk Management Theatre - an excessively costly practice with an artificial sense of value.
Release Testing is high cost, low value Risk Management Theatre
Build Quality In
Continuous Delivery is founded upon the Lean Manufacturing principle of Build Quality In, and the sage advice of Dr. W. Edwards Deming that “we cannot rely on mass inspection to improve quality” is particularly relevant to Release Testing. An organisation should build quality into its product rather than expect testers to inspect quality in at a later date, and that process can be kickstarted by eliminating Release Testing and moving the release testers into the product team.
Folding the release testers back into the product team will remove the handover delays and responsibility barriers imposed by Release Testing. The regression test suite can be audited by the team, with any tests of value retrofitted into acceptance tests or release-time smoke tests. More importantly, the former release testers will be free to work on higher-value activities such as exploratory testing, empowering them to become subject matter experts and uncover missing business requirements as well as defects.
Batch Size Reduction
Given the high cost and limited value of Release Testing it is prudent to consider other risk reduction strategies, and a viable alternative supported by Continuous Delivery is Batch Size Reduction – releasing smaller changesets more frequently into production. Smaller changesets are less probable to contain a defect due to less simultaneous exposure to business and technology change. In addition, if the chance of a defective change in a changeset is a boolean then the probability distribution of a series of changesets is a binomial probability distribution, in which variability decreases proportional to the square root of the number of changesets.
Consider an organisation with an average changeset size of 24 changes and an average lead time of 12 days. The organisation has a new feature in development and wishes to reduce the probability of production defects upon release. This could be accomplished by decreasing the changeset size to 8 changes and releasing the new feature as part of 3 separate changesets, which would decrease defect probability from 50% to 12.5%.
n = number of changesets
n = 3
probability = (1 / 2n) * 100
probability = (1 / 23) * 100 = 12.5%
Assume that batch size reduction is rejected as a solution in this instance, and that a defect D1 is detected 6 days after the subsequent production release. Fortunately batch size reduction also has the ability to reduce the cost of a defect – both the sunk cost incurred between inception and discovery, and the opportunity cost incurred between discovery and resolution. Those costs are a product of cost per unit time and duration, where cost per unit time represents economic impact and duration represents lifetime. Given an estimated cost per unit time of £20,000 per day for D1 this would mean a sunk cost of £120,000 and a potential opportunity cost of £240,000.
sunk cost = cost per unit time * sunk cost duration
sunk cost = £20,000 * 6 days = £120,000
opportunity cost = cost per unit time * opportunity cost duration
opportunity cost = £20,000 * 12 days = £240,000
cost = sunk cost + opportunity cost = £360,000
While the cost per unit time of D1 is governed by external market conditions its opportunity cost duration of 12 days is controlled by Little’s Law, which states that lead time is directly proportional to work in progress. This means opportunity cost duration can be decreased by releasing a fix in a smaller changeset, and that halving the changeset size to 12 changes would halve the opportunity cost of D1 to £120,000.
sunk cost = £120,000
completion rate = work in progress / lead time
completion rate = 24 changes per changeset / 12 days = 2 changes per day
lead time = work in progress / completion rate
lead time = 12 changes per changeset / 2 changes per day = 6 days
opportunity cost= £20,000 * 6 days = £120,000
cost = £120,000 + £120,000 = £240,000
Releasing smaller changesets more frequently can also reduce sunk cost duration, as small batches accelerate feedback. Smaller changesets decrease change complexity, creating faster feedback loops that can speed up defect discovery. For example, in the next release a defect D2 is discovered at an estimated cost per unit time of £10,000 per day, but the recently reduced changeset size means it is found in only 3 days at a sunk cost of £30,000.
sunk cost = 3 days * £10,000 = £30,000
opportunity cost = 6 days * £10,000 = £60,000
overall cost = sunk cost + opportunity cost = £90,000
Release Testing is an obsolete testing practice and the definitive example of Risk Management Theatre in the IT industry today. End-to-end regression testing on the critical path cannot provide any meaningful reduction in defect probability without incurring significant costs that harm product quality and lead times. An alternative approach advocated by Continuous Delivery is for the product team to own responsibility for product quality, with an emphasis upon exploratory testing and batch size reduction to decrease risk.
Tester names have been altered to preserve anonymity
- Leading Lean Software Development by Mary and Tom Poppendieck
- Assign Responsibility And Authority by Shelley Doll
- Integrated Tests Are A Scam by JB Rainsberger
- Test Pyramid by Martin Fowler
- Continuous Delivery by Dave Farley and Jez Humble
- Organisation Antipattern – Release Testing by Steve Smith
- The Essential Deming by W. Edwards Deming
- Explore It! by Elisabeth Hendrickson
- More Releases With Less Risk by Steve Smith
- Principles Of Product Development Flow by Don Reinertsen
Published at DZone with permission of Steve Smith , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.