Agile Failure Has Strengthened, Not Weakened, Software

2024 is promised to be the year of generative AI. Instead, it has been the year of catastrophic software outages. Earlier this year, we saw outages affecting high-street shops, banks, and cloud vendors, whilst those of us in the UK saw the Post Office Horizon IT scandal reach new levels of public outrage.

Having made a living working to investigate and resolve such scandals in recent times, I found myself amid furor after leading a study that found that Agile wasn’t all that it was cracked up to be. After the international crises following the Crowdstrike outage helped underscore the point, I spoke to The Register about how catastrophic takes on Agile feed into failure.

Broadly, I spoke out against interpretations of Agile that focus on getting the latest features as quickly as possible, DevOps metrics that disregard issues so long as they’re fixed quickly, and digital transformation strategies where the informed consent of those being ‘transformed’ is disregarded (despite the low success rates).

However, in truth, at its core, the Agile failure research was speaking to a deeper factor - loss aversion. Back in 1979, now-Nobel-winner Daniel Kahneman and psychologist Amos Tversky coined the phrase to describe how humans feel the pain of a loss to twice the extent of the pleasure of a gain.

I first learned about this when studying cognitive psychology at the University of Cambridge and with this understanding, it isn’t hard to understand why tech scandals can start with a technical cause but snowball into tragedy through coverup attempts.

Through investigating tech project failures and software catastrophes at scale, the pattern of minor problems snowballing to tragedy was blindingly obvious and was the underlying hypothesis of the Agile failure rate study.

The solution? Yes, the psychological safety to discuss and address problems led to an 87% increase in project success, but the one factor that came above this was clear requirements, with a 97% increase in success rates. Something we can all do with less pain than needing to change deep-rooted psychological factors.

In other words, having a gate to discuss and address problems when loss aversion is at its lowest allows for the greatest improvement in success rates.

However, this seemingly obvious proposition might not be universally obvious. Over the past six months, I’ve spoken about technology disasters to audiences of politicians at the UK Parliament, alongside engineers and lawyers. It may well be controversial to say this, but what I’ve found has been that those equipped with the mental models to work across multiple disciplines have been able to engage with these problems and find solutions in a way that will often go over the heads of those who can only operate in one discipline.

This interdisciplinary approach allowed the cybersecurity community to address severe problems of huge public interest in an incredibly effective way by adopting expertise spanning technical, social, and legal factors. Unfortunately, in my experience, there is still an aversion to doing the same in software engineering.

Software engineering is stronger as a result of learning from the catastrophes over the past year, but the longer-term solution rests in us being less averse to having expertise spanning other disciplines alongside software development in our profession.

That way, addressing the next ‘Agile’ won’t hurt like hell.