All agilists are familiar, of course, with The Manifesto for Agile Software Development, a statement of principles developed by a group of influential software developers and consultants, written and released in 2001. Now that agility has gone mainstream, however, we may have forgotten how controversial this document was at the time. I had an interesting personal experience with the heat generated by the introduction of agile methods. In 2000, a few months before the Manifesto was written, I interviewed Jim Highsmith for TechRepublic.com. We had what I thought was a pretty neutral conversation about what we were then calling “light methodologies”. The response was volcanic. Although TechRepublic has removed the comments from the archived version, the responses I got were basically of the “road to hell” variety. The main arguments were either “This is an excuse for hacking” or “This blows up all the project management discipline we’ve been trying to teach the business for twenty years!” I was shocked at the vehemence, and, in some cases, the venom that a simple interview induced.
Agile is now conventional wisdom, but, like all revolutionary ideas, it took a long journey to get here. To really understand and apply agile, I think it’s important that agilists are familiar with some of the experiences and academic ideas that made agile inevitable. Agile development has strong theoretical underpinnings, based on research done at, for instance, Harvard and Berkeley, that examined the software development process. Let’s take a step back and look at some of the theory behind the agile movement.
Like many seemingly radical ideas, the Agile Manifesto was actually the culmination of an incremental realization, among both practitioners and theorists, that then-current methods of software development and project management had serious flaws. These limitations were impeding the development of innovative software applications and causing important projects to take longer, cost more, and to be delivered without the key features that users required.
The famous Standish Group CHAOS Studies, familiar to project managers worldwide, illustrate that the majority of projects fail to comply with schedule and cost projections, and often fail to deliver the benefits predicted when they were approved. Additional research by entities such as the Department of Defense confirmed these findings. At the 5th Annual Joint Aerospace Weapons Symposium in 1999, the results of a Department of Defense (DoD) software spending study were presented. Of $35.7 billion spent by the DoD in 1995 for software, only 2 percent of the software was usable as delivered. The vast majority, 75 percent, of the software was either never used or was cancelled prior to delivery.
Academic research also challenged common IT development and project management methods. In 1998, Harvard Business School academics Robert D. Austin and Richard L. Nolan studied large software projects. Their study questioned many of the fundamental ideas of IT development, and produced these key findings:
- "The first flawed assumption is that it is possible to plan large projects.
- The second flawed assumption is that it is possible to protect against late changes.
- The third flawed assumption is that it even makes sense to lock in big projects early.”
Watts Humphrey, a respected IBM researcher, followed this study with a paper outlining his Requirements Uncertainty Principle, asserting:
"For a new software system, the requirements will not be completely known until after the users have used it."
Hadar Ziv of the University of California followed soon afterwards with hisUncertainty Principle in Software Engineering, which states:
"Uncertainty is inherent and inevitable in software development processes and products."
In an important and influential study that surveyed software development methods of innovative Internet companies, Alan MacCormack, assistant professor at the Harvard Business School, published his “Evolutionary Model of Software Development Methods”:
- Waterfall model – sequential process, maintains a document trail
- Rapid-Prototyping Model – disposable prototype, helps establish customer preference
- Spiral Model – series of prototypes, identifies major risks
- Incremental or Staged Delivery Model – system is delivered to customer in chunks
- Evolutionary Delivery Model – iterative approach in which customers test an actual version of the software
MacCormack also recommended a set of practices that could replace the traditional methods. Many old-school agilists consider this the first articulate explanation of the agile idea:
- Early release of evolving design and code
- Daily build of code and fast turnaround on changes
- Deeply skilled teams
Developments in industry, especially the lean manufacturing systems pioneered by Toyota, validated many of the ideas brewing in the software and project management communities. The Standish Group finding that around 60% of features built in to IT projects are rarely or never used was also influential. Focusing on features the customer actually valued, and building in quality upfront rather than “testing it in” later, resonated with the software development community.
The development of agile methods accelerated in the 1990s, as Scrum was developed at Easel Corporation and Extreme Programming evolved at Chrysler. Dynamic Systems Development Method, a light methodology, was introduced and quickly adopted in Europe. Finally, in 2001, the Agile Manifesto was published, and agile development methods were on their way towards mainstream acceptance.
The connection between these academic ideas and studies and the underlying concepts of agility are clear. If users can't foretell what they'll want until they see it, if predicting and planning substantial IT projects is not possible, and if protecting projects against changes that arise during the development process is unreasonable, the ideas behind existing "waterfall" methods are clearly flawed, and an incremental, collaborative, prototype-based methodology becomes the necessary remedy. The revolutionary Manifesto was based on solid logic, research, and experience. That’s why agile works.