The Standish Group’s CHAOS Report is the largest and longest running research study in the software industry. Standish looks at approximately 34,000 software projects in a range of areas. It includes shrink-wrap applications, operating systems, custom apps, etc. They picked a range of diverse projects to look at. Every ten years the study drops about 3,400 projects from ten years ago and picks up another 3,400 projects in the current year. The analysis of the data is massive. Unfortunately, the study is blemished by many inherent flaws.

One of the flaws of the CHAOS Report is how it defines success. It defines success as “delivered on time and on budget with all the features and functions as originally specified.”

If you deliver exactly what you predicted it means you didn’t learn anything through the process of developing that software. In Agile, software development is a discovery process, with frequent feedback from our customers. We’re learning how to create a more valuable and useful product as we build it. This is one of the primary advantages to doing Agile software development.

But worse still, the Standish definition of success is all about prediction. It encourages bad practices such as padding estimates. Estimates are just guesses, no matter how artfully they’re applied, and sometimes guesses are wrong. Holding teams accountable for their estimates encourages people to be conservative, which erodes trust and discourages innovation.

We tend to throw around terms like “success” and “quality” without validating that everyone shares a common definition of them. These terms sound appealing but when everyone has different definitions of them then it can be very difficult to achieve them consistently.

Any real definition of success in software must encompass both definitions of success:

• Success in the marketplace: the software delivers the features customers want, and maybe some features they didn’t know they wanted but find valuable.
• Success as software: code that supports future extensions so it continues to add value in the future.

These two dimensions of “doing the right thing” and “doing the thing right” are at the core of my definition of success.

Successful software requires many things to be successful, from the code itself to its marketing operations, sales, and so on. It’s a complex process, and with any complex process, there are far fewer ways to do it right than to get it wrong.

Success can’t usually be planned, it’s discovered. And success certainly has little to do with how accurately we were able to estimate a task upfront. To me, success, as defined by Agile, is real success: we want to make our customers happy through early and continuous delivery of valuable software. We want to build features that customers use and find value in. Likewise, we want our teams to be happy and healthy so they can continue to produce valuable software, and we want stakeholders to be happy by keeping the cost of ownership low for the software we produce. Everything else is just playing with numbers.