The universal theory of big software

“Experience alone, without theory, teaches management nothing about what to do to improve quality and competitive position, nor how to do it.” W. Edwards Deming

For some people, software engineering is a solved problem and Agile methods are the solution. If a project is small enough, management enlightened enough, and customers sufficiently supportive or powerless  - in other words, if it’s an easy project - Agile is the way to go. 

But don’t stop thinking about your process:

• Adopting Agile leaves unanswered the question of how to subsequently improve. How do you outperform the other “Agilistas” and build a firm-specific advantage in engineering? 
• As an individual, you compete globally with engineers combining the same Agile methods with a significant cost advantage. How will you feed your kids ten years from now?
• Software projects differ. Complex projects differ from simple ones. Web software differs from medical device firmware. When goals are different, any technique must support some better than others. To customize the process you need to combine a solid theory with intense study of your own situation.
• Agile methods focus heavily on coding and testing and say less about things like requirements gathering and operations. It’s not possible to optimize an entire process while focusing on just one part. 
• Agile is likely not the final advance in development methodology. Open Source, for example, is a far more radical rethinking of the means of producing software, and one with important lessons for developers. 

In manufacturing, the techniques that make up Just-in-Time (to which Agile is often compared) are less important than the realization that every aspect of the process is a control to be manipulated, coupled with a “relentless pursuit of understanding and improvement.” 

This brings to mind another lesson from Just-in-Time manufacturing, the distinction between “pragmatic” JIT - characterized by a patient and exhaustive focus “on the concrete details of the production process” - and the “quasi-mystical hyperbole” of “romantic JIT.”  Much of the writing about Agile (“People over Processes”) tends towards the romantic, making it simultaneously more appealing and less useful.  

None of this makes Agile ‘wrong’; It’s not. The problem is that too many people treat it as a recipe book and when the world asks them to improvise they’re at a loss.  They need to learn to cook, and not just follow recipes. 

Is there an alternate ‘universal theory of big software’?  I think the strands of such a theory, rooted in economics and enriched by systems dynamics and other disciplines, exist.  They’re just waiting for someone to pull them all together. 

That someone is not me.

I’m reminded of William James’s remarks on his own ground-breaking text Principles of Psychology. He called it ”a loathsome, distended, tumefied, bloated, dropsical mass, testifying to but two facts: 1st, that there is no such thing as a science of psychology, and 2nd, that WJ is an incapable.” 

In the posts that follow, I hope to produce something worthy of similar praise. I think I can, I think I can, I think I can.