Alan Kay speculates in this talk that 99% or even 99.9% of the effort that goes into creating a large software system is not productive. Even if the ratio of overhead and redundancy to productive code is not as high as 99 to 1, it must be pretty high.
Note that we’re not talking here about individual programmer productivity. Discussions of 10x or 100x programmers usually assume that these are folks who can use basically the same approach and same tools to get much more work done. Here we’re thinking about better approaches more than better workers.
Say you have a typical stem of 10,000,000 lines of code. How many lines of code would a system with the same desired features (not necessarily all the actual features) require?
- If the same team of developers had worked from the beginning with a better design?
- If the same team of developers could rewrite the system from scratch using what they learned from the original system?
- If a thoughtful group of developers could rewrite the system without time pressure?
- If a superhuman intelligence could rewrite the system, something approaching Kolmogorov complexity?
Where does the wasted effort in large systems go, and how much could be eliminated? Part of the effort goes into discovering requirements. Large systems never start with a complete and immutable specification. That’s addressed in the first two questions.
I believe Alan Kay is interested in the third question: How much effort is wasted by brute force mediocrity? My impression from watching his talk is that he believes this is the biggest issue, not evolving requirements. He said elsewhere
Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves.
There’s a rule that says bureaucracies follow a 3/2 law: to double productivity, you need three times as many employees. If a single developer could produce 10,000 lines of code in some period of time, you’d need to double that 10 times to get to 10,000,000 lines of code. That would require tripling your workforce 10 times, resulting in over 57,000 programmers. That sounds reasonable, maybe even a little optimistic.
Is that just the way things must be, that geniuses are in short supply and ordinary intelligence doesn’t scale efficiently? How much better could we do with available talent if took a better approach to developing software?