The original study that found huge variations in individual programming productivity was conducted in the late 1960s by Sackman, Erikson, and Grant (1968). This study has been repeated at least 8 times over 30 years and the results have not changed! (see below)
Sackman et al studied professional programmers with an average of 7 years’ experience and found that:
- the ratio of initial coding time was about 20 to 1
- the ratio of debugging times over 25 to 1
- program execution speed about 10 to 1
- program size 5 to 1
They found no relationship between a programmer's number of years of experience and code quality or productivity. That is there was NO correlation between experience and productivity (i.e. ability to produce code) and there was NO correlation between experience and quality (i.e. minimizing defects) .
That is the worst programmers and the best programmers made distinct groups and each group had people of low and high experience levels. Whether training helps developers or not is not indicated by these findings, only that years of experience do not matter. If training does makes a difference then there are very few developers getting training out there.
Results Have Been Confirmed for 30 Years!
There were flaws in the study, however, even after accounting for the flaws, their data still shows more than an order of magnitude difference between the best programmers and the worst, and that difference was not related to experience. In years since the original study, the general finding that "There are order-of-magnitude differences among programmers" has been confirmed by many other studies of professional programmers (full references at the end of the article):
- Curtis 1981
- Mills 1983
- DeMarco and Lister 1985
- Curtis et al. 1986
- Card 1987
- Boehm and Papaccio 1988
- Valett and McGarry 1989
- Boehm et al 2000
Technology is More Sophisticated, Developers are not
You might think that we know much more about software development today than we knew in 1968, after all today:
- we have better computer languages
- we have more sophisticated technology
- we have better research on effective software patterns
- we have formal software degrees available in university
It turns out that all these things are true, but we still have order of magnitude differences among programmers and the difference is not related to years of experience. That means that there is some other x-factor that drives productive developers; that x-factor is probably the ability to plan and make good decisions.
Developers face making decisions on how to structure their code every day. There is always a choice when it comes to:
- laying out code pathways
- packaging functions into classes
- packaging classes into packages/modules
Because developers face coding decisions, many of which are complex, the best developers will plan their work and make good decisions. Bad developers just 'jump in'; they assume that they can always rewrite code or make up for bad decisions later. Bad developers are not even aware that their decision processes are poor and that they can become much better by planning their work.
Solution might be PSP and TSP
Watts Humphrey tried to get developers to understand the value of estimating, planning development, and making decisions in the Personal Software Process (PSP) for individuals and the Team Software Process (TSP) for teams, but only a handful of organizations have embraced it. Capers Jones has done analysis of over 18,000 projects and discovered that1:
All of these findings should have a profound effect on the way that we build our teams. Rather than having large teams of mediocre developers, it makes much more sense to have smaller teams of highly productive developers that know how to plan and make good decisions.
We inherently know this, we just don't do it. If the senior management of organizations only knew about these papers, we could make sure that the productive people get paid what they are worth and the non-productive people could seek employment in some other field. This would not only reduce the cost of building software but also increase the quality of the software that is produced.
Unfortunately, we are doomed to religious battles where people debate methodologies, languages, and technologies in the foreseeable future. The way that most organizations develop code makes voodoo look like a science!
- Stop It! No… really stop it.
- 5 common worst practices to stop immediately
- Software Professionals do Inspections
- Evidence on why software inspections improve quality dramatically
- Inspections are not Optional
- Visual evidence why software inspections can remove up to 97% of defects
Boehm, Barry W., and Philip N. Papaccio. 1988. "Understanding and Controlling Software Costs." IEEE Transactions on Software Engineering SE-14, no. 10 (October): 1462-77.
Boehm, Barry, et al, 2000. Software Cost Estimation with Cocomo II, Boston, Mass.: Addison Wesley, 2000.
Card, David N. 1987. "A Software Technology Evaluation Program." Information and Software Technology 29, no. 6 (July/August): 291-300.
Curtis, Bill. 1981. "Substantiating Programmer Variability." Proceedings of the IEEE 69, no. 7: 846.
Curtis, Bill, et al. 1986. "Software Psychology: The Need for an Interdisciplinary Program." Proceedings of the IEEE 74, no. 8: 1092-1106.
DeMarco, Tom, and Timothy Lister. 1985. "Programmer Performance and the Effects of the Workplace." Proceedings of the 8th International Conference on Software Engineering.
Washington, D.C.: IEEE Computer Society Press, 268-72.
1Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
Mills, Harlan D. 1983. Software Productivity. Boston, Mass.: Little, Brown.
Valett, J., and F. E. McGarry. 1989. "A Summary of Software Measurement Experiences in the Software Engineering Laboratory." Journal of Systems and Software 9, no. 2 (February): 137-48.