Say Goodbye to Software Engineering (And Why Software Projects Never End)
Say Goodbye to Software Engineering (And Why Software Projects Never End)
If you consider the computer your canvas, and your keyboard, your paints, then take a look at why creativity is being stifled in development and the effects it has.
Join the DZone community and get the full member experience.Join For Free
Anyone who has ever done any project management training will have been taught something like this:
"A project is a temporary endeavor undertaken to create a unique product, service, or result. The temporary nature of projects indicates that a project has a definite beginning and end." — PMBOK
In the training room, this seems utterly logical and obvious — common sense even. Of course, a project is temporary and therefore must have a beginning and an end. But for me, many years on from the training room, the common sense of this dictum seems to be less and less common!
Certainly, projects do have a start, although it may not be quite as "definite" as implied by the definition. It’s the “and end” that I am struggling with! If I was building a road bridge, for example, I guess I would know when I had reached the end (of the project, not the bridge!). The structure would be standing and vehicles able to cross. It must be highly satisfying to see the first crossing on the bridge and know that the project has been successfully completed. But sadly, for me and I suspect many other project managers, such satisfaction is simply never achieved. And this is because we're building software and not bridges, and the fact is that software projects never end.
I know it's a contentious assertion and based purely on circumstantial evidence from many years of software development, but please bear with me! When I say they never end, that’s not strictly true; they do end but they don’t have a “definite” end. Usually, they just fizzle out or come to a stop when everyone is tired of the project or the funding runs out. When I look back over my projects, if I am honest, I find it hard to identify a single project that didn't end with a somewhat reluctant acceptance of the final product. Without exception, there was always some significant compromise between what we set out to deliver and what we actually delivered. Usually, this meant that either features or functionality were de-scoped.
The next part of the PMBOK definition says:
“The end is reached when the project’s objectives have been achieved…”
Now unless one of the objectives is “to de-scope a certain amount of functionality at a convenient moment so that we can say the objectives have been met and therefore end the project,” I think it is safe to say that most if not all software projects never reach the end according to this definition. It's a rare software project that delivers on all its promises!
Does this matter? Well yes, I believe it does because it demonstrates the collective misconception that software development follows the rules of mechanical engineering.
The problem is inherent in the medium we are working in. Mechanical engineering has some inviolable constraints — the laws of physics for starters. Software engineering is not constrained by the laws of physics (once you move above the machine level) or any other laws in fact. Of course, there are best practices and standards, but as all software architects know these are not always inviolable!
Software project failures are legendary — just Google it (1.8 million hits) and you will see that there is a whole industry both academic and commercial based on the subject or the avoidance of it! And worse, failure (or partial failure) is something that software professionals internalize – by this I mean that we expect to be underwhelmed by our software deliveries. It is a core part of our shared values – we know through experience that the delivered software will not match the “vision” at the start of the project. When we talk about managing users’ expectations, are we ever suggesting that they should expect more? Software project managers are like parents hosting a children’s party, desperately trying to keep a lid on the excitement in case it all gets out of hand!
Of course, this is why methods were developed. PRINCE2 evolved from the UK governments attempt to thwart the spectacular onslaught of software failures it had spawned throughout the 80s and 90s. The method (and a handful of others) is well established in the software world and in essence, takes traditional engineering principles and applies them to software. Until Agile came along this “waterfall” approach was the accepted standard for managing software projects. But it’s fair to say that it was never universally accepted within the software community and certainly not by the developers themselves. And although it may have prevented the more spectacular failures, it’s hard to demonstrate convincingly that it has improved the performance of software development. The reason for this is simple: developing software is not engineering — or if it is, it’s a new type of engineering that needs new ways of controlling it.
Software engineering is a misnomer. The process of creating software is more akin to writing a piece of music or painting than it is to engineering. This may be an uncomfortable comparison for the people who need to manage or account for software projects but nevertheless, it is accurate. When a painter decides to stop adding paint to a canvass, is the painting finished? In reality, only the painter can say. A composer will play around with many harmonic variations until the best sound is found. But who’s to say that the piece of music is finished? Similarly, a coder is likely to play around with different approaches before finding a suitable solution and committing the code. But again is the code finished? Could it be improved? A mechanical engineer (our bridge builder) has a design specification to comply with. An artist doesn’t and neither should a software developer.
The Agile movement recognizes that the waterfall method doesn’t really work when it comes to software. A reductionist approach works well in mechanical engineering but software needs something more holistic and organic. The various Agile methods attempt to create a more holistic environment for coders. Certainly, the approach is much more acceptable to developers because, in the end, Agile is just a formalization of the way that they work! By formalizing their idiosyncrasies (of which a lack of formalism is one) we have a method that the software community can embrace with some enthusiasm. Whereas not so long ago, a Scrum Master was synonymous with the highest level of geekiness, this is no longer true. Scrums, sprints, and burn-downs are rapidly becoming ubiquitous and even mainstream. Even if the business benefits compared to Waterfall are hard to prove, Agile certainly makes a project more inclusive, responsive, and ultimately more fun.
The concept of the minimum viable product is also very much part of the same zeitgeist. To reduce the time to market the product is developed with the smallest feature set possible to meet the user requirements and create the desired user experience rather than pack it full of features. This seems like a pragmatic approach to prevent “over-engineered” products and helps to get the product in front of the customer earlier. This all sounds very sensible, but imagine commissioning a luthier to build a minimum viable guitar — “as long as it has strings and looks like a guitar I don’t care if it’s made with plywood”! The minimum viable guitar produced might meet the user requirements but certainly wouldn’t create the desired user experience. (In actual fact, some guitars are made with plywood soundboards and sound reasonably good – but you get my point). Fundamentally the concept of Minimum Viable Product is anathema to the software developer’s mindset which strives to create beautiful, rather than adequate, code.
Technological change is rapid. The smartphone has invaded almost every aspect of our lives with an app for practically anything you could think of. Apps are free or low cost. There is an abundance of choice and the purchase transaction so insignificant it can be added to your phone bill. Software programmes are now diminutively referred to as “apps.” All of this undermines the perception that software is a high-value product. In fact, digital asset values are fast approaching zero and the zeitgeist also promotes the idea that everything digital is free. All of this impacts the traditional software development lifecycle and project – not least because our “users” perceive themselves as tech-savvy and will no longer accept the delays and failures endemic in many software development projects. Neither will they accept that their business “apps” do not constantly evolve and re-invent themselves just like their smartphone apps. Software development methods must evolve to embrace these trends.
So, the zeitgeist is confused! Whilst we are promoting a “coder friendly” workplace sympathetic to the idiosyncrasies of geeks, we are at the same time taking away the craftsmanship and artistry which is also an inherent aspect of a good coder. It’s true that code generation is increasingly being automated removing much of the “grunt” work but automation will never replace the creativity needed to produce the killer app!
Creativity will be recognized more and more as the single most important critical success factor. The good thing is that it is already there in good supply! Every software team I have worked with has had at least one (but usually more) brilliantly creative software genius. We just need to recognize them and work out how to harness all that creativity. We should be nurturing the inherent creativity of the geek! Let’s start by never using “software” and “engineering” in the same sentence – how about being a software composer? Or software artist?
Opinions expressed by DZone contributors are their own.