Emergent Design: The Evolutionary Nature of Professional Software Development

[img_assist|nid=2526|title=|desc=|link=none|align=right|width=148|height=146]Software development projects are notoriously late. We still have difficulty living up to promised releases. Many methodologies, techniques, processes and practices have been developed over the years, yet a view that software development is still an immature field is widely held, and strongly so.  Emergent Design: The Evolutionary Nature of Professional Software Development tackles this issue, and drawing on patterns, refactoring, and test-driven development, author Scott Bain offers a blueprint for moving efficiently through the entire software lifecycle, smoothly managing change, and consistently delivering systems that are robust, reliable, and cost-effective. 

During a discussion with Scott, we covered how to change developers mentality towards TDD, how to avoid overdesign and some tips to avoid failure in software projects.

James Sugrue: What is your background in software development?

Scott Bain: I worked in broadcast television and radio for over 20 years, doing all kinds of digital automation within that domain, which is quite broad. In recent years I have been a consultant, mentor, coach, and trainer for companies all over the world including Microsoft, Adobe, Sun, Vanguard, Boeing, Texas Instruments, GMAC, etc.

Sugrue: What was your motivation for writing the book?

Bain: Basically, my students suggested it. When I started as a teacher, I realized that I needed lots of feedback if I was going to improve, so I got into the habit of asking for it after every class. After doing this for a number of years, the feedback was usually pretty positive, but I started hearing a new comment fairly frequently.

They pointed out that the reach of a course is very limited (I can only teach so many people per year, not everyone goes to training, not everyone speaks the language I do), but that a book can go places, be translated, therefore can reach many more people.

Sugrue: What is your target audience?

Bain: The sort of people who would come to my classes in patterns, TDD, and so forth, if they could. Basically I am trying to put together a paradigm of development that shows the synergy between these powerful points of view. I think most developers, testers, and those that lead their efforts would benefit from the book. That’s my intent, in any case.

Sugrue: Are you proposing a new methodology in this book, or advocating some of the principles from eXtreme Programming?

Bain: A bit of both. I think the practices of XP are powerful, but they also have implications that should not be ignored. For instance, if we are to “embrace change” that means our approach to design and implementation must be extremely change-friendly (as patterns are). The implications of incrementalism are manifold; how do we keep test suites from becoming maintenance problems themselves, etc? The notion of an emergent design is implied by XP, no question, but I think one needs to support the notion intellectually.

Sugrue: A difficult question for a lot of developers and architects is "How much design is enough design" Do you have an answer or a guideline for this?

Bain: This is essentially what my book is about. How can we design enough, and follow sufficient code quality practices, that we do not need to overdesign? Overdesign comes from a fear/concern that changing to a different design later will be too risky or expensive. If we fundamentally reduce the risk and eliminate much of the waste of change, then the motivation for overdesign evaporates, and we can work emergently.

Sugrue: In maintaining the "Keep It Simple" principle, is there a place for Design Patterns? Is it true to say that a lot of designs are over-engineered as a result of design pattern overuse?

Bain: I would say misuse, not overuse. Most people think of patterns as plug-in solutions, and this leads to the golden hammer anti-pattern (you pick your favorite pattern, and make the problem fit into it). In this book, as in similarly-minded books (Design Patterns Explained, Head First Design Patterns) we consider patterns in a very different way. I actually think they are far more applicable in a lean, or agile process than they ever were in the waterfall.

Sugrue: In your book you address why software projects fail more than they should. Is there any one thing, or a checklist, that we should look out for as we are running a project?

Bain: Software has value to the degree that it serves a need, or a set of needs, for a business or an individual, or the culture overall. If it meets no need, it has no value, and valueless software is a failure. So, we must have a good way to see the real need that we plan to serve; but also, we know that needs change. They are changing faster all the time. Good software must be able to change so as to continue to meet the need it’s intended for. There are many things that this realization leads us to, and they constitute the focus of my book.

Sugrue: Is the high failure rate in projects due to the immaturity of the software development field? Will it ever get better?

Bain: Yes, and yes. Imagine what medicine what like when it was about 40 years old or so. Add to this the fact that the thing we work on (technology) changes much faster than most domains, and I think we can pat ourselves on the back a little insofar as we can make software at all. A lot of people besides me are asking the right questions, and I see real progress in our industry as a result. We have a long way to go, but so long as we’re moving in the right direction, I believe we’ll get there.

Sugrue: A striking statistic that you quote in the book is that of the Breakdown of Features By Usage from the Standish Group. How does your emergent design proposal solve this problem? Do we just need to lean more on XP practices?

Bain: Emergence is a natural outgrowth of most XP practices, so long as you address resource and risk issues early. XP in and of itself does not do this, but then it does not claim to. If you add patterns (as collections of forces) and decent attention to qualities like cohesion and coupling, I believe you can work efficiently in an environment of constant change, and this means validation-centricity. I think much of the “stuff we don’t really need” will simply never get prioritized in an incremental process. This was XP’s point, and I think they are right.

Sugrue: In Chapter 7, you cover some of the main qualities to pay attention to during design. Which of these have you found to be the most ignored?

Bain: Two. Separation of Use from Construction (and encapsulating the latter), plus the testability of every entity in the system. Testability needs to be considered in a non-trivial way, however. How do we, for instance, test workflows within an object? A proper understanding of techniques like Mocking and Endo-Testing is essential.

Sugrue: I find Unit Testing to be a very difficult thing to convince developers to do. In one paragraph, can you give motivation to the non-believers of Test Driven Development?

Bain: The best way, in my experience, to motivate anyone is to show them what’s “in it for them”. There’s a lot to unit testing that is directly beneficial to the developer; too much to delineate here, but I’ll give an example:

You’re going to do some kind of analysis, most likely. You need to understand the problem well, in order to solve it well. How will you document your analysis?

Tests done in a TDD fashion are analysis, and the tests themselves then form the documentation of the analysis (they are the persistent artifacts). But, unlike most analysis artifacts, they compile, they are concrete, and you can use them for other purposes (coverage for refactoring, for example). One effort, many benefits. This is very compelling for most developers.

Sugrue: Should refactoring be scheduled in per elaboration, or is a Wait & See approach better?

Bain: It depends on the extent of legacy code involved. In Greenfield development, refactoring should never be a lengthy issue, and should be done “just in time” whenever something that is not open-closed needs to be, usually because of a new requirement. Refactoring of legacy systems is a much larger issue, and should be considered in the estimation of the project, definitely. I usually refer folks to Michael Feather’s excellent book “Working Effectively With Legacy Code” when they find themselves in this position. I can do little to improve on his work.