Science and Software Development

"So we seek explanations that remain robust when we test them against those flickers and shadows, and against each other, and against criteria of logic and reasonableness and everything else we can think of. And when we can change them no more, we have understood some objective truth. And, as if that were not enough, what we understand we then control. It is like magic, only real. We are like gods!"

Software Development, Science and Engineering

I think that this philosophy of science has profound impacts on how we should approach software development and even how we view what software development is.

The second thing that made start on writing about this was based on a passing comment that I made on Twitter. I repeated a viewpoint that I have long held that automated testing in software is best thought of  and used as a falsification mechanism. Amongst several others, Bill Caputo replied and included some links to his thoughts on this which very closely aligned with mine and described some of these ideas better than I had.

Then in the Twitter conversation that followed Bill posted this.

This is very close to the way in which I have started to think about software development in general and more specifically, the more scientifically rational approach to the engineering of software that I try to apply and promote.

For me these two ideas collide.

Software Development is an Act of Creativity

David Deutsch's good explanations are deeper and more difficult than they sound. In striving for a good explanation, we are required to gather information to allows us to create knowledge.

I describe software development as an inherently creative process. We don't often consider it as such and much of software development is, incorrectly, treated as an exercise in production rather than creativity and suffers as a consequence. This misconception has dogged our industry and how we undertake the intensively creative task that is software development.

We are trying to create knowledge, in the form of a computer program, that captures our best understanding of the problem that we are trying to address. This is entirely a process of exploration and discovery. The encoding of the knowledge, in the form of something executable, is merely a transcription exercise. So the thinking, the design, the discovery of good explanations that fit our understanding is at the heart of all good software development.

Of course "merely a transcription exercise" underplays the complexity of that part of the process, but my point is that the technicalities of coding, the languages, the tools, the syntax of the instructions themselves have the same relationship to software development that maths does to physics. These things are tools that allow us to grow and extend our understanding. They are not the thing itself. Maths, and coding, are great fun. I completely understand and recognize their appeal, but for me at least, that fun is enormously amplified when I can apply them to something practical. Ideally, something that helps me deepen my understanding. Something that helps me to get to better explanations.

This is kind of obvious if we think in terms of computer science, but kind of missed in much of the discussion and practice that I observe in the software development community.

Software Development is Always a Process of Discovery

If we think back to our computer science studies we know that we only need a Turing machine, any Turing machine, to solve any classically computable problem. So the choice of tools, language, architecture, design are all only choices. These tools are not unimportant, but neither are they fundamental to solving any given problem.

I can write code to solve any computable problem in any language or paradigm. The only difference is how efficient I am in transcripting my ideas. Functional Programming, OO Programming, Ruby on Rails, C++, Java, and Assembler can all only render the same ideas.

Of course, it is a bit more complex than that. Certain programming approaches may help me to think, more easily, of some kinds of solution, others may hinder me. However, I believe that there is something deeper here that matters profoundly to the creation of good software.

It is the act of discovery and of learning, understanding the problem in more depth, that characterizes our work and is the real value of what we do.

I believe that we should optimize our development approach, tools, and processes to maximize our ability to foster that learning and process of discovery. We do this by creating a series of better and better explanations of the problem that we are attempting to solve, and the techniques (code) that we are employing to solve it.

Creating "Good Explanations"

Our good explanations take specific forms. They are the documentation and tests that describe a coherent picture of what our systems should do. They are the code that capture our best current theory of how our code should do the things it should. They are the ideas in our heads, the descriptions and stories that we tell each other, that allow us to understand and diagnose problems, and extend and maintain our systems. These are our good explanations and one of the profound advantages that we have over most disciplines is that we can make many of these "explanations" self-validating for consistency by automating them.

I have been a long-term adherent of Test-Driven Development (TDD). I don't take this stuff lightly and over the years of practicing it I have refined my take on it. It is an old statement, not original to me, that TDD is not really about testing. I was peripherally involved in the birth of Behavior-Driven Development (BDD). The idea was to try and re-focus people's thinking on what is really important in TDD. BDD was born as a means of teaching TDD in a way that led to the higher-value ideas of behavioral focus and the use of "executable specifications" to drive the development of our software. It is a very effective approach and I teach it, and recommend it, to the teams and organizations that I work with.

I now think that there is something more profound going on here, though, and for me David Deutch's good explanations hold the key. When we develop some software, any software for any purpose, we are, nearly always, embarking on a process of discovery.

We need to discover a lot of stuff. We need to learn more about the problem that our software is intended to address. We need to learn about what works for the consumers of our software, and what doesn't. We need to discover what designs work well and give us the behaviours that we desire. We need to discover if our solutions are fast enough, robust enough, scalable enough, and secure enough. We start out knowing little about all this, and begin learning from there. At any given moment, in the life of a software system, all of this stuff only adds up to our best current theory. We can never be certain of any of it.

Optimizing for Learning

For the vast majority of human history, we were really quite bad at learning. Then a few hundred years ago, we discovered how to do it. We call the trick that we learned then science.

Science is humanity's best, most effective approach to learning — Deutsch would say "gaining new knowledge." Fundamental to this approach, according to Deutsch, is the formation of these good explanations and their defining characteristic that "they are hard to vary" without invalidating them.

In trying, at multiple levels, to capture a good explanation of what is going on, we are trying to describe the logic and algorithms that capture behaviors that we are interested in. We are trying to describe the data structures of the information that we deal with and process. We are trying, in some manner, to describe the need that our software is intended to address for our users or the market niche that our cool new idea is hoped to exploit.

All of these descriptions are explanations of our understanding. To transform these explanations into good explanations our explanations need to be more rigourous. The need to include everything that we know and, as far as we are able, check that our explanation fits all of the facts.

An Example of a Good Explanation

A good example of this, taken from Professor Deutsch's book, is the idea of seasons. Some people believe that winter is caused by the Earth having an elliptical orbit and so being further from the Sun for part of the year. This is a good explanation in that I can't vary it without changing it significantly. If the idea is correct, changing the explanation to say "The seasons are caused by Earth having a circular orbit" doesn't work because that completely changes the explanation.

So this seems like a reasonable idea, and, even better, it is easily falsifiable. If this were true, if seasons are caused by the distance of the Earth from the Sun, then it should be winter at the same time of the year all over the planet, because the planet is in the same place in its orbit whether I am in London or Sydney. This isn't the case, so this theory fails. It is a bad explanation because it doesn't fit ALL of the facts.

Let's try again. Observations show that for any given location on the Earth, the Sun will rise and set at different points on the horizon at different times of the year. Ancients, before global travel, knew this. A good explanation for this is that the axis of the Earth's rotation is tilted with respect to its orbit around the Sun. The axis is tilted and precesses as the Earth orbits the Sun. That means that when our part of the planet is tilted toward the Sun we get more energy from the Sun because it is more directly overhead (we call this summer) and when tilted away we get less energy (we call this winter).

So if I was an ancient Greek, and knew about axial tilt as an explanation of seasons, I could make a prediction. When it Is summer here, it will be winter on the opposite side of the planet. This explanatory power is profound. It allows ancient Greeks to predict the seasons in a place that their descendants wouldn't get to travel to for thousands of years!

Engineering - Applied Science

So what has all this philosophy of science got to do with software? Well, this science is humanity's best problem-solving technique. It is the difference between essentially static, agrarian civilisations that lasted for tens of thousands of years with virtually no change and our modern, high-tech civilisation that doubles its knowledge every 13 months. The application of science to solving practical problems is how we solve the most difficult problems in the world. It is also what we call engineering.

I believe that we should apply this kind of thinking, engineering thinking, to software development. What that takes is a significantly more disciplined approach to software development.

The rewards are significant. It means that we can create high-quality software, more efficiently and more quickly than we have before. It means that our software will better meet the needs of our users and it means that the organizations in which we work can be more successful, while we are less stressed by trying to solve insoluble problems like, "When will I be ready to release the new feature and get the product owner off my back?"

So, step 1 is to approach software development as an exercise in learning, of discovery.

If our best way to learn is science, and software development is all about learning, then we should apply the lessons of science to our approach to software development.

An Engineering Discipline for Software

Following Deutsch's model, we should be trying to create good explanations that are "hard to vary" and then we should evaluate our explanations with each other, and with reality to confirm that they are consistent. What does this mean in practice?

We could try to write down some explanations of what we would like our software to achieve. We are not going to understand the totality of what we want our software to achieve at the outset, that is something that we will learn as we progress and understand the problem, and hopefully the demand, in more depth. So we are looking for a way in which we can capture our current intent and expectations in a form that we can later extend. How wonderful would it be if we could somehow capture these explantations of our current understanding in a form that would allow us to confirm that they are consistent with one another and met as we proceed to elaborate and extend our theories.

To me this is pretty much the definition of TDD. It allows us to record an incrementally evolving collection of, hard to vary, explanations that capture our current understanding. If we are smart, we capture them in a way that allows us, with the help of Continuous Integration, to immediately see if our theories, our good explanations, in the form of our code meet our expectations — do the tests pass?

This approach allows us to construct and re-use an automated system of checking that our good explanations are consistent with one another, that the body of our knowledge (of the system) as a whole is self-consistent. This, in turn, means that, as our understanding deepens, we can make small changes to our ideas and quickly and efficiently confirm that everything still makes sense. This approach allows us to stay informed about the state of our system-wide understanding, even as the scope of our system extends beyond our ability to intuitively understand it in its entirety. It means that we can extend and deepen our knowledge in a particular focused area (a new feature of the system).

I believe that the TDD approach, refined and elaborated upon by Continuous Delivery, represents a genuine engineering discipline for software development. I don't mean this in a loose sense. I don't mean that this is analogous to engineering. I mean that it allows us to use a more scientifically rational approach to validating our ideas, measuring their effect and maintaining an ever-increasing and consistent collection of good explanations of our system and its behavior.