We All Have to Start
Every high-profile software house started somewhere, in the middle of nothing (in a garage?), and with a not-so-good codebase.
Amazon was a monolithic system that, in 2002, started a very smart process of re-engineering that lasted years. LinkedIn underwent a 2-month full stop of feature development in 2011 to restructure the codebase. Now they are the example to follow. How is it possible? Is it all because they have money? Maybe, but that's not the entire truth.
The unicorns among the software companies fought their way to the excellence they are. Nothing was easy or free. It's sheer hard work and application of good engineering practices every single day and a lot of courage and commitment.
If you work in a small software company like me, and you're stuck with bad habits and lots of the typical pains of the software world, the good news is that everything can change. Your team- your company- can change.
DevOps is a set of practices (not only tools!) that aims to apply in the software world the lean concepts of the manufacturing world.
At the heart of DevOps there are the Three Ways:
The principles of Flow, which accelerate the delivery of work from Development to Operations to our customers;
The principles of Feedback, which enable us to create ever safer system of work;
The principles of Continual Learning and experimentation, which foster a high-trust culture and a scientific approach to organizational improvement as part of our daily work.
If you follow these Ways, change will happen. I'm not saying that's easy. You're going to face every kind of difficulty: from the technical to the psychological point of view. Your team has to strive every day to make this change happen. What you have to do is to commit to the Three Ways with small changes that provide a feedback quickly (think about days or weeks, not months) over and over again. This process will never stop; your team will be always improving in some aspects of the daily work.
DevOps @ MadLab
At MadLab, we are a small team of six developers. At the beginning of 2016, there were two. This big change exposed all our flaws with no mercy: from how we accepted and started work to the delivery of the applications to the customers. But this was a great opportunity to understand that we had to change. In that period, I was reading the book "The Phoenix Project," and that was the key. I urge you to buy that book right now.
How are the three ways helpful to the MadLab team?
The First Way
Visualize the Work
The first principle says we need to accelerate the work from development to operation, and then to our customers. But what is work in the abstract world of software development? Work is every activity we can think of: bug fixes, development of a new feature, index tuning of the database, refactoring… To accelerate work, we need to see the work, and to see the work, we need to transform something invisible/abstract into something we can visualize.
The first thing we did was to make the work visible. We configured our issue tracking software (JIRA) with a kanban board. Every unit of work is represented as a card on a virtual board. This enables us to see the amount of work we have to do in a visual way.
The kanban board enabled us to realize how much work there was in the system, and the first reaction was: “Woah! All this work to do? How can we manage all this!?”
This was the first big-little change that happened in our team.
The WIP Limit
To accomplish the goals of the first way, we also started to apply the golden rule of a kanban board: the WIP limit. To accelerate the flow from left to right, we have to limit the work in progress. If you have less WIP, the quality will go up; if the quality is higher, the work will have fewer defects, and work with fewer defects is likely to not come back (a feature poorly implemented comes back with a bug, for example). If work doesn’t come back, the flow is improved.
You can observe how we did not make changes to the codebase, to our development tools, or to our roles in the team. Jobs descriptions and responsibilities are the same, too.
Tech/Automation Tools and Practices
Other practices to improve the flow need tools and automation. Over time, we implemented continuous build, integration, and then deployment practices. These implementations happened in the span of about eighteen months, and we’re still working on it to constantly improve.
The Second Way
The second way enables the constant flow of feedback from right to left in the stages of our value stream. It requires that we amplify feedback to prevent problems from happening again, and enable faster detection.
At MadLab, we follow the second way, trying to keep the quality closer to the source. After a developer completes the assigned work, he/she creates a pull-request. This unit of work has to be reviewed by someone else and be approved before it goes back to the master/trunk stream of development. This is a peer-review because the work is validated by a member of the team, and not someone higher in the company hierarchy. With this particular activity, we have lots of room for improvement, but we’re working on it. DevOps is a journey that will never end, and we’ll always find something to improve. The important thing is to realize that you cannot achieve all at once, but improvement will come over time.
To keep the quality close to the source, we also apply the rule that whoever breaks a build is responsible for fixing it as soon as possible. This way, we prevent bad code from being pulled from someone else or spreading into other branches of development. Sometimes we also do pair-programming, an extreme implementation of the peer-review.
The advantage of pair programming is its gripping immediacy: it is impossible to ignore the reviewer when he or she is sitting right next to you. Most people will passively opt out if given the choice. With pair programming, that's not possible. Each half of the pair has to understand the code, right then and there, as it's being written. Pairing may be invasive, but it can also force a level of communication that you'd otherwise never achieve. – Jeff “Coding Horror” Atwood
The Third Way
The third way enables the creation of a high-trust culture that supports a dynamic, disciplined, and scientific approach to experimentation and risk-taking, facilitating the creation of organizational learning, both from our successes and failures.
At MadLab, we follow the third way by trying to create a positive environment where errors are not occasions to blame someone, but occasions to understand what happened and how. We try to be transparent with each other and to trust that each member of the team is working as best as he/she can. We encourage creativity; everyone can suggest ideas to improve our way of work and every idea is discussed together.
We reserve time every Monday, about two hours, to decide what we need to improve, how, and to review previous decisions to eventually stop changes that weren’t so good. We strive to improve something every two weeks; it can be a small thing (better comments on commits) or something bigger (let’s start with mandatory peer-review).
Change can happen, and the best way to do it is with evolutionary steps, some big, and some small. DevOps is spreading fast because it is based on the successful principles and lessons learned from manufacturing, high-reliability organization, high-trust management models, and others that have brought us to the state of the art we know today. DevOps is the outcome of applying the most trusted principles from the domain of physical manufacturing and leadership to the IT value stream (Lean, Theory of Constraint, Toyota Production System, and many others). Based on my little experience with DevOps, I can only encourage you to all try to apply the Three Ways with your team.
"The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win" by Gene Kim, Kevin Behr, George Spafford.