DevOps Transformation Using Theory of Constraints - Part 2
DevOps Transformation Using Theory of Constraints - Part 2
Learn how the Theory of Constraints can help developers and DevOps managers examine their policies to adopt DevOps successfully.
Join the DZone community and get the full member experience.Join For Free
Easily enforce open source policies in real time and reduce MTTRs from six weeks to six seconds with the Sonatype Nexus Platform. See for yourself - Free Vulnerability Scanner.
This article is a continuation of Part 1, here. Be sure to read it before you start Part 2!
Q3: What Policies Exist Before Getting Into DevOps?
The third question is of utmost importance, and it has no easy answer. Even when current policies are well known, in many cases, there are just so many policies that cause such a multitude of problems it is hard to find and address all of them. And it is imperative to address them, or else we cannot gain the full benefits that come from lifting the limitation.
For example, if the promise of DevOps is that we can respond to every customer’s whim and request about our platform in less than a day — this is a significant competitive advantage. Can we actually do this if deployments are by policy done once a month? No, that policy must be abolished.
Dr. Goldratt warns that it is critical to find the root of the complexity first, the main thing that if we would change it, everything else falls into place. The ToC method to map all undesirable effects and expose this core conflict is called Logical Thinking Processes, logically finding all the relations of causes and effects. It is similar to the Five Whys method from Lean, but much more rigorous. There is just not enough space here to explain all of the Logical Thinking Processes, but an excellent book that goes into depth to elaborate on these was written by Lisa J. Scheinkopf, called “Thinking for a Change.”
Allow me to try and describe some of the policies that enable success for companies in the “pre-DevOps.” era, where competition had more or less the same limitations as we did.
- There must be a separate operations team (with their own manager) being responsible for the applications in a production environment. Because developers don’t want or don’t know how to operate systems in production.
- The ops team is being put in charge of doing deployments since they can cope with failure faster. Because deployment of changes has a substantial risk, failure rates are often high and restoring a service from failure often takes a long time.
- The introduction of a new role called SQA/QA/QC (a whole team) will ensure high quality of all introduced changes. Since managers are often not expecting high-quality software code to be written, and in practice, it quite often fails in spectacular ways.
- Managers enforce deadlines by adding yet another role called a Release Manager/Engineer. The Release Manager will manage the schedule and deliverables deployed to production. Because developers writing code and testing it for quality take too long and need to be rushed, and releases must happen in a more orderly fashion than simply ad-hoc.
- Deployments once a week (or once a month, a quarter) are mandated as a policy. They are even scheduled on the first day of the week, but the handovers of Dev-Qa-RelEng-Ops constantly move this occurrence to the last day of the week. Often Operators act as heroes and are doing deployments for the duration of the weekend. The reason for this policy would be a reality where rapid change is dictated by high-velocity competitors, each company must deliver features faster.
- Developing new features take priority of time and budget away from reliability or quality improvement and invests all hands in creating more new features instead of more reliability. The reason most likely being managers intuition, which guides the company in a belief that better products and more paying customers must stem from new features.
Why would managers decide to prioritize new features instead of the reliability of their service? Here we actually get a bit closer to the root cause of why organizations rarely get any benefit from DevOps practices. The way managers are deciding in their day-to-day feels right to their intuition, it has always proven to bring success in the past.
The thing is that this way of decision making is by far better than just making arbitrary decisions. Due to the shortage of information, a manager can rarely decide on any better decision and has to compromise on his local optima. A decision made based on local optima will in general not be as bad as a random one. At the same time it is by far worse than prioritizing based on a system-thinking or holistic measurement of the effect of a decision on the bottom line. New competitors are doing just that, measuring the effect of decisions on their bottom line, and are reaping all the benefits from their holistic approach. A great example comes from Etsy who explain how they “Design for Continuous Experimentation.”
As a consultant, I often confess that “Great high-velocity organizations rarely call up experts for help.” From getting around quite a lot visiting many tech companies for the last five years, I can say definitively that the above policies and rules are still in effect in the majority of organizations I had been in contact with. Talking to other consultants in the field reveals that this is a global truth, even companies that do have a “DevOps Team,” or “DevOps Engineer/s,” or jump on the latest and greatest “DevOps Tools.” In most cases keep their old rules and policies, and receive almost no benefit from their adoption of DevOps. The decision making of managers in these organizations did not change, and none of the advantages promised by the DevOps movement are to be found anywhere. A company can adopt Docker, Kubernetes, Cloud, Serverless, but still have a total disconnect between developers and customer needs — and have deployments occur as frequently as a solar eclipse.
In examples narrated by Dr. Goldratt in his audio series “Beyond the Goal,” he explains that early adopters of technology are those who understand the benefit and reap the rewards. When the industry starts to take note, everyone wants a piece of the pie — but they rarely know how to act on it. In the world of DevOps, these early adopters would be companies like Netflix, Flickr, Etsy, Amazon. Once these companies are very successful because of their new technologies, other companies want to replicate the things those have done to compete.
Companies trying to become the next Netflix or Etsy replicate the technology, the tools, the buzzwords. And a neglect to change existing policies and rules leaves them bewildered, not understanding why nothing improved. Now that there is a big new “DevOps Team” full of “DevOps Engineers” using the latest and greatest “DevOps Tools,” they are still unable to deliver software any faster than before. The obvious conclusion? “DevOps does not work,” “DevOps does not bring any benefit,” managers see it just as additional labor and politicking that has no real impact on the bottom line not realizing that they are just doing the wrong thing better.
A multitude of policies hinders companies from adapting to the extremely rapidly changing world. Managers think that just taking this DevOps concept, putting it in, will immediately make the company move much faster, “Agile,” which in most cases is just doing the wrong thing faster without understanding why running faster is not the same as advancing forward.
This conflict of the “Wrong DevOps” inflicted by managers on their employees is causing all kinds of problems for both employees and organizations. I personally have witnessed developers, ops and “devops engineers” live in constant frustration. Stories of burnout and worse are abundant, businesses go bankrupt, services crash and burn. Companies that have a blockbuster legacy product with huge margins keep the fire burning alongside employees who are doing nothing or doing a lot to negative effect just investing in new initiatives.
Keeping old policies and going forward with a “DevOps Team,” “DevOps Engineer,” and “DevOps Tools” will rarely change anything regarding business value. Today everyone is a “DevOps Engineer,” and every company has a “DevOps team.” It was even voted as the second best role based on salary and satisfaction by Glassdoor “50 Best Jobs in America” for 2017! Notice that it didn’t even exist as a profession in the 2016 list.
Do these companies enjoy the advantages of stability, quality and moving forward faster than their competitors? Most of these organizations are still stuck with the same policies and rules that they had before the transformation. It takes some smart management to come up with changes to an organization that will enable the benefits. Because even though talented employees might contribute overall improvements by doing things the right way and giving a personal example. Eventually, the power over changing policy and rules, to set new standards of work, is in the hands of management.
We can do better than this, it only takes some thought and courage to change entrenched behaviors and adopt more suitable ones.
Q4: What New Policies Must We Adopt for a DevOps Transformation?
According to Dr. Goldratt, this is the hardest question of all, and also the most important.
Dr. Goldratt and Dr. Deming teach us to look at an organization as a whole, if the goal of the organization is improving the bottom line results — then decisions across the entire organization need to holistically align to that goal. Continuing to march forward without looking at the data, without measuring how the current practices are serving the system leaves managers in their own bubble of local optima rules. The mentioned companies Netflix, Etsy, Amazon, have a lot of measures and collect data to help the organization choose better decisions and practices that actually improve the bottom line. So before we hire our first “DevOps Engineer,” or start a new “DevOps Team,” or go searching for the best “DevOps Tool,” like containerization, how about starting some measurements? Measure the value delivered to customers and provide clear visibility into these measurements to our managers and employees, this can have a tremendous effect.
Do we believe that quality has an effect on the bottom line? Let us then inspect our policy regarding quality. Do we have a good definition of how quality looks like? Is producing quality, measuring quality and improving quality at the top of our company priorities? Is quality built-in in software development, or is there a policy that tries to hammer quality into software changes after these were created? Does our organization require all our developers to write software unit tests? How about our front-end developers? Are these tests used to provide immediate feedback and increase the trust in our service reliability? Do we measure the effect of bad quality in our service or product has on our bottom line?
The new rules and policies cannot be cookie-cut fit all, we must do some soul searching and find the relevant standards for our organization. Yes, this is hard work, it requires to invest some thinking time instead of spending time in putting out fires. It mandates to learn and to read ideas from incredible men like Dr. Goldratt and Dr. Deming, and try to apply their ideas in our vicinity. Dr. Goldratt, for example, has a big chunk of knowledge on how to handle putting out fires, part of this Engines of Disharmony discussion. There is just so much more in the magnificent books written by Dr. Goldratt, his followers, and his teachers.
I will not write all the new regulations and policies that a company must adopt. This will be left for you, the reader. I do invite you to start a discussion on what these new policies might be. There is a lot of lists and ideas about these around the internet, but none are without drawbacks, and none fit every organization like a glove, context is quite important, and experimenting is part of the fun!
Taking powerful ideas and methods from Giants who form the base of the DevOps movement, can teach us a lot of new interesting ways to think and approach management and engineering.
In this post, I wrote the four questions that evaluate technology in the eye of Dr. Goldratt and tried to put my own interpretation on these to apply to DevOps. Even though I started writing this blog with a different tool from Goldratt which I wanted to discuss, that will have to wait for next time.
Books mentioned in this article:
- The Phoenix Project by Gene Kim and Kevin Behr
- The Goal by Dr. Eliyahu M. Goldratt and Jeff Cox
- Beyond the Phoenix Project at DOES16 by Gene Kim and John Willis
- Necessary but not Sufficient by Dr. Eliyahu M. Goldratt and Eli Schragenheim
- Beyond the Goal by Dr. Eliyahu M. Goldratt
- Thinking for a Change by Lisa J. Scheinkopf
- Interview with Eli Schragenheim discussing N&S
- Design for Continuous Experimentation by Dan McKinley from Etsy
Published at DZone with permission of Evgeny Zislis . See the original article here.
Opinions expressed by DZone contributors are their own.