Fixing a Broken DevOps Culture
Culture is important. Period. Continue reading to get an expert's opinion on how DevOps cultures' break and how to fix them.
Join the DZone community and get the full member experience.Join For Free
DevOps has its own unique culture, and culture is inherently resistant to change. The mantra “this is how it has always been done” is a powerful force.
Despite the fact that DevOps is evolving rapidly with the advent of new tools and services, obsolete ideas and practices that have been ingrained in the IT culture continue to linger. For example, development teams and system administrators for years have clung to the rule of “NEVER deploy on a Friday afternoon” — unless you want to spend your entire weekend fixing bugs. Another example is the division of labor between humans and robots the tools teams create to make their processes less manual. These days you’ll still find many development groups that delegate only menial tasks to automation while reserving the most mission-critical and infrequent tasks to human operators.
The sad truth is, what passes for automation in a DevOps environment today is largely limited to testing. Developers don’t want to wait hours for tests to manually run when deadlines are looming. The initial ways in which automation is usually applied to any DevOps process typically involves repetitive tests that run for multiple builds; tests that span multiple data sets; tests spanning multiple software and hardware platforms; or any test that would otherwise be impossible to conduct manually. That’s not a bad place to start. But limiting automation to tasks developers don’t want to do is not only wrong, it can be crippling to an organization. Applying automation on a limited basis results not just in unnecessary delays to software development, but it leaves an organization open to vulnerabilities that, for want of a patch, could be easily remediated using automation.
Acceptance of cherished fallacies concerning application development and automation are indicative of a “broken culture,” which occurs when culture is deeply misaligned with the interests of the organization, the individuals who compose the culture, and the organization’s stakeholders. That broken culture usually manifests when historic cultural norms coupled with a dose of self-interest prevent organizations from embracing new and better ways of doing things.
Going back to the fear of Friday deployments, this idea has taken hold in DevOps culture because developers haven’t trusted their own software and systems enough, and don’t want to spend their Friday nights debugging a production problem. Fair enough, but there’s got to be a better way.
Let’s go back to the curious division of labor between humans and robots, and take last year’s Equifax breach as a specific example. This incident — which cost the company billions of dollars and continues to haunt consumers to this day — happened because a certain procedure mandated that a specific piece of software must always be updated by a human. Since humans are prone to error, this simple task (purported by years of a broken DevOps culture) resulted in a devastating data breach. Again, there’s got to be a better way.
Advanced automation tools are drastically streamlining application development and deployments, significantly reducing the chance of error, and speeding up the time it takes to identify and correct any errors that do occur. With these tools, companies can deploy any day of the week and any time of the day and can let the deployment date be dictated by what is best for the company and its customers — not the weekend plans of developers. The same concept applies to updating software. A better way is to leverage the automation tools that are now easily accessible and allow organizations to have robust management of compliance requirements without human error or delay.
Besides adopting and then trusting automation tools, here are a few other concepts companies and developers need to embrace to fix the broken DevOps culture:
- Software must be valued as more than just the cost it takes to create it or to replace it. Code should be an asset, not a liability.
- Software should be thought of as constantly changing — a dynamic system with which people, machines, and processes interact continuously. When this never-ending process is supported by the right tools, deployments are easier and managing and updating software is seamless and secure.
- Code by itself is useless. It needs infrastructure to build it, and data to create something meaningful. Therefore, if you only test the code, but are not managing your infrastructure in the same way, you're not really testing the software system.
- Companies can dictate the cadence at which they create new software, but they do not have this same luxury when it comes to fixing broken software. Identifying and fixing problems must be done as soon as possible.
- Every new update or fix must be version controlled so that everything is auditable. That includes changes to infrastructure configurations!
Like most transformations, automation needs to be approached with care. It’s important for an organization to first break down their application development and release management process into a series of discrete tasks. Each of those tasks can then be automated to any desired level. There’s clearly a potential danger that some organizations might attempt to automate processes that are not well understood. But once each discrete process becomes documented, the opportunity to inject automation starts to exponentially increase.
Naturally, there is always going to be resistance to change. Rather than trying to change a culture from the top down, most organizations enjoy more success when they can successfully apply automation to a specific process during an application development sprint that gets measured. Over time, the benefits of automation become apparent to all. The most important thing is to not limit automation to only the things developers are comfortable delegating. DevOps is about making a commitment to continuous improvement, and that’s not going to occur if automation adoption is being constrained by the biases of the DevOps team.
Once companies fully understand these fundamental truths about software and embrace the modern tools available to them, they can then start to shed the aspects of DevOps culture that are limiting or damaging. Like most things worth doing that may take some time and effort, organizations that fully embrace automation have already come to appreciate there is, after all, a much better way.
Opinions expressed by DZone contributors are their own.