Issues Around Scaling DevOps
Culture, skills, tools, coordination, and automation emerge as areas of concern when scaling DevOps.
Join the DZone community and get the full member experience.Join For Free
To gather insights on the current and future state of how DevOps is scaling in the enterprise, we asked IT executives from 32 different companies to share their thoughts. We asked them, "What are the most common issues you see when scaling DevOps?" Here's what they told us:
- Culture is one of the major pieces, especially when you want to use the existing teams. Once they come on board with daily releases, they understand that’s it’s practical and less scary. It takes time. The other issue is when they acquire other companies and integrate the product into the existing set of applications with multiple challenges. The IT team of the acquired company likely has other tools and technology. Bringing the entire company under the same architecture, building practices, and application landscape takes a lot of time and effort. We had to convince people the multi-cloud approach was beneficial based on the application. Help educate them as to why some clouds are appropriate for workloads and capabilities.
- Copying and pasting practices from other shops without considering cultural issues. Check out This American Life “Nummi” how GM and Toyota worked together to fix an auto plant, but they could not replicate with mechanics. It takes cultural change.
- It’s usually more cultural. When scaling anything you will have inflection points. Scaling is rarely linear. In the real world, you need to recognize inflection points. There’s a lot of temptation to do things like monitoring in-house. But these projects tend to be more complex and consume more time and effort in the team and you get mired in setting up an open-source project versus your work.
- The introduction of a DevOps methodology, tooling, and processes requires a cultural shift as well as an investment in time and money. Less discussed is how to handle the development spikes to introduce DevOps. In our situation, we had to figure out how to balance the available resources between day-to-day development and operations, and how to free up and allocate resources to drive the additional DevOps efforts. Long-term, the investment into the DevOps efforts will pay off, free up resources and we can scale more easily, together with the growth of the organization. Of course, for us, the first steps have been the hardest. For example, what we have learned is that the less often a specific task is performed, it is more important to automate it, which can be surprising. Execution of those rare tasks can be associated with higher risk; in a worst-case scenario, the person who did it last time, isn’t even around anymore. At the same time, there is very low risk with regard to development, as there is usually little time pressure in completing them. In this case, the automation is also the documentation for that specific task.
- Some of the most common scaling issues related to personnel and culture. Personnel is one of the biggest challenges due to the current employment market conditions. It is extremely difficult to recruit, hire, and retain talented DevOps practitioners. The other issue is culture. Since effectively scaling anything involves changes, it is human nature to resist. Explaining the “why” in a way where everyone gets it will make execution easier.
- The biggest issue when scaling DevOps is a cultural shift. People are afraid of losing their jobs if you incorporate automation. But humans can focus on doing more interesting tasks to add value to the business when automation is implemented. The cultural shift is necessary when scaling DevOps and having IT teams embrace this methodology.
- Cultural resistance. Resistance to change.
- One of the hardest challenges about adopting and scaling DevOps is not about the technology at all, but rather culturally shifting to new development and deployment models. We were fortunate that cross-functional teams were open to changing the way things were and were willing to try out new and bold approaches. In our early days of DevOps transformation, some of the issues we faced were: managing cross-team coordination, improving internal communications between teams, and reducing long resolution time. While no organization is perfect at this yet, I’m happy to say that we have come a long way and have been successful in accelerating the speed of change significantly.
- The skillset and appropriate training of the newly hired DevOps engineers need to catch up quickly for them to contribute effectively. Sometimes a large number of tools that do almost the same thing may cause analysis paralysis.
- 1) The administration is a big one and one that catches teams off guard quickly once their DevOps adoption curve spikes. What might start as a small cloud instance or a server under someone’s desk can quickly become a critical hub of the development organization, the operations workflow, and even the whole company. Treating the DevOps infrastructure like the business-critical system it is and having appropriate backups, availability, access control, security, and scale is both something that should be considered from the outset and something that will need to be re-evaluated throughout a DevOps adoption journey. Plan for growth before it arrives and use tools that reduce the overhead required as the teams and usage grow. 2) Expertise is another challenge to scaling DevOps, as it takes time to both teach the methodology and practices of DevOps and also to build expertise in the tools and systems involved. Needs will be different as more teams get involved, as some will jump quickly into the command line to configure pipelines and others will prefer a graphical representation of their workflow in a UI. Choosing solutions that don’t force a homogenous experience for all users will become more critical as more teams are involved, and allowing for some users to be experts in DevOps tools and practices while others simply benefit from the automation without being required to understand it is going to make it much simpler to scale DevOps across the organization. 3) Disparate tools, especially across functional boundaries, if the tools are not integrated and integrated with an automated fashion then teams will always be at least somewhat out of sync, operating on stale, differing data. In this case slow, manual reconciliation is always required, at best. At the worst time is wasted delivery the “wrong thing.” 4) Culture. DevOps requires a culture alignment between groups that historically and justifiably have different cultures. If these groups meet in the middle, align on shared objectives and develop a common understanding and language and implement that in common or integrated tooling then that sets the stage for overcoming other cultural differences.
- Many organizations are siloed using their own tools and processes. Another is if you try to enforce different tools for silos you get pushback. That slows things down and blocks the adoption. How to manage and choose the right tools for the job. Choose the right tool for the job. With the cloud, things get even more confusing.
- The most common issue is related to the tooling and picking the "right" tool. There is no right tool and there are a plethora of tools that are helpful. If you aren’t careful, those helpful tools can cause pain if you try to use them in a way that they’re not designed. Teams need to focus on ownership, education, access, and data to further enhance their ability to help build and support their solutions.
- One of the big issues is balancing standardization with the features that DevOps applications teams demand. I have found each app team has different needs from the DevOps platform, including the app stacks, automation, cloud capabilities, and more. These needs must be balanced with what the platform can provide, how many similar features are already provided, and operational support. For example, we currently offer ten different open-source application stacks. If an application team needs another one, how does that need get evaluated, approved or declined, delivered, and supported? Where does this stop? It is a big issue when scaling DevOps. Second, while automation and cloud give you the technology to scale, organizing the people involved with DevOps can be a challenge. Culture change is just one part of it. Building a real continuing operating model around Agile IT concepts and then executing that operating model to improve the DevOps platform itself requires a lot of effort, time, and training for all people involved.
- As leaders focus on creating a collaborative environment and tearing down silos, automation and processes are critical. To improve code coverage quality engineering is pushed upstream as developers write unit test code cases to achieve 90% coverage. Have all automation in place with high levels of code coverage so the code is tested using 90% code coverage.
- Getting everyone on the same page is very important as sometimes it is difficult to achieve. But with the right approach, it is definitely possible. Documentation is a key aspect of DevOps that many fall short on. When scaling needs arise, proper documentation can really help in worst-case scenarios.
Here’s who shared their insights:
- Nancy Wang, CEO, Advancing Women In Product
- Mick Morrissey, Director of Engineering, Asavie
- Lyon Wong, Founder & COO, Blameless
- Patrick Reister, Senior Build Engineer and Ivan Szatmári, Head of Release Management, Bohemia Interactive Simulations (BISim)
- Rob Zuber, CTO, CircleCI
- Brian Dawson, Director of Product Marketing and Brian Nash, Director of Product Marketing, CloudBees
- Eric Robertson, V.P. Product Marketing Management & Strategy Execution, CollabNet VersionOne
- Jeff Williams, Co-founder & CTO, Contrast Security
- Mike Rose, VP Engineering, Cybera
- OJ Ngo, CTO, DH2i
- Chris DeRamus, Co-founder & CTO, DivvyCloud
- Tobi Knaup, Co-Founder & CTO, D2iQ
- Andi Grabner, DevOps Activist, Dynatrace
- Antony Edwards, COO, Eggplant
- Kris Lahiri, Co-Founder & VP Operations and Chief Security Officer, Egnyte
- Chris Michael, DevOps Engineer, FileCloud
- Tamas Cser, Founder & CEO, Functionize
- Justin Stone, Senior Director of Secure DevOps Platforms, Liberty Mutual
- Mark Levy, Director of Strategy, Software Delivery, Micro Focus
- Phaedra Divras, Chief Operating Officer, Mission
- Michael Morris, Senior Director of IT Cloud & DevOps Platforms, NetApp
- Tori Wieldt, Developer Advocate & Sr. Solutions Marketing Manager, New Relic
- Bob Davis, CMO, Plutora
- Veejay Jadhaw, CTO, Provenir
- Vishnu Nallani Chekravarthula, V.P. Head of Innovation, Qentelli
- Anurag Goel, Founder & CEO, Render
- Davy Hua, Director of DevOps, Security & ITOps, ShiftLeft
- Dave Karow, CD Evangelist, Split
- Ben Newton, Director of Product Marketing, Sumo Logic
- Adityashankar Kini, V.P. Engineering, Sysdig
- Neil Barton, CTO, WhereScape
- Dan Beauregard, DevOps Evangelist, Xebia Labs
Opinions expressed by DZone contributors are their own.