Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Issues Around Scaling DevOps

DZone 's Guide to

Issues Around Scaling DevOps

Culture, skills, tools, coordination, and automation emerge as areas of concern when scaling DevOps.

· DevOps Zone ·
Free Resource

To gather insights on the current and future state of how DevOps is scaling in the enterprise, we asked IT executives from 32 different companies to share their thoughts. We asked them, "What are the most common issues you see when scaling DevOps?" Here's what they told us:

Culture

  • The biggest challenge is scaling from the initial teams to the rest of the organization that may be comfortable with the old way of doing things. The only way this works is from the top-down, with executive sponsorship and mandate. It has to come with education and support. People need coding skills, need to invest in reeducation, it takes time and money to see the results at scale. Invest in job security for employees. Repetitive education and telling the whole organization why you are doing what you are doing, and then showcasing internal and external successes.
  • There are tons of technical challenges and those tend to be more easily handled. The harder part is the cultural and organizational issues are ongoing even as DevOps becomes more ingrained.
  • Culture is one of the major pieces, especially when you want to use the existing teams. Once they come on board with daily releases, they understand that’s it’s practical and less scary. It takes time. The other issue is when they acquire other companies and integrate the product into the existing set of applications with multiple challenges. The IT team of the acquired company likely has other tools and technology. Bringing the entire company under the same architecture, building practices, and application landscape takes a lot of time and effort. We had to convince people the multi-cloud approach was beneficial based on the application. Help educate them as to why some clouds are appropriate for workloads and capabilities.
  • Copying and pasting practices from other shops without considering cultural issues. Check out This American Life “Nummi” how GM and Toyota worked together to fix an auto plant, but they could not replicate with mechanics. It takes cultural change.
  • It’s usually more cultural. When scaling anything you will have inflection points. Scaling is rarely linear. In the real world, you need to recognize inflection points. There’s a lot of temptation to do things like monitoring in-house. But these projects tend to be more complex and consume more time and effort in the team and you get mired in setting up an open-source project versus your work.
  • The introduction of a DevOps methodology, tooling, and processes requires a cultural shift as well as an investment in time and money. Less discussed is how to handle the development spikes to introduce DevOps. In our situation, we had to figure out how to balance the available resources between day-to-day development and operations, and how to free up and allocate resources to drive the additional DevOps efforts. Long-term, the investment into the DevOps efforts will pay off, free up resources and we can scale more easily, together with the growth of the organization. Of course, for us, the first steps have been the hardest. For example, what we have learned is that the less often a specific task is performed, it is more important to automate it, which can be surprising. Execution of those rare tasks can be associated with higher risk; in a worst-case scenario, the person who did it last time, isn’t even around anymore. At the same time, there is very low risk with regard to development, as there is usually little time pressure in completing them. In this case, the automation is also the documentation for that specific task. 
  • I see similar patterns with DevOps as a separate role and job title rather than a shared cultural mindset. A culture of continuous learning. You need to be able to learn fast to produce more quickly and effectively. Understand where you need to improve, place additional resources or tooling to deliver the value. Promote a culture of continuous learning. 
  • Some of the most common scaling issues related to personnel and culture. Personnel is one of the biggest challenges due to the current employment market conditions. It is extremely difficult to recruit, hire, and retain talented DevOps practitioners. The other issue is culture. Since effectively scaling anything involves changes, it is human nature to resist.  Explaining the “why” in a way where everyone gets it will make execution easier. 
  • The biggest issue when scaling DevOps is a cultural shift. People are afraid of losing their jobs if you incorporate automation. But humans can focus on doing more interesting tasks to add value to the business when automation is implemented. The cultural shift is necessary when scaling DevOps and having IT teams embrace this methodology. 
  • Cultural resistance. Resistance to change. 
  • One of the hardest challenges about adopting and scaling DevOps is not about the technology at all, but rather culturally shifting to new development and deployment models. We were fortunate that cross-functional teams were open to changing the way things were and were willing to try out new and bold approaches. In our early days of DevOps transformation, some of the issues we faced were: managing cross-team coordination, improving internal communications between teams, and reducing long resolution time. While no organization is perfect at this yet, I’m happy to say that we have come a long way and have been successful in accelerating the speed of change significantly.

Skill

  • DevOps is a discipline that requires you to practice every day, or else your DevOps powers get rusty. The most common issue is failing to maintain balance. DevOps practices require spending as much time on building the business context and strategic awareness needed to inform SMART direction setting as it does on all of its other disciplines. 

  • The skillset and appropriate training of the newly hired DevOps engineers need to catch up quickly for them to contribute effectively. Sometimes a large number of tools that do almost the same thing may cause analysis paralysis. 
  • 1) The administration is a big one and one that catches teams off guard quickly once their DevOps adoption curve spikes. What might start as a small cloud instance or a server under someone’s desk can quickly become a critical hub of the development organization, the operations workflow, and even the whole company. Treating the DevOps infrastructure like the business-critical system it is and having appropriate backups, availability, access control, security, and scale is both something that should be considered from the outset and something that will need to be re-evaluated throughout a DevOps adoption journey. Plan for growth before it arrives and use tools that reduce the overhead required as the teams and usage grow.  2) Expertise is another challenge to scaling DevOps, as it takes time to both teach the methodology and practices of DevOps and also to build expertise in the tools and systems involved.  Needs will be different as more teams get involved, as some will jump quickly into the command line to configure pipelines and others will prefer a graphical representation of their workflow in a UI.  Choosing solutions that don’t force a homogenous experience for all users will become more critical as more teams are involved, and allowing for some users to be experts in DevOps tools and practices while others simply benefit from the automation without being required to understand it is going to make it much simpler to scale DevOps across the organization. 3) Disparate tools, especially across functional boundaries, if the tools are not integrated and integrated with an automated fashion then teams will always be at least somewhat out of sync, operating on stale, differing data.  In this case slow, manual reconciliation is always required, at best. At the worst time is wasted delivery the “wrong thing.” 4) Culture. DevOps requires a culture alignment between groups that historically and justifiably have different cultures. If these groups meet in the middle, align on shared objectives and develop a common understanding and language and implement that in common or integrated tooling then that sets the stage for overcoming other cultural differences.

Tools

  • Many organizations are siloed using their own tools and processes. Another is if you try to enforce different tools for silos you get pushback. That slows things down and blocks the adoption. How to manage and choose the right tools for the job. Choose the right tool for the job. With the cloud, things get even more confusing. 
  • The most common issue is related to the tooling and picking the "right" tool. There is no right tool and there are a plethora of tools that are helpful. If you aren’t careful, those helpful tools can cause pain if you try to use them in a way that they’re not designed. Teams need to focus on ownership, education, access, and data to further enhance their ability to help build and support their solutions.

Coordination

  • One of the big issues is balancing standardization with the features that DevOps applications teams demand. I have found each app team has different needs from the DevOps platform, including the app stacks, automation, cloud capabilities, and more. These needs must be balanced with what the platform can provide, how many similar features are already provided, and operational support. For example, we currently offer ten different open-source application stacks. If an application team needs another one, how does that need get evaluated, approved or declined, delivered, and supported? Where does this stop? It is a big issue when scaling DevOps. Second, while automation and cloud give you the technology to scale, organizing the people involved with DevOps can be a challenge. Culture change is just one part of it. Building a real continuing operating model around Agile IT concepts and then executing that operating model to improve the DevOps platform itself requires a lot of effort, time, and training for all people involved.  
  • As leaders focus on creating a collaborative environment and tearing down silos, automation and processes are critical. To improve code coverage quality engineering is pushed upstream as developers write unit test code cases to achieve 90% coverage. Have all automation in place with high levels of code coverage so the code is tested using 90% code coverage. 
  • Coordination between teams. It’s getting better but coordination is still a challenge. Most companies have occasions where coordination is misaligned. Push to prod before staging is ready. Coordinate to smooth the process quickly and easily. 
  • Getting everyone on the same page is very important as sometimes it is difficult to achieve. But with the right approach, it is definitely possible. Documentation is a key aspect of DevOps that many fall short on.  When scaling needs arise, proper documentation can really help in worst-case scenarios.

Automation

  • Resistance to change/removal of manual process steps that prevent full automation. 
  • Too often processes are manual and therefore inconsistent in their use and quality. You have to use automation. 

Other

  • The temptation to let ad hoc work occur outside of the established DevOps process is the biggest issue I see. Once you start letting any type of work occur outside, it’s a slippery slope for your team as its easy to justify other ad hoc work. It doesn’t take much to undermine a DevOps environment.
  • It's most difficult to have invested a lot to move quickly, you don’t want to slow things down. You put so much into going fast, but you have to be able to slow down to maintain reliability.
  • People struggle because DevOps is not cheaper than not doing DevOps. You are delivering more often and more reliability. Improvements in IT does not equate to doing things cheaper. Another problem is people playing with tools. The average DevOps pipeline has 32 tools. There is a lack of cross-functional teams. Environment management is underestimated as it is amazingly difficult to solve all of the different environments, spin up, spin down, customizations.
  • Security is squarely in the category and it doesn’t go away when you move to DevOps. The issue is more at the enterprise level doing DevOps with advanced projects but not everywhere – legacy apps being maintained in waterfall ways. We see the weird curve of new code built with DevOps but 90% of code is legacy. Once you get the basic pipeline in place, the DevOps process will improve over time.
  • The most common issues when scaling DevOps all pertain to visibility, which is key because if you don’t have visibility into the software delivery process, you will be unable to improve it – which is one of three main problems. The others are: 1) lack of visibility and collaboration across the delivery pipeline, making handoffs along the way is difficult and error-prone. 2) lack of visibility across multiple pipelines. With most "product releases" being comprised of multiple pipelines across multiple teams – all of which need to be synchronized in order to test and release – lack of visibility results in certain failure, or at the least, delays and scope changes.
  • DevOps is a mindset of bringing operational knowledge to development, yes, but it also comes with a panoply of processes, approaches, and ways of using tools that set it apart. The leaders of mature DevOps organizations simply think about engineering differently. The bad news is you can’t wake up one morning and decide to become a fully operational DevOps organization. But the good news is that moving toward DevOps is a journey that is never done. Which means that any team can make the move to become a more mature DevOps organization.
  • A common issue I see is that organizations don’t invest in proper monitoring early enough. For those instances when things go wrong, it’s essential to know when a new code or configuration was deployed, by whom, and what the changeset was. Without this tooling, it may take too long to identify the root cause of a failed deploy, causing people to lose trust in continuous deployment.
  • The real issue here is that more good engineers need to understand that DevOps is important, and they need to take on projects there as well.
  • First, an organization needs to be willing to change their mindset around how they implement their business operations. It’s very easy for customers to fall back into more of a project-based, Waterfall methodology for delivery when faced with urgent needs. This ultimately plays a large role in continuing to silo development and operations ultimately defeating the ability to build a culture which allows for “fast failure.”
  • Reduced visibility into systems because of lack of instrumentation. Difficulty keeping internal docs up to date in a rapidly evolving architecture. Low bus factors, where only one engineer knows how to work with a specific part of the stack.

Here’s who shared their insights:

Topics:
devops

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}