Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

How DevOps Scales

DZone 's Guide to

How DevOps Scales

Automation and culture mature as DevOps scales in the enterprise.

· DevOps Zone ·
Free Resource

To gather insights on the current and future state of how DevOps is scaling in the enterprise, we asked IT executives from 32 different companies to share their thoughts. We asked them, "How has your DevOps methodology changed as it has scaled?" Here's what they told us:

Automation

  • When doing a small DevOps effort it’s possible to leave some things ad hoc but not as you scale. CD and version control are automated with Jenkins, CI, and build servers. Many still don’t automate the feedback loop as rigorously as they could. How are the users and features behaving? When you scale, wizards don’t scale very well, and you have to help people leverage smart tools. Consistency between teams is a delicate tension. LinkedIn has a shared infrastructure of key components and established a trusted centralized system to make a change and watch how it going to determine if they’re moving the needle in the right direction. You need a common infrastructure for experimentation and sense-making to plug in and share and have the same version of the truth.
  • Companies take the best practices they see in their coding environment and bring in industry-standard tooling, such as Jenkins. The industry has seen customers moving DevOps from manual to fully automated. We have seen that shift frequently in standard software development. However, in the data warehouse industry, this shift has been a bit behind, but it is becoming more common.
  • Our main product is a software distribution which includes many components. For us, scaling DevOps is not just scaling the number of people working on the product, but also the number of components that make it into the distribution and require testing and integration. As we scaled as an organization, we invested more in building sophisticated automation for build/test/integration/release which significantly reduced the amount of time it takes to cut a new release. Today, releases are almost non-events and minimally disruptive for individual teams, allowing them to stay focused.
  • We’ve continued to invest in automation, end-to-end testing, and building redundancy into our infrastructure.

Culture

  • People see the benefits and stop fighting it. The cultural environment is getting better as the DevOps technology matures and people are better educated and less terrified. Start doing things better, deploy, and secure. Educational evolution. Less fear of failure, more anticipation of success.
  • It was a cultural shift for us. We had to adjust our culture, in conjunction with our processes. The introduction of tools like Ansible and OpenStack caused visible changes in our organization. The tools have been chosen to support the DevOps methodology, but don’t drive it. From a purely practical perspective, those tools provided us with the ability to develop and deploy ever-larger changes, with increasing iteration speed, clear ability to control and audit those changes. This was the deciding factor in scaling DevOps for us.
  • When I ran my consulting business, DevOps was owned by my developers. We didn’t have a separate sales and marketing department, so my developers effectively owned all of the operations. As my consulting business shifted from a dozen developers into a startup with 80+ employees, my approach to DevOps had to shift. Now, my product management team and customer success team give vital feedback — not just at a technical level — but at the level of the customer. Having my developers in the same room with product management and customer success has buoyed the kind of culture required to make DevOps successful.
  • The longer you go on with DevOps the more integrated and truer it becomes. Move from separate product and DevOps teams to more integrated where everything is in the scrum team, there is not a dedicated DevOps team. Go through steps of DevOps and silos to friendly comingled teams. There's a gradual blurring of the lines between dev and ops until it does not exist anymore?
  • We have realized that as more teams get involved in DevOps, two things happen:  1. We start sharing a common language and an understanding of the same workflow, making it easier to communicate and coordinate product ideas, innovations, and issues that affect customers.  2. More teams start asking to be included in the DevOps process, which is a great problem to have. This can mean more administration and overhead for our DevOps platforms but it also means more parts of the organization leveraging the DevOps mindset and using automation and orchestration to streamline their work, so it’s a win for the company but also one that requires some effort to keep up with.  Looking for ways to reduce administration effort will make it easier to keep saying yes to new on-board requests.

Feedback

  • From an outcome perspective, you promote more collaboration between product and engineering. More focus on continuous-release, deployment, and monitoring. Implement processes and automation with automated feedback. 
  • Continuous feedback loops with continuous improvement. It takes time for feedback loops to become clear and shorter for the engineering teams and the pipelines they have implemented. Once they’ve achieved maturity enterprises set up self-service DevOps practices where they implement the infrastructure or operations with a self-service low ops approach. More frequent releases. Building more of microservices architecture and deploying on the cloud while the other pieces stay in the on-prom model. Transform piece by piece ensuring there is no negative business impact.

Process

  • Things get easier as time progresses. You typically start following a more embedded model. As the DevOps team grows you end up embedding people into dev times, so they are participating in the early stages of the process. In the beginning, you’re just paying down debt. Now we’re spending a lot of time creating a central focus on the best way to layout K8s manifests, set on the right alerts, on file locations. Teams get close to their DevOps counterparts and you have a sense of ownership of both sides. 
  • As you mature, you go from looking at individual projects to scaling up to handle thousands of projects. It's difficult to take projects and turn them into DevOps overnight. You need to introduce practices over time, one at a time. Put in a stricter sprint schedule, then add a "show and tell" for developers to show their work to others on the team. Over the course of years, the DevOps matures in the organization. The dev pipeline starts out pretty simple. Over time, you keep building out the pipeline and increasing resilience. Add quality tools and advanced testing. Look at combinations of technology and run test suites to ensure not breaking things. It takes time to build and add sophistication around testing. 
  • The same way our codebase has scaled. Initially, you might have a small team running their containers on Docker Compose. As the complexity of infrastructure grows, it becomes increasingly complicated to deal with every single resource. Like orders of magnitude increase, the only way this can scale is to think of standardization as policies, e.g., every VM with this tag need to be snapshotted. And as you further scale, you'll see technologies such as Istio that specifically deal with orders of magnitude increase of what's being managed.

Education

  • We set aside a team to play by the new DevOps rules to figure out what it means. What it means for a new cloud-native project, what it means for the team members. We then saw what worked well and fixed what didn’t work well. We applied automation to processes and learned how to operate other software projects within our own. We started with one project, learned, and then rolled it out to different parts of the organization. Constantly evolve as technology, people, and software changes with continuous innovation and experimentation. Feedback loops monitor data and processes. Teach new people to run in a DevOps organization. Take into account the geographical background of distributed workforces. Get everyone on board while considering the unique background of the lab or the people. Be flexible and adapt to change, resources, and people. 
  • It’s really acted as a kick-starter. We had a “DevOps” team before we built our DevOps platform, but they had not changed the application development culture. Building the platform, advocating for DevOps, and educating the entire IT organization on cloud-native application development got DevOps moving for us.  Executives are now supportive, employees are motivated, and change is now expected. We have learned three valuable lessons: 1) Getting the base environment built and ready was a year-long effort from the initial vision to the first release. We realized early on that it would take time because of the complexity involved in getting DevOps workflows automated across developer tools, platform software, and IaaS. 2) Concurrently, with the base environment built, we took time to educate the application development teams on cloud-aware apps and our DevOps platform. By taking the time to educate the teams, we were able to establish a more trusted working relationship between the infrastructure and business apps teams to create a pipeline of new and existing apps to build or migrate. 3) We’ve had to keep the key platform and automation engineers focused. In IT, distractions are always plentiful. I think of our DevOps platform as a train which is rolling down a never-ending track, taking on new cars (features) and passengers (applications), with no end in sight. I see it as a service which will go on for years, so our engineers need to stay engaged throughout.

Other

  • Principles more than methodology with a whole new challenge. DevOps originated with unicorns who were designed differently than large enterprises with hundreds of applications, geographically dispersed teams in highly-regulated environments. It becomes a more transformational challenge. It’s like designing and building a jet while flying. Optimize the enterprise while driving and bringing new revenue streams online.
  • Most organizations start small with one app and one team. They start with their most experienced teams or "rock stars" and they're pretty successful. As organizations scale and duplicate across the rest of the team and apps, they don’t have the same knowledge and expertise as they did with the initial team. You need to find ways to standardize across teams. Reduce and simplify for ease of replication. Then you’re able to scale up. It’s not easy. People don’t like to change the tools they’re using. You can standardize your release pipeline if you can build a pipeline that allows you to swap out tools, you’ll have greater success and fewer bottlenecks.
  • Like any optimization, maintaining the strategic benefits of DevOps is a practice that requires vigilance and unwavering leadership support, throughout the organization. Maintaining the correct balance, coverage, and the maturity of DevOps team competencies becomes a crucial resilience metric for the business as DevOps is scaled up.
  • The foundation of an effective DevOps strategy is the CI/CD pipeline. As an organization evolves and grows, oftentimes CI/CD infrastructure gets neglected and becomes a bigger issue down the road when scaling is needed. It is important to scale and evolve the CI/CD as the organization matures. We’ve had to continuously evolve our CI/CD as the teams grew. The changes have been in the form of improvement to the availability, robustness, and ease of use of the CI/CD tools for new and existing developers alike.
  • As we have scaled, we have realized that it can become more difficult at scale, especially with different tools and technologies. That really drove us to centralize the support and tools being leveraged by the teams. And we’re still on that journey today.
  • When different teams start growing and the scale of operations are rapidly increasing, there comes a time when new DevOps strategies are necessary to keep up the agile methodology in pace. So, it's just a matter of time until every organization has adopted the DevOps way of software development practice.
  • At its core, DevOps methodologies stay consistent as they scale with teams being sized for efficiency (e.g. “two-pizza teams”). Successfully scaling DevOps requires an organization to overcome obstacles that are (most) notably present outside of the DevOps teams.
  • Better tooling for multi-site team collaboration. Tighter cycles for code review and using more security tools in the CI/CD pipeline.
  • What we’ve heard and seen with our customers is mainly the need for enterprise architectural guidance, along with execution capabilities through the agile DevOps methodology of delivery. We have found this allows us to partner alongside our customers through an iterative development process that ultimately supports their need for critical agility.
  • DevOps is pretty natural with small teams, but much more difficult as the engineering team grows. We have seen significant growth in platform engineering, site reliability engineering, and quality engineering functions as the team has gotten bigger. For example, as our company has grown we have organically seen some of our senior engineers develop specific expertise in developing and maintaining our core platform. We have also created specialized teams that focus on specific stages of the software development lifecycle, like Site Reliability Engineering for production platform support as well as Quality Engineering for supporting our testing processes and tools. The key is that those teams still have an engineering mindset and write and support code of their own, rather than just running and supporting tools.
  • Since we added our first SRE team seven years ago, we have morphed into two defined SRE functions with a “Pure SRE” role and an “Embedded SRE” role. Pure SREs are engineers who focus on and are recognized primarily for improving the reliability of systems in our platform. Conversely, Embedded SREs deliver on the reliability of specific products. This means they are a core part of the product team — while the entire team is responsible for product reliability, Embedded SREs provide an added and unique layer of expertise to the team.

Here’s who shared their insights:

Topics:
devops ,automation ,education ,continued learning ,collaboration ,cultural transformation

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}