The original post can be found on the Electric Cloud blog.
Our expert panel included: Gary Gruver, co-author of “Leading the Transformation, A Practical Approach to Large-Scale Agile Development,” and “Starting and Scaling DevOps in the Enterprise”; Mirco Hering, a passionate Agile and DevOps change agent; Rob Hirschfeld, CEO at RackN; Steve Mayner, Agile coach, mentor and thought leader; Todd Miller, delivery director for Celerity’s Enterprise Technology Solutions; and, our very own Sam Fell.
Scaling Agile vs. Scaling DevOps
Lesson #1: Leadership
Hirschfeld on technical debt and DevOps: “There’s an element that can cause external observers, especially in DevOps, to get frustrated which is what we call Yak shaving. It’s the idea that before you can do the thing you want to do, you got to do something and there’s a prerequisite for that and a prerequisite for that and a prerequisite for that. So one of the challenges you get into is that you can be in a DevOps case, where your team feels like it’s doing nothing for a couple of sprints because it has to do all this prep in automation or refactoring so that you can put it into a CI/CD pipeline. My advice for people is to go in armed up with understanding that there’s a certain amount of technical debt paid to understand what technical debt means. Accept that your team might look like they’re completely at a standstill because they’re paying down technical debt.”
DevOps leaders need to give up the illusion of control, per Wallgren: “The leadership has to learn to get comfortable with, I was going to say giving up control, but it’s really just the illusion of control. I had a great epiphany a few months ago when I drove for the first time on the Autobahn. I say drive, but actually I was driven. I was in a car that steers itself and the first few times I took my hands off the steering wheel I was like, ‘Oh crap.’ I was really nervous, foot over the brake, all that kind of stuff. But what you realize is the machine works perfectly, it’s going be a lot better at paying attention to the car in front of me and braking when necessary than I am. So I’m not giving up control, I’m giving up the illusion of control because I’m human and I’m going get distracted and computers, as a rule, tend to not get that distracted. You also have to be willing to relax and learn to live with DevOps.”
Mayner explains that it’s not enough to want quality, you need to know how to get there: “As Deming said, it’s not enough for leaders to know that they need higher quality, they have to know the way. They have to know what it is that we must do because who else has the ability to change the system? And it’s not that we have bad people, we have great people, we’ve hired them. It’s the system. We have to know where it is we are going and even more importantly to get to my OCM hat on, that’s going to require people to change at every level. Down from the team level to the mid-management layer. The role of the leader is to create the safe space.”
Agile and DevOps can be challenging, but don’t let it stop you from reaching your goals, advises Hering: “I think leadership is a fantastic first lesson. I’m not sure whether you’ve seen any of the DevOps Enterprise Summit talks with Damon Edwards, but he goes with this curve where you’re improving and then it goes a little bit back or gets hard and then you would transition into a new phase. A lot of people get stuck in that first bump, and that’s where leadership grid is required. We’re okay, we’ve done Agile for this small digital application, we have done DevOps for our relatively trivial stack, but now we have the tricky, complicated application architectures that everyone refers to as the big monolith. That’s not three months of quick re-architecting, that’s a few years of loading the monolith. If you don’t have the platform leadership then you will get stuck or worse, you will start looking for an achievable goal. People might say, ‘We’ve got you three months deployments, we’re done here. We don’t need to invest any further, this was hard enough.’”
Leadership style is dependent on the type of architecture you have, says Gruver: “I think leadership style depends on if you have a loosely coupled architecture, or not. If you do it’s about empowering the team and removing roadblocks. If you don’t, you have a tightly coupled system, and somebody needs to be coordinating the work across teams. How the teams work is a second order of fact; how the teams come together to deliver value is a first order of fact and that’s a job that’s traditionally given to executives and they need to play that role. Executives need to prioritize things that are slowing down flow through the organization. Microservices will always go faster, small teams will always go faster. But I think the DevOps principles have the potential to add the most value in large, tightly coupled organizations because it’s about coordinating the work across teams.”
Treat your pipeline like a product, per Fell: “We have a concept here at Electric Cloud of treating your pipeline like a product. You should approach DevOps in an Agile fashion, you should approach Continuous Delivery in an Agile fashion, you should approach your pipeline in an Agile fashion. It’s your Minimum Viable Pipeline instead of Minimum Viable Product. These are all things that you can do to help your team start small, prove value, make sure you’re going in the right direction, make sure that when you are doing your minimum viable that it’s going to handle as much of the different use cases that you might run into.”
Don’t get caught up in speed, make sure you are doing things right, says Miller: “One thing I always like to be really cautious with about speed, is that it’s faster time to market. It doesn’t necessarily mean that you’re going to be building software faster. It’s the fact that Agile means that one part of our intention is to get software to production faster. It doesn’t mean that if you undergo an Agile transformation from a leadership perspective, that you should be focusing all your time and attention on measuring how the organization is being able to produce software faster. Because you’re always going to get the answer you want, right? There’s this false impression out there that we’re going to do, we’re going to do DevOps, and we are going be developing five times as fast as what we’re used to. No, we want be developing the right things.”
Lesson #2: Teams and the Pipeline
Running a bit over, we turned to the panel to share their lessons learned and proven patterns and tactics for scaling Agile and DevOps- either from the team’s perspective, or from the perspective of the pipeline – processes and tool-set.
Make sure your teams own not only code, but infrastructure as well, advises Hirschfeld: “If you’re looking at DevOps, teams are going to own their code into production but they don’t want to own the infrastructure. You really need to be looking at site reliability engineering, so that your DevOps teams can focus on their job owing code into production. Other teams that have done DevOps and collaborations but focusing on owning your infrastructure makes it that much more efficient.”
Leaders should help their teams solve problems, not just point them out, per Wallgren: “We have a phrase here at Electric Cloud where we say don’t just be a problem pointer-outer. Grab a shovel and help dig the trench. It’s not leadership to just poke at things and say that’s wrong, that’s broken. You have to come with a proposed solution or come with an opinion that you can help out. I think in any large scale organization, it’s that times a hundred. And if you’re in a situation like open source where you only have a carrot, but not a stick. For better or worse, you better get used to carrots.”
Hering notes the importance of autonomy in teams: “You need to enable teams to be autonomous. The key to this is easy consumption. So it’s not, we’re going to create this DevOps framework for our organization and there is a formal 300 page document to fill in if you want to do a deployment. It needs to be on an API base around this loosely coupled system; the same is true for the DevOps tooling. To make these services really easily consumable, create a DevOps platform that is common language across the organization that people can leverage, that provides common language, and gives you the flexibility to build on top and there’s a platform team that maintains that platform.”
Again, how you define team and how teams function depends greatly on architecture, explains Gruver: “When I think of team, it depends on how tightly coupled your architecture is and what you’re trying to do, and how often you are trying to raise it. If that can be a team of five people, great, you’re going to go fast, you’re going to move quickly, this stuff is easy. If that’s a group of 1,000 people, then I think the team is the executive leadership over that thousand people group. They need to map out how they are getting ideas to the customer and what the business’ biggest sources of inefficiency are. When I think of the team, I think of the team that has to coordinate the work to independently develop, qualify, and deploy code and manage it in production. Make that team as small as you can because it’s hard to coordinate work. If you can get it semi-small but you can’t get it any smaller, look for an architectural change that would maybe decouple two big parts of it.”
DevOps is a great way to give teams the ability to be creative, says Miller: “We have to keep in mind that it is the teams decoupling monolithic architectures. In my experience, I’ve had a lot of success in highlighting that there are a lot of dependencies across these teams to manage. Let them creatively think on the best ways to do that. DevOps is one of the most amazing ways to do this. You think about dependencies and integration issues, that’s really the idealistic solution. Regardless of leadership, the boots on the ground and creating teams that have all the skills necessary to get it into production is ideal. Let the smart people on the ground solve problems.”