Many years ago, as a young software developer in IT, I had the opportunity to work on a team of senior level professionals. We had a few developers, a couple of QA engineers, a systems administrator, and a DBA. We worked closely with each other in QA, Staging, and PROD environments. Stable systems were launched when the deadline was due. We had authority and autonomy to make things happen.
It was DevOps, I guess, before the word was invented, but we referred to ourselves as an IT Navy Seal Team. Years later when I was in leadership roles, I repeated this same team concept.
Now, there’s a quantum leap between what we were doing as a DevOps team then and what the Navy Seals do - obviously. But there are parallels. And those parallels are as relevant to DevOps today as they were 20 years ago.
Navy Seals work as a tight-knit specialized team. They are all ‘A’ players—there are no ‘B’ or ‘C’ players. Each person on the Seals team has a specialization, a dedicated role, and a particular purpose. They collaborate and coordinate closely. They trust one another implicitly to get things done.
Larger organizations will likely struggle to scale the Navy Seal team concept because one of the things that make Special Forces special is that they are small. Communication is much more fluid in smaller teams and it is cheaper and easier to acquire a few ‘A’ players than it is to find and pay top talent in bulk. DevOps demands a level of collaboration and coordination similar to what my IT Navy Seal teams enjoyed. How do you proceed if you are an enterprise? The supposed ‘unicorns’ – small companies or large tech companies that began within the last fifteen years – do not have the same combination of legacy systems, diversity of SMEs and technical debt that most enterprises do.
Making the Ideal DevOps Team
I will offer a few points for consideration. I will begin by referring to these small, autonomous, and authorized teams as DevOps teams. Defining these teams by functionality, product line, micro service and so forth is, in my view, critical to success. Making this determination will be different for everyone company, but I hope the following guiding principles will help in making that determination.
First, recognize that smaller teams are more effective than larger ones. To implement a DevOps team, you need to organize a number of small, ‘self-contained’ teams that have the autonomy, authority, and credentials to warrant and promote changes into escalating environments—including production.
Second, these teams need to include the separate disciplines necessary to architect, develop, test and deploy the change. This will include at least one developer, a QA engineer, a systems administrator and a DBA for each team.
Obviously other roles can be included – the app (or COTS) architecture will dictate the participants required for each DevOps team. The point is these teams should have within them the technical expertise to understand the scope and dependencies of designed changes, developing those changes (if needed), testing and validating the changes, performing dependant work such as database tasks, and have access to the target environments and be able to ensure compliance.
Third, recognizing that the workload of SAs, DBAs, and other SMEs may not lend itself to full workweek utilization, you can organize these roles into supportive or secondary teams that support two, but no more than three primary teams. To ensure collaboration is high, these supportive teams must be a defined and dedicated team and not a rolling roster of specialists. Likewise, supportive teams must be attached to the same primary teams, and not bounced around from this team or that team – that project or this project. Continuity and consistency will promote collaboration, innovation and reduce human error.
Fourth, consideration for legacy systems and the people who support them must be considered. This is because new, innovative or agile systems are in fact dependent on legacy systems though the rate of change and conflict is less frequent. There must be regular communication between legacy teams and the dependent DevOps teams. Again it is important that the same people are communicating and collaborating with each other in order to build trust and lessen miscommunications. Knowing the scope and impact of scheduled changes to legacy systems will allow the affected DevOps teams to reprioritize feature/function timelines if needed, in order to optimize an eventual interdependent release.
Fifth, begin with the end in mind. The whole point of IT and the objective of each DevOps team is to safely and consistently release changes into production and to quickly return to stability if an error should arise. In all my years of writing code when I was younger no-one ever said, “If it works in development it will work in production.” Always have the production use case in mind. Always.
Lastly, one of the Navy Seals’ mottos is, “There are two ways to do something ... the right way, and again”. Companies that fail to bring together the right blend of skills in their DevOps teams risk harming their enterprise continuous deployment pipeline….and starting again.