DevOps Security at Scale (Part 2): Why Separation of Duties Matters

This is the second blog post in a series discussing how high-performing DevOps teams build secure systems at scale. The first post is available here.

Operators should be operators and be focused on operations. Developers should be developers and should focus on… well, on development.

Sadly, in many software teams, developers and operators are also tasked with security duties: maintaining service accounts, defining security controls, controlling access to sensitive data, and more. Doing so places additional burdens on staff who have already got enough work on their plate. It’s also unreasonable to ask a developer to become an expert in an entirely new domain - security - when they’re already expected to be experts in software engineering or systems administration.

Separation of Duties Is a Velocity Enabler

Most articles available about Separation of Duties mention its benefits in preventing conflicts of interest and fraud, and how applying it can reduce the power of any one individual beyond what they should have. To some, that might seem constraining, as it invokes images of bureaucratic process and overbearing committees required to get anything done. However, a high-performing DevOps team will see Separation of Duties as an opportunity to focus their team members on what they’re best at, and optimize the team’s workflow appropriately around everyone’s abilities.

Sophisticated teams have well-defined roles and a clear separation of duties in their groups. Security and oversight is handled by dedicated security staff, developers focus on writing code, and operators make sure the production infrastructure stays healthy. The “handoff” interface between each group is codified in a Security Policy: developers create the security policy that declares what privileges their application or service requires, security staff then review and approve this code change, and operators make sure the application’s deployment goes as expected.

Combining Duties: A Painful Anti-Pattern

When any of Dev, Sec, and Ops roles are combined in an organization, it will inevitably lead to an issue where someone accidentally causes a serious problem. For developers, this is usually because they aren’t aware of the power they may have in a production system, and therefore, the extent of the damage they may inadvertently cause. In 2017, a junior engineer accidentally deleted his company’s production database on his first day. This happened because he was given full access to his employer’s production systems.

This was an implicit, unspoken combination of roles: by virtue of him having access, that unfortunate developer was also acting as an Operator, whether he knew it or not. The fault lay with his employer, not the employee: they failed to separate those duties and prevent the whole situation from occurring in the first place.

The Responsibilities of Each Role

• Developers should be able to write code and tests without fear of accidentally tearing down production, and are responsible for justifying and communicating their hosting and security needs for their applications to the Operations and Security groups, respectively.
• It is the Operator’s job to fulfill the hosting needs of the applications being produced and ensure that they remain available.
• The Security staff are responsible for granting the permissions to applications that Developers request, and ensuring sufficient security practices are put in place system-wide.

Just as embracing DevOps should not mean that Developers also take on the role of Operator, embracing “DevSecOps” does not mean that developers have to additionally become security experts. High-performing organizations have learned this, and doing so has led them to stop using the guideline of “move fast and break stuff”. Instead, their philosophy has matured to “move fast because you know you’re not going to break stuff”.

In the next post in this series, we’ll learn how high-performing organizations focus on flow and velocity while also maintaining a very strong security posture.

DevOps Security at Scale Series

• Part 1: Security Policy as Code
• Part 3: Focus on Flow and Velocity
• Part 4: Security as First-Class Citizen
• Part 5: Automation
• Part 6: Technology Adoption