10 Tips for Integrating Security Into DevOps

Imagine a world where product owners, Development, QA, IT Operations, and Infosec worked together, not only to help each other but also to ensure that the overall organization succeeds. By working toward a common goal, they enable the fast flow of planned work into production (e.g. performing tens, hundreds or even thousands of code deploys per day), while achieving world-class stability, reliability, availability, and security.

In this world, Infosec is always working on ways to reduce friction for the team, creating the work systems that enable developers to be more productive and get better outcomes. By doing this, small teams can fully leverage the collective experience and knowledge of not just Infosec, but also QA and Ops, in their daily work without being dependent on other teams, deploying safely, securely and quickly into production.

This enables organizations to create a safe system of work, where small teams are able to quickly and independently develop, test, and deploy code and value quickly, safely, securely, and reliably to customers. This allows organizations to maximize developer productivity, enable organizational learning, create high employee satisfaction, and win in the marketplace.

Instead of injecting security into our product at the end of the process, we will create and integrate security controls into the daily work of Development and Operations, so that security is part of everyone’s job, every day.

The Need for Force Multiplication

One interpretation of DevOps is that it came from the need to enable developers’ productivity, because, as the number of developers grew, there weren’t enough Ops people to handle all the resulting deployment work.

The ratio of engineers in Development, Operations, and Infosec in a typical technology organization is 100:10:1. When Infosec is that outnumbered, without automation and integrating information security into the daily work of Dev and Ops, Infosec can only do compliance checking, which is the opposite of security engineering—and besides, it also makes everyone hate us.

Getting Started

1. Integrate Security Into Development Iteration Demonstrations

Here’s an easy way to prevent Infosec from being a blocker at the end of the project: invite Infosec into product demonstrations at the end of each development interval. This helps everyone understand team goals as they relate to organizational goals, see their implementations during the build process, and gives them the chance to offer input into what’s needed to meet security and compliance objectives while there’s still ample time to make corrections.

2. Ensure Security Work Is in our Dev and Ops Work Tracking Systems

Infosec work should be as visible as all other work in the value stream. We can do this by tracking it in the same work tracking system that Development and Operations use daily so it can be prioritized alongside everything else.

3. Integrate Infosec Into Blameless Post-Mortem Processes

Also, consider doing a post-mortem after every security issue to prevent a repeat of the same problem. In a presentation at the 2012 Austin DevOpsDays, Nick Galbreath, who headed up Information Security at Etsy for many years, describes how they treated security issues, “We put all security issues into JIRA, which all engineers use in their daily work, and they were either ‘P1’ or ‘P2,’ meaning that they had to be fixed immediately or by the end of the week, even if the issue is only an internally-facing application.

4. Integrate Preventive Security Controls Into Shared Source Code Repositories and Shared Services

Shared source code repositories are a fantastic way to enable anyone to discover and reuse the collective knowledge of the organization, not only for code but also for toolchains, deployment pipeline, standards—and security. Security information should include any mechanisms or tools for safeguarding applications and environments, such as libraries pre-blessed by security to fulfill their specific objectives. Also, putting security artifacts into the version control system that Dev and Ops use daily keeps security needs on their radar.

5. Integrate Security Into the Deployment Pipeline

To keep Infosec issues top of mind of Dev and Ops, we want to continually give those teams fast feedback about potential risks associated with their code. Integrating security into the pipeline involves automating as many security tests as possible so that they run alongside all other automated tests. Ideally, these tests should be performed on every code commit by Dev or Ops, and even in the earliest stages of a software project.

6. Protect the Deployment Pipeline From Malicious Code

Unfortunately, malicious code can be injected into the infrastructures that support CI/CD. A good place to hide that code is in unit tests because no one looks at them and because they’re run every time someone commits code to the repo. We can (and must) protect deployment pipelines through steps such as:

• Hardening continuous build and integration servers so we can reproduce them in an automated manner.
• Reviewing all changes introduced into version control to prevent continuous integration servers from running uncontrolled code.
• Instrumenting the repository to detect when test code contains suspicious API calls.

7. Secure Your Applications

Development testing usually focuses on the correctness of functionality. InfoSec, however, often focuses on testing for what can go wrong. Instead of performing these tests manually, aim to generate them as part of an automated unit or functional tests so that they can be run continuously in the deployment pipeline. It’s also useful to define design patterns to help developers write code to prevent abuse, such as putting in rate limits for services and graying out submit buttons after they’ve been pressed.

8. Secure the Software Supply Chain

It’s not enough to protect our applications, environment, data, and our pipelines — we must also ensure the security of our software supply chain, particularly in light of startling statistics* about just how vulnerable it is. While the use of and reliance on commercial and open source components is convenient, it’s also extremely risky. When selecting software, then, it’s critical to detect components or libraries that have known vulnerabilities and work with developers to carefully select components with a track record of being fixed quickly.

9. Secure Your Environments

We must ensure that all our environments are in a hardened, risk-reduced state. This involves generating automated tests to ensure that all appropriate settings have been correctly applied for configuration hardening, database security, key lengths, and so forth. It also involves using tests to scan environments for known vulnerabilities and using a security scanner to map them out.

10. Integrate Information Security Into Production Telemetry

Internal security controls are often ineffective in quickly detecting breaches because of blind spots in monitoring or because no one is examining the relevant telemetry every day. To adapt, integrate security telemetry into the same tools that Development, QA, and Operations use. This gives everyone in the pipeline visibility into how application and environments are performing in a hostile threat environment where attackers are constantly attempting to exploit vulnerabilities, gain unauthorized access, plant backdoors, and commit fraud (among other insidious things!).

*See Sonatype’s 2015 “State of the Software Supply Chain” Report and Verizon’s 2014 “Data Breach Investigations Report.”

(Adapted from portions of The DevOps Handbook)