DevOps is a modern approach to software development. As companies today are being born into DevOps — and companies with more traditional approaches to software development are starting to experiment with, and even adopt, DevOps — the way in which security is being done at these organizations is constantly changing.

If we're going to talk about why security matters for DevOps, the first thing we should talk about is why DevOps matters. Security is about protecting value, and so first, you must figure out what you have that is valuable enough to protect. For example: You've been tasked with building and/or improving a DevSecOps program. How do you go about it? There are many frameworks and models that you could use (e.g., BSIMM, CSA CCM, ISO27017). Upon closer inspection, you might find these to be extremely long, overly complicated, and often challenging to implement.

Web apps have become more complex, cloud apps are increasingly API driven, and code is being deployed faster and faster. The attack surface has changed, and traditional DevSecOps has evolved. Someone might ask, "Won't the security team just take care of it?" In DevOps, developers have more power, and changes happen quickly, which means there is not always time for the traditional large security review from the security team. This means that it is critical for security to be built into development culture and into the development and deployment processes.

Over the last several years, security shifted from being viewed as a cost center to being an explicit business driver. SaaS companies are often described as "easy come, easy go," and business transactions now involve security as part of the actual sales process in order to evaluate security, compliance, and trust. A DevOps company that wants to do business with an enterprise client is going to encounter vendor security assessments, which insist upon meeting certain security requirements before a business deal is completed.

For DevOps companies, secure software is business-critical. Old-school enterprise software companies consider security to be a cost center, but DevOps requires security as a strategic business driver — both in terms of preventing unplanned work and rework and during the actual sales process when security requirements are specified as part of a vendor security assessment.

Organizations often insist that their software vendors perform DevSecOps activities because these vendors extend their attack surface and change their risk profile. DevSecOps is a common talking point during the sales process, and vendors may be required to provide evidence (e.g., penetration test results) before formal business agreements are put in place.

Cybersecurity has become a must-have for organizations that want to establish and maintain trust with their customers and partners, and DevSecOps has become a differentiator among competing software vendors. Security breaches, and associated news headlines, damage customer trust and have the potential to devastate businesses. Regulations like PCI, HIPAA, GDPR, FedRamp, and SOC2 suggest or require DevSecOps practices to be in place for organizations in particular industry verticals or those that handle certain types of data.

Security: Then vs. Now

Cybersecurity has changed dramatically in the last few decades. In Table 1, learn about some of the key differences between security "then" and security "now." The "then" timeframe refers to before–2010, and the "now" timeframe refers to 2011–present.

Table 1

A Simple DevSecOps Framework

Fundamentally, DevSecOps is about designing, building, and maintaining secure software. Good software helps organizations, and bad software hurts organizations. There are four main categories of DevSecOps activities — Govern, Find, Fix, and Prevent.

Govern

To do DevSecOps well, you must govern the DevSecOps program. There are a number of high-level factors to consider when you are thinking about DevSecOps, including compliance regulations, relationships with other organizations, and having a solid understanding of what you're supposed to be securing in the first place. It's also important to define metrics up front so that you can demonstrate the success of your program over time.

Find

To do DevSecOps well, you must find security issues. There are many ways to find security problems at different points in any SDLC, whether your organization follows a Waterfall, Agile, or DevOps methodology. Security problems exist in two broad categories — bugs and flaws. You can think of bugs as code-level security issues and flaws as design-level security issues.

Fix

To do DevSecOps well, you must fix security issues. It is not good enough to just focus on finding security issues — the quality of software does not improve until the problems are addressed or eliminated. Fixing security issues requires effective communication, coordination, and integration with development teams and processes.

Prevent

To do DevSecOps well, you must prevent security issues from happening in the first place. The people who build software must understand why vulnerable code is insecure. Developers must be empowered with tech-stack-specific knowledge and tools to help them avoid creating security bugs and flaws. Ideally, good programming practices and well-designed frameworks make it easier for developers to write secure software by default and harder for them to make mistakes.

Cloud environments must be configured correctly in order to prevent security vulnerabilities from being exploited, and attacks must be discovered and stopped as early as possible in order to minimize damage.

There are several key methods and techniques that can be used to incorporate DevSecOps into the SDLC. Less mature DevSecOps programs use a few; more mature DevSecOps programs use several.

Success Metrics

In DevSecOps, one size doesn't fit all. Standards and controls are built based on years of practical security experience at real organizations, but this is something that is constantly changing. Today, every DevSecOps practitioner needs to know how to optimize their unique program using data, metrics, and risk management objectives.

Like any business initiative, a DevSecOps program should have objectives and measurements to determine if those objectives are being met. An example of a risk management objective for DevSecOps: "Reduce the probability of attackers causing critical applications to stop functioning." A typical measurement that organizations use to help measure DevSecOps success is defect density, meaning the number of vulnerabilities divided by lines of code (lines of code are typically indicated by the 1000s).

A security metric often measures activities to provide decision support to perform functions better in the future. This data can help answer questions that an executive or operator might have about a particular area, such as source code review, using evidence-based information instead of opinion or anecdotes.

It's often said, "If you can't measure it, you can't manage it." It is indisputable that measuring results and performance is crucial to an organization's effectiveness, and this definitely applies to DevSecOps.

Defect Discovery and Testing

Penetration tests (also known as "pen tests") are a type of manual security testing that provides insight into an app's security by systematically reviewing its features and components. This type of exercise improves the security coverage of an app because the test is intended to explore the complete app rather than just focus on one type of vulnerability or one particular section. Pen tests follow methodologies related to topics like input validation, authentication, and access controls in order to identify flaws in the app's implementation.

Closed-box testers operate with limited knowledge and open-box testers utilize as much information as they can to inform their approach.

Pen Testing as a Service (PTaaS) provides on-demand manual penetration testing for web applications, mobile applications, and APIs. Findings are delivered through a platform that integrates with developer tracking systems like JIRA and GitHub. A SaaS platform also facilitates collaboration between pen testers, security team members, and development teams to not only find but also fix issues.

Security scanners can be programmed to identify certain kinds of vulnerabilities automatically. DevSecOps scanners come in two flavors:

• SAST scanners ("S" for "static" DevSecOps testing) examine the source code, binary, or byte code of an application.
• DAST scanners ("D" for "dynamic" DevSecOps testing) examine the application from the outside when it is running.

The most interesting and important security findings cannot be discovered via automated means alone. Human intelligence and creativity are necessary to discover security flaws in business logic. There are entire classes of security issues (authentication and session management) that cannot be discovered using automated tools.

Code review is the manual review of one developer's code by another developer. It's intended to find mistakes and improve code quality. Similarly, secure code review is a manual code review by a security expert. This is intended to find coding errors that may introduce security vulnerabilities. Secure code review is a manual process that often leverages SAST technology. Every so often, a security researcher not directly associated with an organization will discover and report a security vulnerability, which is called vulnerability disclosure.

A bug bounty is a type of vulnerability disclosure program that leverages a crowd of globally sourced researchers in competition. In a public bug bounty, anyone in the world can submit a potential security vulnerability to an organization, and the first to find a valid bug will be paid a "bounty."

Teamwork

Once you have performed security testing to find as many issues as possible, the next step — and by no means a trivial one — is to communicate issues to the development team. The development team is a critical stakeholder when it comes to prioritizing the fixes, remediating the issues, and ideally, preventing the same issues from coming up again.

One of the best ways to get security bugs fixed is to integrate with developer tools and processes. Now that you have your software inventory and identified security issues, you'll want to keep track of what has been tested, by what means, and when. Keep track of the findings from each security test and prioritize any necessary bug fixes or feature enhancements. In doing so, use business context to better understand which issues matter the most and work with development teams to fix those first.

Make sure you always know which issues are open and which have been addressed and can be closed. Then, report summary information to the relevant stakeholders so that everyone is always aware of the current status.

Proactive Techniques

The best DevSecOps training for developers is based on real security findings, whether these trainings are demonstrated during an actual security incident or found via manual penetration testing. The OWASP Top 10 contains a list of common DevSecOps risks; however, each organization will have its own unique top 10 list. Within your own organization, use this information to prevent entire categories of security vulnerabilities by implementing developer-focused training.

There are two common types of DevSecOps problems: bugs and flaws. Bugs are code-level mistakes, and flaws happen at the design level. Threat modeling is a type of design-level security assessment that is intended to examine the way an application system works to identify potential flaws. The process involves analyzing assets, security controls, and threat agents in the context of an application system. When flaws are detected using threat modeling before software implementation, some security problems can be avoided.

A few examples include:

• Cross-site request forgery tokens prevent cross-site request forgery attacks.
• A content security policy defines open-list assets that the browser allows to load and execute, thus minimizing the impact of cross-site scripting exploits.
• HTTP Strict Transport Security encrypts data in transit and prevents fallback to non-HTTPS traffic.

Other security issues can be avoided by securely configuring the software environment, for example, by following the Amazon CIS benchmark to harden AWS accounts and cloud services.

There are various tools meant to protect an application by identifying and stopping malicious activity while the application is running:

• Web application firewall (WAF) examines web traffic to identify and block suspicious activity, such as comment spam, XSS, and SQL injection attacks.
• Runtime application self-protection (RASP) operates in the runtime environment to monitor, detect, and alert in real time.
• Interactive application security testing (IAST) works inside an application, typically in a QA environment, to analyze code and report vulnerabilities.

Both WAF and RASP can be run in either "detect and alert" or "detect, alert, and block" mode. They are most effective at preventing security issues when running in "detect, alert, and block" mode. However, this forces the business to risk blocking legitimate application activity as well as malicious activity.

The "Three Ways" of Security

For decades, both software and security have struggled with poor quality results, cost overruns, and processes that require experts. While DevOps has shown promise on the software side, security is still practiced in very traditional ways. DevSecOps is not just shoving traditional security practices and tools into DevOps.

Instead, we must rethink the security work. We will need new practices and technologies to perform this work. We can give this transformation structure using the "Three Ways" from The Phoenix Project. By framing the problem this way, we can see that we need to:

• Get security work flowing – Most security work is monolithic and attempts to cover all risks in a single task, like a complete security architecture or security scan.
• Ensure instant security feedback – Security is one of the most common causes of technical debt, and the cost of this work increases dramatically the further it progresses across the SDLC. Several reasons include a lack of knowledge and limited security specialists.
• Create a security culture – Many organizations have a security culture of blind trust, blame, and hiding that prevents developers and operations from working with security.

Let's learn how to tackle the "Three Ways" of security:

1. Get your security work flowing
   • Make the work visible
   • Work a single security challenge at a time
   • Limit work in progress and reduce handoffs
   • Automate everything
2. Ensure instant security feedback
   • Increase awareness about the importance of security
   • Identify potential problems
   • Make problems instantly visible
   • Swarm on the problem and seek the cause
   • Ensure security "findings" are designed for easy consumption
   • Focus on providing a solution rather than exaggerating the problem
3. Encourage a security culture
   • Empower everyone to challenge security design and implementation
   • Take every opportunity to make security threats, policies, architecture, and vulnerabilities visible
   • Allow everyone on the team to participate in security
   • Trust that engineering teams want to do the right thing
   • Celebrate the knowledge gleaned from security issues rather than blaming those involved
   • Spend more effort on upgrading practices and preventive measures than vulnerability remediation and incident response
   • Plan trainings and conduct workshops to maintain continuous security throughout all teams

By following these core approaches, teams will see security as a concrete output from the development process. It is a combination of security features and assurance captured in a tangible way. By applying DevOps concepts, we can produce this concrete security continuously and effectively as a part of standard software development.

The "Five Ideals"

Six years after The Phoenix Project was released, The Unicorn Project was published in 2019. The Unicorn Project is not the sequel to The Phoenix Project. In fact, the stories of both novels take place along the same timeline and provide two different perspectives. The Phoenix Project introduces the "Three Ways" of security, whereas The Unicorn Project introduces "The Five Ideals."

Gene Kim, the author of both books, introduces "The Five Ideals" to frame today's modern business and engineering challenges. Let's further explore these five concepts:

1. Locality and simplicity relates to the degree to which a development team can make local code changes in a single location without impacting various teams.
   • We need to design things so that we have locality in our systems and the organizations that build them; we need simplicity in everything we do.
   • The last place we want complexity is internally, whether it's in our code, organization, or processes.
2. Focus, flow, and joy is all about how our daily work feels.
   • Is our work marked by boredom and waiting for other people to get things done on our behalf?
   • Do we blindly work on small pieces of the whole, only seeing the outcomes of our work during deployment when everything blows up, leading to firefighting, punishment, and burnout? Or do we work in small batches, ideally single-piece flow, getting fast and continual feedback on our work?
   • These are the conditions that allow for focus and flow, challenge, learning, discovery, mastering our domain, and even joy. This is what being a developer means.
3. Improvement of daily work addresses paying down technical debt and improving architecture.
   • When technical debt is treated as a priority and paid down, and architecture is continuously improved and modernized, teams can work with flow, delivering better value sooner, safer, and happier.
   • The business ultimately wins when developers can deliver on enterprise performance goals.
4. Psychological safety is one of the top predictors of team performance.
   • When team members feel safe to talk about problems, problems can not only be fixed but also prevented. Solving problems requires honesty, and honesty requires an absence of fear.
   • In knowledge work, psychological safety should be treated with the same importance as physical safety is in manufacturing.
5. Customer focus relates to the difference between core and context as defined by Geoffrey Moore.
   • Core is what customers are willing and able to pay for, the bread and butter of your business, while context is what they don't care about and what it took to get them that product, including all of an organization's back-end systems (e.g., HR, marketing, development).
   • It's critical to look at these context systems as essential, as mission-critical, and fund them appropriately. Context should never kill core.

Here are some key criteria for how to set up a DevSecOps program:

• Check for accurate software inventory. It's difficult to properly secure a piece of software if you don't know it exists or you're not keeping track of it. Make a list of all your web, mobile, and cloud applications and APIs. You also need to have an up-to-date, comprehensive list of various software components, dependencies, versions, and open-source tools. 
• Conduct risk ranking. Assign a risk rank to each piece of software. Use criteria such as business criticality, data type, and accessibility to group applications. Important and high-risk apps should have more (and perhaps different) security activities applied to them than less important and/or low-risk apps.
• Adhere to compliance and governance requirements. Many organizations are subject to security requirements because of an application's business function (e.g., payments), type of data stored or processed (e.g., healthcare data), or geographical location (e.g., regional requirements for data privacy and protection).

Others may be required to perform specific security activities due to contractual obligations. Due to the nature of compliance requirements, certain security controls may not be optional. Determine these requirements and ensure your organization meets the appropriate standards.

• Beware third-party components. While it'd be nice to say, "Vendor security goes both ways," an organization's DevSecOps is affected by both buyer requirements and by vendors' risk profiles. An application that uses third-party software components — including open-source components — takes on the risk of potential vulnerabilities in those dependencies. These should be identified, tracked, and accounted for in the same way as every other software component, as described above. 

If a software vendor is selling their application to a buyer, that buyer may require specific security activities for the application, such as manual penetration test results, a response to a vendor security questionnaire, or evidence of certain security policies.

DevSecOps is very much a team effort. Security professionals can't do it alone — they must collaborate with development teams. A holistic approach that matches DevSecOps talent and resources with the right technology is required to run the program. This is both a science and an art. Each organization will have a business-specific context for different resource allocation trade-offs, program-level metrics, and KPIs. Today's requirements for DevSecOps activities include:

• Cost that enables higher frequency testing and greater coverage across an application portfolio
• Access to quality security talent who can perform manual testing on demand
• Strong integration with development processes and tools in order to fix security issues and prevent them in the future

In short, today's DevSecOps activities must be agile, actionable, and cost effective.

Additional Resources

Here are some industry recognized standards and frameworks that describe DevSecOps practices:

• Building Security in Maturity Model (BSIMM) – I like the BSIMM because it's specific. The measures in the BSIMM are activities that a development organization actually does to produce secure software. Another key differentiator of the BSIMM is that it's real-world.  The creators of the BSIMM made a rule that no activity would be included in the BSIMM unless at least one of the organizations they interviewed actually performed that activity.
• ISO and IEC standard for information security (ISO/IEC 27001) – The ISO27017 and ISO27034 standards are examples of where controls, additional to those in ISO/IEC 27001, have been defined to meet the requirements of the specific sectors concerned. These recommendations provide controls and implementation guidance for both cloud service providers and cloud service customers.
• Cloud Security Alliance Cloud Controls Matrix (CSA CCM) – The CCM, the only meta-framework of cloud-specific security controls, is mapped to leading standards, best practices, and regulations. CCM provides organizations with the needed structure, detail, and clarity relating to information security tailored to cloud computing. CCM is currently considered a de facto standard for cloud security assurance and compliance.

Additional DZone Refcards on DevOps and CI/CD security for further reading:

• Software Supply Chain Security: Core Practices to Secure the SDLC and Manage Risk
• Getting Started With CI/CD Pipeline Security
• Cloud-Native Application Security Patterns and Anti-Patterns
• Threat Modeling: Core Practices to Securing Applications
• Threat Detection Core Practices
• Advanced Cloud Security: Continuous Security Strategies for Cloud Infrastructure
• IaC Security: Core DevOps Practices to Secure Your Infrastructure as Code
• Continuous Delivery Pipeline Security Essentials