Here are a few of the attributes to consider when choosing security tools and technologies to build your DevSecOps pipeline across the entire SLC. Please note that there is no one set of best tools for DevSecOps. The tools you choose should match the way that you build software, your goals, your culture, and the other technologies you use.
| Policy Coverage |
First and foremost, you must confirm that the tool actually covers the risks you need it to cover. Many products have surprising shortcomings in this area. See the OWASP Benchmark Project for help. |
| Accuracy |
Accuracy (eliminating both false positives and false negatives) is critical. Inaccuracy means humans have to fix results which will destroy your pipeline. You should carefully test your tools to be sure they accurately verify what you need. |
| Speed |
You need to test whether tools are fast enough to work as a part of your DevSecOps pipeline. That may be microseconds, seconds, or minutes…but probably not hours and certainly not days. |
| Scale |
Consider the size of your application portfolio and whether the tools you select are capable of operating continuously, in parallel, across that entire portfolio. Be sure to factor in the number of people you will need to make that work. |
| Process Fit |
You should verify that the tools are useful without complex installation and process. When well-meaning security folks buy tools for development and operations teams it can cause friction if they aren’t compatible with their technology stack or workflow. Engage the actual users in the evaluation process and conduct pilots to confirm they will be easy to install and use. |
| Integrations |
Verify that the tool integrates with the tools that people in your DevOps tool chain are already using. Look for well-documented, supported REST APIs and SDKs in a variety of languages, IDE plugins, webhook support, ChatOps integrations like HipChat and Slack, notifications with PagerDuty and VictorOps, and SIEM integrations like Splunk. |
DevSecOps Core Practices
DevSecOps takes a very agile approach to security, breaking down massive security tasks into incremental improvements that are performed as normal development tasks. These small batches of work include continuous verification, so that security builds over time instead of repeatedly starting over from scratch.
Once we’ve identified the next security challenge, our normal engineering process can execute on the improvement. In this section, we explore four core practices to any DevSecOps initiative. Of course, your DevSecOps process might be considerably more complex. See the next section for more ideas, or add your own practices to this basic cycle.
Fundamentally, it’s the constant tension between creating defenses and attempting to break them that actually makes organizations more secure. The faster you can repeat this DevSecOps cycle, the faster you can improve security. Over time, you’ll build a complete security story that will provide assurance both internally and externally.
1. ANALYZE: Identify Your Next Most Critical Security Challenge
Why should you always focus on your most critical security challenge? Generally, working on anything else won't change your security posture very much. It doesn't help to close the attic window when the garage and front door are wide open. In DevSecOps, we get the work flowing by creating small batch sizes. So, in most cases, we want to work on the most critical security challenge to our enterprise first. Still, don’t be afraid to choose a partial measure or a tiny improvement to your most critical challenge. Working in small increments makes sure we stay on track.
When deciding what to work on next, the team looks at all the potential security “work” available and makes it visible. The team might add new features, pay down some technical debt, make an architectural improvement, fix defects/vulnerabilities, or do something to improve the team’s tools or practices to improve quality, security, or productivity.
It’s important that the team use their threat model and security architecture to make an informed decision about the next most critical security challenge. What is the cost to the company of certain kinds of attack? What is the cost of implementing preventive measures for those attacks? Try to use data from both internal and external sources to figure out the next thing to do that will most effectively reduce risk.
You may find yourself without a threat model or security architecture. Fortunately, in DevSecOps, these artifacts are created one step at a time. Your best bet is to start with a system architecture diagram and consider threats across the entire application stack. When you’re starting out, it’s easy to identify your top challenges – the problems that are the most likely to be found, exploited, and cause serious damage, like injection, authorization problems, known vulnerabilities, etc… Consider all the different layers of your application stack;
| Applications and APIs |
Do you have proactive controls in place? Are you susceptible to common vulnerabilities? |
| Libraries and Frameworks |
Are your libraries and frameworks up to date and properly configured? Do you have a complete up-to-date inventory (with exact version numbers) of all the software you are running across all your servers? |
| Container and Cloud |
Have you hardened your platform configuration and kept it up to date? Are your cloud environments configured correctly? |
| Network |
Do you have strong network security defenses in place and monitored for attacks? |
Use a risk-based approach to deciding what to work on next. Be sure to consider whether there’s a viable connection between a threat agent, attack vector, weakness, technical impact, and business impact in your enterprise. OWASP depicts this connection as follows:
Going forward, you should practice a combination of threat intelligence and security research to continuously zero in on your next most critical security challenge. Note that this process is significantly different from trying to assess ALL your threats. Many organizations get overwhelmed trying to protect against everything at once.
DEVSECOPS THREAT INTELLIGENCE, SECURITY RESEARCH, AND SECURITY ARCHITECTURE |
| Threat Intelligence |
External sources: ISACs (STIX/TAXII), OWASP, SANS, BlackHat, DefCon, LASCON, DevSecCon, CISO events, friendly peer companies Internal sources: Monitor your systems for attacks and learn from the data. Understanding actual attacks is a major factor in prioritizing. |
| Security Research |
Security research should focus on challenging security architectures and identifying new strategic ways to improve security. Where possible, work with development to enhance the DevSecOps pipeline with new testing methods. |
| Security Architecture |
There is a dearth of great threat modeling and security architecture tools. But some interesting projects include OWASP Threat Dragon, IriusRisk, Chaos Engineering (Chaos Monkey, ChaoSlingr). |
You’ll have to work out your own cadence for re-examining your threat model. So, increasing your cycle speed will have a direct effect on the level of security you are able to achieve. And because DevSecOps adds to your security incrementally with continuous security verification (discussed below) you have protection against backsliding.
2. SECURE: Implement a Defense Strategy
Once you’ve decided on a security challenge to tackle, you’ll need to choose a defense strategy. A defense strategy isn’t a single security mechanism or product. A defense strategy can combine technical security mechanisms, secure coding practices, procedural controls, supporting processes, training, background checks, and more. We are using the term “defense strategy” broadly to include anything that you rely on to prevent a breach. Your defense strategy for a particular challenge can (and probably should) comprise one or more primary defenses and a set of supporting defenses as well.
You can capture each security strategy for implementation in a JIRA ticket that covers each security enhancement that you want to make, including:
| Challenge Description |
The goal is to concisely justify why this is an important security challenge. This might take the form of a security story or misuse case. And the description should cover the elements shown in the OWASP diagram above. |
| Defense Story |
This story should detail exactly how the defense should work. For technical defense mechanisms, the story should clearly detail how the threat is countered and why this defense is effective. For other defenses, how they work to provide protection should be argued. |
| Operational Guidelines |
As part of the defense strategy, you should also consider how to configure, operate, and use these defenses. This guidance should apply to end users, operations staff, and even indicate how developers should use the defense effectively. |
| Security Testing Approach |
The final part of the defense strategy is to detail how you will continuously verify the correctness and effectiveness of the defense strategy. |
Your strategy is right when you can easily answer with confidence when anyone asks, “How do you protect against X?” Having a clear, concise, defensible answer to this kind of question can not only provide an easy path to compliance but can also provide business advantage over competitors.
Your defense strategy doesn’t have to be perfect from the very start. It’s far better to start with a basic defense and evolve it over time. In fact, after you implement a basic defense, you may choose to work on another more pressing threat. The key to DevSecOps is to continuously reprioritize based on the threat and your existing defenses. The ability to respond quickly is critical for a world of continuously changing threats.
The work of actually implementing the security defense shouldn’t be any different than any other feature, and should to the maximum extent possible, be delivered as code or configuration with everything in source control. Managing security in this way makes it possible to test and redeploy at any time, ensuring that defenses are in place, working correctly, and properly configured.
3. VERIFY: Automate Security Testing
A key part of DevSecOps is ensuring that the defense strategies have been properly implemented, configured, and operated. Security testing is the way to verify that your actual security controls match your intended defenses. In DevSecOps, we focus on automating those tests by “turning security into code,” so that we can run them frequently without requiring humans, particularly security experts, in the critical path.
There are many ways to automatically verify the security of a system. There is no possible way to list them all, but we provide a few examples of popular tools that have proven themselves to be DevSecOps compatible.
| Automate Everything |
It’s not automated if you need humans in the loop. Don’t fall into the trap of thinking that you’ve automated security, when all you really did was automate the “scan button.” Think about the entire process. Does the tool require human expertise to configure or run? Does it require an expert to interpret and triage the results? We are looking to eliminate the involvement of humans in the critical path, so that we can push code to production with both speed and assurance. |
| Avoid “Tool Soup” |
Every tool that you adopt means additional process steps, a full set of integrations, and a team of people to configure, run, and interpret. Choose powerful platforms that will allow you to use address many different types of security challenge using an integrated framework. |
| Test Your Testing Tools |
Security testing tools vary greatly in their ability to test real applications for a broad range of issues. The only way to know how well a particular tool will work on your applications and APIs is to try it. Consider temporarily adding “tool canaries” in your applications to verify that real vulnerabilities are being discovered and false alarms are not being flagged. See the OWASP Benchmark project for details. |
Below is a DevSecOps security testing funnel to help you choose a security verification technique for the particular security challenge we are working on. This may seem obvious, but don’t blindly rely on the wrong tool. Take a minute to select the simplest, fastest, most accurate way to check that your defense implementation is correct, complete, and configured.
For example, testing for proper clickjacking protection is simple if you simply examine HTTP responses for the proper security headers. But it would be very difficult to verify this by looking at the source code, as there are so many ways that this might be accomplished.
DEVSECOPS SECURITY TESTING FUNNEL |
| What are we trying to test? |
Think carefully about exactly what you want to test and the results you want. Direct complete verification of application behavior is always best, but you can use sampling, fuzzing, design analysis, and other techniques. |
| Do we need to test it? |
Probably yes. But you should clearly prioritize things that are the most critical to security and the most likely to be discovered and exploited by an attacker. |
| Positive or negative testing? |
Are we able to verify that the application always follows a known good pattern of execution (positive security) or will we have to resort to trying to verify that the application never follows any of the known bad patterns of execution (negative security). |
| Do we already test for it? |
You may be using a security testing tool that covers this risk. It’s critical to verify that your tool does a good job of accurately and efficiently verifying your security defense. |
| Do we already have a platform that will allow us to test this easily? |
Perhaps you just need to enable a rule in a tool you’re already using. Or maybe you can use an existing tool as a platform for creating a custom rule. |
| Can we test it by writing custom tests? |
If we can’t use a security testing platform, can we create a custom test case? |
| Is there another tool on the market that can help test this? |
Network: nmap, sslyze, ssh_scan, Tenable, Qualys Cloud/Container: Aqua, Twistlock, Redlock, ThreatStack. Libraries/Frameworks: OWASP Dependency Check, retire.js, Contrast Assess, Snyk, Sonatype, BlackDuck Application: OWASP ZAP, Arachni, sqlmap, Burp, Contrast Assess, Microfocus, CA, Synopsys, Checkmarx |
| Do we need human experts to test it? |
While the goal of DevSecOps is to minimize the number of things that you need human experts to test. Bug bounties, red team exercises, and manual penetration testing can provide useful insight into defenses that are difficult to test automatically. Ensure that these efforts deliver rules, test cases, and other automation, not PDF reports. Examples: BugCrowd, HackerOne |
| Are we also testing for things we didn’t think of? |
It’s impossible to know everything, so certain kinds of testing rely on volume and randomness to try to force applications to behave badly. Fuzz testing and chaos engineering tools can help here. |
Any security issues discovered during testing should feed into the DevSecOps management infrastructure described above, to notify all the people that need to know through the tools they are already using.
4. DEFEND: Detect Attacks and Prevent Exploits
DevSecOps organizations recognize that you can never test yourself secure, so they adopt a balanced approach that focuses on minimizing vulnerabilities during development, and on identifying and preventing vulnerabilities from being exploited in production. While these two activities have traditionally been separate, DevOps has brought them together and DevSecOps projects support the full software lifecycle.
DEVSECOPS ATTACK DETECTION AND PREVENTION TOOLS |
| Runtime Application Self Protection (RASP) |
RASP uses application instrumentation to add attack detection and exploit prevention directly to applications regardless of where or how they are deployed. DevSecOps projects can use RASP to achieve accurate attack blocking and the flexibility to easily deploy in cloud/container environments. Examples: Contrast Protect, Immunio, Prevoty |
| Web Application Firewall (WAF) |
WAFs have been on the market since the early 2000s and have a history of complex configuration and spotty protection. Nevertheless, a DevSecOps project might be able to use a WAF for basic protection or as a platform for virtual patches. Examples: ModSecurity, Signal Sciences, Imperva, F5 |
| Network Intrusion Detection and Prevention (IDS/IPS) |
There are a variety of network, container, and host level protections against attacks. Seek out products that can be deployed and managed as part of your standard technology stack. Examples: Snort, Suricata, Bro, Kismet |
| Security Information and Event Management (SIEM) |
SIEM tools provide real-time analysis of security alerts generated by applications and network hardware, and are important to handling attacks in DevSecOps. Examples: Splunk, ELK, SumoLogic, LogRhythm, ArcSight |
The threat and attack data gathered should feed directly into the next DevSecOps cycle, to be used by security research to help choose the next most critical security challenge.
DevSecOps Additional Practices
There are an additional set of security challenges that emerge when an enterprise has hundreds or thousands of applications in their portfolio. Doing security at this scale is far beyond what a small dedicated security team can accomplish. DevSecOps is a technique for distributing this work effectively across development and operations.
It’s worth noting that in most organizations, only a small percentage of projects are very far along in their DevOps journey. So, managing the transition to DevSecOps across an entire application portfolio is a key part of the challenge
Standard Defenses and Enterprise Security Architecture
Generally, DevSecOps organizations, particularly those with large application portfolios, prefer standard security defenses that are heavily tested for correctness and effectiveness. Don’t reinvent authentication, authorization, encryption, etc… for every application and API you build. This reduces the amount of security work that has to be done and simplifies the organization’s security architecture. Achieving effective enterprise security architecture in a DevSecOps manner is beyond the scope of this document, but these guidelines are good first steps.
| Security Libraries |
You should generally use popular well-tested libraries for security features as it is dangerously easy to make small but disastrous mistakes writing your own. Probably the best approach here is to assemble your own Enterprise Security API that implements, extends, wraps, or is based on existing security libraries like Spring Security, OWASP ESAPI, BouncyCastle, OWASP Encoder, AntiXSS, AntiSamy, Jasypt |
| Standardize on a Standard Software Stack |
To the maximum extent possible, you want a standard approach to security, and you should use the security features provided by your software stack. This makes security invisible or automatic and reduces the likelihood of mistakes. Don’t assume they’re working, though – you have to continuously test these defenses. |
| Security Services |
Consider the extent to which you can turn security defenses into high assurance services that can be invoked by your applications. This creates the possibility of upgrading security across many applications without having to recode, retest, and redeploy. In the spirt of DevOps, make sure there is a self-service way for your empowered engineering teams to consume them without needing central approval or provisioning. |
Manage Software Supply Chain
Libraries have serious vulnerabilities, most of which have not yet been discovered by the good guys. And attacks now start within a day or so of new vulnerabilities being disclosed. So, every DevSecOps project must pay attention to the security of their supply chain.
Your software stack is composed of thousands of libraries, frameworks, modules, and components written by unknown developers all over the world. Using this software can allow you to create software much more rapidly, but the cost is that you must take full responsibility for the security of all that software.
In practice, this translates to a few best practices:
| Show Some Restraint |
Be cautious about the libraries and frameworks that you adopt. Stick to projects that demonstrate a solid security program, evidence of security testing, and effective response to new vulnerabilities. |
| Self-Inventory |
DevSecOps organizations must know the exact version of every library, framework, application, and API that is running on every server in every environment. The best approach is to enable all your systems to self-inventory by reporting their “bill of materials” to a central database. This “always up to date” inventory will enable you to respond quickly to novel attacks. |
| Establish Secure Coding Guardrails |
When you decide to take on a new library, standardize how you will use it safely (both positive and negative rules) and turn them into code so they can be continuously tested. |
| Test for Latent Vulnerabilities |
Before you trust your business to someone else’s code, you must verify that the defenses that are provided by the libraries and frameworks work as advertised and don’t contain undiscovered vulnerabilities. Very few libraries receive adequate security testing. |
| Continuously Monitor for New Vulnerabilities and Respond |
Modern applications are only fully composed at runtime, as their dynamic dependencies, plugins, and injections are fully realized. RASP tools (discussed above) can proactively prevent both known and unknown vulnerabilities from being exploited. |
Explode, Offload, Reload
While it’s not strictly necessary, many DevOps projects use cloud, containers, and APIs. Many organizations have already discovered that this is, in fact, the fastest path to securely achieving digital transformation. DevSecOps projects should strongly consider Ed Amoroso’s advice to “explode, offload, reload.”
| Explode |
Perimeters are no longer effective at stopping attacks, so the first step is to break up your monolithic internal infrastructure into smaller distributed workloads. |
| Offload |
Second, take advantage of security and cost advantages by moving these new smaller workloads to virtual cloud and container infrastructure. |
| Reload |
Finally, re-establish continuous security for each of your newly virtualized workloads by selecting and deploying modern protection technology at application, library, container, and network layers. |
This strategy is very compatible with DevSecOps, as it allows for efficient centralized security management with distributed enforcement for speed and accuracy.
Create Security Culture
Some companies simply seem to have the ability to take security seriously, focus, and do a great job. But others, even companies that seem to be doing all the same practices and using the same tools, are simply terrible at security. The difference is culture. And while culture is a difficult thing to change, there are a few key practices that have worked in organizations and should be part of a DevSecOps program.
| Executive Sponsorship |
You must have support from the executive level. They need to make it clear that security is everyone’s responsibility and that simply getting past the compliance audit is not the goal. |
| Micro-Training |
Everyone needs to understand exactly what their security responsibilities actually are. The best way to achieve this is by providing instant feedback while they are doing their job. |
| Accountability |
Development and operations team members should be responsible for the security of what they produce and operate. Security specialists shouldn’t be in the critical delivery path and should act as coaches and toolsmiths instead. |
| Security in Sunshine |
Make security as visible as possible. Be sure that vulnerability and attack data is never used to shame people. Instead, celebrate security risks as the fastest path to learning and improving. Only when security is visible can you achieve a culture of making informed risk decisions. |
What's Next for DevSecOps?
DevSecOps is still in the formative stages. The best way for you to get involved is to start implementing DevSecOps in your own organization and publish your experiences. Here are some sources of additional information.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}