You’re probably familiar with DevOps by now. It’s the collaboration between Development and Operations teams by leveraging the same tools and processes to get things done more efficiently. Now, Security is being brought into the fold, and this is called DevOpsSec.
Since DevOpsSec is a much newer term and development practice, we wanted to take the opportunity to discuss how companies can get started with many of its foundational elements. There are no two people better equipped to talk about it than Threat Stack’s own Head of Operations, Pete Cheslock, and CTO, Sam Bisbee.
Rather than walking you through a polished slide deck, Pete and Sam wanted to open up the discussion in an AMA (Ask Me Anything) format. We fielded questions from Twitter, LinkedIn, Facebook, as well as email, and received hundreds of submissions. On Tuesday, March 14, in the middle of a blizzard here in Boston, they sat down for an hour to answer many of these questions live.
If you missed the AMA, you can listen to the full webinar above. If you want to read a quick recap, we’ve captured it all for you below.
What is DevOpsSec?
With different definitions being used for the same term, we wanted to start the AMA by finding out what DevOpsSec really is. “DevOps is about getting development and operations teams working together by leveraging continuous integration and delivery tools, so DevOpsSec is about bringing security into the fold,” explained Pete. By integrating everyone’s tools and processes into the same delivery pipelines, silos disappear, security becomes a part of normal conversation, and everyone hums along together at the same pace.
Have you seen a concrete improvement in security practices, and/or a reduction in cost, after implementing DevOpsSec?
Much in the same way DevOps simplified processes and streamlined tools to make things more efficient, DevOpsSec can benefit security teams. To that end, Sam stated there are definite improvements and cost reductions that can be seen. “With a shared language and understanding among teams, you begin to see efficiencies,” he said. “It’s not that there becomes fewer people to hire or fewer tools to use, but you get more out of them, as well as a more secure and compliant organization.”
Do you have to choose between speed and security with DevOpsSec? Or can you achieve both?
“In most circumstances, you don’t have to choose one or the other,” explained Pete. “DevOpsSec helps you identify changes in your environment as they’re made so you can continuously improve the security posture of your entire company.” In this way, security and speed go hand-in-hand. Sam adds that “Once you embrace the same tools and processes, you notice issues quicker, it becomes easier to respond, and you to cut down your mean time to detection.”
What does basic security hygiene look like?
“What we try to get people to think about first is categorizing what your common and likely threats are,” said Sam. Common threats include ransomware, botnets, and known vulnerabilities that apply to all organizations, and likely threats include the threats your unique organization is prone to.
If you handle PHI, and are HIPAA compliant, for example, a likely threat is data loss. “Once you know your common and likely threats, that should inform the type of protections you need and the metrics you should be watching to ensure that you’re following good security hygiene,” explained Sam. “The problem is people often jump right to the data because it’s never been easier to collect it,” continued Sam.
Basic security hygiene requires getting back to the basics and taking an inward look at your own security threats before getting carried away with data. “It’s about prioritizing your risks and then creating a plan,” said Sam. He recommends prioritizing threats on a scale from likely to unlikely and common to uncommon — a process often called threat modeling or risk-based analysis. “Then, balance that list against all the other work your team has to do so that you can create a comprehensive plan,” Sam advised.
What are the best ways to break down the walls between InfoSec and engineering teams?
While the answer to this question will change based on the size of your company, Pete explained that “Training will go a long way in integrating your security team with the rest of the business.” Training should include bringing DevOps tools and processes to the security team and ensuring that everyone understands what is going on and why. “One thing I’ve found to work well at a previous company was showing the security team the tools our development and operations team used on a daily basis,” explained Pete. From there, “I showed them what each tool does and how it integrates with the rest of the environment.”
Pete explained how they spun up a sample environment and sat down with the security team to show how they deployed code using configuration management. They then discussed how they could better monitor and audit those environments so that security would always know what ops was doing. “Once they understood our workflows and tools, our teams were able to work right alongside each other so that as we made changes, security could see what was happening.”
“Next, security showed us tools they were using that could help us be safer with code deployments,” continued Pete. Sam added, “If you don’t create this inclusive culture and share tools, metrics, and insights, you’ll land yourself in a situation where no one knows what the other person is doing.” Or, you’ll end up in a situation where teams bypass security altogether to get things done — a practice that is still all too common today. Sam explained that it’s most often fear that separates the security team from the rest of the organization, but if you take the opportunity to develop an inclusive culture, security becomes a part of the process.
What are the best ways to talk to management about changing the network perimeter model?
There are a lot of discussions out there today about the perimeter being dead, but the reality is, it isn’t — even in the cloud. What has changed is the definition of the perimeter. “With endpoints like mobile phones, laptops, and cloud servers, users can log in from anywhere,” said Pete. “You need to ensure you’re monitoring all these places for changes.”
Where companies often go wrong is only focusing on protecting the perimeter. “Attackers will eventually get in, so if you don’t also have protections inside your application, you’ll have no idea what they’re doing,” explained Sam. The same goes for insider threats — people inside your company already have access to sensitive systems and data within the perimeter, so without monitoring embedded within your systems and applications, you won’t know what they’re doing, either. “People are starting to recognize that they need to secure both the perimeter and the inside,” said Sam.
If a security pro is starting at a new company that has bad or no security practices, what’s the best way to begin implementing protections and policies?
If you walk through the door and immediately lock everything down, chances are you’ll quickly become the least liked person at the company. “While of course, you want to walk into a company and effect change, if the company has been doing things a certain way, they’ll be resistant to change,” said Pete. “Instead, take the time to ask questions, understand how applications work, and how people and tools work together.” Once you understand, then you can bring in new and better security practices. “That’s not to say if you come in and find a terribly egregious firewall rule that you shouldn’t change it or turn it off,” Pete explained. “But you have to strike a balance.”
And chances are the company isn’t neglecting or mishandling security intentionally; they’re either doing things a certain way for a reason (which you’ll soon find out) or perhaps they simply haven’t had the time to address it yet.
Once you get the lay of the land, “the easiest way to begin implementing security is by enabling monitoring,” Pete advised. This will help you see who is doing what so you can identify what needs to be done and start creating a roadmap to make the organization a more secure place. “Don’t expect to get everything fixed on day one,” Sam added. “Security should always be evolving and changing, so don’t think of it as one and done — it’s a slow and continuous process,” said Pete.
What are some best practices for using IDS, IPS, and firewalls on AWS, and are they necessary?
“While the type and implementation of IDS has changed, it is still necessary,” said Sam. Whereas years ago most IDSs were network based, today they're host-based. With IDS embedded at this deeper level, you can see activity across your entire cloud environment.
“Firewalls are also still necessary; it’s a network enforcement tool,” Sam continued. “Because the perimeter is still a thing, we always need to be thinking about it and what systems your applications should and shouldn’t be talking to.” Even though the days of plugging in an appliance are gone, the tools we’ve always used to secure infrastructure still apply. They’re just in new and better forms, such as IP tables and AWS security groups.
How do you protect secrets in production?
Keep it simple when it comes to managing and protecting secrets. “Start by understanding the type of secret before you start protecting it,” advised Pete. Begin by using:
- A shared password vault like 1Password or LastPass to manage passwords.
- Ansible or Puppet for storing SSH keys.
- Amazon Key Management Services (KMS) to make it easy to render files onto your host.
- HashiCorp's Vault Project to give developers temporary access to a database.
“There are so many great tools for managing secrets. Just be sure to choose the right one for you,” advised Pete. “If you’re currently storing secrets in a wiki file, for example, focus on moving them somewhere a little safer and build from there.” Sam adds, “Order your secrets by the level of criticality and frequency of use so you can determine where to store them.” If your SSL private certification is used a lot, for example, moving it to a shared password tool like 1Password is fine. For secrets you use infrequently, but which have high impact, such as code signing keys, place them in IronKey or even on a USB stick you can lock in a safety deposit box in your lawyer’s office or somewhere else that is highly secure.
How do you prioritize vulnerability assessment activities?
“Start by comparing your vulnerabilities to the common vulnerabilities and exposure (CVE) list,” explained Sam. “You can either build your own local tooling or use a third-party vendor to help you do that.”
From there, it’s important to continuously monitor for activity so you can see where, when, why, and how something happened. With this level of detail, you can narrow in on whether the vulnerability is an external or internal threat, or perhaps a server misconfiguration.
There are many tools that can help make monitoring and detecting vulnerabilities a part of your daily workflow. Pete explained that “Many DevOps tools can help to improve your security posture by providing centralized logging without any additional configurations.” This way, when a security event occurs, you can refer to these logs. Another way to do it if you’re on AWS is by using CloudTrail. “CloudTrail collects a lot of great data; you can then use vendors like Threat Stack to provide you with actionable intelligence based on that information,” explained Pete.
To become more proactive about patching vulnerabilities, many companies implement nightly security upgrades. “If your team feels it’s too risky to conduct upgrades at night, move the updates to your normal deployment and testing schedule so that as updates are made to code, and vulnerability improvements are implemented, too.
Big thanks to those of you who tuned into our AMA this week and to Sam and Pete for leading this excellent discussion. If you have any follow-up questions for us, feel free to tweet them to us @ThreatStack.