Rachel King writes in her CIO blog on WSJ.com that Netflix engineers deploy code thousands of times per day. To maintain this pace, the streaming media company uses DevOps.
It’s a pace that’s not for the faint of heart, particularly when it comes to security. Netflix has more than 62 million subscriber accounts to protect worldwide. “The majority of security practitioners view DevOps as a huge threat,” said Gene Kim, co-author of a book about IT and DevOps called “The Phoenix Project.” But companies such as Netflix are using automation and other approaches to mitigate potential security problems while maintaining high velocity, he said.
It’s a problem Jason Chan wrestled with when he began working at Netflix in April 2011 as a cloud security architect and started building a team. The IT environment was starting to get quite big and the company was moving fast. “The only realistic way of maintaining security in an environment that grows so rapidly and changes so quickly is to make it automation-first,” said Chan, who is now director of engineering in cloud security at the company.
Netflix has since introduced a variety of security tools such as Security Monkey, Scumblr, and Fully Integrated Defense Operation (FIDO) that automate everything from finding compromised subscriber accounts to responding to security incidents. “When you move faster, it’s just logical that the ability for humans to operate effectively diminishes,” he said.
Security Monkey is a system that Netflix built and open-sourced in June 2014 that looks internally at the security of configurations. Since Netflix operates its infrastructure within Amazon Web Services, the tool continuously monitors and tracks AWS security configurations. It has a rules engine that can let Netflix know when things change and someone needs to look at it. It might be that a developer created a firewall rule that allowed access from a suspicious IP address or created an access control policy on a storage resource that might provide world-readable access, he said.
There are three components to Security Monkey: Watcher, Notifier, and Auditor. Watcher monitors a given AWS account and technology for changes to configurations. The second component, Notifier, lets teams know when something has changed. The third, Auditor, determines the level of risk associated with a particular configuration by running a set of business rules against each configuration. “There’s a continuum of changes and some of those changes are benign and happen all the time and then there’s a small percentage of those changes that we need to look at more closely and have a human get involved,” said Chan.
Another automated intelligence search tool, Scumblr, can search websites such as Pastebin for leaked names and passwords or look for compromised Netflix accounts that criminals try to sell on eBay. It then reports findings back to Netflix so they can help customers regain control of their accounts. Scumblr is essentially a Web application that lets Netflix create searches of sites like Google, Facebook, Twitter, and eBay that can run automatically, similar to Google Alerts. Scumblr works with a tool called Sketchy that collects screenshots and test content from potentially malicious sites. Sketchy lets engineers collect data while isolating their systems from getting infected by malicious software that may occur on these sites.
Netflix has also begun to automate incident response with a system called Fully Integrated Defense Operation (FIDO) that it open-sourced in May. The system can automatically analyze and prioritize security events depending on the severity. In some cases, it can even automatically isolate problems like disabling an employee account that has been compromised by malicious software.
When FIDO receives a security alert from a firewall or intrusion detection system, the system tries to find more context about what’s happening. It will check internal systems to see if something is targeting an executive, domain administrator, or the PCI zone--the most secure part of the network that handles financial transactions. The system also checks with outside threat information to determine if it’s a false positive or a more pervasive problem. FIDO then correlates that information, scores the threat and then takes further action, whether it’s emailing an engineer or disabling an employee account.
Automation is important to flag potential security problems that developers might miss, said Tom Pageler, chief risk officer of DocuSign. But DevOps primarily is an organizational approach, not a technology-based one, and for that reason any security model must include what he calls "security champions," development team members assigned to check each other’s work, according to Pageler, who was formerly a deputy chief information security officer at JPMorgan Chase. The security and DevOps teams are very connected, with security people assigned to the development team and vice versa, said Pageler.
Walmart, which has been steadily moving to a more agile approach to software development, follows a similar approach. It created a program called “security mavens” to increase security awareness among development teams. “In order for infosec and agile to be effective in an organization, you can’t have it locked up with a few people or a few departments that are narrowly looking at their portfolio of work,” said Julie Tsai, director of engineering in information security at Walmart Global eCommerce, speaking at the DevOps Enterprise Summit in October 2014. Instead, she said it needs to be embodied in people’s everyday practices.
By offering to help subsidize a security certification, the company was able to attract workers--from developers to product managers to QA--to become security mavens. About 10% of its engineering organization is now comprised of security mavens, said Kamal Manglani, agile lead at Walmart Global eCommerce, speaking at the same conference.
Over the past couple of years, as Walmart has promoted application security, there has been a 92% reduction in security defects, said Ms. Tsai. “There is no way we would have achieved this without being able to get in at the root level, where developers were creating the code and threat modeling and internalizing these things,” she said.
“We have to make humans more effective via automated decision making, automated data gathering and analysis,” said Chan. Security is a field where there’s a lot of judgments and decisions that people need to make based on a malicious actor and they may change their tactics in real-time, he said. “You really need to help get what’s most important in front of people as quickly and easily as possible, so you’re using your human resources as effectively as possible,” he added.