The most common failures are related to culture, collaboration, and adoption/change.
Join the DZone community and get the full member experience.Join For Free
To understand the current and future state of DevSecOps, we gathered insights from 29 IT professionals in 27 companies. We asked them, "What are the most common DevSecOps fails? How are they rectified?" Here's what they told us:
- Changing the culture and mindset is not easy. Executives are tasked with this. We look to our executives to make DevSecOps and security something that’s infused throughout the organization. Until recently there was still a lot of talk and not a lot of action. 30% had not implemented a DevSecOps model. The executive team needs to create a built-in security culture, every step of the process so security is a priority for personal, team, and organizational goals. You must change mindsets throughout the organization.
- Security staff must embrace the DevOps culture and show that they add value without adding friction. In particular, the security organization must be willing to give up manual release approvals.
- The most common DevSecOps fails are: 1) Lack of assurance to business and project teams: understand the business context; identify and rank risk; engage senior technical people to work on security; integrate security activities (Threat Modeling, SAST, DAST, Pen Testing) in the SDLC. 2) Cultural barriers to collaboration: build collaboration between security, development, and operations teams; build a continuous security mindset. 3) Lack of security as a top priority for the business, auditors, and development teams: aim for long-term retention rather than short-term training; integrate workflows across teams to promote security. Starting from scratch: use existing standards OWASP Top 10, NIST 800-53, ISO 27001. 4) Lack of automation: emphasize automation wherever possible to drive consistency; use of virtualization and containers; continuous application and performance monitoring.
- 1) Security must be an up-front concern for the team. Adding security later is not going to provide enough protection. 2) Don’t assume your developers and testers are experts in security. Train them up and continue to supplement their training on an ongoing basis to keep up with new developments and threats. 3) Partner with security professionals to ensure that your efforts are effective and efficient. 4) Your development and test environments are not necessarily your production environment. Security testing must be performed in production in order to uncover vulnerabilities that might not be apparent during development. 5) If you’re using open source libraries and components, check regularly that the versions you’re using have no known vulnerabilities, and update them if necessary.
- DevOps overall is first and foremost a cultural transformation within an organization. It is aimed at breaking down the barriers between development, support and QA in order to create an environment based on collaboration and shared accountability. Difficulty in communication and collaboration, finger-pointing and a lack of enthusiasm for the common goals of the team are generally early warning signs that the DevOps initiative is not going well. These same principles apply to DevSecOps. In order to be successful, DevSecOps must be embraced by the entire team.
- I would suspect a risk would be getting the security team to move past the culture of “no” to a collaborative environment.
- It goes back to collaboration which is one of the biggest impediments to DevSecOps. Developers have to get features out ASAP. However, every business cares about security after a data breach. A business needs to allocate some percentage of a development cycle to security for security to be involved from the beginning. It’s a business issue, not a technology issue. Allocate time, allocate talent. When you look at good security programs it’s more than tools. The AppSec teams are doing internal training teaching developers to write code secure code from the beginning.
- One of the most common DevSecOps fails is that many of today’s security tools are not being easily adopted into the newer development or ops toolchains. And on the flip side, security tools that are built with these workflows in mind are minimal, so teams instead are bolting on legacy security tools that are not designed to seamlessly assimilate with the newer process. The best approach is to design a strategy with the typical DevOps toolchain in mind first and leverage existing Dev and Ops tools to achieve the same objective wherever possible, and for all teams –– developers, technology operations and security –– to work together to identify which tools will fit best to what they are trying to accomplish.
- The challenge around people embracing DevSecOps. It takes a while for people to realize the benefits. They need to see the results. During Heartbleed, we ingested it in the pipeline and then discovered it in our build platform. It’s really helping. People are not paying enough attention to the value of DevSecOps. It’s more of a people thing than about tools.
- Policies around security are instituted but viewed as a burden. If they are not part of the core way the operation runs and allow people to take shortcuts, then you will need to enforce the importance of the operations and why they are there. Make sure the tools help handle the operations for them and followed in a consistent way. Flag things when processes are not being followed.
- The best security practices are not implemented at every level of the product life cycle, or incompatible practices are implemented at a different level of the product life cycle. One way to rectify this is to adopt an agreed upon best practice and use it everywhere.
- Trying to shoehorn an existing tool into a DevOps process. Aim to be as frictionless as possible. Look to create value. Static analysis code checks can take a long time, dynamic analysis testing tools that take 12 hours won’t work. Don’t be married to tooling that exists. Be open to new ways of doing things.
- One of the most common — and cited by 70% of respondents in our recent DevSecOps survey — is agreeing security is important but not giving anyone enough time to do it. Lack of cross-functional support in the organization for security — for developers getting involved in security automation, and for security teams leaning “left” into development planning — all but guarantees a failure to execute on DevSecOps initiatives. Related to this misstep is another common one: failure to shift mindsets effectively to see security as an enabler for speed and success, rather than “necessary evil.” Delivering fast at any cost is actually more expensive in the long run than delivering well the first time. Even strong DevSecOps automation isn’t a complete security posture. Just because all the automated security tests passed doesn’t mean security is “done” and failure to think about security as a key component of software delivery throughout the end-to-end process creates bigger security headaches later. Finally, all too often DevSecOps is reduced to “automating easy security tasks” and — just like DevOps — it is much more than an automation project. The DevOps method of tearing down walls between Dev and Ops to share insight, knowledge, process, ideas (and yes, also automation) between those groups is critical to what makes DevOps successful, and the same is true for DevSecOps. It’s as much about a way of thinking and working as it is about tooling and orchestration, as tools are only as savvy and forward-thinking as their designers and operators.
- The most common failures happen when DevSecOps is given lip service but not actually applied – commitment is necessary. The most common failures experienced when people aren’t adapting to the new environment and continue doing things as they would in a previous environment. This can include relying on others to secure the network for them, not realizing the impact of their change and an overall lack of change management. A lot of this can be fixed with proper training for users, selecting the right tooling to help develop and enforcing policies, such as firewall rules, patching, configurations, etc. This does require a change in mindset and there is a natural resistance to change in people. Don’t get frustrated if this takes time, be patient and persistent.
- Teams playing whack-a-mole. Also, stressing the importance of usability vs. security. You have to be practical and realistic about whether the weakness is bad and ask what the compensating controls are.
- The most common is to buy a code scanner to incorporate in the CI/CD pipeline and it doesn’t work, checking the box rather than integrating with the security team.
- You know it’s not working when you have slower feature velocity, less scale, and more downtime. Every business wants to get faster. Banks are getting better pushing to the mobile app. If security slows it down, then you know security was the wet blanket. When you add more security to Agile and the culture shifts to Waterfall you know something broke.
- Security teams have cloud-native infrastructure bubbling up from ops. This is a problem they do not understand the need to do a lot of research, to get up to speed. There’s a slight slowdown in how fast can go live, in order to learn.
- 1) Many security teams don’t understand how automated modern pipelines can be, and why it’s critical to their success. Traditionally, security can be a roadblock slowing the pace of development, but in the new DevSecOps culture, the entire team can be part of the automated process. Dev and DevOps want to move quickly but know little about the application and network security. As security “shifts left” to them, it can initially be frightening to hold responsibility for helping to prevent major data breaches and exploits of their applications. 2) To close the gap between DevOps and Security, security teams should learn more about new methods for deploying applications (including security products) using Docker and Kubernetes, in order to better understand how powerful they are. Similarly, teams should learn the basic reasons why security teams need network and application visibility, and brainstorm how to provide these most efficiently within a container pipeline process.
- The most common DevSecOps fail is the reliance on only SAST (static security analysis tools) which tend to create a lot of false positives. Any team implementing DevSecOps should consider making SAST and DAST (dynamic security analysis) part of their pipeline and ensure that they do not rely on generic rules but customize them to ensure they reduce the false positives.
- The most common failure is having an exception process. You can’t fill security gaps with paperwork. The best thing an exception process can do is raise awareness of a flaw and expose a risk profile. However, an attacker or an evil user doesn’t really care if enterprise architecture or a CIO have signed off on knowing that the system can only use eight-character crypt-hashed passwords. They also don’t care if the security issues are fixed in production. They’ll just land in dev — especially if passwords are re-used, or data is replicated from production into test.
- 1) If you are not building security into the process you are not doing it correctly. Failure to institutionalize security as part of the process. Start at the very beginning. 2) Not having an appropriate incident response process. An incident happens, what do you do? How do you communicate internally and with your customers? Own up to it and fix it as quickly as possible. Develop a comprehensive incident response plan.
- Snowflake deployments where developers get involved with new technology and lose track of it and its in production with mission-critical use. For organizations that care about security, you need best practices, a standardized platform, and infrastructure so security holes don’t sneak into production.
- DevSecOps fails when it’s too slow, too manual, or too difficult to comply with. The beauty of DevOps is automation and self-service; really leaning into that by building policy into programmatic guardrails, is key. Don’t make your upstream developers learn new tools or read new rules in a wiki; don’t expect them to become compliance or security experts. Enforce the rules programmatically and provide clear, automated responses when the code isn’t compliant; your developers will produce more code, without introducing risk.
- Minimizing time to value for the customer is a fundamental tenant of DevOps. With that mindset, the delays that can commonly be present when security is poorly implemented will cause a developer to look for workarounds, or worse yet, not embrace the fundamental tenant of security; protect the data. Access controls, logging, and encryption are all necessities for security best practice. Excessive steps the user must go through will not be well received and could potentially create a culture that presents security and views it as an impediment to success.
Here's who provided their insights:
- Anne Baker, V.P. of Product Management and Marketing, Adaptiva
- Steven Aiello, Solutions Principal, AHEAD
- Gadi Naor, Co-founder and CTO, Alcide
- Mike Stahnke, VP of Platform, CircleCI
- Brian Nash, Director of Product Marketing, and Brian Dawson, DevOps Evangelist, CloudBees
- Michael Rose, Vice President of Engineering, Cybera
- Doug Dooley, COO, Data Theorem
- OJ Ngo, CTO and Co-Founder, DH2i
- Kris Lahiri, Co-founder, Egnyte
- Brian Platz, Co-founder and Co-chairman, Fluree
- Javed Shah, Director of Product Management for Cloud and DevOps, ForgeRock
- Malcolm Isaacs, Senior Solutions Manager, Application Delivery Management, Micro Focus
- Gary Duan, CTO,NeuVector
- Yogesh Badwe, Director of Information Security, Okta
- Franklin Mosley, Senior Application Security Engineer/Evangelist, PagerDuty
- David Strauss, CTO and Co-founder, Pantheon
- Jeff Keyes, Director of Product Marketing, Plutora
- Vishnu Nallani, VP & Head of Innovation, Qentelli
- Sheng Liang, Co-founder and CEO, and Shannon Williams, Co-founder and VP Sales & Marketing, Rancher Labs
- Gene Yoo, CEO, Resecurity
- Altaz Valani, Research Director, SecurityCompass
- Jim Hansen, V.P. Products, SolarWinds
- Colby Dyess, Director of Cloud Marketing, Tufin
- Tim Hinrichs, CTO and co-founder, Styra
- Joseph Feiman, CSO, WhiteHat Security
- Andrei Bezdedeanu, VP of Engineering, ZeroNorth
- Tim Reilly, COO and CFO, Zettaset
Opinions expressed by DZone contributors are their own.