A lot of us that become passionate about DevOps, have had experience executing across multiple functions. We understand the challenges that come with developing new features under tight deadlines. We’ve also experienced the pain of being paged at all hours or having to go through a dreaded security audit. One of the key philosophies of DevOps is to automate things that are painful. I’m not sure there is much that is more painful than a security audit? Or hearing that PagerDuty alert at 3:00 am?
Having the opportunity to work across the development lifecycle functions has allowed me to really understand the importance and the relative amount of operational pain we can avoid if we start incorporating more and more operational aspects early.
In most DevOps adoptions, we are routinely and successfully pushing infrastructure to the left with infrastructure-as-code/immutable infrastructure; however, to fully optimize the operational aspects of a system we need to keep pushing all operational aspects left as well. Continuous Integration & Continuous Delivery (CI/CD) practices are allowing us to deliver competitive features faster than ever. However, if we can’t manage and operate those new features/functions along with the supporting infrastructure with that same efficiency then we quickly find the time-to-market and overall cost benefits of DevOps being negated.
I frequently came across a project that has been actively applying DevOps CI/CD practices in their development of features and functionality but has not yet considered (or perhaps had the time to start) applying CI/CD practices beyond the functional aspects of a project or application. It makes sense from a strategy perspective to get the project, service, or feature out as quickly as possible to start attracting and/or keeping customers. DevOps is definitely a journey that likely often begins with features/function. However, if we want to scale a solution at any level we need to make sure we are not neglecting the operational aspects for any extended period of time. The level of re-work, security exposure, and cost you can expect if you don’t include operational aspects early needs to be heavily balanced against the priority placed on features and functions.
A couple of the key operational aspects I see that consistently cause operational inefficiencies and need to be included in your CI/CD model early include but are not limited to:
Security & Compliance
Logging & Monitoring
Security & Compliance
Security is a big topic and really worthy of its own blog series in itself! I know some people find this topic extremely painful (oddly enough…I’m not one) but it is something we all need to be thinking about when identifying our DevOps strategies. In this blog, I’ll keep this section limited to a quick introduction to the bigger topic of incorporating security early into the CI/CD pipeline. Incorporating security means collaborating upfront on security, prioritizing security requirements, and fully incorporating security into your end-to-end CI/CD pipeline by automating any:
System and application hardening
Security scans with automatic alerts and/or remediation
Enforcement of required security settings
Reporting required for compliance
The identification and implementation of your overall security automation framework will largely depend on how your current DevOps CI/CD pipeline solution is set up as well as your specific security and compliance requirements. However, continuously incorporating and building in varying layers of security and compliance early will have enormous operational benefits in terms of cost as well as reducing potential security exposures. Security is vital for brand image and quality but it’s also extremely expensive to do manually:
Remediate issues late in the lifecycle (or worse…in Production!)
Maintain a compliance posture
Apply & validate patches and fixes
The great part about DevOps is that DevOps practices actually enable security and compliance to become a less painful and less costly by providing a repeatable and reliable set of practices to setup security as well as maintain a compliant posture.
There is a lot of progress in the ability to perform patch management via configuration management tools, containers, and immutable infrastructure strategies. These are all great strategies but the key is to make sure you have one identified and proven.
I see patch management as an afterthought in many cases but, again, patching servers manually can be extremely expensive and ineffective. Also, many security issues are exploited by vulnerabilities found in systems with missing patches. Ensuring there is an automated patch management strategy in place with automated validation is key to maintaining your security posture as well as reducing the cost of manually applying patches.
Logging & Monitoring
There are a lot of different types of feedback loops to consider for DevOps adoption but one type includes the ability for all project stakeholders to have (blame-less) transparency into all issues across environments. Including an efficient logging and monitoring solution into your overall CI/CD pipeline can allow us to begin to identify and address potential operational issues early.
There are traditional monitors of the system and application that should be applied across environments to quickly identify and resolve issues while ensuring transparency of issues across environments. It also allows us to start implementing self-healing capabilities as well as identifying valuable monitors to set up and measure.
There is also monitoring to help facilitate the multiple feedback loop information sources that are critical to DevOps adoptions. This allows all stakeholders to understand where there are issues and react quickly. This typically means bringing together a lot of different information sources into a consolidated dashboard. In a lot of cases, this translates into a custom DevOps dashboard.
There is also a relatively new open source DevOps dashboard created by Capital One called Hygeia that I recommend checking out and contributing to (http://www.capitalone.io). I feel like it’s closing a gap that exists by pulling together all of these amazing DevOps tools into a consolidated dashboard for monitoring and transparency.
To Wrap It Up…
I think a lot of the operational aspects above are items we all really want to think about and include but it doesn’t always get priority over features and functions. My goal with this blog is to highlight some of the items we probably all know we should include but they still get left out because it’s sometimes difficult to quantify the operational impacts of allowing them to be prioritized alongside features a customer may want.
My suggestion is to include delivery and operational expertise early in your solution and development activities. Most successful DevOps adoptions have some portion of engineers/developers with deep operations backgrounds that are actively contributing to the development of those operational aspects. In those cases, the inclusion of operational aspects happens almost organically. If there are any operational knowledge gaps, I recommend reaching out to people across the company to try to bridge those gaps before it becomes an operational nightmare that can no longer scale efficiently.
If the items above seem like a headache to include now or seem safe to hold off on for now, I recommend keeping a really close eye on your quality metrics, operational and labor costs.
Would love to hear your experiences and suggestions about including, or better yet—not including, operational aspects early. Feel free to contact me at @Shelbee_SE or leave a comment to continue the discussion!
If you want to learn more about how to manage security in a DevOps enterprise, register now to watch this webinar.