Continuous Security: Delivering Secure Apps on Kubernetes

In this post, I wanted to talk about the need to "shift left" from continuous security practices with apps deployed to Kubernetes. This will touch on some popular, open-source tools and how they can be used in Jenkins X and beyond.

One of the benefits of a shiny new Kubernetes setup, perhaps using something like Jenkins X, is that you have more freedom to ship rapid changes to applications. Weekly deploys become daily, and daily service updates become hourly. The gap in time between someone merging a change and it starts to roll out moves closer to real time.

This is great. To achieve this rapid continuous delivery (CD), no doubt, your team invested a lot of time in automated unit tests and regression tests.

So, what about the security aspects of your application? Traditionally, application level security would involve code audits, reviews, perhaps runtime penetration testing, approvals, code scanning, and so on. The idea is to find problems before they get too far.

However, with the accelerated velocity of application deployment, these tools need to be woven into the pipeline that takes the code to production, capturing problems specific to your app as early as possible. This is what is meant by "shifting left" security practices (you may hear that term increasingly). The "left" and and "right" implied here is the pipeline of software from ideas/source code (left) to production (right).

The continuous security aspects I wanted to briefly cover here can be broken into two areas:

• Static security of your application composition (sometimes known as SAST)

• Dynamic security of our running application (sometimes known as DAST)

The first is where things like source code scanners and "software component analysis" kick in. A common practice here is to look at how your application brings in libraries, especially open-source libraries, as some of those may have vulnerabilities and need to be updated or patched.

In fact, GitHub does some analysis on dependencies and can warn you about problems quite early on.

Super! I should get right on that and fix that (that demo repo is open source, but thankfully, GitHub doesn't shame you publicly for having really old dependencies)

Ok, so, what about the second type: dynamic security? This involves possibly probing the running application for problems, monitoring for faults, or even deliberately trying to break it.

But wait  — I said "shifting left" meant ideally catching things BEFORE they are deployed, but dynamic requires it to be running. This appears a problem, but later I want to demonstrate how the Jenkins X preview apps can help with this before a change is merged or promoted.

Some Handy Security Tools

Some popular (and open source) tools exist to help with these aspects of testing.

Firstly, the Open Web Application Security Project publishes a handy tool called ZAP. ZAP stands for Zed Attack Proxy. This is itself a project that can be used to simulate attacks on web apps, probe for weaknesses, and so on. It is published in many forms (including a Jenkins plugin), and there is a handy command called "zap-baseline." This baseline can be used to scan a web app for a base set of problems and report on serious problems or warnings. Happily, they make this available as a docker image that is published and updated very often. This tool is a handy complement for dynamic security testing.

For image and component (static) type security, Anchore published the Anchore Engine. The Anchore Engine (which is also helpfully published as docker images) works at the container level, looking for vulnerabilities that have made their way into your containers. This is very useful in a Kubernetes world as containers are the unit of deployment. One of the key features of Anchore is that it looks from the lowest level in the image up for problems, instead of just looking at libraries that your application depends on. Typically, applications built on images depend on other images that have many layers, which, in turn, depend on other images and so on (it can almost seem like you have no idea where things really come from).

This tool can apply to images before they make it to production, but you can also use it on stuff already running, in case there are new problems discovered that wasn't known at the time that version of the application was promoted.

Also, and most importantly, Anchore has the all-important nautical theme that seems to surround tools in the Kubernetes and Docker ecosystems.

Applying Continuous Security Tools With Jenkins X

As I have written about before, you can think of Jenkins X as an opinionated way to deliver apps atop Kubernetes. The benefit of this is that it allows it to have very specific hooks or extension points, allowing security aspects to be woven into the pipeline. In fact, it is so opinionated that it makes it possible to do the following to enable image security with the Anchore engine mentioned above:

> jx create addon anchore

This will put all the machinery in place to enable the engine automatically for your team, and then you run things like:

> jx get cve --environment=staging

This will disclose problems in the specified environment. This also, of course, applies to pull requests and in-progress work. In Jenkins X, each change (pull request) gets its own environment, so you can check that it is ok from a static point of view before merging it. Of course, ideally, you capture these at the pull request stage.

For dynamic security, you can use the ZAP tool mentioned above as simple as:

> jx create addon owasp-zap

This adds a hook that runs the "Zed Attack Proxy" on a preview app, as soon as it is deployed. Preview apps are created on every pull request in Jenkins X. Should it pick up any failures on the baseline test, it will automatically fail the build and report to the users that the pull request/change request needs some work. Preview apps are a mini version of the app running in an isolated environment.

Note that, in both of the above cases, I didn't have to change any pipelines or set it up per application/project. This works at the "team" level in Jenkins X. This is a handy way to enforce a kind of governance across projects. Governance is often a dirty word with developers, but in the case of security, maybe it is warranted (and in both cases here, the impact is fairly minimal but the value of it preventing problems could be huge). 

This could be what your pipeline looks like:

What's Next

There is clearly so much more that can be done, and a wealth of tools both commercial and open source.

The Cloud Native Computing Foundation (which is also the governing body for Kubernetes) is considering a proposal for runtime security called "Falco." I am not sure if it has a nautical theme, but it is certainly container specific. Runtime security is another security measure that is closer to monitoring: watching for abnormal behavior (which also means learning what normal behavior is for your apps). This will be a great project to watch as it looks like it can add another layer of continuous security, which has traditionally been considered quite exotic (but will hopefully be easy to apply with Kubernetes) for the benefit of all.

Additional Resources

Learn more about Jenkins X

Read about Jenkins X and Anchore

Watch the webinar on DevSecOps