Securing Developer Workflows

A few weeks ago, Weaveworks and Snyk delivered a webinar, entitled, "Secure GitOps pipelines for Kubernetes." The theme of the webinar was on how to improve the security of your development workflows — from Git to production.

Brice Fernandes, Customer Success Engineer at Weaveworks kicked off the talks with an in-depth look on what GitOps is and how it improves the overall security of your CICD pipelines.

How Do You Create a Secure CI/CD Pipeline?

In the typical CI/CD pipeline, he said, the developer starts off on the left with read and write credentials to the code repository. The CI system reads the code repository for updates, creates a new entity, and writes that new entity into the container registry. The CI system may also have scripts that talk to the cluster API and modify the cluster itself. The scripts that are in place allow developers to deploy a new version or a new set of configuration files from the CI pipeline to the cluster. In addition to this, the developer may also have read/write credentials to push images to the container registry.

The GitOps Model

Brice then described GitOps:

• An operation model
• Derived from computer science and operations knowledge
• Technology-agnostic (name notwithstanding)
• A set of principles that discuss why rather than how it is done.
• A way to speed up your team

For teams who want to implement GitOps into their CICD pipelines, they will need to implement these four concepts.

1. The Entire System Is Declaratively Described.

Kubernetes is one of many modern cloud-native tools out there that are declarative and that can be treated as code. Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. With your application’s declarations versioned in Git, you have a single source of truth. Your apps can then be easily deployed and rolled back to and from Kubernetes. And even more important and critical to the GitOps story is that when disaster strikes, your cluster can also be dependably and quickly reproduced.

2. The Canonical Desired System State Is Versioned (With Git).

With the declaration of your system stored in a version control system, and serving as your canonical source of truth, you have a single place from which everything is derived and driven. This trivializes rollbacks and you can use git revert to go back to a previous state. With git’s excellent security guarantees, an SSH key signs commits to enforce strong security guarantees about the authorship and the code’s provenance.

3. The Ability to Automatically Apply Approved Changes.

Once you have the declared state kept in git, the next step is to have the ability to automatically apply any state changes to your system. What's significant about this is that you don't need specific cluster credentials to make a change to your system. With GitOps, there is a segregated environment that the state definition lives outside of. This allows your team to separate what they actually do from how they are going to do it.

4. Software Agents to Ensure Correctness.

With the state of your entire system kept under version control, you can now employ software agents to inform you whenever reality doesn’t match your expectations. The use of diff and sync tools also ensures that your entire system is really self-healing. And by self-healing, we don’t mean when nodes or pods fail—those are handled by Kubernetes—but in a broader sense, like in the case of human error. In this case, software agents act as the feedback and control loop for your operations.

GitOps in Practice

Once you have all of these pieces in place, this is what your high-level workflows look like. There is the canonical source of truth kept under source control, then developers modify the canonical source of truth, and software agents are available that ensure the cluster’s correctness against the source of truth.

Security and The GitOps Pipeline

A GitOps pipeline contains a repository that represents your canonical desired state. Immediately, you start to see that the permissions to change the cluster exist only within the cluster and never leave the boundary of the cluster. The actual agent that's modifying and updating the cluster is the software agent that lives inside of it.

“Instead of you reacting to the outside world, it’s now your cluster reacting and changing to the outside world in a very structured way.”      —Brice Fernandes, Customer Success Engineer

Instead of you reacting to the outside world, it's now your cluster reacting and changing to the outside world in a very structured way. The operator can make changes to the canonical system’s repository and the system will react to changes in that repository. This allows your organization to set policies and other constraints that prevent an operator, for example, from making changes through the CI system.

Securing Git itself

Since credentials are kept inside of the cluster, GitOps makes your CI/CD pipelines more secure. However, this does shift the burden of security onto git itself. Brice, followed up his discussion with a few of the common attacks on git:

• User impersonation
• Git history rewrites
• Lax settings for git’s security features
• The use of deprecated software

Do You Know Which Dependencies You Have?

Next up, Simon Maple, Director of Developer Relations (@sjmaple) at Snyk spoke on implementing an additional security layer to your pipeline that scans your dependencies for vulnerabilities.

Analyze your repo with Snyk

Snyk statically analyzes your repo to find vulnerable dependencies you may be using and then helps you fix them. You can use Snyk’s UI to drill down on issues in your repos and keep users from accidentally adding vulnerable libraries by automatically scanning pull requests and failing checks when it finds one.

.

View the slide deck or the entire webinar: