Automating the Operation of Stateful Apps in Kubernetes With GitOps

Scale and velocity are the main drivers behind Kubernetes adoption. Kubernetes allows companies today to run thousands of cloud-native applications, including stateful applications like databases. But that also means managing complex workloads within large cloud-native systems can be a daunting task, especially when it comes to rolling updates or migrations. How can software teams better manage stateful applications and their many operational tasks in an efficient, predictable, and reliable manner?

You may also enjoy:  Kubernetes: The State of Stateful Apps

A few months ago, Ryan Wallner (Technical Advocate, Portworx) and Sebastian Bernheim (Customer Success, Weaveworks) spoke on:

• Reliably running stateful applications like databases in a Kubernetes environment
• How to utilize GitOps best practices and workflows to unify deployment, management, and monitoring for containerized clusters and applications
• How stateless and stateful components can be completely recovered with Portworx and GitOps best practices.

What is Portworx?

Portworx maintains stateful data for applications running in Kubernetes. This applies to any data that has a life beyond the lifecycle of a container or a pod. Portworx is similar to AWS’ EBS in the sense that container developers, as well as operators, don't need to know how the physical storage is managed within the cluster. Instead, teams use familiar workflows to define a Persistent Volume Control and to consume Persistent Volumes within pods.

The following types of applications rely on stateful data:

• Databases — Cassandra, Elastic Search, for example
• Content management systems — WordPress
• DevOps Toolchain — for example, Jenkins
• Big Data — Kafka

Portworx runs natively in a cloud-native stack that gets deployed as a DaemonSet to Kubernetes. It resides on top of the same worker nodes as your application stack. And because it integrates directly with your stack, there’s no need to run a separate external compute structure.

Applications that run on Kubernetes are typically stateless. Access to the stateful applications like databases are typically done over the network and reside outside of your cluster.

What Portworx allows you to do is run those stateful services such as Cassandra and Elasticsearch on the same stack next to your stateless applications. When you can run these services on the same stack, performance, speed, and data locality are all improved. You can also scale to thousands of nodes when you can combine your computing power and storage. An additional advantage gained is the high availability for your applications.

High Availability for Your Applications

When a node fails or a pod fails, Portworx works with Kubernetes, taking advantage of its scheduling ability to ensure that your data is always available within a cluster, across clusters or even across availability zones.

How GitOps works together with Portworx to manage stateful apps

Sebastian from Weaveworks then took us through a demo of how GitOps and Portworx works together to give you reliable, persistent applications that run in the cloud. As mentioned, Portworx behaves like a service layer that persists data from a stateful application like a MySQL database.

What is GitOps?

GitOps is often described as an operating model, but it can also be described as a way to get and keep your systems running. With GitOps you have a single source of truth where all of your infrastructure changes and all of your declarative application configurations are stored in git. This allows your team to use the same set of application code change approvals and procedures that you are using in your development processes.

This standardized workflow provides a number of benefits:

1. Audited and secure continuous deployment.
2. Monitoring and drift alerts.
3. Scalable fleet management

When changes are pushed to Git, agents running in your Kubernetes cluster compare its state with what’s in Git and if it detects a difference, it either automatically deploys the changes or it alerts on and waits for a human operator to make the change. Once the changes are observed within the cluster, they can be further iterated on, deployed to production or rolled back as needed.

Portworx and GitOps

The demo showed us how Portworx and GitOps can be used together to recover a stateful application like a MySQL database during a rolling deployment.

Weavework Enterprise Kubernetes Platform helps you spin up a cluster and a development platform anywhere; whether that is on-prem or in a public cloud, but for this demo, we focused on how GitOps workflows can be reliably applied to stateful applications with Weave Cloud and Portworx.

Weave Cloud connects to your existing cluster with the following functional areas:

• Explore - Provides a graphical overview of your cluster, allowing you to drill down into the cluster and right into a container if you need to.
• Deploy - Built-in GitOps workflows can deploy your application updates into the cluster — all it needs is the URL to a git repository with your config. Github is the single source of truth for application configuration. Weave Cloud's UI allows you to compare present and past states.
• Monitor - Prometheus as a service monitoring with built-in graphs, charts and an interactive PromQL builder for application observability and monitoring.

The Sockshop was deployed to a cluster along with Portworx and the Weave Cloud agents. The Sockshop is a microservices demo application. It's an e-commerce store built with a mix of languages and technologies as well as stateful and stateless applications.

Sockshop Architecture

Below is an overview of the architecture of the Sockshop:

Portworx Installed to The Cluster

Weave Cloud lets you filter and drill down on objects in the cluster. In the screen capture below, we searched for the Portworx container to check that it is running properly correctly. In this case, Portworx provides a persistent data layer for items kept in the catalogue, order, payment, user and cart which all depend on databases.

Git as The Source of Truth

Once Weave Cloud is hooked up to the git repo with your manifests and you've identified that as your source of truth, the agents running in your cluster constantly compare the running state with the source kept in git. If it detects a difference between the two, it can send you an alert. When you deploy a new image, the YAML configuration files are automatically updated in the repository and then the corresponding image gets automatically rolled out to the cluster.

After committing a change through the Weave Cloud interface, a commit ID as well as the corresponding SHA-1 hash for the git repo commit is shown in the event log.

If required, you can further match up the git commit to see exactly what was done when and by whom. Git of course verifies the provenance of what you’ve got running in production, and it leaves a full audit trail of what you’ve committed in your git repo. This is very useful for auditing purposes or for tracking particular changes from members of your team.

Rolling update of a stateful application

In the case of the Sockshop, if, for example, the catalogue goes down while someone is browsing, it completely recovers its data since it is persisted with the help of Portworx.

Likewise in the case of a complete or partial cluster meltdown because everything is kept in git, getting your cluster and its applications back up to working order takes minutes rather than hours. With the addition of Portworx, all of your stateful applications can also seamlessly recover.

Have questions on what you need to create a cloud native platform?

The Weaveworks team can help you navigate the vast landscape of cloud-native technologies – OSS and paid. Together we can create a cloud-native reference architecture that fits your business needs. You can benefit from a Weaveworks’ validated design or you can design, review and select technology options with our help.

Contact us for a demo of the Enterprise Kubernetes Platform or check out our Quickstart program to help you get up and running with Kubernetes.