Backup and Disaster Recovery in the Age of GitOps and CI/CD Deployments

Whether you call it GitOps, Infrastructure-as-Code (IaC), or just using a CI/CD pipeline to automatically deploy changes to an application, I think we can all agree that moving to a model where your application and resource configuration is defined as code is beneficial to everyone involved. This code is usually saved in git, the most popular source version control system for cloud-native projects, and provides an automatic and verifiable change capture process, and simplifies application rollout. More importantly, it also prevents “snowflakes” where the configuration of a deployed application differs from the definition because of manual edits.

However, even with these powerful primitives available to developers and operators, the need for Backup and Disaster Recovery is as important as ever. The below post covers some of the core reasons why deploying a backup solution will be critical for safety and scale.

It’s All About the Data

The biggest thing to note is that any automated way of deploying an application will only bring back Kubernetes objects and configuration. Any persistent data or volumes used by applications are not captured in version control, and therefore to bring back any stateful service, such as a relational database or NoSQL system, requires that the entire application stack, its data, and the dependencies of the stack on the data be discovered, tracked, and captured.

Of course, with any application redeployment coming in from an automated system, the backup platform must support intelligent “data-only restores.” K10, Kasten’s Kubernetes-native data management platform, implements this by tracking the relationship between stateful services (including replication) and then bringing back only the volumes of data associated with the application after the application has been redeployed via a CI/CD or GitOps pipeline.

CI/CD at Scale

A number of our customers that run Kubernetes at scale and as shared infrastructure have clusters with hundreds of applications from multiple teams and business units. As different groups make independent technology choices, we see different CI/CD systems and deployment mechanisms in large companies ranging from GitOps and CI/CD-based Helm installs to manual deployments. This is the reality in complex environments.

Faced with this reality, a backup platform will be needed for the teams that are still ramping up on cloud-native development. However, in the fast-moving Kubernetes ecosystem and given the 50+ ways to deploy a containerized application, we have also observed how some applications tend to be long-lived. The systems they were deployed with aren’t being maintained anymore and might not be compatible with the installed Kubernetes version (e.g., applications installed with Helm v2, that will be EOL soon). Being able to quickly restore these important applications (they are long-lived for a reason) in case of accidental or malicious failure will be critical.

Additionally, when multiple CI/CD systems are in use, it is hard for an operations group to reach out to 100s of developer teams to both retrigger deployment in case of failure or when applications need to be moved across clusters to perform a Kubernetes version upgrade. Sopra Steria, one of Kasten's K10 customers, recently spoke about this exact problem when they had to move 170+ applications to upgrade OpenShift versions and couldn’t get responses from some teams that were running applications on their cluster. Thankfully, a 100% transparent backup platform enabled them to capture both stateful and stateless applications and move them over without downtime, and allowed the developer teams to resync their pipelines to the new clusters when they found free time.

Disaster Recovery

As you go through your Kubernetes business continuity planning, you need to ensure that both Kubernetes and all its applications can be quickly restored in case of disaster. Disaster Recovery (DR) can take many forms including cross-cluster, cross-region/data center, and cross-cloud.

While pushing configuration to a version control system and using CI/CD will be beneficial here too, the DR problem doesn’t go away. You will need to ensure that your version control system is itself HA across all fault domains to be able to quickly recover. Kasten’s K10 data management platform today can support DR in both modes as it can perform full backups but only granular restores. When the version control system (e.g. git) has itself been recovered in a separate fault domain, K10 will work with the CI/CD system to deploy the application configuration from the checked-in state while selectively bringing only the data back from its DR copies. However, if the version control system is not available, K10 will also bring the application configuration and all related Kubernetes back from backup along with the data. However, because of its application-centricity and Kubernetes-native knowledge, this state will be quickly reconciled with your GitOps or CI/CD system when the source is available again.

Other Reasons

Finally, there are a couple of other reasons why you might want a backup platform that understands your runtime application state that works in conjunction with your deployment pipeline. If your clusters are in a regulated environment (e.g., US financial services regulated by the SEC) and your backups need to prove that they captured what was really running vs. what the desired state of the world should have been, a platform that can capture the runtime state will be critical. Even if the sync period between configuration changes and deployment can be small on average, the window can drift in case of controller or CI/CD failures, manual additions of untracked resources, or malicious changes made.

Additionally, some application deployments have external side effects (e.g., a DNS update or external Load Balancer creation) that are dynamically named and are non-deterministic. These changes should be captured by a system that understands runtime state, and this should be brought back in on restore in conjunction with the static resources from the automated deployment system.

Conclusion

If you want to learn more, we encourage you to look at how Kasten K10 tackles these issues and interoperates cleanly with your GitOps, IaC, and CI/CD systems. We have a fully-featured forever-free edition available as well as a freely-downloadable enterprise trial for larger clusters. You don’t need to talk to us or sign up for anything, just go ahead and download it! Or, reach out for a personalized demo!