DevOps Backup: Top Reasons for DevOps and Management

Traditional backup of endpoints, servers, or databases has become almost synonymous with cybersecurity. However, there is increasing discussion about the need to secure data stored in SaaS or DevOps solutions, such as GitHub, GitLab, Bitbucket, Jira, or Azure DevOps.

Why are we questioning additional security for DevOps or SaaS data? And what implications might this have on the intellectual property, source code, metadata, and projects stored within these tools?

Here, we come to the most common misconceptions among people working directly with these tools, including DevOps, decision-makers, and management-level personnel. And no offense here — until recently, there were no tools specializing in securing DevOps data. So we got used to coming up with workarounds, utilizing traditional file backups, or writing and maintaining our scripts.

In this article, we will discuss what consequences this approach can bring to your company, especially if you are a part of the management team. 

Reason # 1: SaaS Data Can Also Be Lost, Deleted, or Corrupted

It is estimated that a single midsize business uses around 217 SaaS applications. This growing number of SaaS services and DevOps tools usage does not escape the attention of cybercriminals. In fact, it opens at least that many “doors” to break into company IT systems and steal company data.

Moreover, many IT professionals highlight service outages, malicious deletion resulting from cyberattacks, and accidental deletion as among the most common causes of data loss or corruption in SaaS-based applications.  

It's no use to go into detail about the threats to your GitHub, GitLab, Bitbucket, Jira, and Azure DevOps data. You should already be strongly aware of them.

Reason # 2: The Shared Responsibility Model

All cloud service providers operate within the Shared Responsibility Model (Limited Liability Model), which strictly defines the obligations of each party — the customer’s and the provider’s. Not to be unfounded, let’s look at the abstracts from official documentation:

GitHub Terms of Service

“You are responsible for keeping your Account secure while you use our Service. We offer tools such as two-factor authentication to help you maintain your Account’s security, but the content of your Account and its security are up to you.”

GitLab Subscription Agreement

“Customer will be responsible for … maintaining the security of Customer’s account, passwords (including, but not limited to, administrative and User passwords) and files, and for all uses of Customer account with or without Customer’s knowledge or consent…”

Atlassian Cloud Terms of Service

“We do not warrant that your use of the cloud products will be uninterrupted or error-free, that we will review your data for accuracy, or that we will preserve or maintain your data without loss. You understand that use of the cloud products necessarily involves transmissions of your data over networks that we do not own, operate, or control, and we are not responsible for any of your data lost, altered, intercepted, or stored across such networks.”

This means that the service provider is responsible for its infrastructure security, service availability, accessibility, and backups of its entire ecosystem. At the same time, a customer (You in this case) is responsible for their own DevOps account data, its security, accessibility, availability, and backups.

Moreover, some cloud services even strongly suggest that their customers have backups. For example, look at the abstract from the Atlassian security practices:

“For Bitbucket, data is replicated to a different AWS region, and independent backups are taken daily within each region. We do not use these backups to revert customer-initiated destructive changes, such as fields overwritten using scripts, or deleted issues, projects, or sites. To avoid data loss, we recommend making regular backups.”

Reason # 3: Security Compliance and Data Retention

Depending on the industry, you may need to comply with different security protocols, acts, certifications, and standards. If your company operates in a highly regulated industry, like healthcare, technology, financial services, pharmaceuticals, manufacturing, or energy, those security and compliance regulations and protocols can be even more strict. 

Thus, to meet the compliance stringent security requirements, your organization needs to implement security measures, like role-based access controls, encryption, ransomware protection measures — just to name RTOs and RPOs, risk-assessment plans, and other compliance best practices… And, of course, a backup and disaster recovery plan is one of them, too. It ensures that the company will be able to restore its critical data fast, guaranteeing the data availability, accessibility, security, and confidentiality of your data.

Here are listed only some of the compliance regulations that require backup and DR measures as compulsory ones:

The Importance of Data Retention

Another issue that is closely related to compliance is data retention. Some compliance regulations require organizations to keep their data for a long time. As an example, we can mention NIST’s requirements from its Security and Privacy Controls for Information Systems and Organizations: “… Storing audit records on separate systems or components applies to initial generation as well as backup or long-term storage of audit records…” 

It means that companies should adopt proper mechanisms to store their backup data for longer periods of time. Can Git hosting services like GitHub, Bitbucket, or GitLab solve this? Let’s turn to documentation once again:

“By default, artifacts and log files generated by workflows are retained for 90 days. You can change this retention period to anywhere between 1 and 400 days.”

GitHub Docs – Enforcing policies for GitHub Actions in your enterprise

“Groups and projects remain restorable during the retention period you define. By default, this is 7 days, but you can change it. If you set the retention period to 0 days, GitLab removes deleted groups and projects immediately. You can’t restore them.”

GitLab Docs – Visibility and access controls – Retention period

It means that if the accidental deletion happens, and you don’t notice it on time, let’s say some old data that you need to store for compliance needs, you won’t be able to restore it and, as a consequence, your company will fail to meet the compliance requirements, which, in its turn, can lead to fines. And as you know, these sums can amount to hundreds or even millions, depending on the scale.

Here, backup becomes the answer. However, pay attention to the retention options and settings that a specific vendor allows you. Many of them limit the possibility of recovering data, for example, to a maximum of 365 days.

Reason # 4: The Real Cost of Failures

Recovering from ransomware, downtime, and data breaches can be very costly… Moreover, the cost of recovering from a failure is increasing from year to year. Thus, for example, the average cost of recovering from a ransomware attack is about $2.73 million. What’s more, a ransomware attack can lead to prolonged downtime and, as a consequence, higher expenses.  

The average downtime from a ransomware attack in the enterprise sector can last for up to 12 days, and with every minute of downtime, the company loses money. The average cost of downtime can reach as high as $9K per minute. Yet, that’s not the highest price; depending on the industry and the size of an organization, the amount of money the company can lose in a minute can vary. Let’s look at the cost of downtime for some of the industries:

• IT industry: Depending on its size, a company can lose from $145K to $450K per hour.
• Automotive industry: The cost can rise up to $50K per minute, which is around $3 million per hour.
• Manufacturing industry: An hour of downtime can cost around $260K per hour.
• Enterprise industry: The lost revenue may vary between $ 1 million and $5 million per hour.
• Healthcare: The cost may be as much as $1.1 million per hour.
• Telecommunications: The cost of downtime can rise up to $2.48 million per hour.

The cost of failure may include direct costs that one can measure pretty easily, as they include fines (e.g. for non-compliance), or lawsuits. Another component is indirect costs, which include reputational damage. Moreover, if we add here the impossibility of restoring the company’s critical data, the issue becomes even more devastating. That’s why a properly built backup and disaster recovery strategy is critical for resuming the company’s operations fast and eliminating data loss. 

Choosing Your Backup Option

A reliable backup strategy can help to address such pain points as:

Data Recovery

For example, if a data breach results in data loss or data corruption, with backup, you can restore the affected data to its pre-breach state. As a result, you will be able to minimize the damage caused by the breach and ensure your business continuity.

Ransomware Mitigation

In the event of a ransomware attack, for example, if bad actors encrypt your data and demand a ransom, with backup, you can avoid paying ransom, as you can restore your data from the latest (or any given) copy.

Integrity Verification

When you have regular backups, you can verify the integrity of your data after a breach. By comparing the current data and the backed-up copies, your team can easily identify what data has been altered or compromised.  

The Importance of Backup

To help the organization comply with security protocols and, in the event of a breach, mitigate the consequences of any event of failure, your backup shouldn’t be solely a copy of your production environment. Your data backup strategy should be built within the backup best practices to guarantee your company’s workflow continuity. Hence, your backup should:

• Include all your DevOps data in the backup copy — source code and project management data, e.g., all of GitHub, GitLab, Bitbucket repositories and metadata, and Jira data.
• Allow automatic scheduled backup, and customize them within your needs, e.g., full, differential, and incremental backups, multiple backups, data storage of your choice, test restore to measure your RPO and RTO for DevOps data. 
• Guarantee not only on an everyday basis granular restore but also provide you with every-scenario-ready disaster recovery in case of major failure or service downtime.
• Be kept for as long as you need — long-term and unlimited retention.
• Be stored in multiple locations — Cloud and local storage location — to meet the 3-2-1 backup rule.
• Be ransomware-proof.
• Guarantee restore and disaster recovery in any event of failure,
• Give the possibility to easily monitor backup performance.

And this list we can continue… Here, we focused only on the main requirements the backup should meet.

The Real Cost of DevOps Backup vs. Scripts 

No protection, self-written git backup scripts based on git-clone command, snapshots of local repositories, and on-premise backup — this is how companies try to deal with DevOps backup today. With a DIY backup script leading the way. What is your solution?

Let’s dive deep into the costs of writing and maintaining your internal backup workaround. 

Simple calculation. 

On average, your team may spend around 250+ hours a year on backup processes. And, taking into account that the average DevOps engineer salary in the USA is around $63 per hour, your company will pay around $ 15,750 a year for backup, and at the same time, disturbing its team from their core duties. 

Add the alternative cost of your DevOps time here. You know best the value each developer, DevOps, or security architect brings to your organization and express it in a number. Remember, we’re talking about a year here, try to imagine that.

Moreover, this sum doesn’t include the backup maintenance, costs for passing the security compliance certificate, and the cost of the risk that you may incur if your solution fails. In addition, remember that you do not share responsibility with anyone — the entire shock wave after a potential event will hit your organization. This gives us another position in our equation. 

Costs to consider — costs related to loss of reputation, customers’ trust and loyalty, and the cost of every downtime minute (as we previously discussed, it can reach $9K per minute). 

Because honestly, are you sure your DevOps backup solution is recoverable? Do you have a restore script? Do you test both backups and restores regularly?

Now you will probably argue that a DevOps backup solution does not have to exempt you from these costs. However, a dedicated DevOps data backup solution allows you to perform test recovery. You are sure that your copies are being restored; thanks to disaster recovery technology, you can estimate the time needed to restore critical and entire infrastructure. This way, you can mitigate or even eliminate these potential costs, ensuring business continuity.

What’s more important, professional DevOps backup guarantees almost immediate data accessibility, availability, and recoverability from any point in time in any event of failure and saves your DevOps team’s time for checking backup, recovery, and compliance reporting. And yes, you get rid of part of the responsibility that is so important in the context of building reputation and customer trust.