DevOps backup – top reasons for DevOps and Management
Last Updated on August 26, 2024
Traditional backup of endpoints, servers, or databases has become almost synonymous with cybersecurity. However, there is increasing discussion about the need to secure data stored in SaaS or DevOps solutions, such as GitHub, GitLab, Bitbucket, Jira, or Azure DevOps.
Why are we questioning additional security for DevOps or SaaS data? And what implications might this have on the intellectual property, source code, metadata, and projects stored within these tools?
Here we come to the most common misconceptions among both people working directly in these tools, DevOps, decision-makers, and management level. And no offense here – until recently, there were no tools specializing in securing DevOps data. So we got used to coming up with workarounds, utilizing traditional file backups, or writing and maintaining our scripts. In this article, we want to show you what consequences this approach can bring to your company – especially if you are a part of management team.
Reason # 1: SaaS data can also be lost, deleted, or corrupted
It is estimated that a single midsize business uses around 217 SaaS applications. This growing number of SaaS services and DevOps tools usage does not escape the attention of cybercriminals. In fact, it opens at least that many “doors” to break into company IT systems and steal company data.
Source: TechTarget
According to TechTarget’s Enterprise Strategy Group report called “Data Protection for SaaS”, IT professionals highlight service outages, malicious deletion from cyberattacks, and accidental deletion. as the most common causes of data loss or corruption in SaaS-based applications.
We won’t go into detail about the threats to your GitHub, GitLab, Bitbucket, Jira, Azure DevOps data. We assume you’re strongly aware of them. If not, we encourage you to reach for one of our materials:
Article DevSecOps Myhtbuster – Nothing fails in the cloud
The State of DevOps Threat Report – our latest research which revealed that last year the number of incidents in GitHub grew by over 21%, Jira users were affected every 5 days or 32% of events in FirLav affected service performance.
All cloud service providers operate within the Shared Responsibility Model (Limited Liability Model), which strictly defines the obligations of each party – the customer’s and the provider’s. Not to be unfounded, let’s look at the abstracts from official documentation:
“You are responsible for keeping your Account secure while you use our Service. We offer tools such as two-factor authentication to help you maintain your Account’s security, but the content of your Account and its security are up to you.”
“Customer will be responsible for … maintaining the security of Customer’s account, passwords (including, but not limited to, administrative and User passwords) and files, and for all uses of Customer account with or without Customer’s knowledge or consent…”
“We do not warrant that your use of the cloud products will be uninterrupted or error-free, that we will review your data for accuracy or that we will preserve or maintain your data without loss. You understand that use of the cloud products necessarily involves transmissions of your data over networks that we do not own, operate or control, and we not responsible for any of your data lost, altered, intercepted or stored across such networks.”
This means that the service provider is responsible for its infrastructure security, service availability, accessibility, and backups of its entire ecosystem. At the same time, a customer (You in this case) is responsible for his own DevOps account data, its security, accessibility, availability, and backups.
Moreover, some cloud services strongly even suggest their customers have backups. For example, look at the abstract from the Atlassian Security Practices:
“For Bitbucket, data is replicated to a different AWS region, and independent backups are taken daily within each region. We do not use these backups to revert customer-initiated destructive changes, such as fields overwritten using scripts, or deleted issues, projects, or sites. To avoid data loss, we recommend making regular backups.”
You can read more about the Shared Responsibility Models in our dedicated articles:
📌 GitHub Shared Responsibility Model
📌 GitLab Shared Responsibility Model
📌 Atlassian Cloud Security Shared Responsibility Model
Reason # 3: Security Compliance & Data Retention
Depending on the industry you may need to comply with different security protocols, acts, certifications, and standards. If your company operates in a highly regulated industry, like healthcare, technology, financial services, pharmaceuticals, manufacturing or energy, those security and compliance regulations and protocols can be even more strict.
Thus, to meet the compliance straightened security requirements, your organization needs to implement security measures, like role-based access controls, encryption, ransomware protection measures – just to name RTOs and RPOs, risk-assessment plants plans, and other compliance best practices… And, of course, a backup and Disaster Recovery plan is one of them too. It ensures that the company will be able to restore its critical data fast, guaranteeing the data availability, accessibility, security, and confidentiality of your data.
Here are listed only some of the compliance regulations that require backup and DR measures as compulsory ones:
| Compliance Regulation | Quote from the Documentation on Data Protection Regulations | Link to the Documentation |
| General Data Protection Regulation (GDPR) | “… the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk, including inter alia as appropriate: … the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident..” | Art. 32 GDPR |
| Health Insurance Portability and Accountability Act (HIPAA) | “Administrative procedures – documented, formal practices to manage the execution and selection of security measures to protect data and to manage the conduct of personnel to protect data, i.e. audits, training, disaster recovery.” | Art. VI. Security Requirements, HIPAA |
| Payment Card Industry Data Security Standard (PCI DSS) | “12.10.1 An incident response plan exists and is ready to be activated in the event of a suspected or confirmed security incident. The plan includes, but is not limited to…. Business recovery and continuity procedures. Data backup processes….” | Payment Card Industry Data Security Standard |
| Federal Risk and Authorization Management Program (FedRAMP) | “The service provider maintains at least three backup copies of user-level information (at least one of which is available online). Conducts backups of system-level information contained in the information system [FedRAMP Assignment: daily incremental; weekly full];” “The organization provides for the recovery and reconstitution of the information system to a known state after a disruption, compromise, or failure.” | FedRAMP Moderate Baseline System Security Plan (SSP) Template |
| ISO/IEC 27001 | “The company performs daily backups and tests recovery periodically” | Annex A.12.3 (Backup) ISO/IEC 27001 |
| Financial Industry Regulatory Authority (FINRA) | “Each plan [Business Continuity Plan], however, must at a minimum, address: (1) Data back-up and recovery (hard copy and electronic); (2) All mission-critical systems; …” | FINRA Rule 4370 |
| National Institute of Standards and Technology (NIST) SP 800-53 | “… a. Conduct backups of user-level information contained in [Assignment: organization-defined system components] [Assignment: organization-defined frequency consistent with recovery time and recovery point objectives]; b. Conduct backups of system-level information contained in the system [Assignment: organization-defined frequency consistent with recovery time and recovery point objectives]; c. Conduct backups of system documentation, including security- and privacy-related documentation [Assignment: organization-defined frequency consistent with recovery time and recovery point objectives]; and d. Protect the confidentiality, integrity, and availability of backup information” | NIST Special Publication 800-53 Revision 5 |
| Health Information Technology for Economic and Clinical Health (HITECH) Act | “(A) Data backup plan (Required). Establish and implement procedures to create and maintain retrievable exact copies of electronic protected health information.(B) Disaster recovery plan (Required). Establish (and implement as needed) procedures to restore any loss of data.” | HIPAA Security Rule under the HITECH Act:Contingency Plan (Standard) (§ 164.308(a)(7) |
| NIS 2 Directive | “The measures referred to in paragraph 1 shall be based on an all-hazards approach that aims to protect network and information systems and the physical environment of those systems from incidents, and shall include at least the following: … (c) business continuity, such as backup management and disaster recovery, and crisis management…” | NIS 2 Directive |
| Digital Operational Resilience Act (DORA) | “For the purpose of ensuring the restoration of ICT systems and data with minimum downtime, limited disruption and loss, as part of their ICT risk management framework, financial entities shall develop and document: (a) backup policies and procedures specifying the scope of the data that is subject to the backup and the minimum frequency of the backup, based on the criticality of information or the confidentiality level of the data; (b) restoration and recovery procedures and methods.” | Regulation (EU) 2022/2554 of the European Parliament and of the Council on digital operational resilience for the financial sector and amending Regulations |
The importance of data retention
Another issue that is closely related to compliance is data retention. Some compliance regulations require organizations to keep their data for a long time. As an example, we can mention NIST’s requirements from its Security and Privacy Controls for Information Systems and Organizations: “… Storing audit records on separate systems or components applies to initial generation as well as backup or long-term storage of audit records…”
It means that companies should adopt proper mechanisms to store their backed-up data for longer periods of time. Can Git hosting services like GitHub, Bitbucket or GitLab solve this? Let’s turn to documentation once again:
“By default, artifacts and log files generated by workflows are retained for 90 days. You can change this retention period to anywhere between 1 and 400 days.”
GitHub Docs – Enforcing policies for GitHub Actions in your enterprise
“Groups and projects remain restorable during the retention period you define. By default, this is 7 days, but you can change it. If you set the retention period to 0 days, GitLab removes deleted groups and projects immediately. You can’t restore them.”
GitLab Docs – Visibility and access controls – Retention period
It means that if the accidental deletion happens, and you don’t notice it on time, let’s say some old data that you need to store for compliance needs, you won’t be able to restore it and, as a consequence, your company will fail to meet the compliance requirements, which, in its turn, can lead to fines. And as you know, these sums can amount to hundreds or even millions, depending on the scale.
Here, backup becomes the answer. However, pay attention to the retention options and settings that a specific vendor allows you. Many of them limit the possibility of recovering data, for example, from a maximum of 365 days.
Here in GitProtect.io backup for DevOps tools we provide you with long-term or even unlimited retention that can help your organization meet shared responsibility and security compliance requirements. Moreover, with scheduled automated backups, ransomware protection, in-flight and at-rest encryption with your own encryption key, data residency of choice, SLA, Disaster Recovery technology, notifications, and compliance reports the backup provider can become a solid data protection baseline.
Reason # 4: The real cost of failures
Recovering from ransomware, downtime, and data breaches can be very costly… Moreover, the cost of recovering from a failure is increasing from year to year. Thus, for example, the average cost of recovering from a ransomware attack is about $2.73 million. What’s more, a ransomware attack can lead to prolonged downtime, and as a consequence, higher expenses.
The average downtime from a ransomware attack in the enterprise sector can last for up to 12 days and with every minute of downtime company loses money. The average cost of downtime can reach as high as $9K per minute. Yet, that’s not the highest price, depending on the industry and the size of an organization, the amount of money the company can lose in a minute can vary. Let’s look at the cost of downtime for some of the industries:
- IT industry: depending on its size, a company can lose from $145K to $450 per hour;
- Automotive industry: the cost can rise up to $50K per minute, which is around $3 million per hour;
- Manufacturing industry: an hour of downtime can cost around $260K per hour,
- Enterprise industry: the lost revenue may vary between $ 1 million and $5 million per hour,
- Healthcare: the cost may compel as much as $1.1 million per hour,
- Telecommunications: the cost of downtime can rise up to $2.48 million per hour.
The cost failure may include direct costs that one can measure pretty easily, as they include fines (e.g. for non-compliance), or lawsuits. Another component is indirect costs, which include reputational damage. Moreover, if we add here the impossibility of restoring the company’s critical data, the issue becomes even more devastating. That’s why a properly built backup and Disaster Recovery strategy is critical for resuming the company’s operations fast and eliminating data loss.
Choosing your backup option
A reliable backup strategy can help to address such pain points as:
Data recovery
For example, if a data breach results in data loss or data corruption, with backup you can restore the affected data to its pre-breach state. As a result, you will be able to minimize the damage caused by the breach and ensure your business continuity.
Ransomware Mitigation
In the event of a ransomware attack, for example, if bad actors encrypt your data and demand a ransom, with backup you can avoid paying ransom, as you can restore your data from the latest (or any given) copy.
Integrity verification
When you have regular backups, you can verify the integrity of your data after a breach. By comparing the current data and the backed-up copies, your team can easily identify what data has been altered or compromised.
The importance of backup
To help the organization comply with security protocols and, in the event of a breach, mitigate the consequences of any event of failure, your backup shouldn’t be solely a copy of your production environment. Your data backup strategy should be built within the backup best practices to guarantee your company’s workflow continuity. Hence, your backup should:
- include all your DevOps data in the backup copy – source code and project management data, e.g. all of GitHub, GitLab, Bitbucket repositories and metadata, and Jira data,
- allow automatic scheduled backup, and customize them within your needs, e.g. full, differential, and incremental backups, multiple backups, data storage of your choice, test restore to measure your RPO and RTO for DevOps data,
- guarantee not only every-day-basis granular restore but also provide you with every-scenario-ready Disaster Recovery in case of major failure or service downtime,
- be kept for as long as you need – long-term and unlimited retention,
- be stored in multiple locations – Cloud and local storage location – to meet the 3-2-1 backup rule,
- be ransomware-proof,
- guarantee restore and Disaster Recovery in any event of failure,
- give the possibility to easily monitor backup performance.
And this list we can continue… Here we focused only on the main requirements the backup should meet.
📌 GitHub backup best practices
📌 Bitbucket backup best practices
📌 GitLab backup best practices
📌 Jira backup best practices
The real cost of DevOps backup vs. scripts
No protection (strongly not recommended), self-written git backup scripts based on git-clone command, snapshots of local repositories, and on-premise backup – this is how companies try to deal with DevOps backup today. With DIY backup script leading the way. What is your solution?
Let’s dive deep into the costs of writing and maintaining your internal workaround with professional backup software.
Simple calculation.
On average your team may spend around 250+ hours a year on backup processes. And, taking into account that the average DevOps engineer salary in the USA is around $63 per hour, your company will pay around $15 750 dollars a year for backup, and at the same time disturbing its team from their core duties.
Add the alternative cost of your DevOps time here. You know best the value each developer, DevOps or security architect brings to your organization and express it in a number. Remember, we’re talking about a year here, try to imagine that.
Moreover, this sum doesn’t include the backup maintenance, costs for passing the security compliance certificate, and the cost of the risk that you may incur if your solution fails. In addition, remember that you do not share responsibility with anyone – the entire shock wave after a potential event will hit your organization. This gives us another position in our equation.
Costs to consider – costs related to loss of reputation, customers’ trust and loyalty, and cost of every downtime minute (as we previously discussed it can reach $9K per minute).
Because honestly – are you sure your DevOps backup solution is recoverable? Do you have a restore script? Do you test both backups and restores regularly?
Now you will probably argue that a DevOps backup solution does not have to exempt you from these costs. However, a dedicated DevOps data backup solution allows you to perform test recovery. You are sure that your copies are being restored, thanks to Disaster Recovery technology you can estimate the time needed to restore critical and entire infrastructure. This way you can mitigate or even eliminate these potential costs, ensuring business continuity.
Moreover, you will be informed about the correctness of your copies by configurable notifications – via email or Slack – whichever is more convenient for you. This eliminates the cost of maintaining a backup solution and the time of your employees, gaining their productivity and influence on generating value for the business
To sum it up, by opting for professional DevOps backup software like GitProtect, organizations can reduce their obligations in data protection, ensure compliance, and let their DevOps team keep focused on their core tasks. Thanks to its multi-storage compatibility, automated scheduled backup policies, ransomware protection, AES encryption with your custom key, flexible data restore and Disaster Recovery technology and other best-in-class security measures proven by SOC 2 & ISO 27001, GitProtect.io foresees any disaster scenario, minimizing or even eliminating downtime, and, consequently, greatly reducing organization’s costs on dealing with disasters.
“GitProtect’s support for compliance standards and regulations has simplified our adherence to industry standards and legal requirements.”
Kubilay Verboom, Cloud Engineer at SUE
📚 Read the full story on how SUE adopted GitProtect.io backups for the GitLab environment to guarantee its Disaster Recovery
Finally, let’s compare the prices.
Your DIY “solution” cost = $15 750 + backup maintenance manhour x manhours + alternative cost + cost of downtime (let’s take an average $9K/minute)
The price of the GitProtect backup package starts from 18$ per month and the price per repository decreases together with a number of repos. Conversations with our customers and several years of experience have shown us that we are on average 16x cheaper than other options our customers had.
What’s more important, professional DevOps backup guarantees almost immediate data accessibility, availability, and recoverability from any point in time in any event of failure and saves your DevOps team’s time for checking backup, recovery, and compliance reporting. And yes, you get rid of part of the responsibility that is so important in the context of building reputation and customer trust.
[FREE TRIAL] Ensure compliant DevOps backup and recovery with a 14-day trial 🚀
[CUSTOM DEMO] Let’s talk about how backup & DR software for DevOps can help you mitigate the risks
