Among other reasons, we should mention:<\/p>\n\n\n\n
- \n
- accidental deletions and data loss<\/strong> – as within the Azure DevOps documentation<\/u><\/a>, the service provider can\u2019t restore your data in case you accidentally delete part of your code or other DevOps documentation:<\/li>\n<\/ul>\n\n\n\n
\n
\u201cAccidental deletion here refers to scenarios that arise as a result of an incident on our services. It doesn’t include customers’ accidental deletion of assets (for example, repositories, work items, attachments, or artifacts). We don’t support restoring assets that customers accidentally delete.\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n
- \n
- Shared Responsibility Model<\/strong> – all service providers operate within the Shared Responsibility Model, and according to it your account data is your, customer\u2019s, responsibility, Microsoft is responsible for its services to run smoothly:<\/li>\n<\/ul>\n\n\n\n
![\"Shared](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2024\/10\/Azure-DevOps-1024x587.png\")
<\/figure><\/div>\n\n\nSource: <\/em>Shared Responsibility Model in the Cloud<\/u><\/em><\/a><\/p>\n\n\n\n
- \n
- business continuity<\/strong> – service outages or your own infrastructure downtime can happen anytime. Unfortunately, no one can predict when it can happen, all you can do is to be prepared.<\/li>\n<\/ul>\n\n\n\n
\ud83d\udca1 In 2023 Microsoft Azure DevOps customers in the South Brazil Region experienced an outage that lasted 10+ hours. The incident was caused by a simple typo in code that ended up deleting 17 production databases.
Though Microsoft managed to detect and remediate the \u201cmistake\u201d the same day, the Azure DevOps users in the region couldn\u2019t access their services for hours. The entire case shows the importance of implementing reliable backup practices to reduce or even eliminate users\u2019 reliance on a single service provider.
Source: TechradarPro<\/u><\/a><\/p>\n\n\n\n- \n
- Compliance with security protocols and audits – <\/strong>backup and Disaster Recovery are some of the requirements to meet strict compliance regulations. For example, here is a quote from ISO\/IEC 27001 documentation<\/u><\/a>:<\/strong><\/li>\n<\/ul>\n\n\n\n
\n
\u201cThe company performs <\/em>daily backups<\/em><\/strong> and tests <\/em>recovery<\/em><\/strong> periodically\u201d<\/em><\/p>\n<\/blockquote>\n\n\n\n
PART 1 – BACKUP PERFORMANCE<\/strong><\/h5>\n\n\n\n
Backup with full data coverage <\/strong><\/h2>\n\n\n\n
When selecting a software to back up your Azure DevOps data it is crucial to have full data coverage of your Azure DevOps environment, no matter if you use the Cloud or the Server option. By full data coverage, we mean the ability to back up and restore both Microsoft Azure DevOps repos and metadata. Thus, backup for Azure DevOps should include:<\/p>\n\n\n\n
- \n
- Repositories and their content (commits, branches, tags etc.)<\/li>\n\n\n\n
- LFS<\/li>\n\n\n\n
- Pull requests (including labels and comments)<\/li>\n\n\n\n
- Projects<\/li>\n\n\n\n
- Work items (including comments and attachments)<\/li>\n\n\n\n
- Project wiki<\/li>\n\n\n\n
- Pipelines<\/li>\n\n\n\n
- Environments<\/li>\n\n\n\n
- Variable groups<\/li>\n\n\n\n
- Processes<\/li>\n\n\n\n
- Work item types (including their layouts and states)<\/li>\n<\/ul>\n\n\n\n
![\"Azure](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2024\/10\/cheatsheet_azure_devops_backup-1024x1024.png\")
<\/a><\/figure><\/div>\n\n\n\nDownload the cheat sheet<\/strong><\/a><\/div>\n<\/div>\n\n\n\n<\/p>\n\n\n\n
Moreover, your backup solution should allow you to create many custom backup policies for you to meet your organization, structure, and workflow requirements. One of the best practices to do so is to have a few backup plans. For example, one can be for unused repos that you need to have for compliance requirements and future reference, and another for critical Azure DevOps data that has daily or more frequent changes (use GFS rotation scheme or Forever Incremental). <\/p>\n\n\n\n
Save storage space with incremental & differential backups <\/strong><\/h2>\n\n\n\n
To save your storage space, your backup provider should give you an opportunity to include in your backup only changed blocks of your Azure DevOps data since the latest incremental or full copy. This will help you not only reduce the backup size inside your storage, but also limit the bandwidth, and improve the speed of your backup processes. <\/p>\n\n\n\n
Another important aspect is the possibility to define the retention period (here is a sneak peek – it should be up to unlimited!), and the performance schemes for every type of your backup copy, including full, differential, and incremental ones.<\/p>\n\n\n\n
Choose between SaaS and On-Premise deployments <\/strong><\/h2>\n\n\n\n
No matter what deployment model for Microsoft Azure DevOps you use – Cloud or Server, – your backup solution should allow you to back it up. Moreover, your backup provider has to allow you to choose where to run your backup software: self-hosted on your private infrastructure<\/a>, or in the Cloud<\/a>. The key difference between the two lies in the location where the backup service will be installed and running, as it may be critical for your compliance and security requirements. <\/p>\n\n\n\n
In the SaaS model, the service will be installed and run on the provider’s cloud infrastructure, therefore there is no need for any other device acting as a local server. The same goes for maintenance, administration, and operational continuity – it is all taken care of by the service provider. <\/p>\n\n\n\n
Contrary to the SaaS deployment model, your On-Premise deployment will require you to install the backup software locally, in your environment, on a machine under your provision. Your ideal backup solution should allow you to install the service on any computer (Linux, macOS, Windows), or NAS devices. In this case, you can avoid any issues related to network connectivity. Moreover, due to the backup copies being made inside the local network, your backup processes can be faster and a bit more efficient.<\/p>\n\n\n\n
\ud83d\udca1 Bear in mind that the deployment model should be independent of data storage compatibility. <\/p>\n\n\n\n
Flexible storage options are rather important in the context of backup, right? Well, with GitProtect.io backup and Disaster recovery software for sure DevOps<\/a>, unlimited cloud storage is always included in the license. Want to use your own storage? Not an issue\u2026 you can assign your local or any S3 compatible storage, including Azure Blob storage, AWS, Google Cloud, Wasabi Cloud, Backblaze B2, NAS\/local (On-Premise), local disk resources, iSCSI, SMB, CIFS, NFS. Moreover, you can keep your backup copies of your Azure DevOps data in multiple storage destinations, both local and cloud. <\/p>\n\n\n\n
Meet the 3-2-1 backup rule with multiple storage compatibility <\/strong><\/h2>\n\n\n\n
A robust backup solution for your Azure DevOps should permit you to add an unlimited amount of storage instances (as cloud, as local) and have the possibility of replication between those multiple storages. This way you won\u2019t be limited to a specific number of instances and you will be protected from risks associated with system outages or other disaster scenarios. Moreover, you will be able to meet the 3-2-1 backup rule<\/u><\/a>, which states that you should have no less than 3 backup copies in at least 2 different storage destinations, with at least 1 being off-site. <\/p>\n\n\n\n
GitProtect.io is a multi-storage system, which means that it allows you to store your data in multiple locations. So, you can assign any of:<\/p>\n\n\n\n
- \n
- Cloud storage – GitProtect Cloud, Azure Blob Storage, AWS S3, Wasabi Cloud, Backblaze B2, Google Cloud Storage, or any public cloud compatible with S3,<\/li>\n\n\n\n
- Local storages – local disk resources, NFS, SMB network shares, CIFS,<\/li>\n\n\n\n
- Hybrid environments<\/a>\/multi-cloud<\/li>\n<\/ul>\n\n\n\n
Well, let\u2019s look at an example, imagine your company\u2019s Security department requires that Azure DevOps data must be stored on Azure Blob storage but you take an extra step for better security and use a third-party backup solution to save copies of your data locally. Now, you need to restore data assets from 2 weeks ago immediately but there is an Azure Blob outage. What do you do? Well, you don\u2019t have to worry, as you made a smart choice to backup your work to the local server. Assuming that you backup Azure DevOps with <\/a>GitProtect.io, you just need to log in to your account, select the relevant copy from 2 weeks ago, and restore it. Within minutes, you are back on track and can continue your work as if nothing ever happened. <\/p>\n\n\n\n
The importance of backup replication <\/strong><\/h2>\n\n\n\n
Backup replication is one of the most critical features when it comes to choosing a backup solution. Why is it so? This feature allows you to have consistent copies in multiple locations. Thus, for example, if you follow the 3-2-1 backup rule and backup Azure DevOps data to multiple storage locations all your backup copies are consistent. It helps enable redundancy and ensure workflow continuity. <\/p>\n\n\n\n
Ideally, you should be able to replicate your data from any-to-any data storage. That includes cloud-to-cloud, cloud-to-local, local-to-cloud, and local-to-local without any limitations.<\/p>\n\n\n\n
GitProtect.io backup and Disaster Recovery software for Azure DevOps, for example, offers a central management console where you can manage your replication plan. All you need to do is to specify the source and target storage, along with the agent and schedule. What\u2019s more, with All2All replication<\/a>, you can maintain consistency of data across all of your replication and backup instances, since any changes made to your source code will be automatically and simultaneously sent to all the different locations in your backup replication plan. <\/p>\n\n\n\n
Take advantage of flexible retention options <\/strong><\/h2>\n\n\n\n
When it comes to backups, retention is important for a range of reasons – from mitigating data loss risks to meeting compliance. It\u2019s especially critical if your organization is in one of the strictly regulated industries<\/a>, where you need to meet rigorous compliance requirements and retention policies. For example, HIPAA <\/u><\/a>(Health Insurance Portability and Accountability Act) outlines that: <\/p>\n\n\n\n
\n
\u201cHIPAA-related files must be saved for six years after any documentation in the files was last effective. This means that if a policy was created in 2018, and it was effective until 2021, the policy document must be retained until 2027 (rather than 2024).\u201d<\/em> <\/p>\n<\/blockquote>\n\n\n\n
Your organization\u2019s data retention period requirements<\/a> may depend on the criticality of the data you store in your Azure DevOps environment, legal and compliance standards, or your company\u2019s necessity for archiving its data.<\/p>\n\n\n\n
It\u2019s worth mentioning that most service providers, as well as Microsoft, offer limited retention options – usually, they vary up to 365 days. In Azure DevOps users can set retention<\/u><\/a> for their pipelines, releases, and tests within the number of builds or the number of days (it may vary from 10 to 365 days). But what if you need to store your data for longer periods? As we have already mentioned, it can be guided by your organization, security, or compliance requirements, for example, if you need to pass SOC 2<\/u><\/a>, ISO 27001<\/u><\/a>, or any other security audit.<\/p>\n\n\n\n
Thus, longer retention plans or even unlimited retention are going to be at hand. Your backup software should allow you to set different retention for each of your backup plans by:<\/p>\n\n\n\n
- \n
- time every backup copy is kept in the storage – you should be able to set your retention rules separately for incremental, differential, and full backups, <\/li>\n\n\n\n
- number of copies you want to keep – you should specify the number of versions that should be saved in your storage, <\/li>\n\n\n\n
- or disable all the rules and keep your backup copies infinitely, for example, for archiving purposes.<\/li>\n<\/ul>\n\n\n\n
Monitoring center and alerts – all for effective backup processes <\/strong><\/h2>\n\n\n\n
In DevOps, you need a detailed overview of the processes that are taking place, especially if we speak about backup procedures and restore processes. Ideally, there should be a comprehensive monitoring center to give you insights into relevant information regarding backup tasks and restore operations, compliance reports, security checks, and full data on who and when performs a specific change in the settings.<\/p>\n\n\n\n
For that reason, your backup software should provide you with a central management console that will allow you to manage backup, restore, monitoring, and other system settings. Thanks to visual statistics and data-driven dashboards<\/strong>, which present the most important statistics regarding your DevOps ecosystem protection, the latest backup and restore operations, a quick summary of your compliance<\/u><\/a> strength, actual SLA auditing<\/u><\/a> and monthly SLA reports<\/u><\/a>, an overview of all your protected resources, and backup plans, you will always be in the know. <\/p>\n\n\n\n
Isn\u2019t it easier to start your working day with a clear summary of all your recent backup plans? With Daily Report Summary<\/strong> that you can get directly to your mailbox, it\u2019s possible. Thus, you will be able to easily track which processes are in progress, finished successfully, or with warnings. <\/p>\n\n\n\n
Moreover, email notifications<\/strong> are the easiest way to be up-to-date without even being logged in. Thus, you should have the possibility to get to your email all the necessary backup performance information like backup plan summary, restore verification summary, notification on your storage capacity, SLA report, and your plan status report. Another important thing is to be able to customize your email notifications<\/strong> within:<\/p>\n\n\n\n
- \n
- recipients (you will be informed about backup statuses without setting an account in the backup software infrastructure),<\/li>\n\n\n\n
- details on the backup plan summary, including successfully finished tasks, ones finished with warnings, failed tasks, canceled tasks, or those not started,<\/li>\n\n\n\n
- storage capacity notification,<\/li>\n\n\n\n
- SLA report for compliance requirements,<\/li>\n\n\n\n
- Restore verification summary. <\/li>\n<\/ul>\n\n\n\n
To ensure that you won\u2019t miss any critical changes, it\u2019s important that your backup software allows you to have notifications sent directly to the software you use daily – Slack notifications<\/strong>. In this case, by integrating Slack into the backup solution, you can easily and quickly verify the status of your backup performance without leaving your DevOps communication infrastructure. <\/p>\n\n\n\n
Another way to stay up-to-date with statuses is Webhook notifications<\/strong>. Your backup software should allow you to send a post request with your plan ID, plan name, execution times, etc. to the webhook URL address of your choice. <\/p>\n\n\n\n
Then, advanced audit logs<\/strong>. It\u2019s vital that your backup software provides you with logs containing all the necessary information about the work of applications, services, backup, and recovered data. Thanks to seeing who performed every action, you can prevent any intentional malicious activity. <\/p>\n\n\n\n
A dedicated GitHub user account to bypass throttling<\/h2>\n\n\n\n
Azure DevOps has a very permissible rate-limiting policy. It means that the vast majority of Azure DevOpsbackups should be throttling-free.<\/p>\n\n\n\n
However, even if throttling takes place, your backup software should have mechanisms to resolve it – for example, creating a few dedicated users solely for backup purposes. In this case, if the first exhausts the number of API requests, you shouldn\u2019t be distracted by adding another one, as everything will be done automatically. Thus, even the largest Azure DevOps environment will continue to operate without interruption while backups are performed. <\/p>\n\n\n\n
PART 2 – BACKUP SECURITY<\/h5>\n\n\n\n
How a reliable backup provider can help with compliance<\/strong><\/h2>\n\n\n\n
Security is a fundamental area to take care of, especially when it comes to source code. Therefore, it\u2019s important that your Azure DevOps backup option brings a set of security features that guarantee data protection and accessibility along with recoverability and, as a result, helps to meet the regulatory compliance requirements<\/a>.<\/p>\n\n\n\n
Thus, it should provide proper encryption in place, data replication and reporting, monitoring and alerting capabilities. Additionally, you need flexible restore and recovery capabilities along with automation. <\/p>\n\n\n\n
Not to be unfounded, let\u2019s look at ISO 27001 documentation<\/u><\/a> and its requirements: <\/p>\n\n\n\n
\n
\u201cOrganizations that are required to comply with Annex A.12.3 of the ISO\/IEC 27001 standard must establish, document, implement, and maintain a procedure for periodically backing up information required to be available for reconstruction after a physical or logical incident.\u201d<\/em> <\/p>\n<\/blockquote>\n\n\n\n
Moreover, your backup provider and Data Center where you have your service hosted should comply with international security standards, including SOC 2 Type II, ISO 27001, FISAMA, HIPAA, GDPR, etc. <\/p>\n\n\n\n
Let\u2019s look at the issues, your backup software should effectively meet more precisely:<\/p>\n\n\n\n
- \n
- AES encryption with your own encryption key,<\/li>\n\n\n\n
- in-flight and at-rest encryption,<\/li>\n\n\n\n
- all-in-one central monitoring and management,<\/li>\n\n\n\n
- different levels of retention – flexible, long-term, unlimited<\/li>\n\n\n\n
- the possibility to archive old and unused Azure DevOps repositories to meet your legal and organizational needs,<\/li>\n\n\n\n
- multi-tenancy and privileged-based access controls,<\/li>\n\n\n\n
- ransomware protection<\/a>,<\/li>\n\n\n\n
- strict legal and security measures for Data Centers,<\/li>\n\n\n\n
- restore and Disaster Recovery<\/a>.<\/li>\n<\/ul>\n\n\n
\n![\"DevOps](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2024\/10\/banner_orange-1024x346.png\")
<\/a><\/figure><\/div>\n\n\nAES encryption – in-flight and at-rest<\/strong><\/h2>\n\n\n\n
Another key component that guarantees the security of your Azure DevOps backups is encryption. Your data must be encrypted before it leaves your machine or environment, during the transfer and then once it is at rest in your repo. This helps to ensure your data is safe and cannot be altered or read even when it is intercepted. To be exact, make sure your backup provider has an Advanced Encryption Standard (AES) since it is an industry-standard and a safe way to encrypt data. It is a symmetric-key algorithm that uses the same key for encryption and decryption. Ideally, you should be able to choose the level of encryption since higher levels of encryption will extend the time of the backup process, take a look: <\/p>\n\n\n\n
- \n
- Low<\/strong>: AES algorithm working in CBC (Cipher-Block Chaining), with a 128-bit encryption key. <\/li>\n\n\n\n
- Medium<\/strong>: similarly, the AES algorithm works in CBC but the encryption key here is 192 bits. <\/li>\n\n\n\n
- High<\/strong>: Here the AES algorithm will run with a 256-bit encryption key and will work in CBC. <\/li>\n<\/ul>\n\n\n\n
Keep in mind that all of these encryption levels are considered unbreakable. Moreover, <\/a>your perfect backup solution should allow you to create your own encryption key, which could boost the security of your Azure DevOps environment even further since you are the only person who knows the encryption key and, subsequently, can decrypt it. <\/p>\n\n\n\n
- Medium<\/strong>: similarly, the AES algorithm works in CBC but the encryption key here is 192 bits. <\/li>\n\n\n\n
- Compliance with security protocols and audits – <\/strong>backup and Disaster Recovery are some of the requirements to meet strict compliance regulations. For example, here is a quote from ISO\/IEC 27001 documentation<\/u><\/a>:<\/strong><\/li>\n<\/ul>\n\n\n\n
- business continuity<\/strong> – service outages or your own infrastructure downtime can happen anytime. Unfortunately, no one can predict when it can happen, all you can do is to be prepared.<\/li>\n<\/ul>\n\n\n\n
- Shared Responsibility Model<\/strong> – all service providers operate within the Shared Responsibility Model, and according to it your account data is your, customer\u2019s, responsibility, Microsoft is responsible for its services to run smoothly:<\/li>\n<\/ul>\n\n\n