{"id":3826,"date":"2023-01-25T14:02:29","date_gmt":"2023-01-25T14:02:29","guid":{"rendered":"https:\/\/gitprotect.io\/blog\/?p=3826"},"modified":"2024-11-21T12:22:14","modified_gmt":"2024-11-21T12:22:14","slug":"bitbucket-backup-to-s3","status":"publish","type":"post","link":"https:\/\/gitprotect.io\/blog\/bitbucket-backup-to-s3\/","title":{"rendered":"Bitbucket Backup to S3"},"content":{"rendered":"\n

Companies utilizing Bitbucket or other solutions as the primary Git-based source control must repeat a particular truism ad nauseam. Preventing data loss is vital to maintaining business continuity. One of the most crucial aspects in dealing with such a challenge is ensuring data protection in S3-compatible cloud storage.<\/strong> <\/p>\n\n\n\n\n\n\n\n

More so, as saving the existing repository alone is not enough. It\u2019s crucial the backup plan includes metadata<\/a>, like pull requests (Git command), history (e.g., comments), and other:<\/p>\n\n\n

\n
\"Bitbucket<\/figure><\/div>\n\n\n\n\n

S3 for Bitbucket repositories<\/strong><\/h2>\n\n\n\n

Bitbucket Cloud services don’t inherently offer native backup and restore features, so users must create or implement their own solutions (or tools).<\/p>\n\n\n\n

\n

We do not use these backups to revert customer-initiated destructive changes, such as fields overwritten using scripts, or deleted issues, projects, or sites. To avoid data loss, we recommend making regular backups.<\/p>\n<\/blockquote>\n\n\n\n

Source: Atlassian security practices<\/u><\/a> under the Shared Responsibility Model.<\/em><\/p>\n\n\n\n

A similar problem also affects other Git-based platforms, such as GitLab<\/a> or GitHub<\/a>.<\/p>\n\n\n\n

Given the distributed nature of cloud repositories and the need for quick recovery in the case of information loss, S3s (Simple Storage Services) \u2013 at the moment \u2013 create the best possibilities when it comes to:<\/p>\n\n\n

\n
\"Bitbucket<\/figure><\/div>\n\n\n

Example methods of backing up Bitbucket repository data to S3<\/h2>\n\n\n\n

You can list several ways to store Bitbuckets repos in S3, from manual to fully automated methods. It all depends on your scalability and control needs. The most effective approaches include:<\/p>\n\n\n\n

Manual backup using AWS CLI<\/strong><\/h3>\n\n\n\n

The backup of your Bitbucket instance can be created using the bitbucket.diy-backup.sh.<\/em> You can manually upload it to an S3 bucket with AWS CLI.<\/p>\n\n\n\n

To generate a backup, all it takes is to run the command:<\/p>\n\n\n\n

.\/bitbucket.diy-backup.sh<\/code><\/pre>\n\n\n\n
Then, you can upload the data to S3.<\/p>\n\n\n\n
aws s3 cp \/path\/to\/backup.tar.gz s3:\/\/bucket-name\/<\/code><\/pre>\n\n\n\n
The solution is suitable for small setups or occasional backups with no priority on automating activities. However, it lacks scheduling and is prone to human errors.<\/p>\n\n\n\n
A more broad example of using Git command and AWS CLI:<\/p>\n\n\n\n
#!\/bin\/bash\n# Configure AWS CLI access\nexport AWS_ACCESS_KEY_ID=your-access-key\nexport AWS_SECRET_ACCESS_KEY=your-secret-key\nexport S3_BUCKET=your-s3-bucket\n\n# Clone the Bitbucket repository\ngit clone https:\/\/bitbucket.org\/your-org\/your-repo.gisdt \/tmp\/your-repo\n\n# Sync the cloned repository with the S3 bucket\naws s3 sync \/tmp\/your-repo s3:\/\/$S3_BUCKET\/backups\/ --delete<\/code><\/pre>\n\n\n\n
Automated backup with Cron Job and AWS CLI<\/h3>\n\n\n\n
The method allows you to automate backup and upload utilizing cron jobs on Linux or macOS operating systems or Task Scheduler on Windows.<\/p>\n\n\n\n
It starts with the shell script:<\/p>\n\n\n\n
#!\/bin\/bash\/path\/to\/bitbucket.diy-backup.sh \naws s3 cp \/path\/to\/backup.tar.gz s3:\/\/bucket-name\/<\/code><\/pre>\n\n\n\n
It can be scheduled to run the script regularly (e.g., daily) using cron:<\/p>\n\n\n\n
crontab -e 0 2  * \/path\/to\/backup-script.sh<\/code><\/pre>\n\n\n\n
Such an approach is best for teams looking for a fully automated process without a complex setup.<\/p>\n\n\n\n
S3 sync for multiple backups<\/h3>\n\n\n\n
The AWS s3 sync command helps upload all directories to S3 storage. It guarantees that the Bitbucket backups are synchronized without uploading data individually.<\/p>\n\n\n\n
aws s3 sync \/path\/to\/backup-directory\/ s3:\/\/bucket-name\/<\/code><\/pre>\n\n\n\n
All you have to do is sync the target folder to S3:<\/p>\n\n\n\n
aws s3 sync \/path\/to\/backup-directory\/ s3:\/\/bucket-name\/<\/code><\/pre>\n\n\n\n
It’s a good way to avoid redundant uploads, providing automatic syncs only for new or modified data. Yet, careful storage management is required to avoid high costs.<\/p>\n\n\n\n
CI\/CD pipeline integration<\/h3>\n\n\n\n
Backing up Bitbucket repos can be integrated into a CI\/CD pipeline using Jenkins or Bitbucket Pipelines. It provides more control and flexibility over the schedule and upload process.<\/p>\n\n\n\n
You can create a Jenkins job or Bitbucket Pipeline to run the Bitbucket backup script and use aws s3 cp to upload the backup to an S3 bucket.<\/p>\n\n\n\n
Schedule it to run nightly or based on events, automating backup management and providing centralized lead with logging, but requiring setup and maintenance.<\/p>\n\n\n\n
pipelines: \n   default: \n\t- step: \n\t   name: Build artifact \n\t   script: \n\t\t- mkdir artifact \n\t\t- echo \"Pipelines is awesome!\" > artifact\/index.html \n\t   artifacts: \n\t\t- artifact\/* \n \t- step: \n\t   name: Deploy to S3 \n\t   deployment: production \n\t   script: \n\t\t- pipe: atlassian\/aws-s3-deploy:0.3.8 \n\t\t  variables: \n\t\t\tAWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID\n\t\t\tAWS_SECRET_ACCESS_KEY$ AWS_SECRET_ACCESS_KEY\n\t\t\tAWS_DEFAULT_REGION: 'us-east-1' \n\t\t\tS3_BUCKET: 'bbci-example-s3-deploy' \n\t\t\tLOCAL_PATH: 'artifact' \n\t\t\tACL: 'public-read'<\/code><\/pre>\n\n\n\n
Source: Atlassian<\/a>.<\/em><\/p>\n\n\n\n
AWS SDK for programmatic backup<\/h3>\n\n\n\n
A custom AWS SDK script – such as a Python boto3, Java, or Node.js – might help you manage the backup and upload process in a broader fashion.<\/p>\n\n\n\n
A fine example is the script generating the Bitbucket backup with bitbucket.diy-backup.sh script<\/em>:<\/p>\n\n\n\n
import boto3\ns3 = boto3.client('s3')\ns3.upload_file('\/path\/to\/backup.tar.gz', 'bucket-name', 'backup.tar.gz')<\/code><\/pre>\n\n\n\n
It may fit teams looking for highly customizable solutions or complex workflows. The method provides fine-grained control yet requires programming expertise.<\/p>\n\n\n\n
Another code may be:<\/p>\n\n\n\n
import boto3import os\ns3 = boto3.client('s3',\n   aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),\n   aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY'))\n\nlocal_repo_path = '\/path\/to\/your\/local\/repository'\n\n   for root, dirs, files in os.walk(local_repo_path):\n      for file in files:\ns3.upload_file(os.path.join(root, file), 'your-s3-bucket', f'backups\/{file}')\n<\/code><\/pre>\n\n\n\n
The script recursively uploads all data in the local repository to S3. Therefore, even newly created elements are part of the backup.<\/p>\n\n\n\n
Of course, there are other ways to back up Bitbucket instances, like using:<\/p>\n\n\n\n