![\"\"](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2025\/01\/reported_incidents_v2.png\")
<\/figure><\/div>\n\n\n\n\nDECEMBER<\/strong> In December some Azure DevOps users in Europe and Brazil experienced 3 incidents with degraded impact, totaling almost 40 hours. <\/p>\n\n\n\n Those 3 cases of degradation performance were impacting Azure DevOps services, including Boards, Repos, Pipelines, Test Plans, Artifacts, and other services. <\/p>\n\n\n\n NOVEMBER<\/strong> Azure DevOps experienced 12 incidents in November. All of the incidents Microsoft classified as degraded impact totaled around 20 hours of disruption.<\/p>\n\n\n\n The most significant incident occurred on November 18th, impacting North European users for over 6 hours, who had difficulties accessing Boards, Repos, Pipelines, and Test Plans.<\/p>\n\n\n\n Other incidents were shorter, targeting different regions, including Asia Pacific, the United States, Australia, and Brazil.<\/p>\n\n\n\n Among the services that were impacted in November were pipelines, boards, repos, and other core services.<\/p>\n\n\n\n OCTOBER<\/strong> In October Azure DevOps users globally experienced 7 degraded incidents with a total time for disruptions of over 40 hours.<\/p>\n\n\n\n The incidents included brief availability degradations across test plans, boards, repos, pipelines, and other services. <\/p>\n\n\n\n SEPTEMBER<\/strong> In September Azure DevOps faced 6 degraded incidents, totaling around 168 hours of disruption\u2026 almost 7 days! The most prolonged incidents included a 93-hour 28-minute issue that affected connections in Pipelines. <\/p>\n\n\n\n Other smaller incidents impacted Artifacts and other Azure DevOps services. <\/p>\n\n\n\n AUGUST<\/strong> Azure DevOps users could experience 5 incidents of different severity in August. The total time of disruption compounds 12 hours and 29 minutes.<\/p>\n\n\n\n The major incident, lasting 1 hour and 21 minutes on August 5, caused intermittent errors across core services like Boards, Repos, and Pipelines in Brazil and the United States.<\/p>\n\n\n\n Multiple Azure customers across North and Latin America were taken down due to a Microsoft Azure outage. The company said that the incident impacted services that leverage Azure Front Door (AFD). <\/p>\n\n\n\n As Microsoft stated, the incident happened due to \u201ca recent configuration change made by an internal service team that uses AFD. Once that change was understood as the trigger event, we initiated a rollback of the configuration change which fully mitigated all customer impact<\/em>.”<\/p>\n\n\n Source: <\/em>Azure DevOps Status<\/em><\/a><\/p>\n\n\n\n As a result, for around 2 hours Azure DevOps users reported errors connecting to Azure services and logging problems. <\/p>\n\n\n\n This outage followed another massive incident, an Azure outage, that lasted for 9 hours and impacted many Azure and Microsoft 365 services. The reason for that 9-hour outage was a volumetric TCP SYN flood DDoS attack which targeted many Azure Front Door and CDN sites. <\/p>\n\n\n\n Bleeping Computer<\/a> \/ DCD<\/a><\/p>\n\n\n\n JULY<\/strong> In July, Azure DevOps users around the world saw 22 incidents, all with degraded impact, resulting in a total disruption time of more than 200 hours (more than 8 days!).<\/p>\n\n\n\n The key incidents included an over 275-hour-long disruption affecting test case order changes across multiple regions and a 38-hour availability degradation in Service Hooks in SEA. <\/p>\n\n\n\n Other incidents were affecting core Azure DevOps services like Boards, Pipelines, Test Plans, and Repos across various regions, including Europe, South and North America.<\/p>\n\n\n\n On July 18th Azure DevOps users might experience issues accessing their Azure DevOps organization.<\/p>\n\n\n\n In their post-mortem<\/a>, Microsoft explained that \u201cThis outage lasted roughly from 21:40 to 02:55 UTC. During this outage, customers may have experienced a range of impact from a full outage of their organization to partial degradation with certain functionality unavailable such as Artifacts, Packaging, Test, etc.\u201d<\/em><\/p>\n\n\n\n The outage was caused by VM availability in the Central US region and led to all Azure DevOps organizations located in that region being inaccessible. For over an hour, Azure DevOps users were receiving a 503 \u201cService unavailable\u201d message when they were trying to access their Azure DevOps instances. Moreover, around 1.4% of organizations outside of the Central US region might also be experiencing some kind of performance degradation at that time as well. <\/p>\n\n\n Source: Azure DevOps post-mortem<\/a>: request failure rates<\/em><\/p>\n\n\n\n JUNE The incidents that Azure DevOps users might face in June were all of degraded severity with the total time of disruption of 21+ hours.<\/p>\n\n\n\n The most significant incident was a 7-hour 26-minute service connection degradation affecting Pipelines.<\/p>\n\n\n\n Other incidents were much shorter and impacted such Azure DevOps services as Boards, Pipelines, Repos, and Test Plans. <\/p>\n\n\n\n Ten Azure services, including Azure DevOps, Azure, Azure Load Testing, Azure API Management Application Insights, etc. were found vulnerable to a bug that allowed an attacker to bypass firewall rules and gain unauthorized access to cloud resources by abusing Azure Service Tags.<\/p>\n\n\n\n A researcher from the cybersecurity firm Tenable stated<\/a>:<\/p>\n\n\n\n \u201cThis vulnerability enables an attacker to control server-side requests, thus impersonating trusted Azure services. This enables the attacker to bypass network controls based on Service Tags, which are often used to prevent public access to Azure customers’ internal assets, data, and services.<\/em>\u201d<\/p>\n\n\n\n While there\u2019s no evidence of this vulnerability being exploited in the wild, Microsoft advises customers to review their use of Service Tags, strengthen validation controls, and implement robust authentication to ensure only trusted traffic is allowed.<\/p>\n\n\n\n In its guidance issued on June 3rd, 2024, Microsoft states<\/a> that \u201cService tags are not to be treated as a security boundary and should only be used as a routing mechanism in conjunction with validation. […] Service tags are not a comprehensive way to secure traffic to a customer\u2019s origin and do not replace input validation to prevent vulnerabilities that may be associated with web requests.<\/em>\u201d<\/p>\n\n\n\n The Hacker News<\/a> \/ Bleeping Computer<\/a><\/p>\n\n\n\n MAY<\/strong> In May, Azure DevOps encountered 10 incidents with a degraded impact, amounting to a total disruption time of over 18 hours.<\/p>\n\n\n\n Users from all geographies might experience disruptions while working with Boards, Repos Pipelines, and other Azure DevOps services. The longest incident that month, lasting 6 hours and 56 minutes, involved failures in Pipelines due to Workload Identity federation-based ARM service connections.<\/p>\n\n\n\n APRIL Users of Azure DevOps in different locations might experience 12 incidents with degraded impact, varying from a few minutes to hours. Summing up, Azure DevOps experienced more than 42 hours of disruptions in April. <\/p>\n\n\n\n The longest incident lasted over 19 hours. During that time Azure DevOps users all around the world might experience slowness and disconnections from the service, impacting Boards, Repos, Pipelines, Artifacts, etc.<\/p>\n\n\n\n On April 25th, Azure DevOps users all around the world might experience service availability degradation – the slowness and intermittent failures across the service. The issue stemmed from a severe performance regression in a frequently used stored procedure within the Token Service database, increasing execution time by nearly 5000x.<\/p>\n\n\n Source: <\/em>Azure DevOps post-mortem<\/em><\/a>: Global request failure rate %<\/em><\/p>\n\n\n\n As Microsoft stated in its post-mortem<\/a> later \u201cThe incident was caused by a significant regression in the performance of a stored procedure with a high frequency of calls in the Token Service.\u201d<\/em><\/p>\n\n\n\n Among the affected Azure DevOps experiences were Azure Pipelines with an average delay of over 3 minutes, Personal Access Token (PAT) usage, and increased failure rates for Azure DevOps extensions, with the global request failure rate peaking at around 20%.<\/p>\n\n\n\n Some Azure DevOps users were experiencing network-related errors, unexpected cancellations, and slowness while they were trying to access large files<\/a> over HTTPS for a few days (April, 22nd – April 24th). That led to reduced availability of Azure repositories, boards, and pipelines.<\/p>\n\n\n\n As Microsoft explained in its post-mortem<\/a>, the reason turned out to be an update that the service provider rolled out sometime prior:<\/p>\n\n\n\n \u201cAs part of ongoing work to make Azure Front Door more reliable, the team rolled out an update, which included a specific change for how packets are processed.\u202fDue to this change, some traffic was not processed correctly, which triggered clients to resend the packets, and this led to higher latency. As such, this issue\u202fprimarily affected\u202fscenarios with large downloads and responses.\u201d<\/em><\/p>\n\n\n\n Azure DevOps engineers quickly detected and addressed the issue. <\/p>\n\n\n\n MARCH<\/strong> In March 2024, Azure DevOps experienced seven incidents resulting in degraded service across various regions, totaling over 36 hours of disruption.<\/p>\n\n\n\n The most significant issue happened on March 27\u201328, when Analytics dashboard widgets failed to load globally for over 22 hours.<\/p>\n\n\n\n Other incidents included availability degradations in Central US, West Europe, and the UK, affecting Boards, Repos, Pipelines, and Test Plans. The artifacts were also impacted twice, showing degraded performance in West Europe and the Central US region.<\/p>\n\n\n\n FEBRUARY<\/strong> Azure DevOps experienced 12 incidents which led to over 48 hours of degraded service across multiple regions in February 2024.<\/p>\n\n\n\n The most prolonged disruption lasted over 26 hours, affecting extensions in the United States.<\/p>\n\n\n\n Europe saw the highest concentration of issues, with multiple availability degradations impacting Boards, Repos, Pipelines, and Artifacts, particularly in West Europe.<\/p>\n\n\n\n Other incidents included pipeline delays in the UK, a global issue preventing Pipeline Artifact downloads, and deprecated pipeline tasks becoming unusable.<\/p>\n\n\n\n Microsoft addressed 73 security vulnerabilities and 2 actively exploited zero-days in February. Among the patches for different Microsoft products, there was one for Azure DevOps Server to address the CVE-2024-20667 security flaw tagged as \u201cimportant\u201d within the severity level.<\/p>\n\n\n\n Using the vulnerability, an attacker could run unauthorized code on the user\u2019s Azure DevOps server and potentially compromise sensitive data or disrupt the user\u2019s services. However, there should be specific conditions for an attacker to succeed in an attack within this vulnerability:<\/p>\n\n\n\n \u201cSuccessful exploitation of this vulnerability requires the attacker to have Queue Build permissions and for the target Azure DevOps pipeline to meet certain conditions for an attacker to exploit this vulnerability,<\/em>\u201d is stated in Microsoft\u2019s security updates<\/a>. <\/p>\n\n\n\n The February updates are essential for users to protect their Azure DevOps Server against this remote code execution bug. <\/p>\n\n\n\n Bleeping computer<\/a><\/p>\n\n\n\n JANUARY<\/strong><\/p>\n\n\n\n There is no full information available about the incidents in January.<\/em><\/strong><\/p>\n\n\n\n After analyzing all the events that impacted Azure DevOps and thoroughly going through the Azure DevOps Status<\/a> (here we managed to analyze only the period from February 2024 to December 2024), we counted that the service experienced 111 incidents of degraded performance, which gives us the total time of disruption – 826 hours and 02 minutes. Thus, Azure DevOps experienced some kind of disruption for about 103 working days, including an outage that took place in Brazil and the United States.<\/p>\n\n\n Let\u2019s not forget that Azure DevOps is a secure platform and Microsoft takes all the necessary measures to make its services available. It patches vulnerabilities and provides updates to Azure DevOps users to know how to deal with them. Here we shouldn\u2019t forget about the Shared Responsibility Model that Microsoft follows. Within it, Microsoft is responsible for its services staying available while users should take care of their data and its security, availability, and recoverability. <\/p>\n\n\n\n To have peace of mind that their source code and project data are secured, Azure DevOps users should implement security best practices<\/a>, which include MFA, monitoring, backup, etc.<\/p>\n\n\n\n Moreover, in view of meeting not only security but also compliance requirements, Azure DevOps users should consider backing up their DevOps and PM\u2019s environment. GitProtect backup and DR software for Azure DevOps<\/a> helps admins ensure that their company\u2019s critical data is accessible and recoverable in any event of disruption. The solution helps organizations to follow backup best practices<\/a> that leave no place for data loss. <\/p>\n\n\n\n
Azure DevOps Status: 3 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 3 incidents<\/strong>
\u231b Total time of incidents with degraded performance – 39 hours 21 minutes<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 12 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 12 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 19 hours 55 minutes<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 7 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 7 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 42 hours 55 minutes<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 6 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 6 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 168 hours 34 min<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 5 incidents<\/strong>
\ud83d\udea8 Incidents that had a major impact – 1 incident<\/strong>
\u231b Total time of incidents with unhealthy performance<\/strong> – 1 hour 21 minutes<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 4 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 11 hours 8 minutes<\/strong><\/p>\n\n\n\nServices across North and Latin America are taken down due to Microsoft Azure outage<\/h2>\n\n\n\n
![\"Azure](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2025\/01\/AzureDevOps-threats-1-1024x511.png\")
<\/figure><\/div>\n\n\n
Azure DevOps Status: 22 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 22 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 376 hours 08 minutes<\/strong><\/p>\n\n\n\nStorage incident in the Central US impacted Azure DevOps users all over the world<\/h2>\n\n\n\n
![\"Azure](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2025\/01\/Azure-DevOps-2.png\")
<\/figure><\/div>\n\n\n
<\/strong>
Azure DevOps Status: 15 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 15 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 21 hours 30 minutes<\/strong><\/p>\n\n\n\nThreat actors might gain unauthorized access to Cloud Azure resources<\/h2>\n\n\n\n
Azure DevOps Status: 10 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 10 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 18 hours 38 minutes<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 12 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 12 incidents<\/strong>
\u231b Total time of incidents with degraded severity – 41 hours 25 min<\/strong><\/p>\n\n\n\nFailures and slowness across Azure DevOps in all geographies<\/h2>\n\n\n\n
![\"Azure](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2025\/01\/image-8-1024x359.png\")
<\/figure><\/div>\n\n\nReduced availability of Azure DevOps services across the globe<\/h2>\n\n\n\n
Azure DevOps Status: 7 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 7 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 36 hours 22 min<\/strong><\/p>\n\n\n\n
Azure DevOps Status: 12 incidents<\/strong>
\u26a0\ufe0f Incidents that had a degraded impact – 12 incidents<\/strong>
\u231b Total time of incidents with degraded performance<\/strong> – 48 hours 45 min<\/strong><\/p>\n\n\n\nCritical Azure DevOps flaw could allow remote code execution<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
![\"\"](\"https:\/\/gitprotect.io\/blog\/wp-content\/uploads\/2025\/01\/incidents_by_type_v2.png\")
<\/figure><\/div>\n\n\n