Headlines

Microsoft’s top AI executive says ‘deeply sorry’ for outage, shares three causes for disruption


Microsoft's top AI executive says ‘deeply sorry' for outage, shares three causes for disruption

Microsoft has apologised for a global outage that hit its Microsoft 365 services last week and has explained what went wrong. Charles Lamanna, Microsoft’s executive vice president of Copilot, Agents and Platform, shared the company’s Preliminary Post-Incident Report on the outage that took place last week. The incident happened on Thursday, November 12, 2026. According to Microsoft, three main factors came together to cause a global outage of the networking gateway that sits in front of many Microsoft 365 services. More details on the root cause are available in the Microsoft 365 Admin center, and Lamanna shared a shorter version.“More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations,” Lamanna said.

Three causes of Microsoft 365 services’ global outage

According to the Microsoft executive, three primary issues converged to knock offline the networking gateway that underpins several Microsoft 365 services.Cause 1: A hidden defect in the serversThe first cause was a latent defect. Under certain load conditions, it left servers overwhelmed and unresponsive. Microsoft said the problem was lock contention inside an internal process, which made the machine unavailable. The defect came from a fix that was prepared specifically for Microsoft 365 networking gateway servers. Microsoft said neither the defect nor the fix applies to broader Windows deployments.The change first began rolling out in July this year. It went through safe deployment and spread globally over several weeks, and no issues were seen. For more than three months, the change ran worldwide without any impact.Cause 2: A surge in loadThe second cause was a surge in load, which made it more likely that the gateways would run into the contention. Microsoft 365 uses two independent and heterogeneous network gateway systems, built for isolation and high availability. This allows many services to fail over between them during outages or disruptions.About fifteen hours before the outage, Teams failed over from the alternative gateway to the “primary” gateway, which also supports Exchange Online. That increased the load on the primary gateway and changed its traffic patterns. Microsoft said this kind of failover is part of normal operations and is an integral part of its standard resiliency efforts.Cause 3: Reduced server capacityThe third cause was that nearly 14% of the gateway’s server capacity was out of service at the same time. The cause was an unrelated issue from the previous day. The capacity was on track to return to service through routine maintenance and health processes.

How the Microsoft 365 services outage spread

Microsoft said the higher load and the lower server capacity together caused the gateway to hit the latent bug from July. Requests built up until the servers stopped responding. This happened during the peak window of traffic in the United States.When servers lose connectivity, Microsoft’s traffic management systems automatically send users to other healthy endpoints. This happens often and normally keeps the service resilient. This time, however, the redirected traffic raised the chance of the same problem hitting the remaining endpoints. The same load was spread across fewer servers, which led to a cascading failure from server to server.In less than two hours, the previously unknown defect had hit servers across the fleet and left them unresponsive. At its lowest, more than 80% of servers were unresponsive.

Why some services came back sooner

Several services, including Teams, OneDrive, SharePoint and the Office apps, failed out of the faulty gateway infrastructure. That is why they were restored earlier. For most of the outage, Teams chats and meetings worked without issue. Not every service can do this, though. Exchange Online, for example, cannot fail out of this gateway.

Why recovery took longer than usual

Recovery took longer because servers that came back into service were immediately overwhelmed by waiting traffic and stopped responding again. Microsoft said its normally automated recovery processes were not designed to restore service from such a broad unresponsive state.As the main defect was a latent bug from several months earlier, the company’s normal rollback, flighting and feature flags could not quickly undo the change.Once engineers confirmed the defect as the cause, they built an update that returned the code path to its previously known good state. To apply the fix and slowly bring the service back, they took groups of servers out of service, applied the change and restarted them before returning them to service. They did this carefully on large amounts of infrastructure at a time, such as 50% or more of a region.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *