Orosz reports a latent Windows bug caused Microsoft's email outage
Gergely Orosz reports that a human-written Windows bug saturated machines and escaped regional containment. Microsoft is patching globally before publishing a postmortem.

TL;DR
- A human-written Windows change lay dormant for months before saturating Microsoft’s gateway machines, according to the initial account.
- Microsoft acknowledged failures in regional containment and mitigation speed, as a follow-up describes.
- AI diagnostics helped analyze traces; a developer identified the root cause, according to the clarification.
- A preliminary postmortem has now appeared, advancing the patch-first publication plan described in the original thread.
Microsoft’s preliminary report, reproduced by 5DMail, contains a nasty testing mismatch: the bug worsened on machines with more CPU cores, while load tests favored weaker hardware. Exchange Online could not bypass the broken gateway for scale reasons, although several other services could, according to The Register’s account of the report.
Three triggers at the Microsoft 365 gateway
The outage lasted 23 hours and 50 minutes, from November 12 at 16:20 UTC through November 13 at 16:10 UTC, according to Office 365 for IT Pros.
Gergely Orosz obtained his account from a Microsoft team lead who had been on the incident bridge. Microsoft told him the triggering change had not been AI-assisted, he later clarified.
Microsoft identifies three interacting factors in its preliminary report:
- Windows lock contention. A gateway-specific fix began deployment in July and ran globally for more than three months without observed impact. Under particular load conditions, contention inside an internal Windows process exhausted CPU and memory.
- Teams failover. Roughly 15 hours before the outage, Teams moved from an alternative gateway to the primary gateway shared with Exchange Online. This routine resilience operation increased gateway load and changed its traffic mix.
- Missing capacity. Nearly 14% of gateway server capacity was already unavailable because of an unrelated issue from the previous day. Microsoft says that reduction alone was within available capacity headroom.
The report explicitly limits both the underlying defect and its fix to Microsoft 365 networking gateway servers, excluding broader Windows deployments.
Failed regional bulkheads
The gateway comprises thousands of capacity units organized into regions. Its automatic traffic redirection sent requests from failed servers to surviving endpoints, increasing the likelihood that those endpoints would encounter the same defect, according to the preliminary report.
Within less than two hours, the defect had spread across the fleet. More than 80% of gateway servers were unresponsive at the worst point.
A separate issue prevented the normal partition bulkheads from stopping the cascade. Microsoft acknowledged two failures in the thread:
- Containment: the failure should have remained isolated rather than spreading globally.
- Mitigation: service restoration should have been substantially faster.
AI diagnostics and a developer’s clue
A developer recognized a familiar error in a dump and connected it to the root cause, according to the initial account. AI tools helped analyze traces and flag suspicious behavior during the investigation.
More than 1,000 engineers participated across dozens of parallel workstreams, according to a follow-up. Orosz rejected claims that layoffs had left Microsoft without people who understood the systems.
Rollback and patch rollout
Reverting the change was difficult, Orosz said in a reply. Microsoft’s preliminary report explains that ordinary rollback, flighting and feature flags could not quickly undo the months-old defect.
Restarting servers also failed to produce sustained recovery: waiting traffic immediately overwhelmed them again. Automated recovery processes were not designed to restore service from such a broadly unresponsive state.
Engineers instead:
- Developed an update returning the Windows code path to its previously known-good state.
- Took groups of servers out of service, applied the update and restarted them. Some batches encompassed more than half a region.
- Returned patched infrastructure to service and gradually restored traffic.
The first deployment of that mitigation began at 07:20 UTC on November 13. At 08:00 UTC, engineers confirmed stability on patched infrastructure and began expanding the rollout.
Orosz also reported that Microsoft was adding more powerful machines to the fleet.
Postmortem timeline
Orosz had said his technical follow-up would involve the responsible engineering team. His interview with Microsoft CEO Satya Nadella had been filmed before the outage began.
The preliminary report is dated November 15 and covers incidents EX1550543 and MO1550821, with access through the Microsoft 365 admin center; 5DMail has reproduced its contents.
Microsoft expects to publish the final post-incident report on November 20, according to The Register.