Skip to content
AI Primer
breaking

Orosz identifies database monitoring gaps in Revolut’s five-hour card outage

Gergely Orosz’s analysis identifies missing database-node monitoring and fragile quorum placement in Revolut’s five-hour card outage. He also questions recovery tools that depended on the failed GCP zone.

5 min read
Orosz identifies database monitoring gaps in Revolut’s five-hour card outage
Orosz identifies database monitoring gaps in Revolut’s five-hour card outage

TL;DR

  • Some five-voter coordination groups had already lost a member without an alert, as the highlighted postmortem excerpt records. Losing two more in Zone B left too few voters to authorise database failover.
  • Database monitoring and recovery tools also depended on the failed zone, a dependency questioned in the thread.
  • Card authorisations returned about 6 hours 40 minutes after the stated onset, per Revolut’s timeline, beyond the five hours described in the opening post.

Routine server replacements left some coordination groups incomplete, and replication lag later blocked automatic PostgreSQL failover even after quorum returned. Google’s power-failure report traces the initial outage to generator controllers running identical faulty logic.

Five-voter coordination groups

Revolut’s affected databases had replicas across three availability zones in Google Cloud’s London region, plus a replica in another region for disaster recovery. Separate five-member groups of distributed configuration store (DCS) servers determined which database replica could accept new transactions.

Each group required three votes. The failure sequence was:

  1. Configured membership: five voters. Losing Zone B’s two voters would ordinarily leave the required three.
  2. Before the power cut: four available. One member was already offline in some groups, without an alert.
  3. After Zone B failed: two available. Losing another two members left those groups unable to authorise promotion of a new primary.

Revolut says routine server replacements had failed to restore full DCS membership. The ugly part was the silent degradation: four voters kept serving, so customers saw no interruption while the affected groups lost their ability to survive Zone B’s loss.

Revolut’s postmortem excerpt describes the missing coordination server and the additional losses in Zone B.

Zone B recovery tooling

Revolut’s primary database-metrics viewer was deployed in Zone B. Engineers had to reconstruct database topology using logs, command-line utilities, and newly created standalone monitoring instances in surviving zones before remediation could fully begin.

The dependency affected both diagnosis and restoration, extending the outage beyond the database failover failure itself.

Recovery milestones

Backup processing allowed some card payments to succeed during the disruption. Revolut’s published milestones separate recovery into several stages on 20 October:

  • Around 06:00 UTC: the power failure disrupted services.
  • By 12:40 UTC: card authorisations and in-store payments were working again, approximately 6 hours 40 minutes after onset.
  • From 15:45 UTC: customers began regaining app access, updated balances, and transfers, approximately 9 hours 45 minutes after onset.
  • By 17:15 UTC: all remaining services were back online, approximately 11 hours 15 minutes after onset.

Some payments made during the outage took longer to appear in the app. Revolut says customer funds and data remained safe throughout.

PostgreSQL replication lag

Some PostgreSQL clusters remained read-only after DCS quorum returned because replication lag prevented automatic failover, according to Revolut. Engineers then had to verify replica state and determine a safe recovery path.

Revolut says it is working with Google Cloud to understand a suspected infrastructure brownout and its effect on database recovery.

Cross-zone capacity spillover

Google’s incident report describes physical power loss in europe-west2-b and additional disruption outside that zone:

  • Redirected workloads saturated capacity in neighbouring zones, producing queues, delayed database failovers, and intermittent errors.
  • Cross-zone control-plane requests to components in the failed zone encountered timeouts.

Revolut says its engineers eventually confirmed that Zones A and C remained available. Orosz challenged a podcast explanation that traffic redistribution had knocked Revolut’s other two data centres offline.

Manual failover and disclosure

Gergely Orosz, writer of The Pragmatic Engineer, called Revolut’s account “too hand-wavey” in his response. He questioned why surviving database nodes could not take over and argued that the account suggested no manual failover strategy.

Revolut says early reports suggested multiple regions might be affected, including its backup region. After confirming the power failure was confined to Zone B, engineers chose to continue recovery in Zones A and C under their single-zone recovery plan.

The published overview gives neither a timestamp nor the exact procedure for restoring DCS quorum, leaving the mechanics behind the recovery questions unspecified.

Orosz separately sought details of a software outage at the FAA that grounded all US flights.

Remediation status

Revolut’s 13 November update reports these completed technical changes:

  • Restored degraded coordination servers and full failover functionality.
  • Enhanced database-monitoring availability during single-zone outages.
  • Added immediate processing of card reversals after stand-in processing (STIP) backlogs clear.

Its broader operational and architectural work includes:

  • Failover testing: more frequent zonal exercises covering the incident’s conditions and restoration under load.
  • Redundancy alerts: implemented monitoring of DCS membership, voter health, and failure tolerance; additional work on alert-generation and delivery checks, cloud-quota warnings, and network-address headroom.
  • Recovery tooling: hardening supporting tools and testing them in recovery exercises to reduce manual intervention.
  • Placement rules: changing a deployment process that favoured one zone for each group’s first server, with checks that every group can continue operating after any single-zone failure.

Further reading

Discussion across the web

Where this story is being discussed, in original context.

On X· 4 threads
TL;DR1 post
Five-voter coordination groups2 posts
Recovery milestones1 post
Manual failover and disclosure2 posts
Share on X