Blog

When one zone loses power, the other two can fill up

2026-10-23

On 20 October 2026 a power cut took Google Cloud zone europe-west2-b offline at 06:00 UTC. Failover then filled the other London zones. How to say that while regional APIs are still returning 503.

At 06:00 UTC on 20 October 2026 the batteries ran out in the facility that hosts Google Cloud zone europe-west2-b. Instances in that zone went unreachable. Eight minutes later Google's monitors paged on elevated API errors. Calls that had failed over into europe-west2-a and europe-west2-c were returning 500 and 503 as well. Is London up? That question is too coarse. You need the zone that lost power, and whether the other two still had room.

What Google's preliminary report says

Google's preliminary incident report is on incident Xs2XTHzAVC9YnafnPyzj. The page says the report can still change, and that a final analysis is still to come. The clocks below are the ones on that page, converted to UTC. UK time that morning was BST, one hour ahead.

• 05:53 UTC (06:53 BST): a power-grid disruption at the facility hosting europe-west2-b. The site moved onto UPS batteries. Both the primary and the backup programmable logic controllers failed to start the emergency generators.
• 06:00 UTC (07:00 BST): the batteries were depleted. Hosts, local storage, and the network aggregation hardware in europe-west2-b went offline. This is the incident start on the page.
• 06:08 UTC (07:08 BST): automated monitoring alerted engineering. Elevated API error ratios, job-submission timeouts, and loss of reachability in that zone.
• 11:46 UTC (12:46 BST): new virtual machine creation resumed.
• 11:58 UTC: The Register recorded Google saying full recovery of workloads should be completed, tentatively, by 13:00 UTC.
• 12:10 UTC (13:10 BST): incident end on the page. Google said more than 99% of Compute Engine virtual machines were healthy. Duration on the page: 6 hours and 10 minutes.
• 14:03 UTC (15:03 BST): closure note. Mitigated for most customers, believed to still affect a very small number, with no further updates on that page until the analysis.

Google says the power loss itself stayed in europe-west2-b. Customers with workloads in europe-west2-a and europe-west2-c, and customers on regional services, still saw errors. The preliminary report gives two reasons. Regional routing and client applications failed over onto the zones that still had power, and that surge used up the spare compute. At the same time, control-plane calls from the healthy zones waited on metadata and catalog components in the offline zone and timed out. Google names BigQuery, Looker, Compute Engine, GKE, Regional Persistent Disk, and Cloud SQL among the services that degraded for that reason. Regional APIs returned HTTP 500 and 503. Storage writes in the region had a transient error rate while failover was in progress.

During recovery, an update The Register quoted said the power event was mitigated, existing workloads were the priority, and customers might not yet be able to create new workloads. Creates and serving were different clocks for part of the morning.

What customers were doing at 08:00 BST

The BBC reported that Revolut, Lloyds, and HSBC confirmed disruption that morning. A Revolut spokesperson told the BBC that some Revolut services were disrupted because of the Google Cloud outage, and later that services were restored and that customers' money and data stayed safe. Shortly after 08:00 BST, the BBC said more than 1,500 reports had been filed against Revolut. One of those customers, Rachael Ellen in Glasgow, could not pay for parking because the app was not working, and the in-app help bot told her a transfer from a relative was not pending.

The same piece reported Guy Baybutt of The Veg Box Company in Preston. The firm could not open its website to download and process the morning's orders, could not book parcels onto DPD for collection, and lost sales from customers who could not place an order. Produce that had to leave that day was the thing at risk. The BBC also reported user-flagged trouble at Royal Mail, DPD, and the courts' remote-hearing video platform. Those reports are not a claim that each of those organisations runs in europe-west2-b. Revolut is the company that named Google Cloud.

Leaving the zone was the wrong comfort this time

On 1 September 2026 Google said a network problem was confined to part of us-central1-b and that the rest of that region was unaffected. Leaving the zone was the workaround, and Expo did it. That writeup is when one zone is down and the region still looks up. On 20 October the preliminary report says the other London zones were not spare capacity. Failing over inside europe-west2 landed on zones that were already filling up, and some of those API calls still waited on components in the zone that had no power.

A status update that names a one-zone power event, says you moved to another zone in London, and calls everything else operational, is the 1 September sentence pasted onto the 20 October incident. At 08:00 BST it is already false if your errors in europe-west2-a or europe-west2-c are still 503s.

Copy you can adapt

The weak version, which could be pasted onto any incident:

We are aware of a power issue at a third-party cloud facility. We have failed over to another zone in the same region. All other systems remain operational. We apologize for any inconvenience.

That paragraph costs the parking payment and the morning's parcels. At 08:00 BST a customer in Glasgow cannot pay, support says retry, and those retries are the surge the preliminary report says filled the remaining zones. A reader an hour later cannot tell which zone lost power, whether the other two were also erroring, or when creates started working again.

The version that names both scopes:

As of 07:08 BST, sign-in and payments are failing. Google Cloud's preliminary report says hosts in europe-west2-b lost power at 07:00 BST after the facility generators did not take the load. The same report says customers in europe-west2-a and europe-west2-c also saw HTTP 500 and 503, because failover traffic filled the remaining capacity and some regional APIs waited on components in the offline zone. We will not mark payments operational while those regional errors continue. New virtual machines in this region could not be created for part of the morning. Google says new VM creation resumed at 12:46 BST.

A reader who was not on the bridge can check each sentence against incident Xs2XTHzAVC9YnafnPyzj and against your own errors by zone. The weak paragraph cannot be checked, so it cannot be trusted an hour later.

What not to do

• Do not tell customers the failover inside London worked until error rates in the other two zones agree.
• Do not stop after naming the zone that lost power, once the vendor has also named spillover into the rest of the region.
• Do not treat resumed virtual machine creation as recovered payments. Google kept those clocks apart until 11:46 UTC.
• Do not close your note at the 12:10 UTC incident end if you still have customer errors. The 14:03 UTC note still allowed a very small tail.
• Do not invent a probe of europe-west2. A public hostname answering is not a zone capacity check.

Operator checklist

1. Write two scopes: the zone that lost power, and any impact the vendor names in the other zones of that region.
2. Split your own errors the same way. Unreachable in europe-west2-b is one line. HTTP 500 and 503 from europe-west2-a, europe-west2-c, and regional APIs is another.
3. Keep a third clock for creates. On this incident, existing workloads were the recovery priority. New VM creation resumed at 11:46 UTC. More than 99% of VMs were healthy at 12:10 UTC. The closure note was 14:03 UTC.
4. Before you tell customers you left the sick zone, read whether the vendor says the destination zones still had room. On 1 September they did. On 20 October they did not.
5. Write the customer update so a reader can check it against the incident page and against your errors by zone.

StackWitness does not publish a Google Cloud truth page. /status has no Google Cloud entry, and a probe of a public hostname would not have shown europe-west2-b lose power or europe-west2-a fill up. For a dependency whose host we do measure, that directory keeps the host answer next to the vendor's own words while you read the zone list. Reachable means the host answered. Name the zone that lost power, and say whether the other zones in that region still had room.