CISSP Domain 3.9: Site and Facility Controls and the Generators Nobody Told to Start (378)

A cloud provider's data center in London lost its utility power. The batteries picked up the load exactly the way they were supposed to, ran for about five minutes, and then ran out.

The generators should have taken over in that window. They never got the signal.

Not because they were broken. Not because somebody forgot to fuel them. They never started because the thing whose job it was to notice the power had gone out was wired to the wrong side of a switch.

Nobody attacked anybody here. There are no bad guys in this one — no ransomware, no intrusion, nothing for incident response to chase. It was a cable fault outside the building and a logic error inside it.

And that is exactly why it is a Domain 3.9 episode.

Start with the design, because the design was good

The facility had two completely independent utility feeds coming in from the power company. They call them A and B. Two separate paths, so losing one does not take the building down. Google's own words for that arrangement are a 2N utility power architecture.

Now listen to where that label stops.

Behind both of those feeds sits one central bank of high-voltage generators, run by a dual-redundant controller — a PLC. Two of the thing coming in. One of the thing that was supposed to save them.

And between the utility going away and the generators coming up, a battery system with about five minutes of runtime. That is not a lot, and it is not supposed to be. It is a bridge, and a bridge only has to be as long as the gap.

So when somebody tells you a site is 2N, the only honest follow-up question is: 2N at which layer? A 2N front end feeding one shared plant is not a 2N facility. Count the layer, not the label.

The good decision that created the exposure

Five days before any of this, they ran a quarterly load test on the generators. A real test, under load.

And that test found a problem.

So they did the right thing. They scheduled the repair. But to do the repair, they had to physically open the breaker on feed B and lock it out — which they did properly: keyed interlock, locked out, tagged out.

That is the moment the building stopped being 2N on the utility side. For several days, that site was running on one utility feed. Google's own words for it are a temporary single-feed, 1N, bridged maintenance posture.

Then the cable broke. An underground cable out on the utility network, outside the building entirely, developed a fault.

And it was on feed A. The only feed that was live.

Why the generators sat there

The PLC decides when to start the generators, and it decides it by sensing voltage. If the voltage goes away, start the generators. Simple enough.

But those voltage sensors were wired upstream of the feed breakers. On the line side. And feed B — the one that was open for repairs — was still energized upstream, all the way up to that open breaker. The power company was still feeding it. It just was not connected to the building anymore.

So the PLC looks at its sensor, sees voltage on feed B, concludes that a healthy utility source is available, and never issues a start command.

Google's stated root cause is a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. It checked whether there was voltage. It never checked whether that voltage was actually connected to anything.

Five minutes later the batteries were done, and the data halls went dark.

Two of them, one mistake

Here is the detail that is easy to read past.

That PLC was dual-redundant. There were two of them. Both were powered, both were healthy, and both reached the same conclusion — because they shared a configuration.

Redundancy protects against independent failure. These two failures were not independent. They were one failure, resident in both units, because both units were built from the same unreviewed source.

Duplicating the box does not duplicate the judgment. That is common-mode failure, and it is the reason a redundancy count is not a resilience measurement. The next time somebody hands you a redundancy number, the useful question is not how many. It is what the halves have in common.

Keep two numbers apart

This is the detail most retellings get wrong, so get it right.

The data halls were without power for roughly ninety minutes. The figure you will see quoted — six hours and eight minutes — is how long it took to get the services back on their feet. Both numbers are in Google's report. They measure different things, and the one attached to the wrong noun is the version that will get corrected in your comments.

According to The Register, the companies caught in that window included Lloyds, HSBC, Royal Mail and Revolut. Google published no customer list and no customer count.

What a manager takes from this

Two things, and neither one of them is "test your generators."

The first one. They did test. The quarterly load test is what found the fault in the first place. Testing worked exactly the way it is supposed to.

What nobody did was look at the consequence. Fixing that fault meant opening a breaker, and opening that breaker meant the building ran on one feed for days. As far as anybody treated it, that was a maintenance ticket. It was not a changed risk posture.

So ask yourself — when your people take half of a redundant pair out of service, does anybody decide whether that is acceptable? Or does it just get scheduled?

Name the boundaries while you are at it, because the exam does. Running the breaker work as a change is Domain 7.9, change management. Deciding whether the reduced redundancy is acceptable is Domain 1.9, risk management. The outage spilling into neighboring zones is Domain 7.10, system resilience and fault tolerance. What belongs to 3.9 is the facility's own power design, and whether the automation reading it was ever validated against a degraded state.

The second one, and this is the one to carry into the exam room:

An automatic control is only as good as the thing it can sense.

The generator was perfect. The battery was perfect. The logic was asking a question — is there voltage? — at a place where it could not tell the difference between "the utility is healthy" and "the utility is healthy and no longer connected to us."

The three states of a redundant system

Only one of them is on the drawing.

Designed. Every path available. This is the state the drawing shows, the state the certificate describes, and the state everybody pictures when they say the building is redundant.

Degraded. One or more paths out of service — maintenance, repair, a failed component not yet replaced. Still running. Often for days. Rarely written down as a change in risk.

Failed. No path available. This is the one everybody plans for, and it is the one that almost never arrives on its own. It arrives while you are already degraded.

Go ask your facilities team how long your site spent degraded last year. If nobody can answer, nobody is managing the state the failure will actually arrive in.

The one line to carry into the exam room

Redundancy is not a property of a design. It is a property of a system at a moment — and that moment includes whatever your facilities team happens to have open for repairs today.

Domain 3.9 does not ask you to list fire suppression types. It asks which controls this facility, in this configuration, right now, actually requires — and whether anybody owns the answer.

Source: Google Cloud's own incident report for europe-west2, Incident Xs2XTHzAVC9YnafnPyzj — <https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj>. The customer names come from The Register's coverage, 5 November 2026, not from Google.

Shon Gerber, CISSP — former Commander, 177th IAS. Former CISO, fractional CISO, founder of CISSP Cyber Training.