Designing a Warm Standby for Kubernetes: Lessons from a Failed Restore
If our primary region went dark tomorrow, how long would it take for our critical services on our compute platform to come back? Until recently, we didn’t have a clear answer.
Our production workloads ran on Google Kubernetes Engine (GKE) in a single region: Mumbai. Networking, identity, secrets management, private DNS and the service mesh were all built around that environment. We had a disaster recovery plan, but it existed mostly on paper. We had never actually run it end to end.
The problem became clearer when we tried a full restore. The Kubernetes objects came back, but almost nothing started. The manifests still assumed the production environment, from registry settings to DNS and scheduling. Restoring the objects wasn't enough to restore a working platform.
We needed a Disaster Recovery (DR) setup we could practise, measure and improve before an actual outage forced us to rely on it. We chose a warm standby approach: keep a smaller environment running in another region, then scale it up when we need to recover.
The design looked straightforward at first. It wasn't.
Choosing a Recovery Strategy on Purpose
Once the DR setup had to be something we could actually run, the next question was how much recovery infrastructure was worth keeping around.
The options ranged from backup and restore, where recovery could take days, to pilot light, warm standby and active-active. Each step up meant keeping more infrastructure running in exchange for recovering faster.
Warm standby offered the middle ground.
A second full production environment would be expensive to keep running. Rebuilding everything from backups during an outage, on the other hand, would leave too much work to do when recovery time mattered most. The DR platform could instead run at minimum size and scale up for a drill or actual failover.
The target was platform recovery within hours. For secrets, the RPO was roughly a day, with daily synchronisation and another sync before each drill.
The choice came down to a practical trade-off: recovery time versus the cost of keeping the standby environment running.
We couldn’t use the Region We Wanted
The first preference was straightforward: keep DR compute in India. But the available infrastructure made that harder than it sounded. Some machine types used by production, including ARM machines, weren’t available in the region being considered. At the same time, secrets had to remain in India.
The design had to satisfy both constraints, so compute and secret replicas were split:
This wasn't a particularly neat architecture, but it was practical. Keeping everything in India would have meant either using weaker machines than production or moving secrets outside India.
The network was treated similarly. Instead of rebuilding the entire shared network in DR, we created a separate VPC in its own project. It was faster to build and isolate. A temporary VPN and DNS link back to the primary provided access to a few private services that still lived there.
Designing DR in Layers
With the region and network decisions settled, the DR setup could be broken into four layers. This kept the architecture understandable without trying to recreate production piece by piece.
Foundation is the base: a dedicated DR project, its network and outbound internet access, plus a GKE cluster with machines that match production.
Platform sits above it. This includes secret syncing, policies, the service mesh, gateways and backup/restore tooling. These are the pieces applications depend on before they can run.
Workloads are restored from backup. As they enter DR, inherited settings are adjusted where needed.
Finally, traffic is handled separately. Users are moved to DR by changing DNS, with the change going through a reviewed pull request rather than being made directly in the console.
The layers are also ordered: each one has to be ready before the next, or the cluster looks ready and fails its first restore.
The setup stays warm rather than fully active. Production names and identities already exist, secrets are synced on a schedule, and the platform runs at minimum size. When a drill or real failover starts, it scales up.
The First Restore Taught Us What Backups Don't Restore
The first full restore worked. Then almost nothing started.
The Kubernetes objects came back, but the applications didn’t. The manifests were written for the production cluster, so they carried assumptions about the environment they ran in.
The backup itself wasn't broken. It had restored exactly what it was supposed to restore. The problem was everything around those objects. They depended on details of the cluster where they had been created.
That led to a more useful way of thinking about the problem: backups restore objects, not assumptions.
One option was to ask every application team to make its manifests DR-aware. That would create a second problem. Those changes would need to stay correct as infrastructure evolved, and a large restore could involve hundreds of namespaces.
Instead, the environment-specific changes were moved into the platform. A central set of admission policies rewrites the relevant settings as objects are restored. Application repositories stay unchanged, and the restore doesn't require every service owner to be involved.
What We Deliberately Didn't Solve Yet
This phase was deliberately limited to compute. It covered the Kubernetes clusters, platform components and application recovery, but not the entire stack. Datastores such as Cloud SQL, Postgres and Memorystore still run only in primary. Their DR is the next phase.
AWS access is a separate dependency. Applications in DR can reach S3, SQS and SNS through the existing production network path, using the same short-lived credentials as production. That works, but the path itself is not yet independent.
The temporary VPN and DNS link is another limitation. It connects DR to selected private services in primary, but would not survive a total loss of primary.
There was also no reason to introduce static AWS keys just to make a drill pass. That would solve a temporary problem by creating a permanent security risk.
So this wasn't “DR solved.” It was a working compute recovery layer, with the remaining dependencies clearly identified.
Restoring Kubernetes Objects Wasn't Enough
Our first full restore exposed the gap in our disaster recovery plan. The Kubernetes objects came back, but the applications still depended on settings and services tied to the primary environment. A successful restore didn't mean we had a working platform.
That led to the key design decision: instead of asking every application team to make its manifests DR-aware, we would use central admission policies to adjust environment-specific settings as objects were restored.
But the design alone wasn't enough. We still had to build the recovery process in the right order, handle secrets without exposing their values, and make sure the platform could scale up safely when needed.
In Part 2, we cover how we built that process and tested it through a repeatable, 12-step disaster recovery drill.
Continue to Part 2: Disaster Recovery You Can Rehearse: Building and Rehearsing the System
More fromÂ
Engineering
 category
