Disaster Recovery

Overview At 2:17 AM, my phone buzzed with an alert: core datacenter A network equipment failure, database primary-standby sync interrupted. The on-call SRE switched to the disaster recovery datacenter B, only to find that after applications connected to the new primary, some order data was missing — the async replication window had lost 47 seconds of data. Recovery took 38 minutes, exceeding the RTO target by 8 minutes. This was a real failover incident....

Overview At 3 AM, your phone vibrates. An alert message: a major cloud provider’s East China region storage service is experiencing widespread unavailability. Your core business is fully deployed in this region, with the primary database there too. The outage has lasted 12 minutes, customers are complaining, and your boss is asking “how long to recover” in the group chat. You know the truth: the cross-region disaster recovery plan was reviewed six months ago but got cut due to “high costs....

Overview When your business grows from “serving one city” to “serving the entire country” or even “serving globally,” single-datacenter architecture hits two hard constraints: latency from distance and single point of failure risk. Multi-region active-active architecture is the engineering solution to both problems. But active-active architecture is one of the most complex topics in SRE — it’s not simply “deploy the service in two datacenters.” It involves a series of deep engineering challenges: data consistency, traffic routing, failover, and operational complexity....

Overview When your business scale grows beyond what a single cluster can handle, multi-cluster becomes an inevitable choice. Reasons include: single cluster node limits (5000 nodes), multi-region deployment, hybrid cloud strategy, fault isolation, and compliance requirements. But multi-cluster management complexity grows exponentially—how to deploy applications across clusters, discover services cross-cluster, synchronize configurations, and handle failover. This article systematically covers multi-cluster architecture patterns, mainstream management tool comparisons, and practical solutions for cross-cluster service discovery, CI/CD, and disaster recovery failover....

Overview “Hope for the best, prepare for the worst.” In Kubernetes production environments, disaster recovery is the last line of defense. Whether it’s etcd corruption, accidental deletion, node failure, or even entire cluster loss, having a solid backup and recovery strategy is critical. This article systematically covers K8s disaster recovery—from etcd backup and restore, Velero full-cluster backup, PV data backup, to cross-cluster recovery and disaster recovery architecture design. Based on Kubernetes v1....