Disaster Recovery Is Not Backup: Cross-Datacenter RPO<5min, RTO<30min Architecture Decisions and Lessons Learned
Overview At 2:17 AM, my phone buzzed with an alert: core datacenter A network equipment failure, database primary-standby sync interrupted. The on-call SRE switched to the disaster recovery datacenter B, only to find that after applications connected to the new primary, some order data was missing — the async replication window had lost 47 seconds of data. Recovery took 38 minutes, exceeding the RTO target by 8 minutes. This was a real failover incident....