Disaster Recovery Is Not Backup: Cross-Datacenter RPO<5min, RTO<30min Architecture Decisions and Lessons Learned

Overview

At 2:17 AM, my phone buzzed with an alert: core datacenter A network equipment failure, database primary-standby sync interrupted. The on-call SRE switched to the disaster recovery datacenter B, only to find that after applications connected to the new primary, some order data was missing — the async replication window had lost 47 seconds of data. Recovery took 38 minutes, exceeding the RTO target by 8 minutes.

This was a real failover incident. The postmortem revealed that the system had a DR architecture, data replication configured, and failover scripts written. But when the actual switch happened, things still went wrong. The problem was not whether DR existed, but whether every component could withstand a production-grade failover.

This article breaks down the core architecture decisions for cross-datacenter disaster recovery, covering active-standby, active-active, and two-site three-center models. It covers sync replication performance costs, split-brain protection, async replication window compensation, and the complete failover workflow from decision to execution. Each section includes real incident records and performance data.

In the cross-datacenter DR projects I’ve handled, we achieved RPO<5min and RTO<30min. These numbers aren’t extreme, but they represent the engineering optimum after balancing cost, stability, and business requirements — not a theoretical RPO=0, but a solution that works in production, can be verified, and can be rolled back.

RPO and RTO: The Engineering Truth

They Are Business Metrics, Not Technical Metrics

RPO (Recovery Point Objective) and RTO (Recovery Time Objective) are thrown around so much in tech circles that the meaning gets inverted — they are business metrics first, technical metrics second.

RPO answers “how much data loss is tolerable.” For an e-commerce system, order data RPO must be 0 (cannot lose orders), but user behavior logs can have an RPO of 1 hour (losing some doesn’t affect core transactions). RTO answers “how long can we be down” — core trading systems need RTO <5 minutes, while reporting systems can tolerate RTO of 4 hours.

Key insight: don’t apply a one-size-fits-all approach. I’ve seen too many teams design all systems for RPO=0 and RTO<1min, causing DR costs to multiply several times over when non-critical systems don’t need that level of protection.

Tiered DR targets by business criticality:

The Cost of RPO=0 Is Not Linear

Many assume “smaller RPO is always better,” but the cost jump from 1 minute to 0 is exponential:

• RPO=1min: Async replication, primary writes are non-blocking, no performance loss
• RPO=10s: Semi-sync replication, primary waits for at least one standby ACK, write latency increases 2-5ms
• RPO=0: Full sync replication, primary must wait for all standbys to persist before returning, latency depends on the slowest standby

Measured data: On a ride-hailing platform’s core order database, switching from async to full sync replication dropped write QPS from 12,000 to 8,500 (29% decrease) and increased P99 write latency from 3ms to 12ms. Whether this performance hit is worth it depends on the business’s tolerance for “losing one second of data.”

My recommendation: P0 systems use semi-sync replication (RPO≈0, but may lose milliseconds in extreme cases). P1 and below use async replication. For true RPO=0 scenarios, use sync replication but plan for performance testing and capacity planning.

DR Architecture Evolution: Three Models

Active-Standby

The simplest DR architecture. The primary datacenter runs the business, the standby datacenter is in hot standby mode, and data is synced via async replication. On primary failure, traffic switches to standby.

• Pros: Simple architecture, low cost
• Cons: Standby resources idle, manual failover, RTO typically 15-30 minutes
• Use case: P2/P3 systems, small-to-medium teams

Active-Active (Same-City)

Two datacenters in the same city both serve traffic simultaneously, with data kept consistent via sync replication. Datacenters are typically <50km apart with <5ms network latency.

• Pros: High resource utilization, second-level failover, RPO=0
• Cons: Complex architecture, requires split-brain protection, cannot survive city-level disasters
• Use case: P0/P1 systems

Two-Site Three-Center

Same-city active-active plus remote disaster recovery. Two datacenters in the same city use sync replication, and a remote datacenter uses async replication. This is the standard configuration for financial-grade systems.

• Pros: Balances same-city HA with remote DR
• Cons: High construction cost, complex three-center data consistency management
• Use case: Financial, payment, and other scenarios requiring extreme data safety and business continuity

For clusters under 50 nodes, I recommend same-city active-active over two-site three-center — the operational overhead of remote DR is disproportionately high for small clusters. Only when both business scale and compliance requirements demand it should you go to two-site three-center. (Related: Reliability Design for Multi-Region Active-Active Architecture)

Same-City Active-Active: Sync Replication Costs and Split-Brain Protection

Performance Impact of Sync Replication

The core of same-city active-active is sync replication — after the primary writes, it must wait for standby confirmation before returning success to the client. This guarantees RPO=0 but increases write latency.

Measured comparison (MySQL 8.0, same-city dual datacenter, 2.3ms network latency):

Full sync mode P99 write latency increased 284%. All write operations depending on this database slow down, requiring upstream timeout adjustments.

Incident Record 1: Sync Replication Caused Upstream Timeouts

During a DR drill, switching MySQL from async to semi-sync replication caused the upstream order service write timeout alerts to fire frequently. Root cause: the order service RPC timeout was set to 30ms, but semi-sync replication pushed P99 write latency to 5.8ms. Combined with network and serialization overhead, total time exceeded 30ms.

Fix: Increased write operation RPC timeout from 30ms to 100ms, and routed non-critical writes (like log recording) through an async write queue.

-- MySQL semi-sync replication configuration (primary side)
-- Install semi-sync plugin
INSTALL PLUGIN rpl_semi_sync_source SONAME 'semisync_source.so';

-- Enable semi-sync and set timeout (milliseconds)
SET GLOBAL rpl_semi_sync_source_enabled = 1;
SET GLOBAL rpl_semi_sync_source_timeout = 3000;  -- 3s timeout, degrades to async

-- Check semi-sync status
SHOW STATUS LIKE 'Rpl_semi_sync_source_status';
SHOW STATUS LIKE 'Rpl_semi_sync_source_yes_tx';  -- Successful semi-sync transactions

Note rpl_semi_sync_source_timeout: semi-sync replication automatically degrades to async after timeout. This is a protection mechanism to prevent standby failure from blocking the primary, but it also means RPO is no longer 0 under extreme conditions.

Split-Brain and Quorum Mechanisms

Split-brain is the most dangerous failure mode in same-city active-active. When the network between two datacenters is interrupted, each datacenter thinks the other has failed, elects its own primary, and starts independent writes. When the network recovers, the two datasets conflict and merging is nearly impossible.

The root cause of split-brain is quorum mechanism failure. Common protection approaches:

Approach 1: Third-Party Quorum Node

Deploy a quorum node in a third independent datacenter. Both datacenters maintain heartbeats with the quorum node. On network partition, the quorum node decides which datacenter continues serving via majority voting.

Datacenter A ←→ Quorum Node ←→ Datacenter B
  ↕                              ↕
  └──── Sync Replication ────────┘

Quorum node placement is critical. If the quorum node shares the same physical network as Datacenter A, it fails together with A, providing no arbitration value. I recommend placing the quorum node in a third independent datacenter or cloud VPC, physically isolated from both production datacenters.

Approach 2: Fencing

When split-brain risk is detected, proactively isolate one datacenter — using STONITH (Shoot The Other Node In The Head) to forcibly shut down the standby datacenter’s database instance. More aggressive than quorum voting but more reliable.

#!/bin/bash
# Fencing script: detect split-brain and isolate standby datacenter
# Shut down standby DB instance + cut traffic via API

STANDBY_HOST="10.0.2.10"
STANDBY_API="http://${STANDBY_HOST}:8080/db/failover"

# Heartbeat check: 3 consecutive failures before fencing
HEARTBEAT_FAIL=0
for i in 1 2 3; do
    if ! ping -c 1 -W 2 ${STANDBY_HOST} > /dev/null 2>&1; then
        HEARTBEAT_FAIL=$((HEARTBEAT_FAIL + 1))
    fi
done

if [ ${HEARTBEAT_FAIL} -ge 3 ]; then
    echo "[$(date)] Split-brain detected: 3 consecutive heartbeat failures, isolating standby"
    # 1. Cut standby DNS traffic
    curl -s -X POST http://dns-api.internal/failover?disable=standby
    # 2. Force shutdown standby database
    curl -s -X POST ${STANDBY_API}
    echo "[$(date)] Standby isolated, all traffic routed to primary"
fi

Approach 3: Application-Layer Dual-Write Verification

Instead of relying on underlying quorum, implement dual-write verification at the application layer. Before each write, check database connection status in both datacenters — if only one is reachable, reject the write. This sacrifices availability for consistency, suitable for P0 financial systems.

My recommendation: Use quorum node + fencing in combination. The quorum node makes the decision, fencing executes the isolation. Single approaches all have failure scenarios — the quorum node itself can fail, and fencing can fail to execute. Combined, even if one fails, the other provides backup.

Remote DR: Async Replication Window and Data Compensation

The Nature of Async Replication

Remote DR uses async replication because cross-city network latency is typically 10-30ms, making sync replication unacceptably slow. Async replication means the primary returns success immediately after writing, without waiting for standby confirmation. Data is transferred to the standby via binlog/WAL logs asynchronously.

This means RPO > 0 — when the primary fails, data corresponding to logs not yet received by the standby will be lost. The RPO size depends on log transfer latency, typically seconds to minutes.

Log Catch-up Mechanism

After primary failure, the standby must catch up on logs before reaching a consistent state. This process is called “log catch-up”:

Timeline:
  Primary: T1(write 100 rows) → T2(write 200 rows) → T3(failure)
  Standby: T1(synced)         → T2(syncing)         → T3(150 rows behind)

Failover process:
  1. Freeze primary writes (middleware intercept + DNS offline)
  2. Wait for standby to catch up (monitor GTID diff → zero)
  3. Verify key table row counts and checksums
  4. Switch traffic to standby

Incident Record 2: GTID Gap Caused Catch-up Failure

During a remote DR drill, after primary failure and switch to remote standby, the standby reported a GTID gap error — it was missing certain GTID transactions that the primary had committed. Root cause: a large transaction on the primary (batch update of 500,000 rows) was truncated during transfer, and the standby only received partial logs.

Fix:

1. Enable binlog_transaction_compression=ON on primary to reduce log fragmentation for large transactions
2. Set slave_pending_jobs_size_max=256M to increase standby receive buffer
3. Add GTID completeness verification before failover

-- Primary: check GTID completeness
SHOW MASTER STATUS\G
-- View executed and pending GTID sets

-- Standby: check GTID sync status
SHOW SLAVE STATUS\G
-- Focus on Retrieved_Gtid_Set and Executed_Gtid_Set
-- If they differ, there are unexecuted GTIDs

-- Standby: calculate GTID diff
SELECT @@global.gtid_executed;
-- Compare with primary's gtid_executed, diff = unsynced transactions

-- Large transaction detection
SET GLOBAL binlog_transaction_compression=ON;
SET GLOBAL binlog_transaction_compression_level_zstd=6;

Data Verification: Don’t Trust Replication Status

“Replication successful” in async replication only means log transfer succeeded, not that data is consistent. Logs can fail during replay on the standby — type incompatibility, character set issues, unique key conflicts — these don’t error during transfer but silently corrupt standby data.

Regular verification is mandatory. I recommend hourly automatic checksum comparison of key tables between primary and standby:

#!/usr/bin/env python3
"""Primary-standby data consistency verification script"""
import hashlib
import mysql.connector
import sys

def table_checksum(host, user, password, database, table):
    """Calculate table checksum"""
    conn = mysql.connector.connect(host=host, user=user, password=password, database=database)
    cursor = conn.cursor()
    cursor.execute(f"SELECT COUNT(*), COALESCE(SUM(CRC32(CONCAT_WS('#', {get_columns(cursor, table)}))), 0) FROM {table}")
    count, checksum = cursor.fetchone()
    cursor.close()
    conn.close()
    return count, checksum

def get_columns(cursor, table):
    """Get all column names of a table"""
    cursor.execute(f"SHOW COLUMNS FROM {table}")
    return ', '.join([f'`{row[0]}`' for row in cursor.fetchall()])

if __name__ == '__main__':
    primary = {'host': '10.0.1.10', 'user': 'check_user', 'password': sys.argv[1], 'database': 'orders'}
    standby = {'host': '10.0.2.10', 'user': 'check_user', 'password': sys.argv[1], 'database': 'orders'}
    
    tables = ['t_order', 't_order_detail', 't_payment', 't_user']
    for table in tables:
        p_count, p_checksum = table_checksum(**primary, table=table)
        s_count, s_checksum = table_checksum(**standby, table=table)
        
        if p_checksum != s_checksum:
            print(f"[ALERT] {table}: checksum mismatch! primary={p_checksum}, standby={s_checksum}")
            print(f"  primary count={p_count}, standby count={s_count}")
        else:
            print(f"[OK] {table}: checksum matched, count={p_count}")

This script runs hourly and triggers an alert on checksum mismatch. In production, I’ve caught 3 instances of silent data inconsistency through this script — all caused by character set issues during async replication replay.

Failover: The Complete Workflow From Decision to Execution

Failover Is Not a Single Command

Many people think DR failover is just “switching DNS from datacenter A to B.” In reality, failover is a controlled operation chain involving databases, caches, message queues, scheduled tasks, and more.

Complete failover workflow:

Phase 1: Fault Detection & Decision (0-2 minutes)
  → Alert triggers, SRE confirms fault scope
  → Decision: switch or not? To which datacenter?

Phase 2: Failover Preparation (2-8 minutes)
  → Freeze primary writes (middleware intercept + DNS offline)
  → Wait for standby log catch-up (GTID diff → zero)
  → Verify key table data consistency
  → Notify upstream/downstream dependencies

Phase 3: Traffic Switch (8-15 minutes)
  → DNS weight adjustment, cut 5% traffic first for validation
  → On validation pass, cut all traffic
  → Update service registry, refresh connection pools

Phase 4: Validation & Observation (15-30 minutes)
  → Monitor core metrics: error rate, latency, throughput
  → Manual verification of core business flows
  → Confirm stable, notify business stakeholders

DNS Switch Traps

DNS is the most common traffic switching mechanism, but it has inherent latency — client DNS cache, browser DNS cache, OS DNS cache, each layer causes some traffic to continue hitting the failed datacenter.

Incident Record 3: DNS Cache Caused Incomplete Traffic Switch

During one failover, DNS TTL was set to 60 seconds, but 15 minutes after switching, 5% of traffic still hit the failed datacenter. Root cause: some Java services cached DNS resolution results in JVM (default 30s, but some frameworks cache permanently). Worse, some mobile app HTTP clients also cached IPs.

Fix:

1. Reduce DNS TTL to 10 seconds 24 hours before failover
2. Use smart DNS clients in application layer (like dnsjava) that periodically refresh resolution
3. Health checks at the LB layer — failed datacenter LB rejects connections, forcing client retry to the new datacenter

// Java application: fix JVM DNS cache issue
// Set DNS cache TTL to 10 seconds at application startup
import java.security.Security;

public class DnsCacheConfig {
    static {
        // Set JVM DNS cache TTL (seconds)
        Security.setProperty("networkaddress.cache.ttl", "10");
        Security.setProperty("networkaddress.cache.negative.ttl", "0");
    }
}

Connection Pool Unawareness

Incident Record 4: Database Connection Pool Unaware of Primary-Standby Switch

After failover, the application’s database connection pool was still connected to the old primary IP. When the pool detected broken connections and reconnected, it reconnected to the same old IP — because the pool cached the IP address at initialization and doesn’t re-resolve DNS.

Fix:

1. Use connection pools that support dynamic datasource refresh (like Druid’s DynamicDataSource)
2. Notify applications to refresh connection pools via API during failover
3. Use database proxies (like ProxySQL, ShardingSphere) for transparent switching

# ProxySQL configuration: automatic backend update on failover
# proxysql.cnf
mysql_servers:
  - hostgroup_id: 1        # writer group
    hostname: "10.0.1.10"   # primary
    port: 3306
    max_connections: 200
  - hostgroup_id: 2        # reader group
    hostname: "10.0.2.10"   # standby
    port: 3306
    max_connections: 200

mysql_galera_hostgroups:
  writer_hostgroup: 1
  backup_writer_hostgroup: 2
  active_writer: "10.0.1.10"  # current primary
  # When primary is unavailable, ProxySQL auto-switches to backup_writer

Duplicate Scheduled Task Execution

In same-city active-active, both datacenters deploy scheduled tasks (like reconciliation, settlement). Normally, a distributed lock ensures only one instance executes. But during failover, the Redis distributed lock also switches, and after the lock is released, both datacenters’ scheduled tasks start simultaneously, causing duplicate execution.

Fix:

1. Use database row locks instead of Redis distributed locks, binding DB failover with task scheduling
2. Add “stop scheduled tasks, then switch DB, finally start tasks” to the failover procedure
3. Design tasks to be idempotent — duplicate execution produces no errors

# Idempotent scheduled task: unique task ID + database row lock
import hashlib
from datetime import datetime

def run_settlement_task():
    """Daily settlement task - idempotent design"""
    task_id = f"settlement_{datetime.now().strftime('%Y%m%d')}"
    task_hash = hashlib.md5(task_id.encode()).hexdigest()
    
    # Try to acquire task lock (database row lock)
    cursor.execute(
        "INSERT INTO task_log (task_id, status, created_at) VALUES (%s, 'running', NOW()) "
        "ON DUPLICATE KEY UPDATE status=IF(status='done', 'running', status)",
        (task_hash,)
    )
    
    if cursor.rowcount == 0:
        print(f"Task {task_id} already running or completed, skipping")
        return
    
    try:
        # Execute settlement logic
        do_settlement()
        cursor.execute("UPDATE task_log SET status='done', finished_at=NOW() WHERE task_id=%s", (task_hash,))
    except Exception as e:
        cursor.execute("UPDATE task_log SET status='failed', error=%s WHERE task_id=%s", (str(e), task_hash))
        raise

Message Queue Offset Loss

After failover, Kafka consumer groups may connect to the new datacenter’s Kafka cluster, but offset information is stored in the old cluster’s ZooKeeper/KRaft. The new cluster doesn’t know where consumers last read — either consume from the beginning (duplicates) or from the latest position (data loss).

Fix:

1. Store consumer offsets in the database rather than ZooKeeper, bound to the data sync pipeline
2. Implement idempotent consumption — duplicate messages produce no errors
3. Record current offsets before failover, resume from recorded positions after switching

# Kafka consumer offset persistence to database (recoverable after failover)
import json
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'order_events',
    bootstrap_servers='kafka-primary:9092',
    group_id='order-processor',
    enable_auto_commit=False  # Disable auto-commit, manage offsets manually
)

def save_offset_to_db(topic, partition, offset):
    """Persist offset to database, recoverable after failover"""
    cursor.execute(
        "INSERT INTO kafka_offset (topic, partition_id, consumer_group, offset_val, updated_at) "
        "VALUES (%s, %s, %s, %s, NOW()) "
        "ON DUPLICATE KEY UPDATE offset_val=VALUES(offset_val), updated_at=NOW()",
        (topic, partition, 'order-processor', offset)
    )

# Consume and manually save offset
for message in consumer:
    process_message(message.value)
    save_offset_to_db(message.topic, message.partition, message.offset)
    consumer.commit()  # Also commit to Kafka, dual insurance

Failover Automation: From Manual to Self-Healing

The failover process described above involves many steps and components. Fully manual operation is too slow for RTO<30min targets. Failover must be automated.

But full automation has risks — if fault detection misfires, automatic failover creates unnecessary downtime. My approach is “semi-automatic”: the system automatically detects faults and executes failover preparation (freeze writes, data verification), but the final switch decision requires human confirmation.

# Failover orchestration config (Ansible Playbook excerpt)
---
- name: DR Failover Orchestration
  hosts: localhost
  vars:
    primary_dc: "dc-a"
    standby_dc: "dc-b"
    switch_reason: "{{ switch_reason }}"

  tasks:
    # Phase 1: Fault confirmation
    - name: Check primary datacenter health
      uri:
        url: "http://{{ primary_dc }}-health:8080/health"
        method: GET
        return_content: yes
      register: health_check
      failed_when: health_check.status != 200

    - name: Confirm failover (requires human confirmation)
      pause:
        prompt: "Primary health check failed, confirm switch to {{ standby_dc }}? (yes/no)"

    # Phase 2: Failover preparation
    - name: Freeze primary writes
      uri:
        url: "http://{{ primary_dc }}-proxy:8080/maintenance"
        method: POST
        body: '{"action": "freeze_writes"}'

    - name: Wait for standby log catch-up
      shell: "mysql -h {{ standby_dc }}-db -u monitor -e 'SHOW SLAVE STATUS\\G'"
      register: slave_status
      retries: 30
      delay: 10
      until: "'Seconds_Behind_Master: 0' in slave_status.stdout"

    - name: Verify key table data consistency
      shell: "python3 /opt/dr/checksum_verify.py --primary {{ primary_dc }}-db --standby {{ standby_dc }}-db"
      register: checksum_result
      failed_when: "'MISMATCH' in checksum_result.stdout"

    # Phase 3: Traffic switch
    - name: Switch DNS (5% traffic for validation first)
      shell: "python3 /opt/dr/dns_switch.py --to {{ standby_dc }} --weight 5"

    - name: Validate 5% traffic is healthy
      uri:
        url: "http://{{ standby_dc }}-monitor:9090/api/v1/query?query=up"
      register: monitor_check
      retries: 6
      delay: 10
      until: monitor_check.status == 200

    - name: Full DNS switch
      shell: "python3 /opt/dr/dns_switch.py --to {{ standby_dc }} --weight 100"

    # Phase 4: Notification
    - name: Send failover completion notification
      uri:
        url: "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key={{ webhook_key }}"
        method: POST
        body_format: json
        body:
          msgtype: text
          text:
            content: "DR failover completed: {{ primary_dc }} → {{ standby_dc }}, reason: {{ switch_reason }}"

Cost Model: DR Is Not Free

There’s an unavoidable truth about DR: each additional 9 of availability costs 3-5x more. Going from 99.9% to 99.99% isn’t adding one more server — it’s a complete architecture redesign.

Cost Breakdown

Using a mid-size e-commerce system (200 microservices, 50 database instances, daily QPS 8,000) as an example:

Same-city active-active doubles cost over active-standby but increases resource utilization from 50% to 100%. Two-site three-center doubles cost again, but the remote datacenter only serves read traffic or hot standby, with 10%-20% resource utilization.

Key decision point: Remote DR resource utilization is the bottleneck of cost efficiency. If the remote datacenter only does DR without serving production, it’s paying 2x for 1.1x compute.

Cost Optimization Strategies

1. Run read-only analytics on remote DR: Don’t let the remote datacenter sit idle. Use it for BI reporting and data analysis. Keeps data sync active and produces business value. One e-commerce platform ran T+1 reports on the remote DR, increasing utilization from 15% to 60%.

2. On-demand cloud DR: Don’t build a self-owned remote datacenter. Use cloud provider DRaaS (Disaster Recovery as a Service). Pay only for storage normally, spin up compute on demand during disasters. Suitable for small-to-medium teams.

3. Tiered DR: P0 systems use two-site three-center, P1 uses same-city active-active, P2/P3 use active-standby or backups. Not all systems need three-center — invest by business criticality.

Measured data from a ride-hailing project: switching from “full two-site three-center” to “tiered DR” reduced annual DR cost from $780K to $420K, a 46% reduction. Core system DR capability remained unchanged — only non-critical systems were downgraded to active-standby.

DR Drills: Why “Tested” Means “Real”

Drills Are Not Process Walkthroughs

The biggest misconception about DR drills is “following the steps” — executing predetermined steps to confirm you can switch over. Real drills should simulate actual fault scenarios, including unexpected failures and operational errors.

Google SRE Book states a principle: “Hope is not a strategy.” No matter how beautiful the DR plan looks, without testing it’s just paper.

Drill Tiers

L4 chaos engineering injection is the most valuable — injecting faults without notifying anyone, observing whether the system can automatically detect and switch. (Related: System Resilience Engineering: From Reactive Firefighting to Proactive Defense)

Common Drill Issues

In the 20+ DR drills I’ve organized, the most common issues ranked:

1. Connection pool not refreshed (80% occurrence): Described above
2. DNS cache (60% occurrence): Described above
3. Duplicate scheduled tasks (40% occurrence): Described above
4. Cache data inconsistency (35% occurrence): After switch, Redis cache is empty in the new datacenter, massive requests penetrate to the database
5. Message queue offset loss (30% occurrence): After switch, consumer group offsets are wrong, causing duplicate or missing consumption

Issue #4 is worth detailing. In same-city active-active, both datacenters have Redis clusters. Normally, cache updates are synced via message queue. But after failover, the new datacenter’s Redis cache may be empty — because the message queue sync has latency, and the last few seconds of cache updates haven’t synced yet.

Fix: Pre-warm the new datacenter’s cache before failover. Load hot data from the database to Redis, then switch traffic after pre-warming completes.

#!/bin/bash
# Cache pre-warming script: load hot data from DB to new datacenter Redis
# Execute before failover

REDIS_NEW="10.0.2.20:6379"
MYSQL_HOST="10.0.2.10"

echo "[$(date)] Starting cache pre-warming..."

# Pre-warm user info cache (Top 10000 active users)
mysql -h ${MYSQL_HOST} -u cache_loader -p${DB_PASS} -e "
  SELECT CONCAT('user:', user_id) AS key, 
         JSON_OBJECT('id', user_id, 'name', nickname, 'level', level) AS value
  FROM t_user 
  WHERE last_login_at > DATE_SUB(NOW(), INTERVAL 7 DAY)
  ORDER BY last_login_at DESC 
  LIMIT 10000
" --batch --raw | while IFS=$'\t' read -r key value; do
    redis-cli -h ${REDIS_NEW} SET "${key}" "${value}" EX 3600
done

echo "[$(date)] User cache pre-warming complete"

# Pre-warm product info cache
mysql -h ${MYSQL_HOST} -u cache_loader -p${DB_PASS} -e "
  SELECT CONCAT('product:', product_id), product_name, price, stock
  FROM t_product WHERE status=1 LIMIT 5000
" --batch --raw | while IFS=$'\t' read -r key name price stock; do
    redis-cli -h ${REDIS_NEW} HSET "${key}" name "${name}" price "${price}" stock "${stock}"
    redis-cli -h ${REDIS_NEW} EXPIRE "${key}" 1800
done

echo "[$(date)] Product cache pre-warming complete"
echo "[$(date)] All cache pre-warming complete"

Architecture Tradeoffs: Which Model for Which Scenario

Decision Framework

DR architecture selection is not about picking the “best” option — it’s about matching business requirements and cost constraints. My decision framework:

Business Requirements Analysis
  ├─ Data loss tolerance (RPO)
  │   ├─ RPO=0 → Sync replication (same-city active-active)
  │   ├─ RPO<1min → Semi-sync replication
  │   └─ RPO<30min → Async replication
  ├─ Downtime tolerance (RTO)
  │   ├─ RTO<1min → Auto-switch (active-active)
  │   ├─ RTO<15min → Semi-auto switch
  │   └─ RTO<2h → Manual switch
  ├─ Compliance requirements
  │   ├─ Financial/Medical → Two-site three-center
  │   └─ General business → Same-city active-active or active-standby
  └─ Cost budget
      ├─ Abundant → Two-site three-center + full sync
      ├─ Moderate → Same-city active-active + semi-sync
      └─ Tight → Active-standby + async

Comparison

My Practical Recommendations

Based on my experience across multiple projects:

1. Clusters under 50 nodes: Active-standby + async replication. Remote DR operational overhead is disproportionately high for small clusters (managing three centers can consume 30% of the ops team’s capacity). Not worth it.

2. Clusters of 50-500 nodes: Same-city active-active + semi-sync replication. At this scale, same-city active-active offers the best balance of resource utilization and failover speed. Remote DR is optional — if budget allows, add an async remote node as backup.

3. Clusters of 500+ nodes: Two-site three-center. At this scale, both business requirements and compliance demand a full two-site three-center architecture. Focus on same-city sync replication performance and remote DR failover automation.

4. Financial-grade systems: Two-site three-center + full sync replication + fencing + quarterly chaos engineering drills. This is a compliance requirement, not a choice.

On cloud DR: If your system runs on the cloud, prioritize the cloud provider’s managed DR services (like Alibaba Cloud DTS, AWS RDS Multi-AZ). These services encapsulate the complexity of sync replication, fault detection, and automatic failover. But managed doesn’t mean hands-off — you still need to understand the underlying mechanisms and configure failover strategy, monitoring, and rollback. (Related: Microservice Rate Limiting, Circuit Breaking, and Degradation)

Monitoring and Alerting: DR System Observability

The DR system itself needs monitoring. Many people only monitor the business system and forget to monitor the DR system’s health — data sync latency, replication link status, standby data consistency. When these metrics go abnormal, the DR system may have already failed.

Core Monitoring Metrics

PromQL Monitoring Examples

# MySQL replication lag monitoring
mysql_slave_status_seconds_behind_master{job="mysql-exporter"} > 30

# GTID diff monitoring (custom exporter)
mysql_gtid_diff{job="mysql-exporter"} > 0

# Semi-sync replication status
mysql_global_status_rpl_semi_sync_source_yes_tx{job="mysql-exporter"} == 0
# Semi-sync success count is 0, may have degraded to async

# Quorum node heartbeat
probe_success{job="blackbox", instance="quorum-node:8080"} == 0

Alert Rule Configuration

# Prometheus alert rules: DR system health monitoring
groups:
  - name: disaster_recovery
    rules:
      - alert: MySQLReplicationLag
        expr: mysql_slave_status_seconds_behind_master > 30
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "MySQL replication lag high: {{ $value }}s"
          description: "{{ $labels.instance }} replication lag exceeds 30s, RPO may exceed target"

      - alert: MySQLReplicationBroken
        expr: mysql_slave_status_slave_io_running == 0 or mysql_slave_status_slave_sql_running == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MySQL replication link broken"
          description: "{{ $labels.instance }} IO/SQL thread stopped, DR system may be compromised"

      - alert: SemiSyncDegraded
        expr: mysql_global_status_rpl_semi_sync_source_yes_tx == 0 and mysql_global_status_rpl_semi_sync_source_no_tx > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Semi-sync replication degraded to async"
          description: "No semi-sync transactions in last 5min, may have degraded, RPO no longer 0"

      - alert: DataChecksumMismatch
        expr: dr_data_checksum_mismatch > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Primary-standby data checksum mismatch"
          description: "{{ $labels.table }} table data inconsistent, investigate immediately"

Failback: More Dangerous Than Failover

After failing over to the DR datacenter, business resumes. But when the failed datacenter is repaired, you need to fail back. Failback is more dangerous than failover — because the DR datacenter is now the primary, and the repaired datacenter is the standby. Failback is essentially another failover, but data flows in reverse.

Failback Risks

1. Data loss risk: New data generated while the DR datacenter served as primary must be synced back to the original primary. If sync is incomplete, data will be lost on failback.
2. Secondary failure risk: During failback, the system is in a fragile transition state. Another failure at this point could make both datacenters unavailable.
3. Business impact: Failback requires a brief write freeze. Business stakeholders must be notified in advance.

Failback Process

Phase 1: Primary datacenter recovery (verify repair complete)
  → Confirm original primary infrastructure is healthy
  → Establish reverse replication (DR → original primary)
  → Wait for data sync to catch up

Phase 2: Failback preparation
  → Execute during low-traffic window (typically 2-4 AM)
  → Notify business stakeholders of upcoming failback
  → Freeze DR datacenter writes

Phase 3: Execute failback
  → Confirm original primary data has caught up
  → Verify key table data consistency
  → Switch DNS and traffic to original primary
  → Verify business is normal

Phase 4: Restore DR configuration
  → Re-establish forward replication (original primary → DR)
  → Update monitoring and alerting configuration
  → Notify business stakeholders of failback completion

My principle: Failback must have business stakeholder sign-off, execute during low-traffic periods, and have a complete data backup before starting. I’d rather spend an extra hour verifying than gamble on data consistency.

Summary

Cross-datacenter DR architecture design is not about choosing the most expensive option — it’s about matching business requirements, cost constraints, and team capabilities.

Key practical takeaways:

RPO and RTO are business metrics, not technical metrics. Design tiered DR targets by business criticality. Don’t apply one-size-fits-all. P0 systems use sync replication, P2/P3 systems are fine with async or scheduled backups.

Sync replication has performance costs. Full sync replication increases write latency 200%-300%. Performance testing and capacity planning are mandatory. Semi-sync replication is the optimal choice for most scenarios — RPO≈0 with controllable performance impact.

Split-brain protection needs a combined approach. Quorum node + fencing. Single approaches all have failure scenarios. The quorum node must be in a third independent location, not sharing physical networks with production datacenters.

Async replication requires regular verification. Replication status “normal” doesn’t mean data is consistent. Hourly checksum verification of key tables catches silent inconsistency.

Failover is a workflow, not a single command. It involves databases, caches, message queues, scheduled tasks, connection pools, DNS — every component must be verified. Any one not properly switched will cause issues.

DR drills are mandatory and must be realistic. L4 chaos engineering injection is the most valuable — unannounced fault injection, observing whether the system can automatically detect and switch. If it hasn’t been tested, it doesn’t exist.

Failback is more dangerous than failover. Must execute during low-traffic periods, must have business confirmation, must have data backup. Better safe than sorry.

In the projects I’ve handled, we achieved RPO<5min and RTO<30min, verified through 20+ real drills. Each drill found problems and optimized processes. DR is not a one-time project — it’s a continuous optimization process.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

1. Google SRE Book - Disaster Recovery — Google SRE Team, disaster recovery and business continuity engineering methodology
2. Oracle Disaster Recovery Introduction — Oracle official documentation, RPO/RTO definitions and DR architecture design reference
3. Azure Well-Architected Framework - Architecture strategies for disaster recovery — Microsoft Azure, multi-region DR architecture strategies
4. Two-Site Three-Center DR Deep Dive — Tencent Cloud Developer Community, sync/async replication mechanisms and database solution comparison
5. Same-City Active-Active and Remote Multi-Active Architecture Design — Baidu Tianchi Community, dual-DC coordination and RPC localization optimization
6. etcd Split-Brain Incident Postmortem — K8s production etcd network partition causing Raft majority failure real postmortem
7. Split-Brain in DR Architecture Explained — Split-brain causes, detection mechanisms, and protection approaches