SRE Practice

Overview Fourteen days before Double 11 one year, an e-commerce platform initiated a full change freeze—all non-emergency changes were frozen, CI/CD pipeline gates were closed, and only a security patch channel remained. The ops team breathed a sigh of relief. This time, it would be stable. At 2 AM on the day of the big sale, the payment pipeline P0 alert blew up. The root cause wasn’t new code. It was a configuration change deployed 3 days before the freeze—a rate-limiting rule’s whitelist had the wrong IP range....

Overview Netflix says chaos engineering reduced major production incidents by 70%. Many teams have seen this number, but those who have actually run fault injection in production are rare. The reason is simple: fear. Inject a network partition and Pods freeze—what then? Simulate disk full and the database crashes—then what? Manually injecting faults at 3 AM and scrambling to roll back—that’s not a drill, that’s manufacturing an incident. During a K8s migration for a new-energy logistics platform with 120+ microservices, I needed to validate disaster recovery capabilities before the full cutover....

Overview At 2:17 AM, my phone buzzed with an alert: core datacenter A network equipment failure, database primary-standby sync interrupted. The on-call SRE switched to the disaster recovery datacenter B, only to find that after applications connected to the new primary, some order data was missing — the async replication window had lost 47 seconds of data. Recovery took 38 minutes, exceeding the RTO target by 8 minutes. This was a real failover incident....

Overview 3 AM, phone vibrating. The alert channel exploded with 200 messages, and user complaints are already flooding customer service. You scramble to your laptop, log into the bastion host, and discover a core API endpoint’s P99 latency spiked to 8 seconds. For the next 30 minutes, you dig through logs, check Grafana dashboards, and ping upstream and downstream teams—only to find that a config center pushed an incorrect parameter....