SRE Incident Preparedness and Drills: From Paper Plans to Muscle Memory

Overview It’s 2 AM. Your phone screams. The monitoring dashboard is a sea of red — core transaction P99 latency just hit 8 seconds, upstream services are timing out and circuit-breaking, and customer support chat is flooding with screenshots. You’re VPN-ing in while your brain runs at full speed: have we seen this scenario in a drill? Is it covered in the runbook? Do I remember the failover steps? If you’re still searching the wiki for documentation at this moment, it means one thing: your runbook was written but never practiced....

July 16, 2026 · 20 mins · 4112 words · Xu Baojin

Incident Response Framework: SEV Classification and Escalation Flow

Overview Incidents are inevitable, but the quality of your incident response determines the impact scope and duration. A mature incident response framework can bring order to chaos — ensuring the right people do the right things, information flows where it needs to go, and recovery happens as fast as possible. The reality many teams face during incidents: alert bombardment, chat channel flooding, unclear ownership, duplicate investigations, inconsistent external messaging, and inability to explain what happened after recovery....

May 23, 2024 · 20 mins · 4176 words · XuBaojin