Incident Management and On-Call Mechanism Design

Overview There’s a saying in SRE: “Systems will fail — the question is whether you’ll be woken up by them or actively managing them.” Incident management is not “dealing with things after they break” — it’s a complete engineering system spanning prevention, detection, response, and learning. This article systematically covers how to build a practical on-call system across five dimensions: incident grading, on-call rotation, incident response processes, postmortem culture, and alert governance....

August 23, 2024 · 13 mins · 2559 words · XuBaojin

Capacity Planning and Elastic Scaling in Practice

Overview Capacity planning is a core SRE responsibility. The Google SRE Book treats capacity planning as “proactive work,” emphasizing data-driven forecasting over experience-based guessing. A team without capacity planning either gets crushed by traffic during peaks or wastes significant resource costs during valleys. This article systematically covers capacity planning engineering practices across four dimensions: metric collection, data modeling, Kubernetes elastic scaling configuration, and pitfall avoidance. For a systematic methodology on capacity planning, see Google SRE Book - Capacity Planning and its discussion of capacity planning and cascading failures....

June 27, 2024 · 10 mins · 1970 words · XuBaojin