SRE

Overview The traditional approach to reliability is “try not to have failures” — add monitoring, add alerts, add redundancy. But this passive defense has a fundamental flaw: you don’t know how the system actually behaves during a failure until one actually occurs. Chaos engineering takes the opposite approach: proactively and controllably inject failures to discover system weaknesses before they become incidents. It’s not about causing destruction — it’s a scientific experimental method: form a hypothesis (“the system should be able to withstand a node failure”), design an experiment (kill a node), verify the hypothesis (is the service still working?...

Overview In modern microservice architectures, a seemingly simple user request may traverse dozens of service nodes. When an incident occurs, the first question an SRE engineer faces is often not “how to fix it” but “what is the scope of impact.” Without a fast answer to this question, incident recovery gets bogged down in endless investigation. Service Dependency Maps and Failure Domain Analysis are the engineering methodologies that address this problem....

Overview Every incident is a free learning opportunity — provided you have a mechanism to extract lessons from it. A Postmortem is not about writing a confession or finding a scapegoat. It is a structured engineering methodology for converting incident experience into systemic improvements. One of the core tenets of Google SRE is: “Blameless postmortem.” The focus of a review is always on “why did the system fail,” not “who messed up....

Overview The core philosophy of SRE is: manage reliability through engineering methods. The most important tools are SLI, SLO, and Error Budgets. SLI: Service Level Indicator SLI is a quantitative metric for system reliability. Common SLIs include: Availability: successful requests / total requests Latency: P99 response time < 200ms Throughput: QPS > 10000 Correctness: data consistency check pass rate Key principle for choosing SLIs: start from the user’s perspective. Users don’t care about your CPU usage — they care whether requests succeed and are fast enough....

Overview Many teams treat SRE as just “operations with a new name” — hire a few people who can write scripts, change their titles, and call it a transformation. This mindset ignores a fundamental truth: SRE is an engineering methodology, not a toolchain. When Google created the SRE function in 2003, the core idea was “treat operations problems with software engineering methods,” which fundamentally changed the positioning, workflow, and culture of operations....

Overview Performance issues are among the most common scenarios every SRE encounters: users report “it’s slow,” alerts say “P99 latency exceeds threshold,” monitoring shows “CPU is almost maxed.” But many teams handle performance issues with a “tune wherever it’s high” approach — add machines when CPU is high, add indexes when SQL is slow, add cache when latency is high. This symptomatic treatment may work short-term, but over time it makes the system increasingly complex, costs keep rising, and problems become harder to troubleshoot....

Overview There’s a saying in SRE: “Systems will fail — the question is whether you’ll be woken up by them or actively managing them.” Incident management is not “dealing with things after they break” — it’s a complete engineering system spanning prevention, detection, response, and learning. This article systematically covers how to build a practical on-call system across five dimensions: incident grading, on-call rotation, incident response processes, postmortem culture, and alert governance....

Overview Alerting is the “last mile” of a monitoring system — and the hardest to get right. A common predicament: servers run dozens of alerting rules generating hundreds of alert notifications daily. On-call engineers, bombarded by WeChat/DingTalk/email, gradually become desensitized — truly urgent alerts are drowned in noise until customer complaints reveal the system has been broken for hours. The golden rule of SRE: every alert must have a clear action....

Overview The Google SRE Book contains a frequently cited principle: SRE teams should spend no more than 50% of their total work time on toil. This principle seems simple, but in practice, many SRE teams’ toil ratio far exceeds 50% — some even reach 80% or more. Why should SRE take “toil” so seriously? Because toil is the invisible killer of reliability: Toil consumes enormous amounts of time, leaving engineers no energy for work that genuinely improves reliability Toil typically involves manual operations that are error-prone, actually introducing new incidents Toil leads to burnout and attrition of talented engineers Toil doesn’t scale — when the system grows 10x, toil grows 10x too This article systematically covers how to manage operational work across toil definition and identification, source analysis, automation elimination paths, the 50% cap principle, measurement and tracking methods, and team practices....

Overview Woken up by an alert at 3 AM, facing an unfamiliar service — how fast can you recover? If you need to scroll through chat logs, ask colleagues, and dig through code to figure out what to do, your team is missing one thing — a Runbook. A Runbook is the most fundamental yet most easily overlooked engineering practice in the SRE framework. It bridges the gap between “alert” and “action” — an alert tells you “something is wrong,” and a Runbook tells you “what to do....