SRE Fundamentals

Overview It’s 2 AM. Your phone screams. The monitoring dashboard is a sea of red — core transaction P99 latency just hit 8 seconds, upstream services are timing out and circuit-breaking, and customer support chat is flooding with screenshots. You’re VPN-ing in while your brain runs at full speed: have we seen this scenario in a drill? Is it covered in the runbook? Do I remember the failover steps? If you’re still searching the wiki for documentation at this moment, it means one thing: your runbook was written but never practiced....

Overview 3 AM. The core trading system is down. The on-call engineer frantically flips through the Runbook. The DBA says it’s not a database issue. The network team says the links are fine. The developers say nothing changed. Three teams point fingers at each other. Incident recovery drags on for 47 minutes. This is the reality of operations in many companies. The problem isn’t that people aren’t trying hard enough. The problem is there’s no engineering-driven reliability team to decompose the problem....

Overview When your business grows from “serving one city” to “serving the entire country” or even “serving globally,” single-datacenter architecture hits two hard constraints: latency from distance and single point of failure risk. Multi-region active-active architecture is the engineering solution to both problems. But active-active architecture is one of the most complex topics in SRE — it’s not simply “deploy the service in two datacenters.” It involves a series of deep engineering challenges: data consistency, traffic routing, failover, and operational complexity....

Overview In modern software engineering, the relationship between SRE (Site Reliability Engineering) and development teams is one of the most critical and delicate aspects. Development pursues “fast delivery of new features,” while SRE pursues “stable system operation” — these two goals are inherently in tension. If the collaboration model is poorly designed, the result ranges from inefficiency and finger-pointing to frequent production incidents and collapsed trust between teams. The Google SRE Book contains a classic observation: “The central SRE contradiction is that we need to simultaneously allow development teams to move quickly while maintaining the reliability of the system....

The Role of Change Management in SRE Google SRE identified an iron rule: approximately 70% of production incidents are directly caused by changes. Whether it’s code deployment, configuration modification, infrastructure adjustment, or dependency upgrades, every change injects uncertainty into the system. Change management is therefore not bureaucratic red tape — it’s the first line of defense in SRE reliability engineering. The core objectives of change management can be summarized in three points:...

Reliability Engineering: More Than Just “Not Breaking” The goal of reliability engineering is not to pursue zero failures — that’s neither realistic nor economical. The real goal is: given that failures are inevitable, make the system capable of fast detection, automatic recovery, and continuous learning. Google SRE proposes a core formula: MTTR << MTBF / (MTBF + MTTR) × (1 - SLO) This formula reveals a key insight: when the time between failures (MTBF) is much greater than the repair time (MTTR), system availability naturally approaches the SLO target....

Overview The traditional approach to reliability is “try not to have failures” — add monitoring, add alerts, add redundancy. But this passive defense has a fundamental flaw: you don’t know how the system actually behaves during a failure until one actually occurs. Chaos engineering takes the opposite approach: proactively and controllably inject failures to discover system weaknesses before they become incidents. It’s not about causing destruction — it’s a scientific experimental method: form a hypothesis (“the system should be able to withstand a node failure”), design an experiment (kill a node), verify the hypothesis (is the service still working?...

Overview In modern microservice architectures, a seemingly simple user request may traverse dozens of service nodes. When an incident occurs, the first question an SRE engineer faces is often not “how to fix it” but “what is the scope of impact.” Without a fast answer to this question, incident recovery gets bogged down in endless investigation. Service Dependency Maps and Failure Domain Analysis are the engineering methodologies that address this problem....

Overview Every incident is a free learning opportunity — provided you have a mechanism to extract lessons from it. A Postmortem is not about writing a confession or finding a scapegoat. It is a structured engineering methodology for converting incident experience into systemic improvements. One of the core tenets of Google SRE is: “Blameless postmortem.” The focus of a review is always on “why did the system fail,” not “who messed up....

Overview The core philosophy of SRE is: manage reliability through engineering methods. The most important tools are SLI, SLO, and Error Budgets. SLI: Service Level Indicator SLI is a quantitative metric for system reliability. Common SLIs include: Availability: successful requests / total requests Latency: P99 response time < 200ms Throughput: QPS > 10000 Correctness: data consistency check pass rate Key principle for choosing SLIs: start from the user’s perspective. Users don’t care about your CPU usage — they care whether requests succeed and are fast enough....