SRE

Overview A friend of mine complained last year: his company deployed Prometheus, ELK, and Jaeger — three separate tool stacks maintained by a 7-person ops team. They spent 4 hours a day just triaging alerts. One night at 3 AM, the payment service hit a P0 incident, and it took 42 minutes from alert trigger to root cause identification. The root cause was a database connection pool configuration change causing slow query buildup....

Overview 1:47 AM. Phone buzzed. Alert channel exploded. A newly launched recommendation service at a ride-hailing project went OOMKilled on all 3 Pods less than 2 hours after launch. Memory spiked from 800MB to the 2GB limit, the kernel killed the process, Kubernetes restarted the Pod, and it got killed again. Loop. Users were routed to the fallback path. P99 latency shot from 80ms to 2000ms. The postmortem found the root cause....

Overview You’ve probably been here: SLO set at 99.9%, error budget calculated, Grafana dashboard looking sharp. Then Friday afternoon, with the budget 87% consumed, the PM shows up carrying the boss’s mandate: “This feature must ship Monday.” You say no, the budget’s almost gone. PM asks “Then when?” Thirty minutes of arguing later, it ships anyway. Monday night, it blows up. This isn’t an isolated case. I’ve seen too many teams turn error budgets into “ops’ dashboard”—metrics defined, alerts configured, dashboards painted—but the moment a decision needs to be made, the whole thing collapses....

Overview 3 AM. Your phone buzzes. P1 alert — order service 5xx error rate spiking. You drag yourself out of bed, fire up the laptop, pull logs, check Grafana, trace it to a saturated upstream database connection pool. Restart, scale up, recover. It’s 4:30 AM. The next morning, nursing dark circles in the standup, someone mentions that last night’s alert was actually a false positive. If you’ve worked in operations, you know this feeling....

Overview 99.5% availability sounds pretty good — until you do the math. A year has 8,760 hours. 99.5% means you’re allowed 43.8 hours of downtime. Per month, that’s roughly 3.65 hours of service interruption. If this is a core transaction system, those 3.65 hours could mean tens of thousands in lost orders. During a major promotion, the loss multiplies by orders of magnitude. 99.9%? Annual downtime drops to 8.77 hours, less than 44 minutes per month....

Overview 1:47 AM. Phone buzzes. The alert group explodes with 200+ messages. Order success rate drops from 99.8% to 92%. The on-call engineer restarts the service, rolls back the version, and fights for 40 minutes to restore service. Next day’s postmortem: root cause was a one-line database connection pool config change pushed during the afternoon release. This change went through the full approval process — developer submission, test verification, manager sign-off, SRE review....

Overview 3 AM. You get woken up by an alert. You check: CPU normal, memory normal, pods running. But users are complaining the system is unusable. You open Grafana, stare at a wall of charts, and cannot answer one simple question: is this an incident or not? I have encountered this scenario repeatedly across multiple enterprise SRE consulting engagements. The core problem is not insufficient monitoring — it is that the measurement system is designed wrong: we keep monitoring internal system metrics instead of user-perceived service quality....

Overview 3 AM, phone buzzing wildly. Open it — the alert group is already on fire. Payment service P99 latency jumped from 50ms to 12 seconds, error rate past 80%. Worse, the failure cascaded like dominoes: user service timeout → auth failures → inventory service retry storm → message queue backed up → payment callbacks all timed out. The monitoring dashboard went entirely red in thirty seconds. This isn’t a unique disaster....

Overview Anyone who has done production operations has probably lived through this scenario: you get woken up by an alert at 3 AM, fight the fire until service is restored, hold a postmortem the next day, list a dozen improvement actions in the meeting, send the minutes to the group chat, and everyone says “got it.” Then what? A month later you dig it up and find maybe two or three items actually got done....

Overview 3 AM. The core trading system is down. The on-call engineer frantically flips through the Runbook. The DBA says it’s not a database issue. The network team says the links are fine. The developers say nothing changed. Three teams point fingers at each other. Incident recovery drags on for 47 minutes. This is the reality of operations in many companies. The problem isn’t that people aren’t trying hard enough. The problem is there’s no engineering-driven reliability team to decompose the problem....