SLO

Overview When I took over an operations team in 2021, I ran a simple audit: 12 people, 40 hours per week each, averaging 24 hours spent on manual ticket handling, service restarts, and disk space cleanup. That’s 60% toil. Of the remaining 16 hours, half went to status meetings. Actual engineering time: under 8 hours per week per person. Everyone was busy, but the system was still unstable. MTTR sat at 40 minutes, availability at 99....

Overview Your company uses a cloud service with an SLA that clearly states 99.95% monthly availability. One night, the service goes down for 30 minutes. You file a compensation claim with the cloud provider. The calculation goes roughly: downtime as a fraction of total monthly minutes, multiplied by your monthly fee. Assuming you pay 1000 yuan/month, that’s 30 minutes divided by 43,200 minutes times 1000, giving you about 0.7 yuan....

Overview You’ve probably been here: SLO set at 99.9%, error budget calculated, Grafana dashboard looking sharp. Then Friday afternoon, with the budget 87% consumed, the PM shows up carrying the boss’s mandate: “This feature must ship Monday.” You say no, the budget’s almost gone. PM asks “Then when?” Thirty minutes of arguing later, it ships anyway. Monday night, it blows up. This isn’t an isolated case. I’ve seen too many teams turn error budgets into “ops’ dashboard”—metrics defined, alerts configured, dashboards painted—but the moment a decision needs to be made, the whole thing collapses....

Overview 99.5% availability sounds pretty good — until you do the math. A year has 8,760 hours. 99.5% means you’re allowed 43.8 hours of downtime. Per month, that’s roughly 3.65 hours of service interruption. If this is a core transaction system, those 3.65 hours could mean tens of thousands in lost orders. During a major promotion, the loss multiplies by orders of magnitude. 99.9%? Annual downtime drops to 8.77 hours, less than 44 minutes per month....

Overview 3 AM. You get woken up by an alert. You check: CPU normal, memory normal, pods running. But users are complaining the system is unusable. You open Grafana, stare at a wall of charts, and cannot answer one simple question: is this an incident or not? I have encountered this scenario repeatedly across multiple enterprise SRE consulting engagements. The core problem is not insufficient monitoring — it is that the measurement system is designed wrong: we keep monitoring internal system metrics instead of user-perceived service quality....

Overview Alerting is the “last mile” of a monitoring system — and the hardest to get right. A common predicament: servers run dozens of alerting rules generating hundreds of alert notifications daily. On-call engineers, bombarded by WeChat/DingTalk/email, gradually become desensitized — truly urgent alerts are drowned in noise until customer complaints reveal the system has been broken for hours. The golden rule of SRE: every alert must have a clear action....

Overview The Error Budget is the most ingeniously designed mechanism in the SRE framework. It transforms the long-standing “stability vs. iteration speed” debate — previously settled by opinion and politics — into a quantifiable engineering decision framework: your system has an “unavailability allowance,” and when it’s spent, you stop and fix things. In practice, however, many teams define SLOs and error budgets but stop at displaying a percentage number on a dashboard....