Reliability Measurement Is Not About Piling Metrics: Turning 'Is the System Stable?' into Decision-Driven Numbers with a Four-Layer Model

Overview

3 AM. You get woken up by an alert. You check: CPU normal, memory normal, pods running. But users are complaining the system is unusable. You open Grafana, stare at a wall of charts, and cannot answer one simple question: is this an incident or not?

I have encountered this scenario repeatedly across multiple enterprise SRE consulting engagements. The core problem is not insufficient monitoring — it is that the measurement system is designed wrong: we keep monitoring internal system metrics instead of user-perceived service quality.

During a postmortem at a ride-hailing platform, the team discovered an ironic fact: the alerting system triggered 12 minutes after the incident started, while users began complaining 30 seconds in. The reason was simple — alerts were configured on CPU usage above 80%, but the actual failure was a database connection pool exhaustion where CPU sat at 35%, never touching the threshold.

The essence of reliability measurement is transforming the question “is the system stable?” from a subjective feeling into a quantifiable number. This article does not cover basic SLI/SLO concepts (those are thoroughly explained in Related: SRE Core Concepts: SLI, SLO and Error Budgets). Instead, it focuses on how to design a complete, practical, decision-driving reliability measurement system from scratch.

Why Traditional Monitoring Cannot Answer “Is the System Stable?”

Let me give you the conclusion first: traditional monitoring cannot answer this question because it measures the wrong things.

Three Blind Spots of Traditional Monitoring

Blind Spot 1: Monitoring Resources Instead of User Experience

Most teams’ dashboards look like this: CPU line chart, memory line chart, disk IO line chart, network traffic line chart. These metrics reflect “how busy the server is,” not “how the user feels.”

A real example: an e-commerce order service had CPU usage consistently at 70%. Ops felt something was off and planned to scale up. But P99 latency was 120ms, request success rate was 99.97% — user experience was perfectly fine. Meanwhile, a cache service sat at 20% CPU, looking “healthy,” but cache hit rate dropped from 95% to 60%, hammering the database and degrading actual user experience significantly.

Blind Spot 2: No Quantified Boundary Between “Good” and “Bad”

“Is CPU at 80% good or bad?” — traditional monitoring cannot answer this. 80% might be normal batch processing load, or it might be a precursor to saturation. Without SLOs, 80% is just a number, not a basis for decisions.

Google’s SRE Book puts it well: “If you can’t measure it, you can’t manage it.” But the more accurate statement is: if you measure the wrong thing, it is worse than not measuring at all — because you will make decisions based on wrong signals.

Blind Spot 3: Static Threshold Alerts Do Not Sense Business Impact

Traditional alerting logic is “trigger when metric exceeds threshold.” The problem is that the same threshold means completely different things in different business contexts:

Static thresholds cannot distinguish these scenarios, resulting in either too many alerts (alert fatigue) or alerts that come too late (users already complaining before alerts fire).

The Four-Layer Reliability Measurement Model

I have deployed this model across multiple enterprises. The core idea: start from the user’s perspective, drill down layer by layer to infrastructure, with clear measurement targets and alerting rules at each layer.

┌─────────────────────────────────────────────┐
│  Layer 4: Error Budget Governance (Decision) │
│  "Can we still deploy?" "Should we freeze?"  │
├─────────────────────────────────────────────┤
│  Layer 3: SLO Target Setting (Governance)    │
│  "Availability ≥ 99.9%" "P99 ≤ 300ms"       │
├─────────────────────────────────────────────┤
│  Layer 2: SLI Metric Collection (Measure)    │
│  "Request success rate" "Latency" "Throughput"│
├─────────────────────────────────────────────┤
│  Layer 1: User Journey Decomposition (Base)  │
│  "What does the user do?" "Critical path?"   │
└─────────────────────────────────────────────┘

Layer 1: User Journey Decomposition

Many people skip this step and jump straight to selecting SLIs, ending up with metrics that have no connection to user perception. User journey decomposition is the foundation of the entire system — figure out what users care about first, then decide what to measure.

The method is straightforward: map out the critical paths users take through your system, marking “success” and “failure” criteria for each step.

Example for an e-commerce system:

Key principle: Do this decomposition for every service, but only define SLOs for critical path services. Not all services need SLOs — internal tools can use a “best effort” model. Forcing SLOs on everything just increases operational burden.

In a logistics platform engagement, I classified 120+ microservices into three tiers:

• S0 (critical path, 8 services): SLO 99.95%, multi-window burn rate alerting, On-Call response < 5min
• S1 (important support, 30 services): SLO 99.9%, threshold + burn rate alerting
• S2 (internal tools, rest): Best effort, basic monitoring only

This tiering let the team focus on the 8 services that actually impact business, cutting alert volume by 60%.

Layer 2: SLI Metric Collection

SLI (Service Level Indicator) is the specific metric used to measure service quality. The most common mistake when selecting SLIs: treating infrastructure metrics as SLIs.

Good SLIs look like this:

• HTTP request success rate: non-5xx requests / total requests
• API response P99 latency
• Message processing success rate
• Order creation success rate

Metrics that should NOT be SLIs:

• CPU utilization (users do not care about your CPU)
• Memory utilization (same)
• Disk IO (same)
• Database connection pool size (this is a diagnostic metric, not an experience metric)

These diagnostic metrics are not useless — they are extremely valuable during troubleshooting. But they should not be the basis for determining “is the system stable.” See Related: Alerting Strategy Design: From Noise to Signal for the distinction between diagnostic and experience metrics.

SLI count control: No more than 4-5 SLIs per service. Beyond that, monitoring and alert configuration complexity grows exponentially, and the team cannot maintain it.

PromQL implementation for SLIs:

# Availability SLI: request success rate (excluding 5xx)
sum(rate(http_requests_total{status!~"5.."}[5m])) 
  / 
sum(rate(http_requests_total[5m]))

# Latency SLI: P99 response time
histogram_quantile(0.99, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# Throughput SLI: requests per second
sum(rate(http_requests_total[5m]))

A pitfall here: latency SLIs must use percentiles (P95/P99), never averages. Averages mask tail latency — 1000 requests averaging 50ms might include 10 requests at 5 seconds. Those 10 users have a terrible experience that gets diluted by the average.

I once saw a team use average latency as their SLI. During a peak sale event, P99 spiked to 3 seconds but the average was only 200ms — no alert fired, users flooded in with complaints, and the team only discovered the problem after the fact. Since then, I have enforced one rule: latency SLIs use percentiles only, averages are banned.

Layer 3: SLO Target Setting

SLO (Service Level Objective) is the expected value for an SLI. For example, “availability ≥ 99.9% (30 days)” or “P99 latency ≤ 300ms”.

How to Set SLO Values

This is the most frequently asked question. My method has three steps:

Step 1: Pull Historical Data

Pull the past 90 days of SLI performance data and observe historical fluctuation ranges and incident frequency.

# Availability over the past 90 days
1 - (
  sum(rate(http_requests_total{status=~"5.."}[90d])) 
    / 
  sum(rate(http_requests_total[90d]))
)

Step 2: Set a Target Slightly Above Historical Performance

If historical availability is 99.89%, do not jump straight to 99.99% (the team will give up trying to reach it). Do not set it at 99.8% either (no challenge). I recommend 99.9%-99.95% — achievable with some effort, but requires continuous improvement to maintain.

Step 3: Align with Product Stakeholders

SLOs are not set by ops alone. I recommend a 30-minute meeting with ops, dev, and product to confirm:

• Product explains “at what point does user experience degradation impact business”
• Ops explains “with current architecture, is this target achievable”
• Dev explains “what technical improvements are needed to reach this target”

Multi-Dimensional SLO Design

A single-dimension SLO is not enough. I recommend a multi-dimensional SLO approach:

Key insight: Higher SLO is not always better. 99.999% SLO means only 5.26 minutes of downtime per year — in most business scenarios, this is over-engineering with maintenance costs far exceeding benefits. I once saw a team set 99.99% SLO on an internal tool used by a handful of people, spending 10 hours per week maintaining it. Classic “SLO for the sake of SLO.”

Here is a practical reference for availability numbers:

Layer 4: Error Budget Governance

This layer is the decision engine of the entire system. Error budgets transform “is the system stable” into “can we still deploy” — a decision that business stakeholders can understand.

Error Budget = 1 - SLO. If SLO is 99.9%, the error budget is 0.1%. Over a 30-day period, 0.1% unavailability = 43.2 minutes.

The core value of error budgets is not “telling you how many minutes are left” — it is establishing a decision mechanism based on budget consumption:

The value of this mechanism: transforming “should we deploy” from subjective judgment to data-driven decision. Product manager pushing you to release? Check the error budget — if it is healthy, deploy; if tight, hold. No arguments needed.

When I introduced this mechanism at an e-commerce platform, the biggest resistance came from the dev team: “why were the old metrics fine but now we cannot deploy?” The key to explaining: error budgets do not restrict releases, they make release decisions evidence-based. When budget is healthy, deploy boldly; when tight, be conservative. One quarter later, the team found that release frequency actually increased 30% — because “deploy freely when budget is healthy” encouraged faster shipping during safe periods.

Error budget consumption strategies and action guidelines are discussed in more detail in Related: Error Budget Consumption Strategies and Action Guidelines.

Multi-Window Multi-Burn-Rate Alerting: From Theory to Implementation

Traditional threshold alerting (“alert when CPU > 80%”) has two fatal flaws: thresholds are hard to set, and it cannot distinguish “slow degradation” from “sudden failure.”

Google SRE’s Multi-Window Multi-Burn-Rate alerting solves this. The core idea: use two different time windows of burn rates in combination to detect sudden failures fast while catching gradual degradation.

What Is Burn Rate

Burn Rate = actual error rate / allowed error rate.

Assuming SLO is 99.9% (allowed 0.1% error rate):

• If current error rate is 0.1%, burn rate = 0.1% / 0.1% = 1 (normal consumption speed)
• If current error rate is 1%, burn rate = 1% / 0.1% = 10 (10x consumption)
• If current error rate is 10%, burn rate = 10% / 0.1% = 100 (100x consumption)

A burn rate of 1 means the budget will be exactly consumed by the end of the SLO period. A burn rate of 10 means the budget is consumed in 1/10 of the time — a 30-day budget burned in 3 days.

The Magic of Multi-Window Combination

A single window cannot distinguish between:

• Sudden failure: error rate spikes within 5 minutes → needs immediate alerting
• Slow degradation: error rate consistently elevated over 6 hours → needs attention but not urgent

The multi-window approach uses “short window × long window” combinations:

The short window confirms “the problem is happening now,” the long window confirms “it is not a transient blip.” Both windows must exceed the threshold simultaneously to trigger — this is the essence of multi-window: short window for speed, long window for precision.

Why 14.4 instead of a round number? For a 30-day SLO period, 14.4x consumption rate = 30 / (14.4 × 2) ≈ 1 day. This number comes from Google SRE Workbook’s derivation: trigger Page-level alerts at 2% budget consumption. See Google SRE Workbook Chapter 5 for the full derivation.

Complete Prometheus Configuration

Here is a production-ready Prometheus alerting rule based on the multi-window multi-burn-rate approach:

# slo-burn-rate-alerts.yaml
# SLO: 99.9% availability, 30-day period
# Error budget: 0.1% × 30 days = 43.2 minutes

groups:
  - name: slo-burn-rate
    interval: 30s
    rules:
      # ============ Page-level alerts ============
      # Short window 5min + Long window 1h, burn rate 14.4
      # Trigger: error rate in both 5min and 1h windows exceeds 1.44% (14.4 × 0.1%)
      # Meaning: at this rate, 30-day budget exhausted in 2 days
      - alert: SLOBurnRateCritical
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
              / 
            sum(rate(http_requests_total[5m])) by (service)
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
              / 
            sum(rate(http_requests_total[1h])) by (service)
          ) > (14.4 * 0.001)          
        for: 2m
        labels:
          severity: page
          slo_severity: critical
        annotations:
          summary: "{{ $labels.service }} SLO burn rate critical"
          description: "{{ $labels.service }} burn rate exceeds 14.4 in both 5min and 1h windows. At this rate, error budget will be exhausted in 2 days."

      # ============ Ticket-level alerts ============
      # Short window 30min + Long window 6h, burn rate 6
      # Trigger: error rate in both windows exceeds 0.6%
      # Meaning: at this rate, budget exhausted in 5 days
      - alert: SLOBurnRateWarning
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[30m])) by (service)
              / 
            sum(rate(http_requests_total[30m])) by (service)
          ) > (6 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[6h])) by (service)
              / 
            sum(rate(http_requests_total[6h])) by (service)
          ) > (6 * 0.001)          
        for: 5m
        labels:
          severity: ticket
          slo_severity: warning
        annotations:
          summary: "{{ $labels.service }} SLO burn rate elevated"
          description: "{{ $labels.service }} burn rate exceeds 6 in both 30min and 6h windows. At this rate, error budget will be exhausted in 5 days."

      # ============ Latency SLO burn rate ============
      # SLO: P99 latency ≤ 300ms
      # Requests exceeding 300ms as "errors"
      - alert: SLOLatencyBurnRateCritical
        expr: |
          (
            sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) by (service)
              / 
            sum(rate(http_request_duration_seconds_count[5m])) by (service)
          ) < (1 - 14.4 * 0.001)
          and
          (
            sum(rate(http_request_duration_seconds_bucket{le="0.3"}[1h])) by (service)
              / 
            sum(rate(http_request_duration_seconds_count[1h])) by (service)
          ) < (1 - 14.4 * 0.001)          
        for: 2m
        labels:
          severity: page
          slo_type: latency
        annotations:
          summary: "{{ $labels.service }} latency SLO burn rate critical"
          description: "Proportion of requests exceeding 300ms P99 is sustained high. At this rate, latency SLO budget will be exhausted in 2 days."

A Real Production Incident: Burn Rate Alert False Positives

During the first week of burn rate alerting deployment at a ride-hailing project, the team received 23 Page-level alerts, 18 of which were false positives. Investigation revealed two causes:

Cause 1: Statistical Noise on Low-Traffic Services

An internal service with 1000 daily requests had only 3-4 requests in a 5-minute window. A single 5xx error pushed the error rate to 25%, burn rate to 250, triggering the alert. But the actual impact was negligible.

Fix: Add a minimum request count threshold for low-traffic services:

# Add minimum request count filter to alert condition
- alert: SLOBurnRateCriticalWithTraffic
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
        / 
      sum(rate(http_requests_total[5m])) by (service)
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
        / 
      sum(rate(http_requests_total[1h])) by (service)
    ) > (14.4 * 0.001)
    # New: at least 100 requests in 5 minutes before alerting
    and
    sum(rate(http_requests_total[5m])) by (service) > 100    
  for: 2m
  labels:
    severity: page

Cause 2: Planned Maintenance Triggering Alerts

Weekly Wednesday 2 AM database maintenance caused 2 minutes of connection timeouts, spiking error rates and triggering Page-level alerts. On-call engineers were woken up every time and started becoming immune to alerts — the most dangerous signal.

Fix: Configure silence rules during maintenance windows, and exclude planned maintenance time from SLO calculations:

# Alertmanager silence rule (maintenance window)
# Silence every Wednesday 02:00-02:30
- name: maintenance-silence
  matchers:
    - name: service
      value: "doctor-gateway"
    - name: maintenance_window
      value: "weekly-db"
  startsAt: "2026-07-30T02:00:00+08:00"
  endsAt: "2026-07-30T02:30:00+08:00"

These two fixes reduced false positives from 18/week to 2/week, and the team started trusting the alerting system again.

Implementation Path for the Measurement System

Do not try to do everything at once. My experience across multiple enterprises: three phases, 4-6 weeks each.

Phase 1: Observable (Weeks 1-4)

Goal: Make SLIs visible, but do not rush to set SLOs.

• Select 3-5 core services, deploy HTTP metric collection (request success rate, latency distribution)
• Build an “SLI Overview” dashboard in Grafana
• Do not configure any alerts — just observe data for 2 weeks
• Focus: verify metrics accurately reflect user experience

The biggest value of this phase is discovering monitoring blind spots — you will find many services lack basic HTTP metrics, or latency histograms are poorly configured (too few buckets leading to inaccurate P99 calculations).

Phase 2: Measurable (Weeks 5-8)

Goal: Define SLOs and start measuring.

• Based on Phase 1 data, set initial SLOs (slightly below historical best performance)
• Implement error budget calculation and visualization
• Configure burn rate alerts, but only enable Ticket level (not Page)
• Weekly SLO achievement review

Key action: Put the error budget dashboard where the entire team can see it. My approach at one team: place a large screen in the office area showing real-time error budget consumption bars for each service. This “visual pressure” is more effective than any KPI — dev teams proactively monitor their own service’s budget consumption.

Phase 3: Decision-Driven (Weeks 9-12)

Goal: Use error budgets to drive release decisions.

• Enable Page-level burn rate alerts
• Establish error budget decision process (freeze non-emergency releases when budget < 25%)
• Confirm SLO adjustment mechanism with product (quarterly review)
• Start linking postmortem findings to error budget consumption

At this phase, the reliability measurement system truly comes “alive” — not just dashboards and alerts, but a decision mechanism that drives team behavior.

Common Pitfalls in Measurement Systems

Pitfall 1: Too Many SLOs

Not all services need SLOs. I once saw a team define SLOs for 80 services — maintaining SLO configurations itself became a full-time job.

Recommendation: 5-8 S0 services, 20-30 S1 services, rest on “best effort.” Focus on making core services excellent rather than making all services “passable.”

Pitfall 2: SLOs Never Adjusted

SLOs are not carved in stone. Business changes, architecture changes, user expectations change. A reasonable SLO review cadence:

• Quarterly review: assess whether SLO is still appropriate
• Annual review: re-examine the entire measurement system

One principle when adjusting SLOs: only adjust upward, never downward — unless there is a compelling reason (architecture downgrade, business pivot). Lowering SLOs equals lowering standards, creating a habit of “cannot meet it, so lower the bar.”

Pitfall 3: Only Looking at SLO Compliance, Not Budget Consumption Trends

“SLO met” does not equal “system is healthy.” A service with SLO 99.9% and current availability 99.92% — compliant. But if last month was 99.98%, the trend is deteriorating, and next month may breach.

Recommendation: Add a “budget consumption trend” panel to the dashboard, watching the slope rather than the absolute value:

# Error budget consumption rate over past 7 days (daily)
avg_over_time(
  (1 - (
    sum(rate(http_requests_total{status!~"5.."}[1d])) 
      / 
    sum(rate(http_requests_total[1d]))
  ))[7d:1d]
)

If consumption rate is accelerating, intervene early even if SLO is currently met.

Pitfall 4: Using SLOs as KPIs

This is the most dangerous one. Once SLOs are tied to performance reviews, team behavior distorts:

• SLOs set conservatively (99.8% instead of 99.9%, easier to meet)
• Incidents go unreported (reporting consumes budget, affecting “compliance rate”)
• New features are not deployed (deploying risks incidents, affecting compliance)

Correct approach: SLOs are engineering governance tools, not performance evaluation tools. The purpose of SLOs is to help teams make better decisions, not to punish people for missing targets. Google’s SRE Book states clearly: “When SLO is not met, the correct response is to invest resources in improving the system, not to assign blame.”

Tool Chain Recommendations

The difference between Sloth and Pyrra: Sloth is “configuration as code” mode — define SLOs in YAML, auto-generate Prometheus alerting rules. Pyrra is “visual interface” mode — manage SLOs via Web UI. They can work together: Sloth for CI/CD management, Pyrra for daily visualization.

If you prefer not to introduce additional tools, you can hand-write PromQL. The alerting rules above are pure Prometheus native configuration without Sloth dependency. But the downside of manual maintenance: when SLO count exceeds 10, the rules file becomes very long and hard to maintain.

Building the Grafana Error Budget Dashboard

For a measurement system to deliver value, data must be “seen.” A well-designed dashboard is more useful than 10 alert emails.

Four Core Dashboard Panels

I recommend an error budget dashboard layout with four areas:

Panel 1: SLO Compliance Overview

Use Stat panels to display each service’s SLO compliance status. Green = compliant, yellow = approaching the limit (budget consumption > 75%), red = breached. At a glance, you can see which services need attention.

# Current period (30 days) SLO compliance rate
1 - (
  sum(rate(http_requests_total{status=~"5..", service="$service"}[30d]))
    /
  sum(rate(http_requests_total{service="$service"}[30d]))
)

Panel 2: Error Budget Consumption Gauge

Use Gauge panels to show each service’s remaining error budget percentage. The visual impact: when the needle slides from green to red, the team can intuitively feel “the budget is burning.”

# Error budget remaining ratio (30-day period)
1 - (
  (
    sum(rate(http_requests_total{status=~"5..", service="$service"}[30d]))
      /
    sum(rate(http_requests_total{service="$service"}[30d]))
  ) / (1 - 0.999)  # 1 - SLO
)

Panel 3: Real-Time Burn Rate Curve

Use Time Series panels to show burn rate over time. The value of this curve is seeing trends — a flat curve means healthy, a sudden spike means an incident just occurred.

# Real-time burn rate (5-minute window)
(
  sum(rate(http_requests_total{status=~"5..", service="$service"}[5m]))
    /
  sum(rate(http_requests_total{service="$service"}[5m]))
) / 0.001  # divide by allowed error rate

Add two threshold lines in Grafana: 14.4 (red, Page level) and 6 (yellow, Ticket level), so the team can immediately see the current state.

Panel 4: Daily Error Budget Consumption Detail

Use Bar Chart panels to show daily budget consumption over the past 30 days. The value: discovering patterns — if every Wednesday shows elevated consumption, it indicates a recurring scheduled task that needs targeted optimization.

Variable Configuration

To support multi-service switching, configure a service variable:

{
  "name": "service",
  "type": "query",
  "datasource": "Prometheus",
  "query": "label_values(http_requests_total, service)",
  "refresh": 2,
  "includeAll": true,
  "multi": true
}

This configuration automatically pulls all service names from Prometheus service label, with a dropdown for switching. refresh: 2 updates the label list every 2 minutes, so new services automatically appear in the dropdown.

A Dashboard Pitfall: Timezone-Induced Budget Calculation Errors

Once, the SLO dashboard showed “budget exhausted,” but the system was running fine. Investigation revealed that Grafana timezone configuration and Prometheus query windows were misaligned.

Prometheus rate() calculates based on UTC, while Grafana panels default to UTC display. If the SLO period is “Beijing time 1st of month 00:00 to end of month 23:59,” but Prometheus calculates in UTC, that is “last day of previous month 16:00 to last day of current month 16:00” — an 8-hour data window discrepancy.

Fix: Use consistent UTC offsets in Prometheus queries, set the Grafana panel timezone to Asia/Shanghai, and ensure query time ranges match the SLO period. This seems minor, but when SLOs are tied to team performance, it causes serious disputes — teams will question “is the data accurate?”

Architecture Trade-Offs: Self-Built SLO Platform vs Open-Source Tools

When implementing an SLO system, a common architecture decision: use open-source tools (Sloth/Pyrra) or build a custom SLO management platform. I have practiced both approaches at different team scales — the answer depends on team size and SLO count.

Open-Source Tool Approach (Sloth + Pyrra)

Suitable for scenarios with < 30 SLOs and < 20 team members.

Advantages:

• Quick to set up, 1-2 days to get running
• Sloth uses YAML for SLO definitions, version-controlled, GitOps-friendly
• Pyrra provides Web UI, dev teams can self-serve SLO status
• Active community, native implementation of Google SRE methodology

Disadvantages:

• Limited SLI types (mainly HTTP/gRPC request success rate and latency)
• Custom SLIs require raw PromQL, Pyrra cannot display them
• Weak permission management for multi-team/multi-tenant scenarios
• Fixed error budget calculation periods (7-day/30-day), no custom periods

Self-Built SLO Platform Approach

Suitable for scenarios with > 30 SLOs, multi-team collaboration, and custom SLI type requirements.

Advantages:

• Fully customizable SLI types (supports message queues, batch jobs, databases, non-HTTP scenarios)
• Flexible error budget calculation periods (7-day/30-day/90-day, even custom business cycles like “sale event period”)
• Comprehensive permission management (different teams see only their own services)
• Can integrate with internal ticket systems and release pipelines

Disadvantages:

• High development cost (at least 1 full-time SRE for 2-3 months)
• Ongoing maintenance cost (Prometheus version upgrades may affect PromQL compatibility)
• Requires deep Prometheus expertise in the team

My Recommendation

A middle ground: For teams of 30-50 people with 20-40 SLOs, I recommend the Sloth + Grafana custom dashboard combination. Sloth handles auto-generating Prometheus alerting rules (eliminating hand-written PromQL), Grafana handles visualization (more flexible than Pyrra’s UI, can embed into existing monitoring dashboards). This combination has the lowest maintenance cost while meeting mid-scale team needs.

SLO Propagation in Complex Dependency Chains

In real systems, a service’s availability depends on underlying infrastructure and upstream services. If a database goes down causing API timeouts, whose SLO failure is it? This question causes endless finger-pointing in multi-team environments.

Dependency Chain SLO Model

Core principle: upstream service SLO must be better than downstream service SLO. If a payment service depends on a database, and the payment service’s SLO is 99.95%, then the database SLO must be at least 99.97% — because database unavailability directly causes payment service unavailability.

User Request → API Gateway (99.95%) → Order Service (99.95%) → Database (99.97%)
                                          ↓
                                    Inventory Service (99.9%)

The mathematical basis is the availability multiplication principle: end-to-end availability = product of each component’s availability. If 4 components are all 99.9%, the chain is 99.9%^4 ≈ 99.6% — significantly lower than any single component.

Fault Attribution: Layered SLI Tagging

When SLO is not met, you need to quickly determine whether it is a self-service problem or an upstream dependency problem. The method: tag fault source in SLIs:

# Error rate caused by the service itself (excluding upstream dependency failures)
sum(rate(http_requests_total{status=~"5..", service="order-service", upstream_error="false"}[5m]))
  /
sum(rate(http_requests_total{service="order-service"}[5m]))

# Error rate caused by upstream dependencies
sum(rate(http_requests_total{status=~"5..", service="order-service", upstream_error="true"}[5m]))
  /
sum(rate(http_requests_total{service="order-service"}[5m]))

This implementation requires tagging error sources at the application layer — when catching upstream timeouts or downstream rejections, add an upstream_error label to the HTTP response. This way, error budget consumption can distinguish “self-caused” from “dependency-caused,” preventing cross-team disputes.

Summary

The core of a reliability measurement system is not tools — it is a mindset shift — from “monitoring resources” to “measuring experience,” from “static thresholds” to “dynamic budgets,” from “subjective judgment” to “data-driven decisions.”

Key takeaways from the four-layer model:

1. User journey decomposition is the foundation: figure out what users care about before deciding what to measure. Skip this and everything above is built on sand
2. SLIs focus on user experience: CPU, memory, disk are diagnostic metrics, not experience metrics. Use percentiles for latency, never averages
3. SLOs are data-driven: pull 90 days of historical data, set targets slightly above historical performance. Higher is not always better
4. Error budgets drive decisions: deploy boldly when budget is healthy, be conservative when tight. This is not about restricting releases — it makes release decisions evidence-based
5. Multi-window burn rate alerting: short window for speed, long window for precision. Both must exceed threshold simultaneously — balancing sensitivity and accuracy

In practice, driving availability from 99.5% to 99.9% was not achieved by adding more monitoring and alerts — those were already in place. What actually worked was establishing the error budget decision mechanism: letting the team know “when it is safe to deploy, when to apply the brakes.” When release decisions shift from “gut feeling” to “data,” reliability improvement follows naturally.

If you are building an SLO system, my advice: start with one core service, walk through all four layers end to end, then expand. Do not define SLOs for all services at once — that will trap the team in configuration overhead and the initiative will fizzle out. Remember, the ultimate goal of a measurement system is making system reliability actionable, not filling dashboards with numbers.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Google SRE Workbook - Chapter 5: Alerting on SLOs — Google SRE Team, the original methodology for multi-window multi-burn-rate alerting
2. Scalable Cloud Applications and Site Reliability Engineering (SRE) — Microsoft Azure Team, SLI/SLO implementation practices in cloud architecture
3. SLO Engineering: From Definition to Intelligent Evolution — Tencent Cloud Developer Community, three-layer model definition and enterprise implementation pitfalls
4. SLO Monitoring and Alerting with Prometheus — Zhihu, burn rate alerting Prometheus implementation reference
5. Building a Practical Reliability Measurement Framework from Scratch — CSDN, SLI selection principles and error budget management practices
6. Alert Governance: Reducing 500+ Daily Alerts to 50 — 51CTO, quantified data on alert fatigue and governance solutions