SLO Design in Practice: From Business Goals to Technical Metrics

            ┌─────────────┐
            │ Business     │  "How good does our service need to be?"
            │ Goals        │
            └──────┬──────┘
                   │
            ┌──────▼──────┐
            │ User         │  "What do users care about?"
            │ Experience   │
            └──────┬──────┘
                   │
            ┌──────▼──────┐
            │ SLI          │  "How do we measure user experience?"
            │ Definition   │
            └──────┬──────┘
                   │
            ┌──────▼──────┐
            │ SLO          │  "What target should this metric hit?"
            │ Target Value │
            └──────┬──────┘
                   │
            ┌──────▼──────┐
            │ Alerting     │  "What happens when we miss the target?"
            │ & Actions    │
            └─────────────┘

User Journey: E-commerce Checkout

  Browse Products → Add to Cart → Submit Order → Pay → Confirm

  Each step maps to one or more SLIs:

  Browse Products:
    - SLI: Product page load success rate > 99.95%
    - SLI: Product page P95 load time < 500ms

  Add to Cart:
    - SLI: Cart operation success rate > 99.9%
    - SLI: Cart operation P99 latency < 200ms

  Submit Order:
    - SLI: Order submission success rate > 99.99%
    - SLI: Order submission P99 latency < 1s

  Payment:
    - SLI: Payment success rate > 99.99%
    - SLI: Payment P99 latency < 2s

# CUJ priority matrix
user_journeys:
  - name: "User Login"
    business_value: "high"       # Login failure → immediate user churn
    user_sensitivity: "high"     # Users have zero tolerance for login failures
    priority: P0
    needs_slo: true

  - name: "Product Search"
    business_value: "high"       # Search affects conversion rate
    user_sensitivity: "medium"   # Slightly slower search is acceptable
    priority: P1
    needs_slo: true

  - name: "View Order History"
    business_value: "medium"
    user_sensitivity: "low"
    priority: P2
    needs_slo: false             # Can be covered by default SLO

# SLI specification template
sli_spec:
  name: "Payment Request Success Rate"

  # 1. Measurement subject
  subject: "payment-service"

  # 2. Measurement dimension
  metric: "success rate"

  # 3. Calculation formula
  formula: |
    Successful requests / Total requests
    Success definition: HTTP status code not in [500, 599] range
    Excluded: Client errors (4xx), health check requests (/healthz)    

  # 4. Measurement window
  window: "30d"

  # 5. Data source
  source: "Prometheus http_requests_total metric"

# ❌ Wrong: CPU utilization is not an SLI
sli:
  metric: "CPU utilization < 80%"
  why_wrong: "At 80% CPU, users may not notice anything, or latency may already be severely impacted"

# ✅ Correct: Define from user perspective
sli:
  metric: "Request P99 latency < 200ms"
  why_right: "Directly reflects the experience users perceive"

# ❌ Wrong: Global success rate
sli:
  metric: "All HTTP request success rate > 99.9%"
  why_wrong: "Health check requests and payment requests are mixed together, masking critical path issues"

# ✅ Correct: Define separately for critical paths
sli:
  metric: "Payment API success rate > 99.99%"
sli:
  metric: "Product API success rate > 99.9%"

# ❌ Wrong: Average latency
sli:
  metric: "Average latency < 100ms"
  why_wrong: "Average latency masks the long tail — 1% of users may wait 5 seconds"

# ✅ Correct: Use percentiles
sli:
  metric: "P99 latency < 200ms"
sli:
  metric: "P95 latency < 100ms"

# SLO calibration analysis
def calibrate_slo(historical_sli_data, user_satisfaction_threshold):
    """
    historical_sli_data: SLI data from the past 90 days
    user_satisfaction_threshold: User satisfaction threshold
    """
    # 1. Calculate historical SLI distribution
    p50 = percentile(historical_sli_data, 50)
    p90 = percentile(historical_sli_data, 90)
    p99 = percentile(historical_sli_data, 99)
    p999 = percentile(historical_sli_data, 99.9)

    # 2. Find the user satisfaction inflection point
    # The SLI value where user satisfaction suddenly drops is the SLO lower bound
    satisfaction_inflection = find_inflection_point(
        historical_sli_data,
        user_satisfaction_threshold
    )

    # 3. Set SLO above the inflection point with a safety margin
    suggested_slo = satisfaction_inflection * 1.1  # 10% safety margin

    return {
        "historical_p50": p50,
        "historical_p99": p99,
        "satisfaction_inflection": satisfaction_inflection,
        "suggested_slo": suggested_slo,
        "error_budget": 1 - suggested_slo
    }

Step 1: Determine the service's business tier
  → L1 Critical / L2 Important / L3 Standard / L4 Auxiliary

Step 2: Analyze historical data
  → How has the SLI performed over the past 90 days?
  → What's the P50 / P95 / P99 distribution?

Step 3: Identify user tolerance
  → At what SLI level do users start complaining?
  → Historically, at what SLI level did business metrics get affected?

Step 4: Select initial SLO
  → Based on historical P99 or P99.9 performance, set a target slightly above current level
  → Leave sufficient error budget for releases and innovation

Step 5: Trial run validation
  → Set a 4-6 week trial period
  → Observe feasibility and whether it reflects real user experience

Step 6: Official release and periodic review
  → After trial passes, officially activate
  → Review SLO reasonableness monthly/quarterly

# New service SLO initial setting strategy
new_service_slo_strategy:
  phase_1: "Trial period (first 4 weeks)"
    action: "Monitor only without setting SLO, collect baseline data"
    goal: "Understand the natural distribution of SLIs"

  phase_2: "Initial SLO (weeks 5-8)"
    action: "Set a conservative SLO based on baseline data"
    strategy: |
      Availability SLO = Historical lowest availability + 0.5%
      Latency SLO = Historical P99 × 1.2      
    goal: "Validate SLO feasibility"

  phase_3: "Official SLO (from week 9)"
    action: "Adjust based on trial data and officially release"
    review_cycle: "Monthly review"

# SLO Quarterly Review Template

## Review Scope
- Service name: payment-service
- Review period: 2026 Q2
- Participants: SRE lead, service owner, product manager

## Current SLOs
| SLI | SLO Target | Actual Performance | Error Budget Consumed |
|-----|---------|---------|-------------|
| Availability | 99.95% | 99.97% | 40% |
| Latency (P99) | <200ms | 185ms | 25% |
| Latency (P95) | <100ms | 92ms | 10% |

## Review Questions

### 1. Does the SLO reflect user experience?
- [ ] Is the user complaint volume correlated with SLO violations?
- [ ] Are there user-perceived issues not covered by SLIs?
- Analysis: A latency degradation in March wasn't captured by P99 SLO, but users complained
  → Need to add P95 latency SLI ✅

### 2. Is the SLO too loose or too strict?
- [ ] Is the error budget always used up? → No, 40% consumed
- [ ] Is the error budget always surplus? → Availability budget 40%, latency budget 25%
- Analysis: Latency SLO consumption is low; consider tightening
  → Latency P99 SLO from 200ms to 180ms ⚠️

### 3. Do we need to add or remove SLIs?
- [ ] Are there new critical user journeys to cover?
- [ ] Are there SLIs that are no longer relevant?
- Analysis: Added "payment callback latency" SLI to cover the async part of the payment flow
  → New SLI: Payment callback P99 < 5s ✅

### 4. Is the SLO cost reasonable?
- [ ] What's the infrastructure cost of maintaining the current SLO?
- [ ] What's the cost of upgrading the SLO to the next level?
- Analysis: Upgrading from 99.95% to 99.99% requires multi-active architecture, cost increases 200%
  → Don't upgrade for now; current SLO meets business needs ✅

## Review Conclusions
- Availability SLO: Maintain 99.95%
- Latency P99 SLO: Tighten from 200ms to 180ms
- Add latency P95 SLO: <100ms
- Add payment callback SLO: P99 < 5s
- Next review: 2026 Q3

Error budget consumption rate
  │
  ├─ < 25% (long-term)
  │   → SLO may be too loose
  │   → Evaluate whether to tighten SLO
  │   → Or use the surplus for innovation
  │
  ├─ 25-75%
  │   → SLO is reasonable
  │   → Maintain current SLO
  │
  └─ > 75% (long-term)
      → SLO may be too strict
      → Evaluate whether to relax SLO
      → Or invest resources to improve system capability

┌──────────────────────────────────────────────────┐
│  User Experience SLO                               │
│  "Can users smoothly complete critical operations?"│
│  → Defined from user perspective, business-facing │
└───────────────────────┬──────────────────────────┘
                        │ Decompose
┌───────────────────────▼──────────────────────────┐
│  Service SLO                                       │
│  "Does each service meet its interface contract?"  │
│  → Defined from inter-service call perspective    │
└───────────────────────┬──────────────────────────┘
                        │ Decompose
┌───────────────────────▼──────────────────────────┐
│  Resource SLO                                      │
│  "Are underlying resources healthy?"               │
│  → Defined from infrastructure perspective        │
└──────────────────────────────────────────────────┘

# User Experience SLO example: E-commerce checkout
user_experience_slo:
  journey: "User Checkout"

  sli_1:
    name: "Checkout Success Rate"
    formula: "Successful checkouts / Checkout requests"
    target: 99.99%
    window: 30d
    user_impact: "Checkout failure directly causes transaction loss"

  sli_2:
    name: "Checkout End-to-End Latency"
    formula: "Time from clicking 'Submit Order' to seeing 'Order Successful' page"
    target: "P95 < 2s, P99 < 5s"
    window: 30d
    user_impact: "Latency over 5s may cause user abandonment"

  sli_3:
    name: "Checkout Page Availability"
    formula: "Successful product page loads / Total visits"
    target: 99.95%
    window: 30d
    user_impact: "Page won't load → immediate user churn"

# Service SLO example: Order service
service_slo:
  service: "order-service"

  sli_1:
    name: "API Availability"
    formula: |
      Non-5xx responses / Total requests
      Excluded: /healthz, /metrics      
    target: 99.95%
    window: 30d

  sli_2:
    name: "API Latency"
    formula: "P99 response time"
    target: "< 500ms"
    window: 30d
    breakdown:
      - endpoint: "POST /api/orders"
        target: "P99 < 1s"
      - endpoint: "GET /api/orders/{id}"
        target: "P99 < 200ms"
      - endpoint: "GET /api/orders"
        target: "P99 < 500ms"

  sli_3:
    name: "Message Processing Latency"
    formula: "Time from message enqueue to processing completion"
    target: "P99 < 10s"
    window: 30d

# Resource SLO example: Database
resource_slo:
  resource: "order-db (PostgreSQL)"

  sli_1:
    name: "Database Availability"
    formula: "Time SELECT 1 succeeds / Total time"
    target: 99.99%
    window: 30d

  sli_2:
    name: "Query Latency"
    formula: "P99 query execution time"
    target: "< 50ms"
    window: 30d

  sli_3:
    name: "Replication Lag"
    formula: "Replica lag"
    target: "< 1s"
    window: 30d

  sli_4:
    name: "Connection Pool Health"
    formula: "Available connections / Total connections"
    target: "> 30%"
    window: 5m   # Resource SLOs typically use shorter windows

User Experience: Checkout success rate 99.99%
    ↑ Depends on
Service: Order API success rate 99.95% + Payment API success rate 99.99%
    ↑ Depends on
Resource: Database availability 99.99% + Cache availability 99.95%

# SLO cascading alert example
cascade_alerting:
  # Resource-level alert: early warning
  - level: resource
    condition: "Database connection pool utilization > 80%"
    action: "Notify SRE, don't notify business team"
    purpose: "Intervene before service-level SLO is affected"

  # Service-level alert: impact is occurring
  - level: service
    condition: "Order API error rate > 0.1%"
    action: "Notify SRE + service owner"
    purpose: "Prevent impact on user experience SLO"

  # User experience-level alert: users are affected
  - level: user_experience
    condition: "Checkout success rate < 99.99%"
    action: "Immediate page, notify entire team + management"
    purpose: "Users are感知ing the problem; must restore immediately"

# Service SLO: payment-service

## Service Overview
- Service name: payment-service
- Business tier: L1 (Critical)
- Responsible teams: Payment team + SRE platform team
- SLO review cycle: Quarterly

## Critical User Journeys
1. User initiates payment → Payment processing → Return result
2. Payment callback → Order status update → Notify user

## SLIs and SLOs

### SLI-1: Payment API Availability
- Definition: Non-5xx responses / Total requests (excluding /healthz)
- SLO: 99.99% (30-day window)
- Error budget: 4.3 minutes/month
- Data source: Prometheus http_requests_total

### SLI-2: Payment API Latency
- Definition: P99 response time
- SLO: < 2s (30-day window)
- Data source: Prometheus http_request_duration_seconds

### SLI-3: Payment Success Rate
- Definition: Successful payments / Initiated payments
- SLO: 99.95% (30-day window)
- Data source: Business logs + database statistics

## Alerting Strategy
- Fast burn: 1h window burn rate > 14x → Immediate page
- Slow burn: 6h window burn rate > 6x → Ticket tracking
- Budget exhausted: Monthly budget 100% → Freeze releases

## Dependencies
- Upstream dependencies: order-service, user-service
- Downstream dependencies: payment-db, redis-cluster, third-party payment gateway
- Shared infrastructure: API Gateway, Load Balancer

## History
| Date | Change | Reason |
|------|------|------|
| 2026-04-01 | Availability SLO raised from 99.95% to 99.99% | Business requirement |
| 2026-05-15 | Added payment callback latency SLI | Cover async path |
| 2026-07-01 | Latency SLO tightened from 2.5s to 2s | Historical performance exceeds SLO |

Search is the prerequisite step before users enter product detail pages
  → Slow search → Users abandon browsing → Transaction loss
  → No results → User churn
Business goal: Search experience directly impacts GMV

User enters keywords → Search request → Return results → User clicks product

Key experience dimensions:
  1. Is search available? (Availability)
  2. Is search fast enough? (Latency)
  3. Are results relevant? (Correctness/Relevance)
  4. Does search return results? (Coverage)

search_service_sli:
  sli_1:
    name: "Search API Availability"
    formula: "Non-5xx search responses / Total search requests"
    target: 99.95%

  sli_2:
    name: "Search P99 Latency"
    formula: "P99(search response time)"
    target: "< 300ms"

  sli_3:
    name: "Search P95 Latency"
    formula: "P95(search response time)"
    target: "< 100ms"

  sli_4:
    name: "Zero Result Rate"
    formula: "Searches returning 0 results / Total searches"
    target: "< 5%"
    note: "Zero result rate reflects search quality, not a technical metric but affects user experience"

# Calibration based on historical data
historical_data = {
    "availability_p50": 99.98,
    "availability_p99": 99.93,
    "latency_p99_avg": 250,  # ms
    "latency_p99_max": 450,  # ms
    "zero_result_rate_avg": 3.2,  # %
}

# SLO setting
slo = {
    "availability": 99.95,       # Slightly below P50, leaving budget
    "latency_p99": 300,          # Slightly above average, achievable
    "latency_p95": 100,          # Stricter, drives optimization
    "zero_result_rate": 5,       # Allow some zero results
}

Platform-level SLO (user-facing)
  ├── Checkout success rate > 99.9%
  ├── Search availability > 99.95%
  └── Payment success rate > 99.95%
      │
      ├── order-service SLO
      │   ├── API availability > 99.95%
      │   └── API P99 < 500ms
      │       │
      │       ├── order-db SLO
      │       │   ├── Availability > 99.99%
      │       │   └── Query P99 < 50ms
      │       │
      │       └── redis SLO
      │           ├── Availability > 99.95%
      │           └── Hit rate > 90%
      │
      ├── payment-service SLO
      │   └── ...
      └── inventory-service SLO
          └── ...

Platform-level: Checkout success rate 99.9%
  = order-service(99.95%) × payment-service(99.95%) × inventory-service(99.95%)
  = 0.9995^3 = 0.9985 = 99.85%
  → Doesn't meet 99.9% platform-level SLO!

  Need: Each service at least 99.97% → 0.9997^3 = 0.9991 = 99.91% ✅

# Prometheus + Grafana SLO monitoring
slo_monitoring:
  prometheus_rules:
    # SLO achievement rate
    - record: slo:availability:rate30d
      expr: |
        sum(rate(http_requests_total{status!~"5.."}[30d]))
        /
        sum(rate(http_requests_total[30d]))        

    # Error budget consumption
    - record: slo:error_budget:consumed:rate30d
      expr: |
        1 - (slo:availability:rate30d / 0.9999)        

    # Remaining error budget
    - record: slo:error_budget:remaining:ratio
      expr: |
        1 - (slo:error_budget:consumed:rate30d / (1 - 0.9999))        

  grafana_dashboards:
    - "SLO Overview Dashboard"
      panels:
        - "Current SLI value vs SLO target"
        - "Error budget consumption trend"
        - "SLO achievement history"
        - "SLO status grouped by service"

# slo-spec.yaml - SLO specification as code
apiVersion: sre/v1
kind: ServiceSLO
metadata:
  name: payment-service-slo
  service: payment-service
spec:
  window: 30d
  targets:
    - name: availability
      sli:
        source: prometheus
        query: |
          sum(rate(http_requests_total{service="payment",status!~"5.."}[{{.window}}]))
          /
          sum(rate(http_requests_total{service="payment"}[{{.window}}]))          
      slo: 0.9999
      alerts:
        - burn_rate: 14
          window: 1h
          severity: page
        - burn_rate: 6
          window: 6h
          severity: ticket

    - name: latency_p99
      sli:
        source: prometheus
        query: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{service="payment"}[{{.window}}])) by (le)
          )          
      slo: 2.0
      comparison: less_than