Performance Engineering: A System Optimization Methodology from the SRE Perspective

Team without performance engineering:
  User complains "slow" → Emergency investigation → Find slow SQL → Add index
  → A month later, slow again → Find low cache hit rate → Add cache
  → Another month later, slow again → Find connection pool insufficient → Tune pool
  → Repeat endlessly, system gets more complex, problems keep multiplying

Team with performance engineering:
  Establish performance baseline → Continuous monitoring → Detect P99 slowly rising (before users notice)
  → Proactive analysis → Identify database query pattern change → Optimize query
  → Resolve the issue before users are affected

Relationship of the four golden signals:

  Traffic ↑ → Saturation ↑ → Latency ↑ → Errors ↑
  
  Typical evolution path of performance issues:
  1. Traffic growth
  2. Resource saturation rises
  3. Latency starts degrading
  4. Errors begin after exceeding threshold

  Ideal monitoring catches issues at Step 2-3, not Step 4.

# Common pitfalls in latency measurement
latency_measurement:
  wrong:
    - "Only looking at average latency — averages mask long-tail problems"
    - "Mixing successful and failed requests — failed requests may inflate average"
    - "Only looking at P50 — 50th percentile may not represent the overall picture"

  correct:
    - "Use percentiles: P50/P95/P99/P99.9"
    - "Separate successful and failed request latency"
    - "Focus on the long tail: P99 reflects user experience better than P50"
    - "Use histograms instead of averages — reveals distribution shape"

# Correct latency measurement in Prometheus

# ❌ Wrong: Average
avg(http_request_duration_seconds)

# ✅ Correct: Percentile
histogram_quantile(0.99, 
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# ✅ Correct: Separate success and failure
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])) by (le)
)

Typical latency distribution:

  Request count
  │
  │  ██
  │ ████
  │██████
  │████████
  │██████████
  │█████████████
  │██████████████████████████████████  ████  ██
  └──────────────────────────────────────────────→ Latency
  0ms  10ms  20ms  50ms  100ms  200ms  500ms  1s  5s

  Most requests are 10-50ms, but there's a long tail at 500ms-5s.
  
  P50 = 30ms     ← Looks good
  P95 = 100ms    ← OK
  P99 = 800ms    ← 1% of users have a bad experience
  P99.9 = 3s     ← 0.1% of users are about to timeout

  If you only look at P50, you won't know that 1% of users waited 800ms.

Server resource inventory:
  ├── CPU
  ├── Memory
  ├── Network interface
  ├── Storage devices (disk/SSD)
  ├── Storage controller
  ├── Network controller
  ├── Interconnects (PCIe, NUMA)
  └── Kernel resources (file descriptors, connection tracking table)

# CPU check
# Utilization
top -bn1 | grep "Cpu(s)"
# Saturation (run queue length)
uptime  # load average > CPU core count indicates saturation

# Memory check
# Utilization
free -m
# Saturation (page scanning)
vmstat 1 | grep -E "si|so"  # si/so > 0 indicates swapping
# Errors
dmesg | grep -i "oom"

# Network check
# Utilization
sar -n DEV 1  # Bandwidth usage
# Saturation
ifconfig  # Check drops/overruns
# Errors
ip -s link  # RX/TX errors

# Disk check
# Utilization
iostat -xz 1  # %util
# Saturation
iostat -xz 1  # avgqu-sz (average queue length)
# Errors
dmesg | grep -i "error\|fail" | grep -i "disk\|scsi"

USE Method workflow:

1. List all resources
   → CPU, memory, network, disk...

2. Check U/S/E for each resource
   → Is utilization too high?
   → Is it saturated?
   → Are there errors?

3. Identify the bottleneck
   → The first saturated resource is usually the bottleneck

4. Deep dive
   → Why is this resource saturated?
   → Which process/request is consuming it?

5. Optimize
   → Optimize the largest consumer
   → Or scale up that resource

Complementary relationship between USE and RED:

   RED Method (service perspective)        USE Method (resource perspective)
   ┌───────────────┐                      ┌───────────────┐
   │ Rate          │                      │ CPU           │
   │ Errors        │ ──── why look at resources? ──→       │ Memory        │
   │ Duration      │                      │ Network       │
   └───────────────┘                      │ Disk          │
    "How is the service performing?"       └───────────────┘
                                          "Are resources sufficient?"

   Analysis workflow:
   1. First use RED to discover service issues (high latency? many errors?)
   2. Then use USE to locate resource bottlenecks (which resource is saturated?)
   3. Combine both to find the root cause

# Rate: Requests per second
sum(rate(http_requests_total[5m])) by (service)

# Errors: Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Duration: P99 latency
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (service, le)
)

RED dashboard layout (one row per service):

  Service    | Rate (QPS) | Error Rate | P50 Latency | P99 Latency
  ----------|------------|------------|-------------|------------
  payment   | 2,500      | 0.05%      | 45ms        | 180ms
  order     | 5,000      | 0.02%      | 30ms        | 120ms
  search    | 8,000      | 0.10%      | 80ms        | 350ms  ← P99 is high
  user      | 3,000      | 0.01%      | 25ms        | 90ms

  → Instantly see which service has performance issues

Without a baseline:
  Q: "Is P99 latency of 500ms normal?"
  A: "Don't know, no historical data to compare against."

With a baseline:
  Q: "Is P99 latency of 500ms normal?"
  A: "Baseline P99 is 200ms; current 500ms is 2.5x the baseline, which is abnormal degradation."

performance_baseline:
  service: "payment-service"
  baseline_period: "2026-06-01 ~ 2026-06-30"
  
  # 1. Traffic baseline
  traffic:
    avg_qps: 2500
    peak_qps: 5000
    daily_pattern: "9:00-22:00 is peak, QPS 3000-5000"
    
  # 2. Latency baseline
  latency:
    p50: 45ms
    p95: 120ms
    p99: 200ms
    p999: 500ms
    
  # 3. Resource usage baseline
  resource_usage:
    cpu_avg: 35%
    cpu_peak: 65%
    memory_avg: 4.2GB
    memory_peak: 5.1GB
    disk_io_avg: "500 IOPS"
    network_avg: "50Mbps"
    
  # 4. Error rate baseline
  error_rate:
    avg: 0.03%
    peak: 0.1%
    
  # 5. Dependency performance baseline
  dependencies:
    db_query_p99: "50ms"
    redis_p99: "5ms"
    downstream_api_p99: "100ms"
    
  # 6. Capacity baseline
  capacity:
    max_sustainable_qps: 8000  # Max QPS within SLO
    cpu_per_1000_qps: 12%      # CPU consumed per 1000 QPS
    memory_per_1000_qps: 0.8GB

# Performance baseline calculation
def calculate_baseline(metrics_data, period_days=30):
    """
    Calculate performance baseline from historical monitoring data
    """
    baseline = {}
    
    # 1. Calculate percentile latencies
    latencies = metrics_data['latency']
    baseline['latency'] = {
        'p50': percentile(latencies, 50),
        'p95': percentile(latencies, 95),
        'p99': percentile(latencies, 99),
        'p999': percentile(latencies, 99.9)
    }
    
    # 2. Calculate traffic patterns
    qps_data = metrics_data['qps']
    baseline['traffic'] = {
        'avg_qps': mean(qps_data),
        'peak_qps': max(qps_data),
        'p95_qps': percentile(qps_data, 95),
    }
    
    # 3. Calculate resource usage
    cpu_data = metrics_data['cpu_usage']
    baseline['resource_usage'] = {
        'cpu_avg': mean(cpu_data),
        'cpu_p95': percentile(cpu_data, 95),
        'cpu_peak': max(cpu_data),
    }
    
    # 4. Calculate anomaly thresholds
    # Using 3-sigma rule: values exceeding mean + 3*std are considered anomalous
    baseline['anomaly_thresholds'] = {
        'latency_p99': {
            'normal': baseline['latency']['p99'],
            'warning': baseline['latency']['p99'] * 1.5,
            'critical': baseline['latency']['p99'] * 2.0
        },
        'error_rate': {
            'normal': mean(metrics_data['error_rate']),
            'warning': mean(metrics_data['error_rate']) * 3,
            'critical': mean(metrics_data['error_rate']) * 10
        }
    }
    
    return baseline

baseline_update:
  frequency: "Update monthly"
  
  trigger:
    scheduled: "Update on the 1st of each month, based on previous month's data"
    event_driven: "Re-establish baseline after architecture changes"
  
  versioning:
    - "Keep historical baseline versions for comparison"
    - "Investigate when baseline changes exceed 20%"
    
  alerting_based_on_baseline:
    - alert: "LatencyAnomalyDetected"
      condition: "current_p99 > baseline_p99 * 1.5"
      message: "P99 latency exceeds 1.5x the baseline"
    
    - alert: "ResourceSaturationAnomaly"
      condition: "current_cpu > baseline_cpu_p95 * 1.3"
      message: "CPU usage exceeds 1.3x the baseline P95"

Step 1: Confirm the problem
  → Which metric is abnormal? Latency? Error rate? Resource usage?
  → Is it sudden degradation or gradual degradation?
  → Scope: All users? Some users? Specific APIs?

Step 2: Golden signal check
  → Traffic: Abnormal growth?
  → Errors: What type of errors?
  → Latency: Is P50 or P99 degraded?
  → Saturation: Which resource is near saturation?

Step 3: USE method to locate resource bottleneck
  → CPU saturated? (load > core count)
  → Memory saturated? (swap/OOM)
  → Disk saturated? (high IO wait)
  → Network saturated? (bandwidth/packet loss)

Step 4: RED method to locate service bottleneck
  → Which service has abnormal latency/errors?
  → Is it the service itself or a dependency issue?

Step 5: Distributed tracing to locate specific node
  → Use Jaeger/Zipkin to trace request paths
  → Find the slowest node

Step 6: Deep dive
  → CPU profile (which function consumes CPU?)
  → Flame graph (code-level identification)
  → Slow query log (database level)
  → GC log (memory/GC issues)

Step 7: Validate hypothesis
  → Form hypothesis based on analysis
  → Validate hypothesis with data
  → Fix and verify the effect

# Flame graph generation for Go applications
# 1. Enable pprof
import _ "net/http/pprof"
go func() {
    http.ListenAndServe("localhost:6060", nil)
}()

# 2. Capture CPU profile
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# 3. Generate flame graph
(pprof) web  # Opens flame graph in browser

Flame graph example (simplified):

  ┌─────────────────────────────────────────────────────┐
  │                  main.main                           │
  ├──────────────┬──────────────────────────────────────┤
  │ http.Handle  │          db.Query                    │
  │              ├──────────┬───────────────────────────┤
  │              │ db.Exec  │   pgConn.Query             │
  │              │          ├──────────┬────────────────┤
  │              │          │  network │   parseResult   │
  │              │          │  200ms   │    50ms         │
  └──────────────┴──────────┴──────────┴────────────────┘

  → Instantly see that db.Query takes most of the time
  → Of which network (DB network round-trip) takes 200ms
  → Optimization direction: reduce database queries or use connection pooling

-- 1. Check slow queries
SELECT * FROM pg_stat_statements 
ORDER BY mean_exec_time DESC 
LIMIT 10;

-- 2. Analyze execution plan
EXPLAIN ANALYZE 
SELECT * FROM orders WHERE user_id = 12345;

-- 3. Check index usage
SELECT 
    schemaname,
    relname,
    seq_scan,        -- Full table scan count
    seq_tup_read,    -- Rows read by full table scan
    idx_scan,        -- Index scan count
    idx_tup_fetch    -- Rows fetched by index scan
FROM pg_stat_user_tables
ORDER BY seq_scan DESC;

-- 4. Check lock waits
SELECT 
    pid,
    state,
    wait_event_type,
    wait_event,
    query
FROM pg_stat_activity
WHERE wait_event IS NOT NULL;

Optimization ROI = (Performance improvement × Impact scope × Duration) / (Development cost + Risk cost)

Examples:
  Optimization A: Add index
    Performance improvement: Query from 500ms → 50ms (10x)
    Impact scope: All users
    Duration: Permanent
    Development cost: 0.5 days
    Risk cost: Low
    → ROI is very high, do first

  Optimization B: Refactor data access layer
    Performance improvement: P99 from 200ms → 150ms (25%)
    Impact scope: All users
    Duration: Permanent
    Development cost: 2 weeks
    Risk cost: Medium (refactoring may introduce bugs)
    → ROI is medium, schedule it

  Optimization C: Change programming language
    Performance improvement: Expected 30% but uncertain
    Impact scope: Entire system
    Development cost: 6 months
    Risk cost: High
    → ROI is very low, not recommended

optimization_strategies:
  # Priority 1: Low effort, high return
  database:
    - strategy: "Add/optimize indexes"
      effort: "Low"
      impact: "High"
      example: "Add index on WHERE clause fields"
    
    - strategy: "Optimize slow queries"
      effort: "Low"
      impact: "High"
      example: "Avoid SELECT *, use covering indexes"
    
    - strategy: "Connection pool optimization"
      effort: "Low"
      impact: "Medium"
      example: "Tune pool size and timeout"

  # Priority 2: Medium effort, medium return
  application:
    - strategy: "Cache hot data"
      effort: "Medium"
      impact: "High"
      example: "Cache product info in Redis"
    
    - strategy: "Batch operations instead of loops"
      effort: "Low"
      impact: "Medium"
      example: "Batch INSERT instead of loop INSERT"
    
    - strategy: "Async processing"
      effort: "Medium"
      impact: "Medium"
      example: "Use message queue for non-critical logic"

  # Priority 3: High effort, high return
  architecture:
    - strategy: "Read-write separation"
      effort: "High"
      impact: "High"
      example: "Route reads to replica"
    
    - strategy: "Database sharding"
      effort: "High"
      impact: "High"
      example: "Shard by user ID"
    
    - strategy: "CDN acceleration"
      effort: "Medium"
      impact: "High"
      example: "Serve static assets via CDN"

  # Not recommended
  anti_patterns:
    - "Premature optimization: optimizing without baseline data"
    - "Micro-optimization: spending a week to improve 1ms to 0.9ms"
    - "Over-engineering: considering sharding at 100 QPS"
    - "Only optimizing application layer: ignoring database and infrastructure"

# Performance optimization validation process
optimization_validation:
  before:
    - "Record pre-optimization performance baseline (P50/P95/P99/error rate)"
    - "Ensure monitoring data is complete"
    
  during:
    - "Canary release the optimized code"
    - "Compare metrics before and after optimization"
    
  after:
    - "Verify performance improvement magnitude"
    - "Confirm no functional regression"
    - "Observe stability for at least 24 hours"
    - "Update performance baseline"

  # A/B test validation
  ab_test:
    control_group: "Unoptimized version"
    experiment_group: "Optimized version"
    metrics: ["P50", "P95", "P99", "error_rate", "cpu_usage"]
    duration: "At least 1 hour during high-traffic period"

# Performance regression detection
performance_regression_detection:
  ci_cd_integration:
    # Run performance benchmarks before each release
    pre_deploy:
      - name: "Run performance benchmark"
        script: |
          # Compare current version with baseline performance
          ./benchmark.sh --compare baseline
          # Block deployment if P99 degrades by more than 10%
          if [ $(echo "$P99_REGRESSION > 0.1" | bc) -eq 1 ]; then
            echo "Performance regression detected: P99 degraded by $P99_REGRESSION"
            exit 1
          fi          
  
  continuous_monitoring:
    # Continuously monitor performance metrics
    - alert: "PerformanceRegressionDetected"
      expr: |
        histogram_quantile(0.99,
          sum(rate(http_request_duration_seconds_bucket[1h])) by (le)
        ) > 250        
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "P99 latency exceeds baseline threshold (250ms)"
        description: "Current P99: {{ $value }}ms, baseline P99: 200ms"

# Performance budget example
performance_budget:
  service: "payment-service"
  
  latency_budget:
    total_p99: "500ms"       # End-to-end P99 budget
    breakdown:
      api_gateway: "20ms"    # Gateway
      app_logic: "100ms"     # Application logic
      db_query: "200ms"      # Database
      cache: "10ms"          # Cache
      network: "20ms"        # Network
      downstream: "150ms"    # Downstream dependencies
      # Note: Parts don't sum to the total budget
      # because not all parts peak simultaneously

  resource_budget:
    cpu_per_request: "10ms CPU time"
    memory_per_request: "5MB"
    db_connections: "max 20 concurrent"

  enforcement:
    # Check performance budget in CI
    - "In unit tests, mock dependencies and measure pure application logic time"
    - "In integration tests, measure end-to-end latency"
    - "If budget is exceeded, CI fails"

# Monthly Performance Review Template

## Overview
- Service name: payment-service
- Review period: July 2026
- Owner: Zhang San

## Performance Trends
| Metric | Last Month | This Month | Change | Status |
|------|------|------|------|------|
| P50 latency | 45ms | 42ms | -7% | ✅ Improved |
| P99 latency | 200ms | 230ms | +15% | ⚠️ Degraded |
| Error rate | 0.03% | 0.04% | +33% | ⚠️ Degraded |
| CPU avg | 35% | 42% | +20% | ⚠️ Increased |
| QPS avg | 2500 | 3100 | +24% | Traffic growth |

## Analysis
1. P99 latency degraded 15%, primarily due to 24% QPS growth causing CPU usage to rise
2. Error rate slightly increased, correlated with latency degradation (more timeouts)
3. Current CPU at 42%, still in safe range, but growth trend needs attention

## Action Items
- [ ] Analyze which specific APIs have P99 degradation, see if they can be optimized
- [ ] Evaluate if capacity expansion is needed (current CPU approaching 50% warning line)
- [ ] Do a database slow query analysis next week

performance_culture:
  principles:
    - "Performance is everyone's responsibility, not just SRE's"
    - "Performance issues should be data-driven, not based on feelings"
    - "Establish baselines, continuously measure, proactively discover"
    - "Optimization needs ROI analysis, don't do meaningless optimizations"
  
  developer_practices:
    - "Pay attention to performance impact during code reviews"
    - "Set latency budgets when adding new APIs"
    - "Check execution plans when making database changes"
    - "Include performance assertions in unit tests"
  
  sre_practices:
    - "Maintain performance baselines and monitoring"
    - "Conduct regular performance audits"
    - "Drive cross-team performance optimization"
    - "Establish performance budgets and regression detection"

Myth: Poor performance → Add CPU/memory/nodes
Problems:
  - If it's a slow SQL query, adding machines only delays the problem
  - If it's lock contention, adding machines may make it worse (distributed locks are slower)
  - Costs keep growing, unsustainable

Correct approach:
  → First identify the bottleneck
  → If resources are insufficient → Scale up
  → If it's a code/SQL issue → Optimize code
  → If it's an architecture issue → Change architecture

Myth: Slow → Add cache
Problems:
  - Cache introduces consistency complexity
  - Ineffective when cache hit rate is low
  - Cache itself can become a bottleneck

Correct approach:
  → First optimize the data source (database/indexes)
  → Then consider caching
  → Cache is an optimization technique, not a tool to mask problems

Myth: Average latency is 50ms, no problem!
Problems:
  - Averages mask the long tail
  - 1% of users may have waited 5 seconds

Correct approach:
  → Look at P95/P99
  → Look at latency distribution
  → Focus on the long tail

Myth: System hasn't launched yet, but already obsessing over 1ms optimizations
Problems:
  - Wasting time on non-bottlenecks
  - Increasing code complexity
  - Optimizations may be based on wrong assumptions

Correct approach:
  → First make it work
  → Then make it correct
  → Finally make it fast (based on data)