Incident Management and On-Call Mechanism Design

Overview

There’s a saying in SRE: “Systems will fail — the question is whether you’ll be woken up by them or actively managing them.” Incident management is not “dealing with things after they break” — it’s a complete engineering system spanning prevention, detection, response, and learning.

This article systematically covers how to build a practical on-call system across five dimensions: incident grading, on-call rotation, incident response processes, postmortem culture, and alert governance.

For a systematic methodology on incident management, see Google SRE Book - Managing Incidents and Google SRE Book - Postmortem Culture.

1. Incident Severity Grading Standards

Incident management without grading is no management at all — if every incident is treated as urgent, nothing is truly urgent. Proper severity grading is the cornerstone of the on-call system.

P0-P4 Incident Definitions

Level	Definition	Impact Scope	Response SLA	Example
P0	Production service completely unavailable	All users affected	Immediate, <5min	Core service down, database unavailable
P1	Core functionality severely degraded	Large number of users affected	<15min	Payment failure rate spike, API error rate >10%
P2	Partial functionality degraded	Some users affected	<30min	Latency degradation in a region, non-core service anomaly
P3	Potential risk	No direct impact	<2h (business hours)	Disk usage >80%, single node failure
P4	Optimization suggestion	No impact	Next business day	Alert threshold optimization, documentation update

Grading Principles

The essence of grading is resource scheduling priority — ensuring limited human resources are directed at the highest-impact issues first:

Based on user impact, not technical metrics: CPU at 99% is P3, but if it causes user request timeouts, it’s P1
Clear definitions, avoid ambiguity: Is “large number of users” 30% or 50%? It must be quantified
Automatable determination: Ideally, incident severity should be automatically determined through alert rules

# Alert grading rules example
groups:
  - name: incident-grading
    rules:
      # P0: Core service completely unavailable
      - alert: P0ServiceDown
        expr: up{job="critical-service"} == 0
        for: 1m
        labels:
          severity: P0
          page: true           # Trigger phone alert
          escalation: true     # Auto-escalate to manager
        annotations:
          summary: "Core service unavailable"
          runbook: "https://wiki/incident/p0-service-down"

      # P1: Error rate exceeds SLO
      - alert: P1HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.05          
        for: 2m
        labels:
          severity: P1
          page: true
        annotations:
          summary: "Error rate exceeds 5%"
          runbook: "https://wiki/incident/high-error-rate"

      # P2: Latency degradation
      - alert: P2LatencyDegraded
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 0.5          
        for: 5m
        labels:
          severity: P2
          page: false           # Notify only, no phone call
          slack: "#alerts-prod"

2. On-Call Rotation Mechanism Design

Basic Principles

The core goal of the on-call mechanism is: get the right alerts to the right person at the right time. Design should follow these principles:

Sustainable: On-call is not a punishment; no one should be chronically overburdened
Primary-Secondary: Always have Primary + Secondary as two layers of coverage
Clear boundaries: Response time, scope, and escalation paths must be clearly defined
Respect personal life: Nighttime alerts must have compensation to avoid burnout

Rotation Schedule Design

Primary-Secondary Rotation

# On-Call schedule configuration example
oncall_schedule:
  timezone: "Asia/Shanghai"

  # Primary: First responder
  primary:
    rotation: weekly          # Weekly rotation
    members:
      - name: "Zhang San"
        phone: "+86-138xxxx0001"
      - name: "Li Si"
        phone: "+86-138xxxx0002"
      - name: "Wang Wu"
        phone: "+86-138xxxx0003"
    handoff_time: "10:00"     # Handoff every Monday at 10:00
    response_sla:
      P0: "5min"
      P1: "15min"
      P2: "30min"

  # Secondary: Escalation backup
  secondary:
    rotation: weekly
    members:
      - name: "Zhao Liu"
        phone: "+86-138xxxx0004"
      - name: "Sun Qi"
        phone: "+86-138xxxx0005"
    escalation_after: "10min"  # If Primary doesn't respond in 10min, escalate to Secondary

  # Manager Escalation
  manager:
    escalation_after: "30min"  # If Secondary doesn't respond in 30min, escalate to Manager
    members:
      - name: "Manager Zhou"
        phone: "+86-138xxxx0006"

  # Holiday coverage
  holidays:
    - date: "2026-01-01"
      primary: "Zhang San"
      secondary: "Li Si"

Follow-the-Sun Model

For global teams, a Follow-the-Sun model can keep on-call always during daytime hours:

Timezone       08:00 - 20:00 Coverage
──────────────────────────────────────────
Beijing (UTC+8)   |  Rotation A  |
London (UTC+0)              |  Rotation B  |
SF (UTC-8)                            |  Rotation C  |
──────────────────────────────────────────
            24-hour coverage, no nighttime alerts

Schedule Example

Week of 2026-07-06 ~ 2026-07-12

┌──────────┬──────────┬──────────┐
│   Date   │ Primary  │ Secondary│
├──────────┼──────────┼──────────┤
│ Mon 7/6  │ Zhang S. │ Zhao L.  │
│ Tue 7/7  │ Zhang S. │ Zhao L.  │
│ Wed 7/8  │ Zhang S. │ Zhao L.  │
│ Thu 7/9  │ Li S.    │ Zhao L.  │
│ Fri 7/10 │ Li S.    │ Sun Q.   │
│ Sat 7/11 │ Li S.    │ Sun Q.   │
│ Sun 7/12 │ Wang W.  │ Sun Q.   │
└──────────┴──────────┴──────────┘

Handoff time: Daily 10:00 (Asia/Shanghai)
Response tools: PagerDuty / Opsgenie / custom alerting platform

On-Call Health Metrics

The sustainability of the on-call mechanism requires quantitative monitoring:

Metric	Healthy Range	Warning Threshold
Weekly alert count	<10	>20
Nighttime alert count	<2/week	>5/week
Average response time	<5min (P0)	>15min
Average handling time	<1h	>4h
On-call work ratio	<25%	>50%

If alert counts exceed warning thresholds, the system has a systemic problem that needs immediate treatment rather than adding headcount.

3. Incident Response Process

Google proposed a structured incident response model — the SEV model, dividing incident response into five phases:

Detection → Triage → Mitigation → Resolution → Postmortem

1. Detection

Detection is the first and most critical step in incident response. Detection methods include:

Alerting system: SLO-based proactive alerting (not passive thresholds)
User feedback: Customer support tickets, social media sentiment
Active probing: Regular health checks, chaos engineering

Golden metrics for detection:

Metric	Target
MTTD (Mean Time To Detect)	<1min (automated) / <15min (manual)
False positive rate	<5%
Detection rate	>95%

2. Triage

After confirming an alert, the first thing is not to fix the issue but to triage:

Confirm impact scope: Which users are affected? Which features are affected?
Determine severity level: Assign P0-P4 based on severity standards
Decide response mode: Single responder / form virtual team / escalate

3. Mitigation

The core principle of mitigation is: stop the bleeding first, then treat the disease.

Not root cause analysis: RCA is for the Postmortem phase
Restore service: Rollback, traffic switching, scaling, degradation — whatever it takes to restore the user experience

Common mitigation strategies in priority order:

1. Rollback (fastest)     → Deployment issue
2. Traffic switch (next)  → Regional/nodal issue
3. Scale up (slower)      → Capacity shortage
4. Degrade (fallback)     → Disable non-core features to protect core
5. Restart (last resort)  → Temporary recovery, root cause TBD

4. Resolution

After mitigation comes fundamental resolution — fix the root cause, restore full service, validate system state:

# Incident resolution checklist
- [ ] Root cause confirmed and fixed
- [ ] Affected services restored to SLO range
- [ ] All nodes in healthy state
- [ ] Monitoring alerts returned to normal
- [ ] Affected users confirmed recovery
- [ ] Post-incident action items recorded

5. Postmortem

See the next section.

4. Postmortem Culture

The Blameless Principle

The first principle of Postmortem is Blameless — no blame.

“The purpose of a postmortem is to learn from the incident, not to assign blame.” Source: Google SRE Book - Postmortem Culture

Blameless doesn’t mean not pursuing accountability — it means pursuing system accountability rather than personal accountability. If the same mistake would be made by anyone, the problem lies in the system design, not the person.

Postmortem Template

Here is a production-grade Postmortem template:

# Postmortem: [Incident Title]

## Basic Information

| Field | Content |
|------|------|
| Incident severity | P0 |
| Start time | 2026-07-08 14:32 UTC+8 |
| Recovery time | 2026-07-08 15:48 UTC+8 |
| Duration | 1h16min |
| Impact scope | Site-wide order placement unavailable, ~120K users affected |
| Author | Zhang San |
| Reviewer | Manager Zhou |
| Status | Completed |

## Incident Summary

On July 8 at 14:32, the order service became unavailable site-wide due to database connection pool exhaustion.
The incident lasted 76 minutes, affecting approximately 120K users, with an estimated order revenue loss of ~¥2.3M.
Root cause was a configuration change pushed directly to full production without canary release, resulting in incorrect connection pool parameters.

## Timeline

| Time | Event |
|------|------|
| 14:32 | Alert system triggered P0: order service error rate 100% |
| 14:35 | On-call engineer Zhang San confirmed alert, began investigation |
| 14:38 | Confirmed as database connection pool exhaustion, massive request timeouts |
| 14:42 | Discovered a configuration change at 14:25 (connection pool max reduced from 200 to 50) |
| 14:45 | Executed rollback, restored connection pool configuration |
| 14:52 | Order service began recovering, error rate declining |
| 15:10 | Error rate restored to SLO range |
| 15:48 | Full validation completed, incident closed |

## Root Cause Analysis

### Direct Cause
The configuration change reduced the database connection pool maximum from 200 to 50, causing pool exhaustion under high concurrency.

### Deep Causes
1. Change deployed without canary release, pushed directly to full production
2. Configuration changes lacked automated validation (pool minimum should not be below 100)
3. Change approval process didn't cover configuration-type changes
4. No rollback plan for configuration changes

## Impact Assessment

- Affected users: ~120K
- Business impact: ~¥2.3M in lost orders
- Reputation impact: 47 social media complaints
- SLI violation: Order availability SLO (99.9%) error budget fully consumed for July

## Action Items

| # | Action Item | Owner | Due Date | Priority |
|------|--------|--------|---------|--------|
| #1 | Enforce canary release for configuration changes | Li Si | 2026-07-15 | P0 |
| #2 | Automated validation for connection pool parameters (min ≥ 100) | Wang Wu | 2026-07-20 | P0 |
| #3 | Include configuration changes in approval process | Zhao Liu | 2026-07-25 | P1 |
| #4 | Add connection pool utilization alerting for order service | Zhang San | 2026-07-12 | P1 |
| #5 | Create SOP for configuration change rollback | Sun Qi | 2026-07-18 | P2 |

## Lessons Learned

### What went well
- Alert detection was timely (MTTD < 3min)
- On-call response was rapid (confirm + investigate < 10min)
- Rollback decision was decisive, no over-analysis of root cause

### What didn't go well
- Configuration changes lacked canary mechanism
- Connection pool parameters lacked reasonable minimum protection
- Configuration-type changes were outside the approval process

## Appendix
- Related alert record: alert-20260708-1432
- Related change record: change-20260708-1425
- Related monitoring charts: Grafana dashboard "Order Service Overview"

Postmortem Execution Essentials

Timeliness: Complete the postmortem within 48 hours of incident resolution, while memories are fresh
Participation: All relevant personnel should participate, including development, operations, and product
Trackable: Every Action Item must have an owner and due date
Shareable: Postmortem documents should be visible to the entire organization — institutional knowledge
Closed-loop: Regularly review Action Item completion to ensure follow-through

5. Alert Fatigue Governance

The Danger of Alert Fatigue

Alert fatigue is the number one killer of on-call. When alert volume exceeds human processing capacity, on-call engineers start ignoring alerts — and that’s the most dangerous moment. The root cause of many major incidents includes “ignored alerts.”

Identifying Alert Fatigue

# Weekly total alert volume trend
sum by (week) (rate(ALERTS_FIRED[7d]))

# Alert noise ratio (false positive rate = fired but unacknowledged alerts)
sum(ALERTS_FIRED{alertstate="firing"}) / sum(ALERTS_FIRED) * 100

# Alert response time trend (upward trend means alerts are being ignored)
avg_over_time(ALERTS_ACK_TIME[7d])

Alert Tiering Strategy

Not all alerts need to wake someone up. Establish a three-tier alert classification:

Tier	Notification Method	Trigger Condition	Example
Page	Phone + SMS	Incidents affecting SLO	Service unavailable, error rate exceeds threshold
Ticket	Ticket system	Needs handling but not urgent	Disk usage 80%, single node failure
Info	Slack/DingTalk	For awareness only	Deployment completed, scaling triggered

# Alert routing configuration example (Alertmanager)
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 10s          # Initial alert wait time
  group_interval: 30s     # Same-group alert interval
  repeat_interval: 4h     # Repeat notification interval

  routes:
    # P0/P1: Immediate phone notification
    - matchers: ['severity=~"P0|P1"']
      receiver: 'pagerduty-critical'
      group_wait: 0s
      repeat_interval: 30m

    # P2: Business hours notification
    - matchers: ['severity="P2"']
      receiver: 'slack-alerts'
      group_wait: 30s
      repeat_interval: 2h
      active_time_intervals:
        - business-hours

    # P3/P4: Log only
    - matchers: ['severity=~"P3|P4"']
      receiver: 'null'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: '<key>'
        severity: critical
        send_resolved: true

  - name: 'slack-alerts'
    slack_configs:
      - api_url: '<webhook-url>'
        channel: '#alerts-prod'
        send_resolved: true

  - name: 'null'

time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - { start_time: '09:00', end_time: '18:00' }

Noise Reduction Strategies

1. Alert Aggregation

Multiple alerts triggered by the same root cause should be aggregated into one:

# Alertmanager grouping configuration
route:
  group_by: ['cluster', 'service']  # Group by cluster + service
  group_wait: 10s                     # Wait 10s to collect same-group alerts
  group_interval: 30s

2. Inhibition Rules

When a high-severity alert fires, inhibit lower-severity alerts:

# Alertmanager inhibition rules
inhibit_rules:
  # When P0 service is down, inhibit P2 latency alerts for that service
  - source_matchers: ['severity="P0"', 'alertname="ServiceDown"']
    target_matchers: ['severity="P2"', 'alertname="HighLatency"']
    equal: ['service']

  # When a node is down, inhibit all application alerts on that node
  - source_matchers: ['alertname="NodeDown"']
    target_matchers: ['severity=~"P2|P3"']
    equal: ['node']

3. Alert SLO

Set SLOs for alerts themselves and review them regularly:

# Alert quality SLO
alert_quality_slo:
  precision: 0.95        # 95% of alerts are real issues (false positive rate < 5%)
  recall: 0.95           # 95% of real incidents are detected by alerts
  weekly_volume: 20      # No more than 20 alerts per week
  ack_time_p90: 5m       # 90% of P0/P1 alerts acknowledged within 5 minutes

4. Regular Alert Audits

#!/bin/bash
# Monthly alert audit script: find the noisiest alert rules
# For use with Prometheus + Alertmanager

ALERTMANAGER_API="http://alertmanager:9093/api/v2"

echo "=== Monthly Alert Audit Report ==="
echo "Period: $(date -d '30 days ago' +%Y-%m-%d) ~ $(date +%Y-%m-%d)"
echo ""

# Top 10 noisiest alerts (most fired but not escalated)
echo "=== Top 10 Noisy Alerts ==="
curl -s "$ALERTMANAGER_API/alerts" | \
  jq -r '.[] | .labels.alertname' | \
  sort | uniq -c | sort -rn | head -10

echo ""
echo "=== Never Acknowledged Alerts (Likely False Positives) ==="
curl -s "$ALERTMANAGER_API/alerts?active=true" | \
  jq -r '.[] | select(.status.state != "suppressed") | .labels.alertname' | \
  sort | uniq -c | sort -rn | head -10

echo ""
echo "=== Recommended Actions ==="
echo "1. Alerts fired >50 times without escalation → Consider noise reduction or downgrade to Ticket"
echo "2. Never acknowledged alerts → Review for false positives, consider removal"
echo "3. Alerts firing in clusters at the same time → Consider aggregation or inhibition rules"

Summary

The incident management and on-call system is a core SRE practice. Its design goals are:

Dimension	Design Goal
Incident grading	Quantify impact, allocate response resources rationally
On-call rotation	Sustainable, primary-secondary, respects personal life
Incident response	Stop bleeding first, structured process
Postmortem	Blameless, focused on system improvement
Alert governance	Precise, actionable, not fatiguing

Core principle: The ultimate goal of on-call is not “putting out fires faster” but “making fires fewer.” Every incident is an opportunity to improve the system. Every postmortem should produce action items that reduce future incidents. When your alerts drop from 50 per week to 5 per week, and every single one is worth responding to, your on-call system is truly mature.

For the foundational SRE measurement framework — SLI/SLO and error budgets — see the previous article.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Google SRE Book - Managing Incidents — Google SRE Team, referenced for Google SRE Book - Managing Incidents
Google SRE Book - Postmortem Culture — Google SRE Team, referenced for Google SRE Book - Postmortem Culture

Overview#

1. Incident Severity Grading Standards#

P0-P4 Incident Definitions#

Grading Principles#

2. On-Call Rotation Mechanism Design#

Basic Principles#

Rotation Schedule Design#

Primary-Secondary Rotation#

Follow-the-Sun Model#

Schedule Example#

On-Call Health Metrics#

3. Incident Response Process#

1. Detection#

2. Triage#

3. Mitigation#

4. Resolution#

5. Postmortem#

4. Postmortem Culture#

The Blameless Principle#

Postmortem Template#

Postmortem Execution Essentials#

5. Alert Fatigue Governance#

The Danger of Alert Fatigue#

Identifying Alert Fatigue#

Alert Tiering Strategy#

Noise Reduction Strategies#

1. Alert Aggregation#

2. Inhibition Rules#

3. Alert SLO#

4. Regular Alert Audits#

Summary#

References & Acknowledgments#