Incident Response Framework: SEV Classification and Escalation Flow

# Incident severity determination matrix
severity_matrix:
  dimensions:
    - user_impact:        # User impact
        none: 0
        minimal: 1        # <1% of users affected
        moderate: 2       # 1-10% of users affected
        significant: 3    # 10-50% of users affected
        severe: 4         # >50% of users affected

    - business_impact:    # Business impact
        none: 0
        low: 1            # Non-core functionality, no revenue impact
        medium: 2         # Core functionality degraded, minor revenue impact
        high: 3           # Core functionality failed, clear revenue loss
        critical: 4       # Site-wide unavailability, severe revenue loss

    - duration:           # Duration (elapsed or estimated)
        unknown: 3        # Unknown duration treated as high risk
        brief: 1          # <5 minutes
        short: 2          # 5-30 minutes
        medium: 3         # 30 minutes - 2 hours
        long: 4           # >2 hours

  # The highest dimension determines SEV level
  rule: "max(user_impact, business_impact) + duration_bonus"
  # duration_bonus: if duration exceeds expectations, SEV is upgraded one level

# Prometheus alert auto-grading
groups:
  - name: incident-severity
    rules:
      # SEV1: Critical service completely unavailable
      - alert: SEV1CriticalServiceDown
        expr: |
          up{job="payment-service",env="production"} == 0          
        for: 1m
        labels:
          severity: SEV1
          page: true
          escalation: true
          auto_bridge: true       # Auto-create conference bridge
        annotations:
          summary: "Payment service completely unavailable"
          impact: "All users unable to pay, direct revenue loss"
          runbook: "https://wiki/runbooks/payment-service-down"
          ic_rotation: "primary-oncall"

      # SEV2: Error rate exceeds threshold
      - alert: SEV2HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="payment-service",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="payment-service"}[5m])) > 0.05          
        for: 2m
        labels:
          severity: SEV2
          page: true
          escalation: false
        annotations:
          summary: "Payment service error rate exceeds 5%"
          impact: "Some users experiencing payment failures"
          runbook: "https://wiki/runbooks/high-error-rate"

      # SEV3: Resource water level alert
      - alert: SEV3DiskUsageHigh
        expr: |
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 80          
        for: 5m
        labels:
          severity: SEV3
          page: false
          notify: "#alerts-warn"
        annotations:
          summary: "Disk usage exceeds 80%"
          runbook: "https://wiki/runbooks/disk-cleanup"

Scenario: SEV3 disk alert
  → Investigation reveals disk is about to fill → Service may crash
  → Upgrade to SEV2
  → Disk fills up, service starts erroring
  → Upgrade to SEV1

                    ┌──────────────┐
                    │     IC       │  ← Unified command, no hands-on operations
                    │  Commander   │
                    └──────┬───────┘
                           │
               ┌───────────┼───────────┐
               │           │           │
        ┌──────▼──────┐ ┌──▼───┐ ┌─────▼──────┐
        │    Ops      │ │Scribe│ │   Comms    │
        │ Operations  │ │Notes │ │  Comms     │
        └──────┬──────┘ └──────┘ └────────────┘
               │
        ┌──────▼──────┐
        │    SME      │  ← Join as needed, provide expert judgment
        │  Expert     │
        └─────────────┘

ic_principles:
  do:
    - "Maintain a global perspective; don't get lost in details"
    - "Give clear instructions: who does what by when"
    - "Sync progress regularly: update status every 15 minutes"
    - "Encourage information sharing: 'If you find something, tell me'"
    - "Make decisions and own the consequences: 'Let's rollback first, then investigate'"

  dont:
    - "Don't perform hands-on technical operations"
    - "Don't let multiple people operate the same system simultaneously"
    - "Don't execute irreversible operations without confirmation"
    - "Don't ignore anyone's information — even if it seems unimportant"

IC: "First check recent change records"
Ops: "At 10:00 a developer submitted a config change, modifying the database connection timeout"
IC: "Rollback that change"
Ops: "Rolling back... rollback complete, error rate is dropping"
Ops: "Error rate back to normal, P99 latency recovering to 180ms"
IC: "Good, observe for 10 minutes to confirm stability"

## [SEV1] Payment Service Unavailable - Status Update #3

**Time**: 2026-07-10 15:00
**Level**: SEV1
**Status**: Investigating

### Current Situation
The payment service started returning large numbers of 5xx errors at 14:30, currently affecting approximately 15% of payment requests.
The SRE team is investigating and has初步 identified a correlation with a database configuration change at 14:00.

### In Progress
- Rolling back the database configuration change
- Estimated recovery within 15 minutes

### Impact Assessment
- Affected users: ~15% of payment requests failing
- Estimated revenue impact: ~¥XXK/hour
- Data impact: No data loss

### Next Update
15:15

## Service Disruption - Payment Functionality

We've detected that some users are experiencing issues with the payment functionality. Our technical team is urgently working on a fix.

**Impact scope**: Some users' payment requests may fail
**Start time**: 2026-07-10 14:30 UTC+8
**Current status**: Investigating

We will provide updates every 15 minutes. Thank you for your patience.

---
2026-07-10 14:35 - Issue confirmed, team is working on it
2026-07-10 14:50 - Root cause identified, fix being applied
2026-07-10 15:05 - Fix applied, verifying recovery
2026-07-10 15:15 - Service has been restored to normal

# Incident Timeline Record

## Basic Information
- Incident ID: INC-2026-0710-001
- Level: SEV1
- Start time: 14:30
- Recovery time: 15:15
- Duration: 45 minutes
- IC: Zhang San
- Ops: Li Si
- Comms: Wang Wu

## Timeline

| Time | Event | Operator | Source |
|------|------|--------|------|
| 14:30 | Prometheus alert: Payment service error rate >5% | Auto alert | AlertManager |
| 14:32 | Zhang San responded to alert, confirmed as SEV1 | Zhang San | IM group |
| 14:33 | Zhang San formed response team: IC=Zhang San, Ops=Li Si, Comms=Wang Wu | Zhang San | Conference bridge |
| 14:35 | Li Si began investigation, checking Grafana monitoring | Li Si | Operation log |
| 14:37 | Li Si noticed abnormal database connection count | Li Si | Grafana |
| 14:40 | Li Si checked change records, found 14:00 database config change | Li Si | Git log |
| 14:42 | IC decided to rollback config change | Zhang San | Conference bridge |
| 14:45 | Li Si executed rollback | Li Si | kubectl |
| 14:48 | Error rate started dropping | Li Si | Grafana |
| 14:52 | Error rate back to normal, P99 latency still elevated | Li Si | Grafana |
| 15:00 | Latency returned to normal | Li Si | Grafana |
| 15:05 | Observed for 5 minutes, confirmed stable | Zhang San | - |
| 15:15 | IC declared incident resolved | Zhang San | IM group |

Incident lasts >2 hours:
  → IC hands off to backup IC
  → Ops hands off to backup Ops
  → 5-minute briefing during handoff: current status, what's being done, next steps
  → Outgoing responder rests at least 4 hours before returning

Incident lasts >6 hours:
  → Full team rotation
  → Consider requesting cross-team support

# Escalation matrix
escalation_matrix:
  SEV1:
    step_1:
      delay: 0min
      targets:
        - role: "Primary On-Call"
          contact: "phone + pagerduty"
          timeout: 5min

    step_2:
      delay: 5min   # Primary didn't respond
      targets:
        - role: "Secondary On-Call"
          contact: "phone + pagerduty"
          timeout: 5min

    step_3:
      delay: 10min  # Secondary didn't respond
      targets:
        - role: "SRE Team Lead"
          contact: "phone"
          timeout: 5min

    step_4:
      delay: 15min  # Still no progress
      targets:
        - role: "Service Owner"
          contact: "phone"
        - role: "Engineering Director"
          contact: "phone + im"

    step_5:
      delay: 30min  # Still not resolved
      targets:
        - role: "CTO"
          contact: "phone"

    step_6:
      delay: 60min  # Long unresolved
      targets:
        - role: "All Hands"
          contact: "mass notification"

  SEV2:
    step_1:
      delay: 0min
      targets:
        - role: "Primary On-Call"
          contact: "pagerduty + im"
          timeout: 15min

    step_2:
      delay: 15min
      targets:
        - role: "Secondary On-Call"
          contact: "pagerduty + phone"

    step_3:
      delay: 30min   # No progress
      targets:
        - role: "SRE Team Lead"
          contact: "im"

  SEV3:
    step_1:
      delay: 0min
      targets:
        - role: "On-Call"
          contact: "im only"
      # Handle during business hours, no phone calls

# Escalation Notification Template

## SEV1 Escalation: Payment Service Unavailable

**Incident summary**: Payment service has been returning large numbers of 5xx errors since 14:30, ~15% of users affected
**Current status**: Been investigating for 15 minutes, identified database config change, rollback in progress
**Approaches tried**:
1. Checked service logs - found database connection timeouts
2. Checked change records - 14:00 database config change
3. Rolling back configuration

**Help needed**: Need DBA team to confirm if rollback plan is safe
**IC**: Zhang San
**Conference bridge**: https://meet.example.com/incident-001
**Timeline document**: https://wiki/incidents/INC-2026-0710-001

The War Room is the command center for incident response — it can be a physical conference room or a virtual conference bridge.

War Room rules:
  1. Only response roles enter the War Room (IC, Ops, Comms, Scribe, SME)
  2. Others get information through the IM group, don't enter the War Room
  3. Only discuss the current incident in the War Room, no other topics
  4. All operational instructions come from the IC, no self-directed operations
  5. All key decisions and operations are recorded by the Scribe

# Incident Status Update Template

## [SEV Level] Incident Title - Update #N

**Time**: YYYY-MM-DD HH:MM
**Level**: SEVX
**Status**: [Investigating / In Progress / Recovering / Resolved]
**Duration**: XX minutes

### Current Situation
[One paragraph describing current status]

### Impact Scope
- Affected users: [percentage/count]
- Affected features: [list]
- Business impact: [quantified]

### In Progress
1. [Current action 1]
2. [Current action 2]

### Next Steps
1. [Planned action 1]
2. [Planned action 2]

### Estimated Recovery Time
[Estimated time or "Unable to estimate"]

### Next Update
[Time]

communication_discipline:
  war_room:
    - "Only talk about things related to the incident"
    - "Speak up — even if you think it's not important"
    - "If in doubt, ask the IC directly"
    - "Announce before operating, confirm after operating"

  im_channel:
    - "Only the Comms role posts status updates"
    - "Use thread replies for technical discussions"
    - "Don't spam the channel — important information gets buried"
    - "Screenshots must include timestamps"

  examples:
    good:
      - "Ops: I executed a rollback on server-01, error rate dropped from 15% to 2%"
      - "IC: Acknowledged. Li Si, continue observing for 5 minutes. Wang Wu, send a status update."
    bad:
      - "Ops: I think it might be a database issue"  # ← Unfounded speculation
      - "Someone: Oh no it's down again!!!"          # ← Emotional outburst
      - "Someone: This happened last time too"      # ← Doesn't help current recovery

# Timeline recording tool selection
timeline_tools:
  real_time:
    - tool: "Dedicated incident management tool (e.g., FireHydrant, Rootly)"
      pros: "Structured, auto-correlates alerts and changes"
      cons: "Requires pre-deployment and integration"

    - tool: "Google Doc / collaborative document"
      pros: "Zero deployment cost, everyone can edit simultaneously"
      cons: "Unstructured, requires manual organization"

    - tool: "IM group messages + Bot"
      pros: "Responders already use IM, no extra effort"
      cons: "Information scattered, requires later organization"

  recommended: "IM group + Bot auto-recording, with collaborative document for structured organization"

# Auto-collect timeline events from multiple data sources
class TimelineCollector:
    def __init__(self, incident_id):
        self.incident_id = incident_id
        self.events = []

    def collect_from_alertmanager(self, start_time, end_time):
        """Collect alert events from AlertManager"""
        alerts = query_alertmanager(start_time, end_time)
        for alert in alerts:
            self.events.append({
                "time": alert["starts_at"],
                "type": "alert",
                "description": alert["annotations"]["summary"],
                "source": "AlertManager"
            })

    def collect_from_git(self, start_time, end_time):
        """Collect change events from Git"""
        commits = query_git_log(start_time, end_time)
        for commit in commits:
            self.events.append({
                "time": commit["timestamp"],
                "type": "change",
                "description": f"Code change: {commit['message']} ({commit['author']})",
                "source": "Git"
            })

    def collect_from_ci_cd(self, start_time, end_time):
        """Collect deployment events from CI/CD"""
        deploys = query_deployments(start_time, end_time)
        for deploy in deploys:
            self.events.append({
                "time": deploy["timestamp"],
                "type": "deploy",
                "description": f"Deployment: {deploy['service']} {deploy['version']}",
                "source": "CI/CD"
            })

    def collect_from_chat(self, incident_channel, start_time):
        """Collect manual records from IM group"""
        messages = query_chat_history(incident_channel, start_time)
        for msg in messages:
            if msg.get("type") in ["action", "decision", "finding"]:
                self.events.append({
                    "time": msg["timestamp"],
                    "type": msg["type"],
                    "description": msg["text"],
                    "source": msg["user"]
                })

    def generate_timeline(self):
        """Generate a sorted timeline"""
        return sorted(self.events, key=lambda x: x["time"])

Good timeline recording:
  14:35  Checked Grafana monitoring, found DB connections spiked from 50 to 500  [Li Si]
  14:37  Confirmed connection spike aligns with 14:00 config change              [Li Si]
  14:42  IC decided to rollback config change                                     [Zhang San]
  14:45  Executed kubectl rollout undo deployment/payment-svc                   [Li Si]
  14:48  Error rate dropped from 15% to 2%                                      [Li Si]

Bad timeline recording:
  14:35  Looked at monitoring             ← Looked at what? Found what?
  14:42  Decided to rollback              ← Who decided? Why?
  14:45  Operated                         ← Operated what? On which system?

Wrong approach:
  Incident occurs → Find root cause → 30 minutes searching → Found it → Fix → Recover
  Total time: 45 minutes

Right approach:
  Incident occurs → Quick assessment → Rollback recent changes → Recover → Then find root cause
  Total time: 10 minutes

Incident occurs
  │
  ├─ Changes in the last 30 minutes?
  │   ├─ Yes → Rollback changes → Observe if recovered
  │   │         ├─ Recovered → Incident ends, analyze root cause afterward
  │   │         └─ Not recovered → Continue investigation
  │   │
  │   └─ No → Check monitoring metrics
  │             │
  │             ├─ Resource exhaustion (CPU/memory/disk/connections)?
  │             │   → Scale up/cleanup/restart → Observe if recovered
  │             │
  │             ├─ Dependency service anomaly?
  │             │   → Switch to backup/degrade features → Observe if recovered
  │             │
  │             ├─ Traffic anomaly (spike/attack)?
  │             │   → Rate limit/WAF block → Observe if recovered
  │             │
  │             └─ No obvious clues
  │                 → Restart service (last resort) → Observe if recovered
  │                 → If still not recovered, escalate per escalation matrix

# Recovery verification checklist
recovery_verification:
  technical_checks:
    - "Error rate has returned to normal levels (<0.1%)"
    - "Latency back within SLO (P99 < 200ms)"
    - "All Pods in Running state"
    - "Database connection count back to normal"
    - "Queue backlog has been consumed"

  business_checks:
    - "User complaint volume dropped to normal levels"
    - "Core business metrics recovered (e.g., checkout success rate >99.9%)"
    - "Customer service team confirms user feedback is normal"

  monitoring_checks:
    - "Alerts have auto-cleared"
    - "All metrics on monitoring dashboards are normal"
    - "Black-box monitoring probes pass"

  observation_period:
    duration: "10-15 minutes"
    purpose: "Confirm it's not an intermittent recovery"
    action: "After IC declares recovery, continue observing for 30 minutes"

Post-recovery wrap-up steps:

1. IC declares incident resolved (IM group + conference bridge)
2. Comms publishes recovery announcement (status page + IM group)
3. Scribe finalizes the timeline
4. IC confirms all responders can rest
5. Create Postmortem ticket, set review date (within 3-5 business days)
6. If SEV1/SEV2, submit preliminary Postmortem within 24 hours
7. If there are temporary fixes (e.g., degraded features), record for later permanent fix

# Incident Response Tabletop Exercise

## Scenario
Friday afternoon at 17:00, the payment service suddenly starts returning large numbers of 5xx errors. Simultaneously, customer service reports a surge of user complaints about payment failures.

## Exercise Flow

### T+0 minutes: Alert triggered
- Facilitator: "17:00, AlertManager triggers SEV2 alert, payment service error rate 8%. Who is Primary On-Call?"
- Participant (playing On-Call): "I'm Primary, I received the alert."

### T+2 minutes: Response initiated
- Facilitator: "What do you do?"
- Participant: "Check Grafana to confirm alert accuracy, then..."
- Facilitator probes: "How many minutes until you respond? What if you don't respond in 5 minutes?"

### T+5 minutes: Grading and team formation
- Facilitator: "You confirm error rate is indeed 8%, affecting 8% of users. How do you grade it?"
- Participant: "SEV2. I need to form a response team..."

### T+15 minutes: Investigation
- Facilitator: "You checked monitoring and found database connections are normal, but Redis latency is spiking. What do you do?"
- Participant: "Check Redis status..."

### T+30 minutes: Escalation
- Facilitator: "It's been 30 minutes, problem still not solved. What should happen now?"
- Participant: "Per the escalation matrix, 30 minutes with no progress requires escalation to the SRE lead."

### Exercise Debrief
- Which steps went smoothly?
- Which steps got stuck? Why?
- What needs improvement in process/tools/communication?