Runbook Writing Guide: Making Operational Knowledge Reproducible

Incident response without a Runbook:
  Alert triggers → On-Call engineer sees "Redis memory usage 90%"
           → Doesn't know what to do
           → Calls Zhang San (Zhang San wrote this service)
           → Zhang San is offline
           → Calls Li Si
           → Li Si says "It might be a large key, take a look"
           → Engineer spends 20 minutes finding large keys
           → Eventually resolved, but took 40 minutes

Incident response with a Runbook:
  Alert triggers → On-Call engineer sees "Redis memory usage 90%"
           → Alert includes a Runbook link
           → Opens Runbook, follows the steps
           → Step 1: Run redis-cli --bigkeys to scan for large keys
           → Step 2: Find user_session:xxx taking up 500MB
           → Step 3: Run DEL user_session:xxx
           → Recovered in 10 minutes

runbook_value:
  direct_value:
    - "Reduce MTTR: On-Call engineers don't need to figure out what to do from scratch"
    - "Reduce dependency on specific individuals: Knowledge is documented, not just in someone's head"
    - "Standardize operations: Avoid making different decisions for the same problem"
    - "Lower on-call stress: Know there's a reference, reducing panic"

  indirect_value:
    - "Knowledge transfer: New engineers can learn service operations by reading Runbooks"
    - "Identify automation opportunities: Manual steps in Runbooks are candidates for automation"
    - "Expose design flaws: If a Runbook requires many manual steps, the service design may have issues"
    - "Improve service quality: Runbook reviews can reveal monitoring blind spots"

# Runbook: [Alert Name]

## Alert Information
- Alert name: Redis memory usage above 90%
- Severity: Warning
- Trigger condition: redis_memory_used / redis_memory_max > 0.9

## Impact Assessment
- Affected service: User session service
- Impact: May cause session write failures, user logouts
- Severity: Medium

## Diagnosis Steps
1. Check current Redis memory usage:
   ```bash
   redis-cli INFO memory | grep used_memory_human

### Each Section Explained

#### Alert Information

```yaml
alert_information:
  purpose: "Help On-Call quickly understand what alert triggered"
  required_fields:
    - alert_name: "Clear, descriptive alert name"
    - severity: "Info / Warning / Critical"
    - trigger_condition: "Specific condition that triggers the alert"
    - source: "Monitoring system / metric source"

  example:
    alert_name: "Payment API P99 Latency Above 500ms"
    severity: "Critical"
    trigger_condition: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5"
    source: "Prometheus"

impact_assessment:
  purpose: "Help On-Call judge priority and urgency"
  dimensions:
    affected_service: "Which service is affected"
    user_impact: "What users experience"
    business_impact: "Business impact (revenue, reputation)"
    severity: "Combined assessment of the above"

  example:
    affected_service: "Payment service"
    user_impact: "Some users unable to complete payments"
    business_impact: "Estimated ~1000 orders/hour affected, ~$50k/hour"
    severity: "High"

diagnosis_steps:
  purpose: "Help quickly locate the problem root cause"
  principles:
    - "From surface to deep: check the most obvious cause first, then complex ones"
    - "From quick to slow: run fast-checking commands first, then slow ones"
    - "Provide specific commands: not 'check logs', but 'kubectl logs -n prod payment-service | grep ERROR'"

  common_diagnostic_commands:
    - "Check service status: kubectl get pods -n prod"
    - "Check recent events: kubectl get events -n prod --sort-by='.lastTimestamp'"
    - "Check logs: kubectl logs -n prod -l app=payment-service --tail=100"
    - "Check metrics: PromQL queries"
    - "Check dependencies: database, cache, message queue status"

handling_steps:
  purpose: "Provide concrete, actionable resolution steps"
  principles:
    - "Scenario-based: Different root causes, different handling"
    - "Step-by-step: One command per step, with expected output"
    - "Include validation: After each action, verify if it worked"
    - "Include rollback: If action doesn't work, how to roll back"

  structure:
    scenario_1:
      condition: "If XXX is the case"
      steps:
        - step_1: "Execute command A"
        - step_2: "Verify result B"
        - step_3: "Execute command C"
        - step_4: "Verify final state"

    scenario_2:
      condition: "If YYY is the case"
      steps: [...]

escalation:
  purpose: "When Runbook steps don't solve the problem, know who to contact"
  content:
    - escalation_condition: "Under what circumstances to escalate"
    - contacts: "Specific people/teams"
    - contact_method: "Phone / Slack / PagerDuty"
    - escalation_time: "How long before escalating"

  example:
    - condition: "Unresolved after 30 minutes"
      contact: "@payment-team"
      method: "Slack channel #payment-oncall"
    - condition: "Service unavailable"
      contact: "@sre-lead"
      method: "Phone: +86-xxx-xxxx-xxxx"

runbook_quality_checklist:
  completeness:
    - "Alert information is complete and accurate"
    - "Impact assessment covers user and business impact"
    - "Diagnosis steps cover common root causes"
    - "Handling steps cover all scenarios"
    - "Escalation path is clear"

  actionability:
    - "Every step has specific commands, not vague descriptions"
    - "Commands can be copy-pasted and run directly"
    - "Expected output is described for each step"
    - "Validation steps are included after actions"

  safety:
    - "Destructive operations are explicitly marked"
    - "Rollback steps are provided"
    - "Commands that require confirmation are marked"
    - "Risk warnings for high-risk operations"

  timeliness:
    - "Last reviewed within the last 3 months"
    - "Commands and paths are still valid"
    - "Contact information is up to date"
    - "Alert thresholds are still current"

  formatting:
    - "Uses consistent Markdown format"
    - "Code blocks have language tags"
    - "Step numbers are clear"
    - "Includes table of contents for long Runbooks"

# Common Runbook Quality Issues

## Issue 1: Too Vague
❌ Bad: "Check if the database is normal"
✅ Good: "Execute: kubectl exec -it postgres-0 -- psql -U postgres -c 'SELECT count(*) FROM pg_stat_activity;'"

## Issue 2: Missing Validation
❌ Bad: "Restart the service"
✅ Good: "Restart the service: kubectl rollout restart deployment/payment-service
         Verify: kubectl rollout status deployment/payment-service
         Confirm recovery: curl http://payment-service/health"

## Issue 3: Missing Rollback
❌ Bad: "Update the configuration"
✅ Good: "Update the configuration:
         1. Backup current config: kubectl get configmap payment-config -o yaml > /tmp/payment-config-backup.yaml
         2. Update: kubectl edit configmap payment-config
         3. Verify: kubectl get configmap payment-config -o yaml
         4. Rollback if issue: kubectl apply -f /tmp/payment-config-backup.yaml"

## Issue 4: Missing Risk Warning
❌ Bad: "Delete old data"
✅ Good: "⚠️ WARNING: This will delete data older than 30 days, irreversible!
         Confirm before executing: echo 'I confirm data deletion' | base64
         Execute: ./cleanup-old-data.sh --confirm"

runbook_quality_scoring:
  dimensions:
    completeness: 25 points
      - Alert info complete: 5
      - Impact assessment complete: 5
      - Diagnosis steps complete: 5
      - Handling steps complete: 5
      - Escalation path clear: 5

    actionability: 25 points
      - Specific commands: 10
      - Expected output: 5
      - Validation steps: 5
      - Copy-paste ready: 5

    safety: 25 points
      - Risk warnings: 10
      - Rollback steps: 10
      - Confirmation mechanisms: 5

    timeliness: 25 points
      - Reviewed within 3 months: 10
      - Commands valid: 5
      - Contacts current: 5
      - Thresholds current: 5

  grading:
    excellent: "90-100"
    good: "75-89"
    needs_improvement: "60-74"
    unacceptable: "< 60"

runbook_automation_levels:
  level_1_manual:
    description: "All steps are manual"
    example: "Runbook tells you to manually execute kubectl commands"
    suitable_for: "Low-frequency, high-risk operations"

  level_2_semi_automated:
    description: "Some steps are automated scripts"
    example: "Runbook provides a script that auto-diagnoses, but remediation requires manual confirmation"
    suitable_for: "Medium-frequency operations requiring human judgment"

  level_3_fully_automated:
    description: "Entire process is automated"
    example: "Alert triggers → auto-diagnose → auto-remediate → verify → notify"
    suitable_for: "High-frequency, well-understood operations"

  level_4_self_healing:
    description: "System auto-recovers without Runbook"
    example: "Auto-scaling, circuit breaker, failover — no human intervention needed"
    suitable_for: "Common, well-defined failure scenarios"

#!/usr/bin/env python3
"""
Runbook: Redis Memory Usage Above 90%
Automated diagnosis and remediation
"""

import subprocess
import sys
import requests
from typing import Optional

class RedisMemoryRunbook:
    def __init__(self, redis_host: str, redis_port: int = 6379):
        self.redis_host = redis_host
        self.redis_port = redis_port
        self.actions_taken = []

    def diagnose(self) -> dict:
        """Diagnose Redis memory issue"""
        result = {
            "memory_usage": self._get_memory_usage(),
            "large_keys": self._find_large_keys(),
            "memory_growth_trend": self._check_growth_trend(),
        }
        return result

    def remediate(self, diagnosis: dict) -> bool:
        """Execute remediation based on diagnosis"""
        if diagnosis["large_keys"]:
            return self._handle_large_keys(diagnosis["large_keys"])
        elif diagnosis["memory_usage"] > 0.95:
            return self._handle_memory_expansion()
        else:
            print("No action needed yet, monitor closely")
            return True

    def _get_memory_usage(self) -> float:
        """Get current Redis memory usage percentage"""
        cmd = f"redis-cli -h {self.redis_host} -p {self.redis_port} INFO memory"
        output = subprocess.check_output(cmd, shell=True, text=True)
        for line in output.splitlines():
            if "used_memory:" in line:
                used = int(line.split(":")[1])
            if "maxmemory:" in line:
                maxmem = int(line.split(":")[1])
        return used / maxmem if maxmem > 0 else 0

    def _find_large_keys(self) -> list:
        """Find large keys occupying memory"""
        cmd = f"redis-cli -h {self.redis_host} -p {self.redis_port} --bigkeys"
        output = subprocess.check_output(cmd, shell=True, text=True)
        large_keys = []
        # Parse --bigkeys output
        for line in output.splitlines():
            if "Biggest string" in line or "Biggest list" in line:
                # Extract key name and size
                parts = line.split("'")
                if len(parts) >= 2:
                    key = parts[1]
                    large_keys.append({"key": key, "size": "unknown"})
        return large_keys

    def _check_growth_trend(self) -> str:
        """Check memory growth trend via Prometheus"""
        query = f'redis_memory_used{{instance="{self.redis_host}:{self.redis_port}"}}[1h]'
        response = requests.get(
            "http://prometheus:9090/api/v1/query",
            params={"query": query}
        )
        data = response.json()
        if data["data"]["result"]:
            values = [float(v[1]) for v in data["data"]["result"][0]["values"]]
            if len(values) >= 2:
                growth_rate = (values[-1] - values[0]) / values[0]
                if growth_rate > 0.1:
                    return "rapid_growth"
                elif growth_rate > 0:
                    return "slow_growth"
                else:
                    return "stable"
        return "unknown"

    def _handle_large_keys(self, large_keys: list) -> bool:
        """Handle large key scenario"""
        for key_info in large_keys:
            key = key_info["key"]
            print(f"Found large key: {key}")
            print(f"Memory usage: {self._get_memory_usage():.1%}")

            # Risk assessment
            if key.startswith("user_session:"):
                print(f"⚠️  This is user session data, deletion will log out that user")
                confirm = input("Confirm deletion? (yes/no): ")
                if confirm.lower() != "yes":
                    print("Skipped, needs manual handling")
                    continue

            # Delete large key
            cmd = f"redis-cli -h {self.redis_host} -p {self.redis_port} DEL {key}"
            subprocess.run(cmd, shell=True, check=True)
            self.actions_taken.append(f"Deleted key: {key}")
            print(f"✅ Deleted: {key}")

        return True

    def _handle_memory_expansion(self) -> bool:
        """Handle memory expansion scenario"""
        print("Memory usage exceeds 95%, executing expansion...")
        # Call expansion API or script
        # ...
        self.actions_taken.append("Executed memory expansion")
        return True

    def verify(self) -> bool:
        """Verify remediation effectiveness"""
        usage = self._get_memory_usage()
        if usage < 0.85:
            print(f"✅ Memory usage recovered to {usage:.1%}")
            return True
        else:
            print(f"⚠️  Memory usage still high: {usage:.1%}")
            return False

    def report(self):
        """Generate handling report"""
        print("\n=== Runbook Execution Report ===")
        print(f"Redis: {self.redis_host}:{self.redis_port}")
        print(f"Actions taken: {len(self.actions_taken)}")
        for i, action in enumerate(self.actions_taken, 1):
            print(f"  {i}. {action}")
        print("================================\n")

if __name__ == "__main__":
    runbook = RedisMemoryRunbook(redis_host="redis-prod")

    # 1. Diagnose
    diagnosis = runbook.diagnose()
    print(f"Diagnosis: {diagnosis}")

    # 2. Remediate
    success = runbook.remediate(diagnosis)

    # 3. Verify
    if success:
        runbook.verify()

    # 4. Report
    runbook.report()

Playbook (High-level orchestration)
  ├── Runbook 1: Database failover steps
  ├── Runbook 2: Cache cluster rebuild steps
  ├── Runbook 3: Message queue reconfiguration steps
  └── Runbook 4: Traffic switching steps

# Playbook example: Full region failure recovery
playbook:
  name: "Region Failure Recovery"
  trigger: "Region unavailable, affecting all core services"

  phases:
    phase_1_assessment:
      description: "Assess scope of impact"
      runbooks:
        - "Check all service health endpoints"
        - "Confirm region infrastructure status"
      duration: "5 minutes"

    phase_2_failover:
      description: "Failover to backup region"
      runbooks:
        - "Database read/write failover"
        - "Cache cluster rebuild"
        - "Update DNS records"
        - "Switch traffic to backup region"
      duration: "15-30 minutes"

    phase_3_verification:
      description: "Verify recovery"
      runbooks:
        - "Core service health check"
        - "Data consistency check"
        - "User traffic verification"
      duration: "10 minutes"

    phase_4_communication:
      description: "Internal and external communication"
      runbooks:
        - "Notify incident management team"
        - "Update status page"
        - "Customer notification"
      duration: "Throughout"

runbook_version_control:
  storage:
    repository: "Git repository, same repo as service code"
    structure: |
      service-repo/
      ├── src/
      ├── deploy/
      └── runbooks/
          ├── redis-memory-high.md
          ├── payment-api-latency.md
          └── README.md      

  branching:
    strategy: "Runbook changes follow the same branch strategy as code"
    review: "Runbook changes require review from service owner + SRE"

  ci_cd:
    validation: "CI validates Runbook format, links, and command validity"
    deployment: "Runbooks are deployed to internal documentation site"
    notification: "Runbook changes notify the relevant team"

  versioning:
    changelog: "Each Runbook change records what was modified and why"
    history: "Keep historical versions for reference"
    rollback: "Can roll back to previous version if needed"

runbook_maintenance:
  review_frequency:
    critical_services: "Review every 3 months"
    important_services: "Review every 6 months"
    general_services: "Review annually"

  review_checklist:
    - "Are alert thresholds still current?"
    - "Are commands still valid?"
    - "Are contacts still with the team?"
    - "Does the service architecture still match the Runbook?"
    - "Are there new failure modes not covered?"
    - "Were there recent incidents that revealed Runbook gaps?"

  triggers_for_immediate_update:
    - "Postmortem revealed Runbook gap"
    - "Service architecture changed"
    - "On-Call feedback that Runbook was hard to use"
    - "Alert threshold changed"
    - "New team member joined and found Runbook confusing"

  metrics:
    - "Runbook coverage: % of alerts with corresponding Runbooks"
    - "Runbook freshness: % reviewed within the last 3 months"
    - "Runbook usage: How often each Runbook is accessed"
    - "Runbook effectiveness: % of incidents resolved using Runbook steps"

Create → Review → Publish → Use → Review/Update → Archive

┌─────────┐     ┌─────────┐     ┌──────────┐     ┌──────┐     ┌──────────────┐
│ Create  │────→│ Review  │────→│ Publish  │────→│ Use  │────→│ Review/Update│
└─────────┘     └─────────┘     └──────────┘     └──────┘     └──────────────┘
                       ↑                                          │
                       └──────────────────────────────────────────┘
                              (Feedback loop: use → review → update)

                    ┌──────────┐
                    │ Archive  │ ← Service decommissioned or alert retired
                    └──────────┘

# Runbook: [Alert Name]

## Metadata
- Service: [Service name]
- Alert: [Alert name]
- Severity: [Info / Warning / Critical]
- Last reviewed: [YYYY-MM-DD]
- Owner: [Team or individual]

## Alert Information
- **Alert name**: [Descriptive name]
- **Trigger condition**: [Specific condition]
- **Alert source**: [Monitoring system]
- **Default severity**: [Severity level]

## Impact Assessment
- **Affected service**: [Service name]
- **User impact**: [What users experience]
- **Business impact**: [Revenue, reputation, etc.]
- **Severity**: [Combined assessment]

## Diagnosis Steps

### Step 1: [Quick check name]
**Command**:
```bash
[specific command]

[expected output]

[rollback command]

### Template: High CPU Usage

```markdown
# Runbook: Node CPU Usage Above 85%

## Metadata
- Service: Kubernetes cluster nodes
- Alert: NodeCPUHigh
- Severity: Warning
- Last reviewed: 2026-07-10
- Owner: SRE team

## Alert Information
- **Alert name**: Node CPU usage above 85%
- **Trigger condition**: `100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100 > 85`
- **Alert source**: Prometheus / node-exporter
- **Default severity**: Warning

## Impact Assessment
- **Affected service**: All services running on this node
- **User impact**: Possible increased latency for services on the node
- **Business impact**: If sustained, may cause service degradation
- **Severity**: Medium

## Diagnosis Steps

### Step 1: Identify which node has high CPU
**Command**:
```bash
kubectl top nodes --sort-by=cpu

NAME           CPU(cores)   MEMORY(bytes)
node-prod-01   1850m (92%)  12000Mi (75%)
node-prod-02   900m (45%)   8000Mi (50%)

kubectl top pods --all-namespaces --sort-by=cpu --field-selector spec.nodeName=node-prod-01

NAMESPACE     NAME                    CPU(cores)   MEMORY(bytes)
production    payment-service-xxx     800m (40%)   500Mi
production    analytics-worker-xxx   500m (25%)   300Mi

kubectl top nodes

## 8. Runbook Review and Continuous Improvement

### Review Process

```yaml
runbook_review_process:
  trigger:
    - "Scheduled review (every 3/6/12 months)"
    - "Postmortem revealed Runbook gap"
    - "On-Call feedback"
    - "Architecture change"

  participants:
    - "Runbook owner (service owner)"
    - "SRE representative"
    - "On-Call engineer (recent user)"

  review_items:
    technical_accuracy:
      - "Are commands still valid?"
      - "Are paths and endpoints correct?"
      - "Are alert thresholds current?"
      - "Does the architecture match the Runbook?"

    completeness:
      - "Are all common scenarios covered?"
      - "Are diagnosis steps sufficient?"
      - "Is the escalation path clear?"
      - "Are rollback steps included?"

    usability:
      - "Is it easy to follow under pressure?"
      - "Can commands be copy-pasted?"
      - "Are expected outputs described?"
      - "Is the language clear and concise?"

    automation:
      - "Which steps can be automated?"
      - "Are there scripts available?"
      - "Can this be integrated with alerting?"

  output:
    - "Updated Runbook"
    - "Action items (e.g., 'add monitoring for X')"
    - "Automation tickets"
    - "Review date for next review"

runbook_metrics:
  coverage:
    name: "Runbook coverage"
    formula: "Alerts with Runbooks / Total alerts"
    target: "> 90%"

  freshness:
    name: "Runbook freshness"
    formula: "Runbooks reviewed in last 3 months / Total Runbooks"
    target: "> 80%"

  usage:
    name: "Runbook usage rate"
    formula: "Incidents where Runbook was used / Total incidents"
    target: "> 70%"

  effectiveness:
    name: "Runbook effectiveness"
    formula: "Incidents resolved using Runbook / Incidents where Runbook was used"
    target: "> 80%"

  feedback_score:
    name: "On-Call satisfaction"
    formula: "Average rating from On-Call feedback surveys"
    target: "> 4/5"

  continuous_improvement:
    - "Monthly: Review metrics, identify low-scoring Runbooks"
    - "Quarterly: Review and update all critical service Runbooks"
    - "After each major incident: Review if Runbook helped or had gaps"
    - "Collect On-Call feedback after each on-call rotation"