MTTR 从 40 分钟到 8 分钟：用四层防御把故障定位提速 5 倍

MTTD: 3 min (8%)   ← 监控体系成熟后提升空间有限
MTTA: 5 min (13%)  ← On-Call 机制优化后趋于稳定
MTTI: 25 min (63%) ← 最大变量，优化空间最大
MTTR: 5 min (16%)  ← 恢复操作本身通常很快

# 告警分级脚本（Prometheus AlertManager webhook 后处理）
# 核心逻辑：根据 service criticality 和告警类型自动分级

CRITICAL_SERVICES = {
    "payment-gateway", "order-center", "user-auth",
    "search-engine", "recommendation-engine"
}

def classify_alert(alert: dict) -> str:
    """根据服务关键度和告警类型返回优先级"""
    service = alert.get("labels", {}).get("service", "")
    alertname = alert.get("labels", {}).get("alertname", "")
    
    # P0: 核心服务 + 可用性告警 → 立即打电话
    if service in CRITICAL_SERVICES:
        if "down" in alertname.lower() or "error_rate" in alertname.lower():
            return "P0"
    
    # P1: 核心服务 + 延迟告警 → 短信+IM
    if service in CRITICAL_SERVICES:
        if "latency" in alertname.lower() or "p99" in alertname.lower():
            return "P1"
    
    # P2: 非核心服务告警 → 仅 IM 通知
    return "P2"

# 效果：日均告警从 500+ 降到 80 条，P0 告警准确率从 5% 提升到 70%

# Prometheus SLO 告警规则示例
# 基于错误预算消耗速率，而非固定阈值
groups:
  - name: slo-based-alerts
    rules:
      # 快速消耗：1小时内消耗了 2 小时的错误预算
      - alert: SLOBurnRateFast
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 14.4 * 0.001  # 14.4 = 2h / (30d * 24h) * 100%          
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SLO 错误预算快速消耗"
          description: "过去 1 小时错误预算消耗速率超过正常 14.4 倍"
      
      # 慢速消耗：6小时内消耗了 3 天的错误预算
      - alert: SLOBurnRateSlow
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 3 * 0.001          
        for: 30m
        labels:
          severity: warning

[P0] payment-gateway 错误率异常
服务: payment-gateway
环境: production
当前错误率: 12.3% (阈值: 1%)
影响范围: /api/v1/pay/create, /api/v1/pay/refund
最近变更: config-center 推送 payment-timeout 配置 (5min前)
值班: 张三 → 李四 (升级)
Runbook: https://wiki.internal/runbook/payment-gateway-error
Dashboard: https://grafana.internal/d/payment-gateway

P0 故障 → 5分钟无人响应 → 自动升级到 backup on-call
        → 10分钟无人响应 → 升级到 team lead
        → 15分钟无人响应 → 升级到部门负责人

# 告警路由配置（由部署系统自动维护）
routes:
  - matchers: ['service="payment-gateway"']
    receiver: "payment-team-oncall"
    group_by: ["service", "severity"]
    # 平日发到 team channel + oncall 个人
    receiver_config:
      webhook: "https://im.internal/alert/payment-team"
      sms: "+86138xxxx1234"
    # 升级链
    routes:
      - matchers: ['severity="critical"']
        receiver: "payment-team-lead"
        group_wait: 5m  # 5分钟无响应升级

# 示例：统一标签注入中间件（Go 语言）
# 在所有请求入口注入统一标签，确保 Metrics/Logs/Traces 可关联

func TracingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        
        // 从请求头提取或生成 trace_id
        traceID := r.Header.Get("X-Trace-ID")
        if traceID == "" {
            traceID = generateTraceID()
            w.Header().Set("X-Trace-ID", traceID)
        }
        
        // 注入统一标签到 context
        ctx = context.WithValue(ctx, "trace_id", traceID)
        ctx = context.WithValue(ctx, "service", "payment-gateway")
        ctx = context.WithValue(ctx, "version", buildVersion)
        
        // Metrics 自动记录（含 trace_id 标签）
        metrics.RequestCounter.WithLabelValues(
            r.URL.Path, r.Method, "200",
        ).Inc()
        
        // 日志自动带上 trace_id
        logger.Info(ctx, "request received",
            "path", r.URL.Path,
            "method", r.Method,
        )
        
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

#!/bin/bash
# diag-service-down.sh — 服务不可用快速诊断
# 用法: ./diag-service-down.sh <namespace>/<service>

set -euo pipefail

TARGET="${1:?用法: $0 <namespace>/<service>}"
NAMESPACE="${TARGET%%/*}"
SERVICE="${TARGET##*/}"

echo "=========================================="
echo "诊断目标: ${NAMESPACE}/${SERVICE}"
echo "诊断时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "=========================================="

# 1. Pod 状态检查
echo -e "\n[1] Pod 状态"
kubectl get pods -n "$NAMESPACE" -l app="$SERVICE" \
  -o wide 2>/dev/null || echo "  无匹配 Pod，检查 label"

# 2. Pod 事件（最近 10 分钟）
echo -e "\n[2] Pod 事件（最近 10 分钟）"
kubectl get events -n "$NAMESPACE" \
  --field-selector involvedObject.kind=Pod \
  --sort-by='.lastTimestamp' 2>/dev/null | tail -20

# 3. Endpoints 检查
echo -e "\n[3] Endpoints"
ENDPOINTS=$(kubectl get endpoints "$SERVICE" -n "$NAMESPACE" \
  -o jsonpath='{.subsets[*].addresses[*].ip}' 2>/dev/null)
if [ -z "$ENDPOINTS" ]; then
    echo "  ⚠️  无可用 Endpoints！"
    echo "  可能原因: Pod 未就绪 / Readiness 探针失败 / 标签不匹配"
else
    echo "  可用 Endpoints: ${ENDPOINTS}"
fi

# 4. 最近部署历史
echo -e "\n[4] 最近部署"
kubectl rollout history deployment/"$SERVICE" -n "$NAMESPACE" \
  2>/dev/null | tail -5

# 5. 配置变更检查
echo -e "\n[5] ConfigMap 最近变更"
kubectl get configmap -n "$NAMESPACE" \
  -l app="$SERVICE" \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.annotations.last-modified}{"\n"}{end}' \
  2>/dev/null

# 6. 关联节点状态
echo -e "\n[6] 节点状态"
kubectl get pods -n "$NAMESPACE" -l app="$SERVICE" \
  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' \
  2>/dev/null | sort -u | while read node; do
    echo "  节点 $node:"
    kubectl describe node "$node" 2>/dev/null | grep -A5 "Conditions:" | head -8
done

echo -e "\n=========================================="
echo "诊断完成。建议下一步:"
echo "  - 如 Pod 频繁重启: kubectl logs --previous"
echo "  - 如 Endpoints 为空: 检查 Readiness 探针配置"
echo "  - 如有近期部署: 考虑回滚 kubectl rollout undo"
echo "=========================================="

# Runbook: payment-gateway 错误率飙升

## 症状
- 告警: SLOBurnRateFast / payment-gateway-error-rate
- 现象: /api/v1/pay/create 接口 5xx 错误率 > 5%

## 快速诊断（2分钟内完成）
1. 执行 `./diag-error-rate.sh prod/payment-gateway`
2. 检查"最近部署"部分是否有变更
3. 检查"依赖服务状态"部分是否有红色标记

## 处置决策树
### 情况A: 最近 10 分钟有部署
→ 执行回滚: `kubectl rollout undo deployment/payment-gateway -n prod`
→ 等待 2 分钟，确认错误率下降
→ 如果未恢复，转情况B

### 情况B: 无近期部署，依赖服务异常
→ 联系依赖服务团队（见下方联系人表）
→ 评估是否需要降级（关闭非核心功能，保支付主流程）
→ 降级命令: `./scripts/degrade-payment.sh --disable-coupon`

### 情况C: 无近期部署，无依赖异常
→ 检查数据库: `./diag-db.sh payment-db`
→ 检查网络: `./diag-network.sh payment-gateway`
→ 如果仍无法定位，升级到 P0，拉入更多人员

## 回滚验证
回滚后执行:
curl -s https://api.internal/health/payment-gateway | jq .
预期返回: {"status": "ok", "error_rate": "0.01%"}

# 自动回滚策略（Kubernetes + 自定义 controller）
apiVersion: ops.internal/v1
kind: AutoRollbackPolicy
metadata:
  name: payment-gateway-auto-rollback
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-gateway
  # 触发条件：部署后 5 分钟内错误率 > 10%
  trigger:
    window: 5m  # 部署后观察窗口
    condition: |
      error_rate > 0.1 AND
      requests_per_second > 100      
  # 执行动作：回滚到上一个 revision
  action:
    type: rollback
    notify: ["oncall-payment", "incident-channel"]
  # 安全约束
  constraints:
    maxRollbacksPerHour: 2  # 每小时最多自动回滚 2 次
    requireHumanApproval: false  # 不需要人工确认（因为是低风险场景）

# MTTR 自动统计脚本（从 Prometheus 查询故障时间段）
import requests
from datetime import datetime, timedelta

def calculate_mttr(prometheus_url, service, incident_start, incident_end):
    """
    从 Prometheus 查询数据，计算 MTTR 各阶段时间
    
    Args:
        prometheus_url: Prometheus 地址
        service: 服务名
        incident_start: 故障开始时间 (datetime)
        incident_end: 故障恢复时间 (datetime)
    """
    query_range = {
        "start": incident_start.timestamp(),
        "end": incident_end.timestamp(),
        "step": "15s"
    }
    
    # 1. MTTD: 从故障发生到告警触发
    # 查询第一个异常数据点
    error_query = f'rate(http_requests_total{{service="{service}",status=~"5.."}}[1m])'
    resp = requests.get(f"{prometheus_url}/api/v1/query_range",
                       params={"query": error_query, **query_range})
    error_data = resp.json()["data"]["result"][0]["values"]
    
    first_anomaly = None
    for ts, val in error_data:
        if float(val) > 0.01:  # 错误率 > 1%
            first_anomaly = datetime.fromtimestamp(ts)
            break
    
    # 2. MTTA: 从告警触发到首次响应（从 On-Call 系统获取）
    # 3. MTTI: 从首次响应到根因定位（从 Incident 系统获取）
    # 4. MTTR: 从根因定位到服务恢复
    
    mttd = (first_alert_time - first_anomaly).total_seconds() if first_anomaly else None
    mtta = (first_ack_time - first_alert_time).total_seconds() if first_alert_time else None
    mtti = (root_cause_time - first_ack_time).total_seconds() if root_cause_time else None
    mttr_restore = (incident_end - root_cause_time).total_seconds() if root_cause_time else None
    
    return {
        "MTTD": mttd,
        "MTTA": mtta,
        "MTTI": mtti,
        "MTTR_restore": mttr_restore,
        "Total_MTTR": (incident_end - first_anomaly).total_seconds()
    }

# 诊断脚本降级逻辑
if ! curl -s --max-time 3 "$PROMETHEUS_URL/-/healthy" >/dev/null 2>&1; then
    echo "⚠️  Prometheus 不可用，切换到离线诊断模式"
    OFFLINE_MODE=1
    # 直接查 Pod 日志代替 Metrics 查询
    kubectl logs -n "$NAMESPACE" -l app="$SERVICE" \
      --since=30m | grep -E "ERROR|WARN|panic" | tail -50
fi