混沌工程：主动发现系统弱点

chaos_engineering_benefits:
  direct_benefits:
    - "提前发现系统弱点和单点故障"
    - "验证告警和恢复机制是否有效"
    - "提升团队对故障的响应能力"
    - "验证架构设计假设是否成立"
  
  indirect_benefits:
    - "建立团队对系统弹性的信心"
    - "驱动架构改进（从'看起来能扛'到'验证过能扛'）"
    - "减少真实故障的 MTTR（因为已经演练过类似场景）"
    - "培养'故障不可避免'的工程文化"

# Chaos Monkey 简化逻辑
import random
import schedule

def chaos_monkey():
    instances = get_all_production_instances()
    victim = random.choice(instances)
    
    log.info(f"Chaos Monkey terminating: {victim}")
    terminate_instance(victim)
    
    # 等待并观察
    time.sleep(300)  # 5 分钟
    
    # 验证服务是否正常
    if check_service_health():
        log.info(f"Service survived termination of {victim}")
    else:
        log.error(f"Service degraded after terminating {victim}")
        alert_oncall(f"Chaos Monkey found weakness: {victim} is critical")

# 工作时间每小时执行一次
schedule.every().hour.at(":00").do(chaos_monkey)

# 混沌实验定义模板
experiment:
  name: "支付服务单 Pod 故障容忍验证"
  
  # 1. 稳态假设（什么算"正常"）
  steady_state_hypothesis:
    sli: "支付 API 成功率"
    normal_state: "> 99.9%"
    sli_source: "Prometheus"
    
  # 2. 实验范围
  scope:
    environment: "production"       # 生产环境
    service: "payment-service"
    namespace: "production"
    
  # 3. 故障注入
  fault:
    type: "pod_kill"                # 杀掉 Pod
    target: "random"                # 随机选择
    count: 1                        # 杀 1 个
    
  # 4. 爆炸半径控制
  blast_radius:
    strategy: "percentage"          # 按比例
    percentage: 10                  # 只影响 10% 的实例
    fallback: "auto_abort"          # SLI 恶化超阈值自动中止
    
  # 5. 持续时间
  duration: "5m"
  
  # 6. 回滚方案
  rollback:
    action: "kubernetes_auto_heal"  # K8s 自动重建 Pod
    manual_rollback: "kubectl rollout restart deployment/payment-service"
    
  # 7. 成功/失败标准
  success_criteria: "SLI 在实验期间保持 > 99.9%"
  failure_action: "记录弱点，创建改进工单"

Step 1: 确定实验目标
  → "验证支付服务能容忍单个 Pod 故障"

Step 2: 定义稳态假设
  → "在 Pod 故障期间，支付 API 成功率 > 99.9%，P99 延迟 < 500ms"

Step 3: 选择故障类型
  → "杀掉 1 个 payment-service 的 Pod"

Step 4: 确定爆炸半径
  → "从 1 个 Pod 开始，payment-service 有 10 个 Pod，影响 10%"

Step 5: 设定中止条件
  → "成功率 < 99.5% 或 P99 > 1s 时自动中止实验"

Step 6: 准备回滚方案
  → "K8s 自动重建 Pod；如果未恢复，手动 rollout restart"

Step 7: 执行实验
  → 注入故障 → 持续监控 SLI → 验证假设

Step 8: 分析结果
  → 稳态保持 → 假设成立 → 系统有弹性 ✅
  → 稳态被破坏 → 假设不成立 → 发现弱点 🔍

优先级 1：高频故障场景
  → Pod 崩溃、网络延迟、磁盘满
  → 这些是最常发生的故障，验证弹性优先级最高

优先级 2：核心依赖故障
  → 数据库不可用、缓存不可用、消息队列不可用
  → 验证降级和故障转移机制

优先级 3：区域性故障
  → 可用区隔离、区域不可用
  → 验证多活/容灾能力

优先级 4：组合故障
  → 同时注入多种故障
  → 验证极端场景下的系统行为

爆炸半径控制策略（从小到大）：

Level 1: 单实例
  → 只注入 1 个实例的故障
  → 最小影响，适合初次实验

Level 2: 百分比实例
  → 注入 5-10% 的实例故障
  → 验证负载均衡和自动恢复

Level 3: 单可用区
  → 模拟整个可用区不可用
  → 验证跨可用区容灾

Level 4: 跨可用区
  → 模拟多可用区同时故障
  → 验证区域级容灾（高风险）

# 自动中止机制
auto_abort:
  enabled: true
  
  # 监控的 SLI
  monitors:
    - metric: "payment_api_success_rate"
      threshold: 99.5%        # 低于此值自动中止
      window: "1m"
      
    - metric: "payment_api_p99_latency"
      threshold: 1000ms       # 超过此值自动中止
      window: "1m"
      
    - metric: "error_rate"
      threshold: 5%            # 错误率超标自动中止
      window: "30s"
  
  # 中止动作
  abort_actions:
    - "立即撤销故障注入"
    - "通知 On-Call 工程师"
    - "记录实验中止原因"
    - "如果服务未自动恢复，执行回滚"

# 自动中止实现
class ChaosExperiment:
    def __init__(self, config):
        self.config = config
        self.aborted = False

    def run(self):
        # 1. 记录实验前的稳态基线
        baseline = self.get_current_sli()
        log.info(f"Baseline SLI: {baseline}")

        # 2. 注入故障
        self.inject_fault()
        log.info("Fault injected, monitoring...")

        # 3. 持续监控
        start_time = time.time()
        while time.time() - start_time < self.config.duration:
            current_sli = self.get_current_sli()

            if self.should_abort(current_sli):
                self.abort()
                return

            time.sleep(10)  # 每 10 秒检查一次

        # 4. 实验正常结束，撤销故障
        self.revoke_fault()

        # 5. 验证恢复
        time.sleep(60)
        if not self.is_recovered():
            self.emergency_rollback()

    def should_abort(self, current_sli):
        for monitor in self.config.monitors:
            if current_sli[monitor['metric']] > monitor['threshold']:
                log.error(f"Abort condition met: {monitor['metric']} = "
                         f"{current_sli[monitor['metric']]} > {monitor['threshold']}")
                return True
        return False

    def abort(self):
        self.aborted = True
        self.revoke_fault()
        notify_oncall(f"Chaos experiment aborted: SLI exceeded threshold")
        log.error("Experiment aborted due to SLI violation")

# 实验时间窗口选择
experiment_schedule:
  preferred_window:
    time: "工作日 10:00-16:00"
    reason: "团队在线，能快速响应意外"
    
  avoid:
    - "非工作时间（夜间/周末）"
    - "业务高峰期（如电商大促期间）"
    - "有计划维护窗口的时间"
    - "有重要发布的当天"
    
  frequency:
    initial: "每周 1 次，小爆炸半径"
    mature: "每天自动运行，常规实验"

# 使用 Helm 安装 Chaos Mesh
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  -n chaos-testing \
  --set chaosDaemon.runtime=containerd \
  --create-namespace

# 验证安装
kubectl get pods -n chaos-testing

# pod-kill-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: payment-pod-kill
  namespace: chaos-testing
spec:
  action: pod-kill         # 杀掉 Pod
  mode: one                 # 只杀一个
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  scheduler:
    cron: "@every 1h"      # 每小时执行一次

# network-delay-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-network-delay
  namespace: chaos-testing
spec:
  action: delay             # 注入延迟
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  delay:
    latency: "200ms"        # 200ms 延迟
    correlation: "0"
    jitter: "50ms"          # 50ms 抖动
  duration: "5m"            # 持续 5 分钟
  scheduler:
    cron: "@every 24h"

# network-partition-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-db-partition
  namespace: chaos-testing
spec:
  action: partition         # 网络分区
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  direction: to             # payment → db 方向阻断
  target:
    selector:
      namespaces:
        - production
      labelSelectors:
        "app": "postgres"
    mode: all
  duration: "2m"

# cpu-stress-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: payment-cpu-stress
  namespace: chaos-testing
spec:
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  stressors:
    cpu:
      workers: 2            # 2 个 CPU 压力 worker
      load: 80              # 80% 负载
  duration: "3m"

# disk-io-delay-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: payment-disk-io-delay
  namespace: chaos-testing
spec:
  action: latency
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  volumePath: "/data"
  path: "/data/**/*"        # 影响的数据路径
  delay: "100ms"            # IO 延迟 100ms
  percent: 50               # 50% 的 IO 受影响
  duration: "5m"

# 串行编排：先杀 Pod，再注入网络延迟
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: payment-resilience-test
  namespace: chaos-testing
spec:
  entry: serial
  workflow:
    - template:
        name: phase-1-pod-kill
        templateType: PodChaos
        deadline: 2m
        podChaos:
          action: pod-kill
          mode: one
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
    
    - template:
        name: phase-2-network-delay
        templateType: NetworkChaos
        deadline: 5m
        networkChaos:
          action: delay
          mode: all
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          delay:
            latency: "200ms"
            jitter: "50ms"
    
    - template:
        name: phase-3-stress
        templateType: StressChaos
        deadline: 3m
        stressChaos:
          mode: one
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          stressors:
            cpu:
              workers: 2
              load: 90

# 使用 Chaos Mesh 的 Workflow + Prometheus 验证
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: payment-chaos-with-validation
  namespace: chaos-testing
spec:
  entry: serial
  workflow:
    # 1. 记录实验前基线
    - template:
        name: record-baseline
        templateType: Task
        task:
          container:
            image: curlimages/curl
            command: 
              - /bin/sh
              - -c
              - |
                echo "Recording baseline..."
                curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status!~\"5..\"}[1m])" > /tmp/baseline.json                
    
    # 2. 注入故障
    - template:
        name: inject-fault
        templateType: NetworkChaos
        deadline: 5m
        networkChaos:
          action: delay
          mode: all
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          delay:
            latency: "500ms"
    
    # 3. 验证 SLI
    - template:
        name: validate-sli
        templateType: Task
        task:
          container:
            image: curlimages/curl
            command:
              - /bin/sh
              - -c
              - |
                echo "Validating SLI..."
                ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~\"5..\"}[1m]))/sum(rate(http_requests_total[1m]))" | jq -r '.data.result[0].value[1]')
                echo "Current error rate: $ERROR_RATE"
                if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
                  echo "SLI violated! Error rate too high."
                  exit 1
                fi
                echo "SLI OK"                

# 弱点分类与处理策略
weakness_categories:
  architecture:
    description: "架构设计缺陷"
    examples:
      - "单点故障：某个 Pod 死了整个服务不可用"
      - "无冗余：数据库没有从库"
    action: "架构改造，增加冗余"
    priority: "P0"
  
  configuration:
    description: "配置不当"
    examples:
      - "健康检查配置错误：Pod 死了 K8s 没发现"
      - "没有配置 PDB（Pod Disruption Budget）"
    action: "修正配置"
    priority: "P0"
  
  monitoring:
    description: "监控盲区"
    examples:
      - "故障发生了但告警没触发"
      - "告警触发了但定位信息不足"
    action: "补充监控和告警"
    priority: "P1"
  
  recovery:
    description: "恢复机制不足"
    examples:
      - "Pod 被杀后没有自动重启"
      - "自动扩容没有生效"
    action: "完善自动化恢复"
    priority: "P1"
  
  process:
    description: "流程缺陷"
    examples:
      - "Runbook 缺失，排查时不知道该做什么"
      - "escalation 不清晰，不知道该找谁"
    action: "完善流程和文档"
    priority: "P2"

# 混沌实验发现的弱点跟踪

## 实验：支付服务 Pod Kill
**日期**：2026-07-10
**结果**：稳态被破坏

### 发现的弱点
| # | 弱点 | 类型 | 影响 | 优先级 | 状态 |
|---|------|------|------|--------|------|
| 1 | payment-service 只有 2 个 Pod，杀 1 个后 P99 延迟翻倍 | 配置 | P99 从 200ms 到 500ms | P0 | 已修复（扩到 5 个 Pod） |
| 2 | Pod 被杀后负载均衡 30 秒后才摘除流量 | 配置 | 30 秒内请求发到死 Pod | P1 | 已修复（调短健康检查间隔） |
| 3 | 告警在 Pod 重启后才触发，不够及时 | 监控 | 延迟告警 30 秒 | P2 | 待修复 |

### 改进项
- [x] 扩大 payment-service 副本数到 5
- [x] 调整健康检查：liveness probe 间隔从 10s 调到 5s
- [ ] 增加 Pod 重启告警，阈值从 3 次/小时调到 1 次/小时

Phase 1: 试点（1-2 个月）
  → 选择非核心服务在测试环境做实验
  → 目标：熟悉工具，建立流程
  → 产出：实验模板、操作手册

Phase 2: 生产小范围（2-3 个月）
  → 选择 1 个核心服务在生产做实验
  → 从最小爆炸半径开始（1 个 Pod）
  → 目标：验证生产环境的安全性
  → 产出：爆炸半径控制方案、自动中止机制

Phase 3: 扩大范围（3-6 个月）
  → 覆盖所有核心服务
  → 增加故障类型（网络、磁盘、CPU）
  → 目标：发现并修复主要弱点
  → 产出：弱点清单和修复计划

Phase 4: 自动化（6-12 个月）
  → 建设自动化实验平台
  → 实验自动调度、执行、验证、报告
  → 目标：持续运行，而非一次性
  → 产出：自动化混沌平台

Phase 5: 常态化（12 个月+）
  → 混沌实验融入开发流程
  → 新服务上线前需要通过弹性验证
  → 目标：弹性成为工程文化的一部分

culture_preparation:
  common_resistance:
    - "在生产环境注入故障？你疯了吗？"
    - "这会影响用户体验的"
    - "我们没时间做这个"
  
  strategies:
    - "从非核心服务开始，用成功案例建立信心"
    - "先在测试环境做，团队适应后再上生产"
    - "让管理层参与实验设计，理解价值"
    - "公开分享发现的弱点和修复成果"
    - "把混沌实验纳入 SRE 团队的 OKR"
  
  success_indicators:
    - "开发团队主动要求做混沌实验验证新架构"
    - "发现的弱点数量在减少（说明系统在变强）"
    - "真实故障的 MTTR 在缩短（因为演练过）"
    - "团队对故障的恐慌感在降低"

chaos_engineering_metrics:
  experiment_metrics:
    - name: "实验执行频率"
      target: "核心服务每周至少 1 次"
    
    - name: "实验覆盖率"
      formula: "已做实验的服务 / 核心服务总数"
      target: "> 80%"
    
    - name: "故障类型覆盖率"
      formula: "已测试的故障类型 / 计划测试的故障类型"
      target: "> 70%"
  
  outcome_metrics:
    - name: "发现的弱点数"
      direction: "初期增多，后期减少"
    
    - name: "弱点修复率"
      formula: "已修复弱点 / 发现的弱点"
      target: "> 80%"
    
    - name: "同类故障复发率"
      formula: "混沌实验发现并修复的弱点类型在真实故障中复发的比例"
      target: "< 10%"
  
  business_metrics:
    - name: "真实故障 MTTR"
      direction: "持续下降"
    
    - name: "真实故障导致的 SEV1/SEV2 数量"
      direction: "持续下降"