Chaos Engineering: Proactively Discovering System Weaknesses

chaos_engineering_benefits:
  direct_benefits:
    - "Proactively discover system weaknesses and single points of failure"
    - "Validate that alerting and recovery mechanisms are effective"
    - "Improve the team's incident response capability"
    - "Verify that architecture design assumptions hold"

  indirect_benefits:
    - "Build team confidence in system resilience"
    - "Drive architecture improvements (from 'seems like it can handle it' to 'verified it can handle it')"
    - "Reduce real incident MTTR (because similar scenarios have been rehearsed)"
    - "Cultivate an engineering culture of 'failures are inevitable'"

# Chaos Monkey simplified logic
import random
import schedule

def chaos_monkey():
    instances = get_all_production_instances()
    victim = random.choice(instances)

    log.info(f"Chaos Monkey terminating: {victim}")
    terminate_instance(victim)

    # Wait and observe
    time.sleep(300)  # 5 minutes

    # Verify service health
    if check_service_health():
        log.info(f"Service survived termination of {victim}")
    else:
        log.error(f"Service degraded after terminating {victim}")
        alert_oncall(f"Chaos Monkey found weakness: {victim} is critical")

# Execute every hour during business hours
schedule.every().hour.at(":00").do(chaos_monkey)

# Chaos experiment definition template
experiment:
  name: "Payment Service Single Pod Failure Tolerance Validation"

  # 1. Steady-state hypothesis (what counts as "normal")
  steady_state_hypothesis:
    sli: "Payment API success rate"
    normal_state: "> 99.9%"
    sli_source: "Prometheus"

  # 2. Experiment scope
  scope:
    environment: "production"       # Production environment
    service: "payment-service"
    namespace: "production"

  # 3. Fault injection
  fault:
    type: "pod_kill"                # Kill a Pod
    target: "random"                # Random selection
    count: 1                         # Kill 1

  # 4. Blast radius control
  blast_radius:
    strategy: "percentage"          # By percentage
    percentage: 10                   # Only affect 10% of instances
    fallback: "auto_abort"          # Auto-abort if SLI degrades beyond threshold

  # 5. Duration
  duration: "5m"

  # 6. Rollback plan
  rollback:
    action: "kubernetes_auto_heal"  # K8s auto-rebuilds Pod
    manual_rollback: "kubectl rollout restart deployment/payment-service"

  # 7. Success/failure criteria
  success_criteria: "SLI maintained > 99.9% during experiment"
  failure_action: "Record weakness, create improvement ticket"

Step 1: Define experiment goal
  → "Validate that payment service can tolerate a single Pod failure"

Step 2: Define steady-state hypothesis
  → "During Pod failure, payment API success rate > 99.9%, P99 latency < 500ms"

Step 3: Choose fault type
  → "Kill 1 payment-service Pod"

Step 4: Determine blast radius
  → "Start with 1 Pod; payment-service has 10 Pods, affecting 10%"

Step 5: Set abort conditions
  → "Auto-abort if success rate < 99.5% or P99 > 1s"

Step 6: Prepare rollback plan
  → "K8s auto-rebuilds Pod; if not recovered, manual rollout restart"

Step 7: Execute experiment
  → Inject fault → Continuously monitor SLI → Verify hypothesis

Step 8: Analyze results
  → Steady state maintained → Hypothesis confirmed → System is resilient ✅
  → Steady state broken → Hypothesis disproved → Weakness found 🔍

Priority 1: High-frequency failure scenarios
  → Pod crash, network latency, disk full
  → These are the most common failures; validating resilience has the highest priority

Priority 2: Core dependency failures
  → Database unavailable, cache unavailable, message queue unavailable
  → Validate degradation and failover mechanisms

Priority 3: Regional failures
  → Availability zone isolation, region unavailable
  → Validate multi-active/disaster recovery capabilities

Priority 4: Combined failures
  → Inject multiple failures simultaneously
  → Validate system behavior under extreme scenarios

Blast radius control strategies (from small to large):

Level 1: Single instance
  → Inject fault into only 1 instance
  → Minimal impact, suitable for initial experiments

Level 2: Percentage of instances
  → Inject fault into 5-10% of instances
  → Validate load balancing and auto-recovery

Level 3: Single availability zone
  → Simulate entire AZ unavailability
  → Validate cross-AZ disaster recovery

Level 4: Cross-AZ
  → Simulate multi-AZ simultaneous failure
  → Validate regional disaster recovery (high risk)

# Auto-abort mechanism
auto_abort:
  enabled: true

  # Monitored SLIs
  monitors:
    - metric: "payment_api_success_rate"
      threshold: 99.5%        # Auto-abort below this value
      window: "1m"

    - metric: "payment_api_p99_latency"
      threshold: 1000ms       # Auto-abort above this value
      window: "1m"

    - metric: "error_rate"
      threshold: 5%            # Auto-abort if error rate exceeds
      window: "30s"

  # Abort actions
  abort_actions:
    - "Immediately revoke fault injection"
    - "Notify On-Call engineer"
    - "Record experiment abort reason"
    - "If service doesn't auto-recover, execute rollback"

# Auto-abort implementation
class ChaosExperiment:
    def __init__(self, config):
        self.config = config
        self.aborted = False

    def run(self):
        # 1. Record pre-experiment steady-state baseline
        baseline = self.get_current_sli()
        log.info(f"Baseline SLI: {baseline}")

        # 2. Inject fault
        self.inject_fault()
        log.info("Fault injected, monitoring...")

        # 3. Continuous monitoring
        start_time = time.time()
        while time.time() - start_time < self.config.duration:
            current_sli = self.get_current_sli()

            if self.should_abort(current_sli):
                self.abort()
                return

            time.sleep(10)  # Check every 10 seconds

        # 4. Experiment completed normally, revoke fault
        self.revoke_fault()

        # 5. Verify recovery
        time.sleep(60)
        if not self.is_recovered():
            self.emergency_rollback()

    def should_abort(self, current_sli):
        for monitor in self.config.monitors:
            if current_sli[monitor['metric']] > monitor['threshold']:
                log.error(f"Abort condition met: {monitor['metric']} = "
                         f"{current_sli[monitor['metric']]} > {monitor['threshold']}")
                return True
        return False

    def abort(self):
        self.aborted = True
        self.revoke_fault()
        notify_oncall(f"Chaos experiment aborted: SLI exceeded threshold")
        log.error("Experiment aborted due to SLI violation")

# Experiment time window selection
experiment_schedule:
  preferred_window:
    time: "Weekdays 10:00-16:00"
    reason: "Team is online, can respond quickly to surprises"

  avoid:
    - "Non-business hours (nights/weekends)"
    - "Business peak periods (e.g., major e-commerce promotions)"
    - "Scheduled maintenance windows"
    - "Days with important releases"

  frequency:
    initial: "Once per week, small blast radius"
    mature: "Daily automated runs, routine experiments"

# Install Chaos Mesh using Helm
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  -n chaos-testing \
  --set chaosDaemon.runtime=containerd \
  --create-namespace

# Verify installation
kubectl get pods -n chaos-testing

# pod-kill-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: payment-pod-kill
  namespace: chaos-testing
spec:
  action: pod-kill         # Kill Pod
  mode: one                 # Kill only one
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  scheduler:
    cron: "@every 1h"      # Execute every hour

# network-delay-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-network-delay
  namespace: chaos-testing
spec:
  action: delay             # Inject latency
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  delay:
    latency: "200ms"        # 200ms latency
    correlation: "0"
    jitter: "50ms"          # 50ms jitter
  duration: "5m"            # Last 5 minutes
  scheduler:
    cron: "@every 24h"

# network-partition-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-db-partition
  namespace: chaos-testing
spec:
  action: partition         # Network partition
  mode: all
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  direction: to             # Block payment → db direction
  target:
    selector:
      namespaces:
        - production
      labelSelectors:
        "app": "postgres"
    mode: all
  duration: "2m"

# cpu-stress-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
  name: payment-cpu-stress
  namespace: chaos-testing
spec:
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  stressors:
    cpu:
      workers: 2            # 2 CPU stress workers
      load: 80              # 80% load
  duration: "3m"

# disk-io-delay-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: payment-disk-io-delay
  namespace: chaos-testing
spec:
  action: latency
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      "app": "payment-service"
  volumePath: "/data"
  path: "/data/**/*"        # Affected data path
  delay: "100ms"            # IO delay 100ms
  percent: 50               # 50% of IO affected
  duration: "5m"

# Serial orchestration: first kill Pod, then inject network latency
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: payment-resilience-test
  namespace: chaos-testing
spec:
  entry: serial
  workflow:
    - template:
        name: phase-1-pod-kill
        templateType: PodChaos
        deadline: 2m
        podChaos:
          action: pod-kill
          mode: one
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"

    - template:
        name: phase-2-network-delay
        templateType: NetworkChaos
        deadline: 5m
        networkChaos:
          action: delay
          mode: all
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          delay:
            latency: "200ms"
            jitter: "50ms"

    - template:
        name: phase-3-stress
        templateType: StressChaos
        deadline: 3m
        stressChaos:
          mode: one
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          stressors:
            cpu:
              workers: 2
              load: 90

# Using Chaos Mesh Workflow + Prometheus validation
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
  name: payment-chaos-with-validation
  namespace: chaos-testing
spec:
  entry: serial
  workflow:
    # 1. Record pre-experiment baseline
    - template:
        name: record-baseline
        templateType: Task
        task:
          container:
            image: curlimages/curl
            command:
              - /bin/sh
              - -c
              - |
                echo "Recording baseline..."
                curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status!~\"5..\"}[1m])" > /tmp/baseline.json                

    # 2. Inject fault
    - template:
        name: inject-fault
        templateType: NetworkChaos
        deadline: 5m
        networkChaos:
          action: delay
          mode: all
          selector:
            namespaces: [production]
            labelSelectors:
              "app": "payment-service"
          delay:
            latency: "500ms"

    # 3. Validate SLI
    - template:
        name: validate-sli
        templateType: Task
        task:
          container:
            image: curlimages/curl
            command:
              - /bin/sh
              - -c
              - |
                echo "Validating SLI..."
                ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~\"5..\"}[1m]))/sum(rate(http_requests_total[1m]))" | jq -r '.data.result[0].value[1]')
                echo "Current error rate: $ERROR_RATE"
                if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
                  echo "SLI violated! Error rate too high."
                  exit 1
                fi
                echo "SLI OK"                

# Weakness classification and handling strategy
weakness_categories:
  architecture:
    description: "Architecture design flaw"
    examples:
      - "Single point of failure: one Pod dying makes the entire service unavailable"
      - "No redundancy: database has no replica"
    action: "Architecture redesign, add redundancy"
    priority: "P0"

  configuration:
    description: "Improper configuration"
    examples:
      - "Health check misconfigured: Pod died but K8s didn't notice"
      - "No PDB (Pod Disruption Budget) configured"
    action: "Fix configuration"
    priority: "P0"

  monitoring:
    description: "Monitoring blind spots"
    examples:
      - "Failure occurred but alert didn't trigger"
      - "Alert triggered but insufficient diagnostic information"
    action: "Add monitoring and alerting"
    priority: "P1"

  recovery:
    description: "Insufficient recovery mechanisms"
    examples:
      - "Pod didn't auto-restart after being killed"
      - "Auto-scaling didn't trigger"
    action: "Improve automated recovery"
    priority: "P1"

  process:
    description: "Process gaps"
    examples:
      - "Missing Runbook, don't know what to do during investigation"
      - "Unclear escalation, don't know who to contact"
    action: "Improve processes and documentation"
    priority: "P2"

# Weakness Tracking from Chaos Experiments

## Experiment: Payment Service Pod Kill
**Date**: 2026-07-10
**Result**: Steady state broken

### Weaknesses Found
| # | Weakness | Type | Impact | Priority | Status |
|---|------|------|------|--------|------|
| 1 | payment-service only has 2 Pods; killing 1 doubles P99 latency | Configuration | P99 from 200ms to 500ms | P0 | Fixed (scaled to 5 Pods) |
| 2 | Load balancer took 30 seconds to remove dead Pod traffic | Configuration | Requests sent to dead Pod for 30s | P1 | Fixed (shortened health check interval) |
| 3 | Alert triggered only after Pod restart, not timely enough | Monitoring | 30s alert delay | P2 | Pending |

### Action Items
- [x] Scale payment-service replicas to 5
- [x] Adjust health check: liveness probe interval from 10s to 5s
- [ ] Add Pod restart alerting, threshold from 3/hour to 1/hour

Phase 1: Pilot (1-2 months)
  → Select non-core services in test environment
  → Goal: Familiarize with tools, establish process
  → Output: Experiment templates, operation manuals

Phase 2: Small-scale production (2-3 months)
  → Select 1 core service for production experiments
  → Start with minimal blast radius (1 Pod)
  → Goal: Validate production safety
  → Output: Blast radius control plan, auto-abort mechanism

Phase 3: Expand scope (3-6 months)
  → Cover all core services
  → Add fault types (network, disk, CPU)
  → Goal: Discover and fix major weaknesses
  → Output: Weakness inventory and fix plan

Phase 4: Automation (6-12 months)
  → Build automated experiment platform
  → Auto-schedule, execute, validate, report
  → Goal: Continuous operation, not one-time
  → Output: Automated chaos platform

Phase 5: Normalization (12+ months)
  → Integrate chaos experiments into development workflow
  → New services must pass resilience validation before launch
  → Goal: Resilience becomes part of engineering culture

culture_preparation:
  common_resistance:
    - "Inject failures in production? Are you crazy?"
    - "This will affect user experience"
    - "We don't have time for this"

  strategies:
    - "Start with non-core services, build confidence with success stories"
    - "Do it in test environment first, move to production after team adapts"
    - "Have management participate in experiment design to understand the value"
    - "Publicly share discovered weaknesses and fix results"
    - "Include chaos experiments in the SRE team's OKRs"

  success_indicators:
    - "Development teams proactively request chaos experiments to validate new architectures"
    - "Number of discovered weaknesses decreasing (system getting stronger)"
    - "Real incident MTTR decreasing (because scenarios have been rehearsed)"
    - "Team's fear of failures decreasing"

chaos_engineering_metrics:
  experiment_metrics:
    - name: "Experiment execution frequency"
      target: "At least once per week for core services"

    - name: "Experiment coverage"
      formula: "Services with experiments / Total core services"
      target: "> 80%"

    - name: "Fault type coverage"
      formula: "Tested fault types / Planned fault types"
      target: "> 70%"

  outcome_metrics:
    - name: "Discovered weaknesses count"
      direction: "Initially increases, later decreases"

    - name: "Weakness fix rate"
      formula: "Fixed weaknesses / Discovered weaknesses"
      target: "> 80%"

    - name: "Similar incident recurrence rate"
      formula: "Proportion of weakness types found and fixed in chaos experiments that recur in real incidents"
      target: "< 10%"

  business_metrics:
    - name: "Real incident MTTR"
      direction: "Continuously decreasing"

    - name: "SEV1/SEV2 incidents from real failures"
      direction: "Continuously decreasing"