Overview

Capacity planning is a core SRE responsibility. The Google SRE Book treats capacity planning as “proactive work,” emphasizing data-driven forecasting over experience-based guessing. A team without capacity planning either gets crushed by traffic during peaks or wastes significant resource costs during valleys.

This article systematically covers capacity planning engineering practices across four dimensions: metric collection, data modeling, Kubernetes elastic scaling configuration, and pitfall avoidance.

For a systematic methodology on capacity planning, see Google SRE Book - Capacity Planning and its discussion of capacity planning and cascading failures.

1. The Core Philosophy: Data-Driven, Not Guesswork

Three Levels of Capacity Planning

  1. Current capacity assessment: How much can the system handle now? What’s the utilization?
  2. Capacity trend forecasting: Based on current growth trends, when do we need to scale?
  3. Elastic scaling strategy: How to automatically respond to traffic spikes?

Prerequisite: Observability

You can’t manage what you don’t measure. The foundation of capacity planning is a comprehensive monitoring system that continuously collects the following metrics:

Metric CategorySpecific MetricsCollection Tool
CPUUtilization, load (1m/5m/15m)node_exporter / cAdvisor
MemoryUsage, available, OOM countnode_exporter / cAdvisor
NetworkInbound/outbound bandwidth, connections, packet lossnode_exporter
Disk I/OIOPS, read/write latency, queue depthnode_exporter
ApplicationQPS, latency distribution, error ratePrometheus / custom metrics
MiddlewareConnection pool utilization, queue length, cache hit rateExporter / custom metrics

Capacity Threshold Definition

Not all metrics are equally important. You need to define capacity thresholds for critical resources:

# Capacity threshold definition example
capacity_thresholds:
  cpu:
    warning: 60%    # Start paying attention at 60%
    critical: 80%   # Need to scale at 80%
    limit: 90%      # Emergency scaling at 90%
  memory:
    warning: 70%
    critical: 85%
    limit: 95%
  disk_io:
    warning: 60%
    critical: 80%
    limit: 90%
  connection:
    warning: 60%    # Connection pool utilization
    critical: 80%
    limit: 90%

Key principle: Thresholds shouldn’t be set arbitrarily — they should be based on load test data and historical failure analysis. If your application starts degrading latency at 85% CPU, then the warning threshold should be set below 70%.

2. Capacity Metric Collection

Prometheus Metric Collection Configuration

Here is a commonly used Prometheus scrape configuration for production environments, covering node-level and application-level metrics:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: "production"

scrape_configs:
  # Node-level metrics
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  # Container-level metrics
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  # Kubernetes service discovery
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

  # Application custom metrics (for HPA)
  - job_name: 'app-metrics'
    static_configs:
      - targets: ['app-metrics-service:9090']

Key Capacity Query Expressions

# CPU utilization (node-level)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory utilization
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

# Disk I/O utilization
rate(node_disk_io_time_seconds_total[5m]) * 100

# Pod CPU utilization (relative to limit)
sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod) /
  sum(kube_pod_container_resource_limits{resource="cpu"}) by (pod) * 100

# Connection count monitoring
node_netstat_Tcp_CurrEstab

# HTTP request QPS (application-level)
sum(rate(http_requests_total[5m])) by (service)

3. Data-Driven Capacity Modeling

Trend Analysis

The first step in capacity forecasting is identifying trends. Using QPS growth as an example, linear regression can predict future resource requirements:

#!/usr/bin/env python3
"""
Capacity trend analysis: predict future capacity needs based on historical monitoring data
Data source: Prometheus Query API
"""
import requests
import numpy as np
from datetime import datetime, timedelta
from sklearn.linear_model import LinearRegression

PROMETHEUS_URL = "http://prometheus:9090/api/v1/query_range"

def fetch_qps_history(days=30):
    """Fetch daily peak QPS for the last N days from Prometheus"""
    end = datetime.now()
    start = end - timedelta(days=days)
    query = 'sum(rate(http_requests_total[1h]))'

    resp = requests.get(PROMETHEUS_URL, params={
        'query': query,
        'start': start.timestamp(),
        'end': end.timestamp(),
        'step': '3600s'  # One data point per hour
    })
    data = resp.json()['data']['result'][0]['values']
    return [float(v[1]) for v in data]

def predict_capacity(qps_history, days_ahead=30):
    """Linear regression to predict future capacity needs"""
    x = np.arange(len(qps_history)).reshape(-1, 1)
    y = np.array(qps_history)

    model = LinearRegression().fit(x, y)

    future_x = np.arange(len(qps_history), len(qps_history) + days_ahead).reshape(-1, 1)
    predictions = model.predict(future_x)

    # Calculate growth rate
    growth_rate = (predictions[-1] - qps_history[-1]) / qps_history[-1] * 100

    return predictions, growth_rate

def check_capacity(predictions, current_capacity, threshold=0.7):
    """Check when capacity threshold will be reached"""
    for day, pred in enumerate(predictions, start=1):
        if pred > current_capacity * threshold:
            return day, pred
    return None, None

if __name__ == "__main__":
    qps_history = fetch_qps_history(30)
    predictions, growth_rate = predict_capacity(qps_history, 30)

    current_capacity = 8000  # Current total capacity
    threshold_day, threshold_qps = check_capacity(predictions, current_capacity, 0.7)

    print(f"Current peak QPS: {qps_history[-1]:.0f}")
    print(f"Predicted QPS in 30 days: {predictions[-1]:.0f}")
    print(f"Growth rate: {growth_rate:.1f}%")
    if threshold_day:
        print(f"⚠️ Day {threshold_day} will reach 70% threshold (QPS: {threshold_qps:.0f}), recommend scaling in advance")
    else:
        print("✅ No scaling needed within 30 days")

Seasonal Fluctuations

Many businesses have seasonal patterns — e-commerce spikes during Double 11 and 618; social platforms see doubled traffic during evening peaks. Capacity planning must account for these cyclical patterns:

"""
Seasonal capacity analysis: identify periodic traffic patterns
"""
import numpy as np

def detect_seasonality(data, period=24):
    """
    Detect periodic patterns in data
    data: list of QPS data points, one per hour
    period: cycle length (24 = one day)
    """
    if len(data) < period * 7:  # Need at least 7 cycles
        return None

    # Calculate average for each time slot
    pattern = []
    for i in range(period):
        values = [data[j] for j in range(i, len(data), period)]
        pattern.append(np.mean(values))

    avg = np.mean(data)
    peak_ratio = max(pattern) / avg  # Peak/average ratio

    return {
        'pattern': pattern,
        'peak_ratio': peak_ratio,
        'peak_hour': pattern.index(max(pattern))
    }

# Example usage
data = fetch_qps_history(30 * 24)  # 30 days, hourly
result = detect_seasonality(data, period=24)
if result:
    print(f"Peak hour: {result['peak_hour']}:00")
    print(f"Peak/average ratio: {result['peak_ratio']:.2f}x")
    # If peak/average ratio > 2, traffic varies significantly, elastic scaling is beneficial
    if result['peak_ratio'] > 2:
        print("💡 Traffic varies significantly, consider elastic scaling over fixed scaling")

4. Kubernetes Auto-Scaling

HPA (Horizontal Pod Autoscaler)

HPA is the most commonly used auto-scaling mechanism in Kubernetes, automatically adjusting Pod replica counts based on CPU/memory or custom metrics.

CPU Utilization-Based HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3          # Minimum replicas (ensure base capacity)
  maxReplicas: 20          # Maximum replicas (prevent runaway scaling)
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70  # Target CPU utilization 70%
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80  # Target memory utilization 80%
  behavior:
    # Scale-up behavior: fast scaling
    scaleUp:
      stabilizationWindowSeconds: 0  # No wait, scale immediately
      policies:
        - type: Percent
          value: 100                  # Max scale up 100% each time
          periodSeconds: 15
        - type: Pods
          value: 4                     # Or max 4 Pods at a time
          periodSeconds: 15
      selectPolicy: Max                # Take the maximum of both policies
    # Scale-down behavior: slow scaling
    scaleDown:
      stabilizationWindowSeconds: 300  # 5-minute stabilization window to prevent flapping
      policies:
        - type: Percent
          value: 10                    # Max scale down 10% each time
          periodSeconds: 60

Custom Metric-Based HPA

For latency-sensitive services, scaling based on custom metrics like QPS or latency is more appropriate:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-server-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 4
  maxReplicas: 30
  metrics:
    # Scale based on QPS
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "1000"  # Target 1000 QPS per Pod
    # Scale based on latency (P99)
    - type: Pods
      pods:
        metric:
          name: http_request_duration_p99
        target:
          type: AverageValue
          averageValue: "200"  # P99 latency target 200ms
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 50
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 20
          periodSeconds: 120

Using custom metrics requires deploying Prometheus Adapter to expose Prometheus metrics as Kubernetes custom metrics API:

# Prometheus Adapter rule configuration
# rules.yaml
rules:
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_total"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

VPA (Vertical Pod Autoscaler)

VPA automatically adjusts Pod CPU/memory requests and limits, suitable for stateful services that can’t scale horizontally:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: database-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: StatefulSet
    name: postgresql
  updatePolicy:
    updateMode: "Auto"  # Auto: automatic adjustment | Off: recommendation only
  resourcePolicy:
    containerPolicies:
      - containerName: postgresql
        minAllowed:
          cpu: 500m
          memory: 2Gi
        maxAllowed:
          cpu: 4000m
          memory: 16Gi
        controlledResources: ["cpu", "memory"]

Cluster Autoscaler

When Pods are Pending due to insufficient resources, Cluster Autoscaler automatically adds nodes to the cluster:

# Cluster Autoscaler configuration (AWS example)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.27.0
          name: cluster-autoscaler
          command:
            - ./cluster-autoscaler
            - --cluster-name=production-cluster
            - --max-node-provision-time=15m     # Max node provisioning time
            - --balance-similar-node-groups       # Balance similar node groups
            - --scale-down-enabled=true
            - --scale-down-delay-after-add=10m   # No scale-down within 10 minutes after scaling up
            - --scale-down-unneeded-time=15m    # Node must be idle for 15 minutes before scale-down
            - --scale-down-utilization-threshold=0.5  # Node utilization below 50% for scale-down

5. Pitfalls and Best Practices in Elastic Scaling

Pitfall 1: Cold Start Latency

Problem: HPA detects high load → creates new Pods → pulls image → starts application → health check passes → joins load balancer. This chain can take 2-5 minutes in extreme scenarios, and peak traffic has already overwhelmed existing instances by then.

Best Practice:

# 1. Pre-warming: Use CronHPA to scale before predictable peaks
apiVersion: autoscaling.alibaba.com/v1
kind: CronHorizontalPodAutoscaler
metadata:
  name: ecommerce-pre-scaling
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  jobs:
    - name: "morning-peak"
      schedule: "0 9 * * 1-5"    # Weekdays at 9:00
      targetReplicas: 15
    - name: "evening-peak"
      schedule: "0 19 * * 1-5"   # Weekdays at 19:00
      targetReplicas: 20
    - name: "scale-down"
      schedule: "0 23 * * 1-5"   # Weekdays at 23:00, scale back to normal
      targetReplicas: 5

# 2. Reserve buffer replicas: set minReplicas to 60%-70% of peak demand
spec:
  minReplicas: 6  # Keep 6 replicas in steady state, not the bare minimum of 2
  maxReplicas: 30

Pitfall 2: Cascading Scaling Storm

Problem: When traffic spikes, multiple services simultaneously trigger HPA scaling, causing cluster resources to become instantly tight, insufficient nodes, Pending Pods, and a cascading failure.

Best Practice:

  1. Set scaling priorities: Core services scheduled first
  2. Tiered scaling strategy: Different services with different scaling rates
  3. Overprovisioning: Use low-priority placeholder Pods to reserve resources
# Placeholder Pod: use low-priority Pods to reserve resources, automatically evicted during peaks
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
  value: -1               # Lowest priority
  globalDefault: false
  description: "Placeholder Pod for resource reservation"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
  namespace: production
spec:
  replicas: 3
  template:
    spec:
      priorityClassName: overprovisioning
      containers:
        - name: reserve
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 2
              memory: 4Gi

Pitfall 3: Scale-Down Protection

Problem: Traffic peak just passed, HPA immediately scales down, but residual traffic is still being processed, and scaling down causes in-flight requests to be interrupted.

Best Practice:

# 1. Scale-down stabilization window: set a sufficiently long stabilizationWindow
behavior:
  scaleDown:
    stabilizationWindowSeconds: 300  # 5-minute stabilization window
    policies:
      - type: Pods
        value: 1                      # Only scale down 1 Pod at a time
        periodSeconds: 120            # Max 1 scale-down per 2 minutes

# 2. Graceful termination: give the application enough time to complete in-flight requests
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: app
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "15"]  # Wait 15s after LB removal before exiting

# 3. PDB (Pod Disruption Budget) prevents too many Pods from being evicted simultaneously
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
  namespace: production
spec:
  minAvailable: 50%   # Keep at least 50% replicas available
  selector:
    matchLabels:
      app: web-app

Summary

Capacity planning is not a one-time exercise but a continuous engineering practice:

DimensionKey Practice
MeasurementComprehensive metric collection system with clearly defined thresholds
ForecastingData-driven trend analysis + seasonal pattern identification
AutomationCoordinated HPA/VPA/Cluster Autoscaler multi-layer scaling
ProtectionCold-start pre-warming, scaling storm prevention, scale-down protection

There’s one core principle: make data-driven decisions and use automation to handle change. When you can anticipate capacity bottlenecks before failures occur and scale automatically, you’ve truly achieved “managing reliability with engineering methods.”

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

  1. Google SRE Book - Capacity Planning — Google SRE Team, referenced for Google SRE Book - Capacity Planning