Kubernetes Resource Management: Requests and Limits

Semantics of Requests and Limits

In Kubernetes, each container can be configured with CPU and Memory requests and limits. Many developers fail to distinguish between the two, leading to frequent Pod evictions or OOMKilled events.

apiVersion: v1
kind: Pod
metadata:
  name: api-server
spec:
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests:
          cpu: "250m"    # 0.25 core
          memory: "256Mi"
        limits:
          cpu: "500m"    # 0.5 core
          memory: "512Mi"

Core Differences

Dimension	Requests	Limits
Phase	At scheduling time	At runtime
Meaning	Minimum guaranteed resources for the Pod	Maximum resource cap for the Pod
Scheduler behavior	Scheduler uses requests to determine if a node has sufficient resources	Scheduler ignores limits
Runtime behavior	Guaranteed share in cgroups	CPU is throttled; Memory triggers OOMKilled
Overcommit allowed	Yes (sum of all Pod limits on a node can exceed node capacity)	Memory overcommit is not recommended

In simple terms:

Requests are a scheduling promise—“this Pod needs at least 0.25 CPU cores and 256Mi of memory, find a node that can accommodate it”
Limits are a runtime constraint—“this Pod can use at most 0.5 CPU cores; if memory exceeds 512Mi, kill it”

CPU vs Memory: A Fundamental Difference

CPU is a compressible resource: when a container exceeds its CPU limit, the kernel throttles it via CFS (Completely Fair Scheduler). The container slows down but is not killed.

Memory is an incompressible resource: when a container exceeds its Memory limit, the kernel directly triggers the OOM Killer to terminate the container process, with exit code 137.

This article references the official Kubernetes resource management documentation

QoS Classes

Kubernetes automatically assigns a QoS (Quality of Service) class to each Pod based on its requests and limits configuration. QoS determines the Pod’s eviction priority when node resources are tight.

Three-Level QoS Determination Rules

                    ┌─────────────────────────┐
                    │  All containers' requests │
                    │    == limits (CPU+Mem)?  │
                    └──────┬──────────┬───────┘
                        Yes│          │No
                           ▼          ▼
                    ┌──────────┐  ┌──────────────────┐
                    │Guaranteed│  │Any container has  │
                    │ (highest) │  │  requests set?   │
                    └──────────┘  └────┬─────────┬─────┘
                                   Yes│         │No
                                      ▼         ▼
                               ┌───────────┐ ┌────────────┐
                               │Burstable  │ │BestEffort  │
                               │ (medium)  │ │ (lowest)    │
                               └───────────┘ └────────────┘

Guaranteed

Condition: Every container in the Pod has both CPU and Memory requests and limits set, and requests == limits.

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Guaranteed Pods are last to be evicted when node resources are tight. They are suitable for critical services (databases, API gateways, etc.).

Burstable

Condition: The Pod doesn’t meet Guaranteed conditions, but at least one container has requests set.

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "500m"
    memory: "1Gi"

Burstable Pods can use more resources when idle and are evicted at medium priority when the node is under pressure. Most microservices fit this class.

BestEffort

Condition: No container in the Pod has requests or limits set.

resources: {}

BestEffort Pods are first to be evicted and have no resource guarantees. Only suitable for temporary tasks or test Pods.

Checking QoS

# Check a single Pod's QoS
kubectl get pod <pod-name> -o jsonpath='{.status.qosClass}'

# Check QoS for all Pods in a namespace
kubectl get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

OOMKilled Diagnosis and Troubleshooting

Two Types of OOMKilled

Type	Cause	Symptom
Container OOMKilled	Container memory exceeds limits.memory	`Exit Code: 137`, `Reason: OOMKilled`
Node OOMKilled	Node overall memory exhausted, kernel kills processes	Pod evicted, node enters `MemoryPressure`

Diagnostic Steps

# 1. Check Pod status, confirm OOMKilled
kubectl describe pod <pod-name> | grep -A5 "Last State"

# Example output:
# Last State: Terminated
#   Reason: OOMKilled
#   Exit Code: 137

# 2. Check exit code
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# 137

# 3. Check node resource usage
kubectl describe node <node-name> | grep -A10 "Allocated resources"

# 4. Check if node has MemoryPressure
kubectl describe node <node-name> | grep -i "MemoryPressure"

Troubleshooting Approach

# Scenario 1: Container OOMKilled, but node has sufficient memory
# → limits.memory is set too low, needs to be increased

# Scenario 2: Container OOMKilled, node memory is also tight
# → Node is heavily overcommitted, check sum of all Pod requests on the node

# Check total memory requests of all Pods on the node
kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].resources.requests.memory}{"\n"}{end}' --field-selector spec.nodeName=<node-name>

Common Root Causes

Memory leak: The application has a memory leak, with memory continuously growing over time until the limit is triggered. Identify this by monitoring whether the memory curve rises monotonically.
Limit set too low: JVM applications without proper heap memory settings (-Xmx), where the container limit is smaller than the JVM’s default heap size.
Node overcommit: The sum of all Pod requests.memory on a node far exceeds the node’s actual memory, meaning that while scheduling “satisfies” the requests, the actual capacity is insufficient.

# Monitor memory trends (with Prometheus)
# Query: Container memory usage over the past 6 hours
container_memory_working_set_bytes{pod="<pod-name>"} 
  / container_spec_memory_limit_bytes{pod="<pod-name>"} 
  * 100

ResourceQuota and LimitRange

ResourceQuota: Namespace-Level Quotas

ResourceQuota limits the total resources a namespace can request, preventing a single team from exhausting cluster resources.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: production
spec:
  hard:
    requests.cpu: "20"          # Max total CPU requests in namespace
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    persistentvolumeclaims: "10" # Max PVC count
    count/deployments.apps: "20" # Max Deployment count
    count/pods: "50"

# Check namespace quota usage
kubectl describe resourcequota team-quota -n production

# Output:
# Name:            team-quota
# Resource         Used  Hard
# --------         ----  ----
# requests.cpu     12    20
# requests.memory  28Gi  40Gi
# limits.cpu       24    40
# limits.memory    56Gi  80Gi
# pods             18    50

LimitRange: Pod/Container-Level Defaults and Constraints

LimitRange sets default requests/limits for Pods in a namespace and constrains the resource range a single container can request.

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
    - type: Container
      default:          # Default limits when not set
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:   # Default requests when not set
        cpu: "100m"
        memory: "128Mi"
      max:              # Maximum a container can request
        cpu: "4"
        memory: "8Gi"
      min:              # Minimum a container must request
        cpu: "50m"
        memory: "64Mi"

LimitRange addresses:

Forgotten resource settings—Pods without configuration automatically get defaults, avoiding BestEffort status
Over-requesting—max limits a single container to no more than 4 cores and 8Gi
Under-requesting—min ensures at least 50m CPU, preventing starvation

# Verify: create a Pod without resources, observe auto-injected defaults
kubectl run test-pod --image=nginx
kubectl get pod test-pod -o jsonpath='{.spec.containers[0].resources}'
# {"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

Vertical Pod Autoscaler (VPA)

VPA automatically adjusts Pod requests and limits, recommending appropriate values based on historical resource usage data.

Installing VPA

# Clone the autoscaler repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA components
./hack/vpa-up.sh

# Verify
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxx   1/1 Running
# vpa-recommender-xxx            1/1 Running
# vpa-updater-xxx                1/1 Running

VPA Three Modes

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-server-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Auto"  # Off | Initial | Auto
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 2
          memory: 4Gi

Mode	Behavior	Use Case
`Off`	Only provides recommendations, doesn’t modify Pods	Evaluation phase, observe if recommendations are reasonable
`Initial`	Only sets requests at Pod creation, doesn’t modify running Pods	Conservative strategy, avoiding restarts
`Auto`	Automatically modifies requests, evicts and recreates Pods	Production use (must accept restarts)

Understanding VPA Recommendations

kubectl describe vpa api-server-vpa -n production

# Output:
# Recommendation:
#   Container Recommendation:
#     Target:    cpu 250m, memory 384Mi    # Recommended value (most suitable)
#     Lower Bound: cpu 100m, memory 256Mi  # Lower bound (minimum acceptable)
#     Upper Bound: cpu 500m, memory 768Mi  # Upper bound (maximum possibly needed)
#     Uncapped Target: cpu 250m, memory 384Mi  # Recommendation without minAllowed/maxApplied constraints

VPA Considerations

VPA evicts Pods—modifying requests in Auto mode requires recreating Pods, which may cause brief unavailability. Do not use VPA Auto mode for stateful services that cannot tolerate restarts.
VPA and HPA cannot target the same resource—if HPA scales based on CPU and VPA also adjusts CPU requests, they will interfere with each other. Let VPA manage Memory and HPA manage CPU.
Start with Off, then Auto—for newly deployed VPAs, use Off mode to observe recommendations for 3-7 days before switching to Auto.
Set minAllowed / maxAllowed—prevent VPA from recommending extreme values.

Production Practices: Resource Recommendation Methodology

Rules of Thumb

There’s no one-size-fits-all formula, but the following methodology works for most scenarios:

Step 1: Baseline Measurement

# After deploying the app, initially set no limits (but set requests as a floor)
resources:
  requests:
    cpu: "100m"
    memory: "256Mi"

# Run for 7 days, collect Prometheus metrics
# P95 CPU usage, P95 Memory usage, peak Memory

Step 2: Apply Formulas

requests.cpu    = P95 CPU usage × 1.5
requests.memory = P95 Memory usage × 1.2
limits.cpu      = requests.cpu × 2 (or P99 CPU × 2)
limits.memory   = requests.memory × 1.5 (or peak Memory × 1.2)

Step 3: Validate and Iterate

# Observe for 7 days after deployment
# - CPU throttle rate < 5%?  → limits.cpu is reasonable
# - Memory usage < 80%?     → limits.memory is reasonable
# - Pod OOMKilled?           → increase limits.memory
# - Pod scheduling failed?   → check cluster total requests capacity

Recommendations by Application Type

Application Type	requests.cpu	requests.memory	limits.cpu	limits.memory	QoS
API service	250m	256Mi	500m	512Mi	Burstable
Database (MySQL/PG)	1000m	2Gi	2000m	4Gi	Guaranteed
Message queue (Redis)	500m	1Gi	1000m	2Gi	Guaranteed
Log collector (Fluentd)	100m	128Mi	200m	256Mi	Burstable
Batch processing	500m	512Mi	2000m	2Gi	Burstable
Frontend static server	50m	64Mi	100m	128Mi	Burstable

Set Core Services to Guaranteed

# Production core services: requests == limits, QoS = Guaranteed
resources:
  requests:
    cpu: "1000m"
    memory: "2Gi"
  limits:
    cpu: "1000m"
    memory: "2Gi"

Core services (databases, gateways, authentication services) should be set to Guaranteed to ensure:

100% resource guarantee at scheduling time, no resource contention due to overcommit
Last to be evicted when node resources are tight
CPU is never throttled (because requests == limits, CFS quota equals the guaranteed value)

Monitoring Resource Utilization

# CPU usage (relative to limits)
sum(rate(container_cpu_usage_seconds_total{pod=~"api-.*"}[5m])) by (pod)
  / 
sum(kube_pod_container_resource_limits{resource="cpu", pod=~"api-.*"}) by (pod)
  * 100

# Memory usage (relative to limits)
container_memory_working_set_bytes{pod=~"api-.*"}
  /
container_spec_memory_limit_bytes{pod=~"api-.*"}
  * 100

# CPU throttle detection
rate(container_cpu_cfs_throttled_seconds_total{pod=~"api-.*"}[5m]) * 100

Summary

Kubernetes resource management is the cornerstone of production stability. Key takeaways:

Requests determine scheduling, limits determine runtime—both are essential; a Pod without resource settings is a ticking time bomb
QoS class directly affects survival—set core services to Guaranteed, normal services to Burstable, and use BestEffort only for temporary tasks
OOMKilled is the most common container failure—exit code 137, troubleshooting path is describe pod → check limits → check monitoring curves → distinguish leak vs. low config
ResourceQuota + LimitRange are cluster-level guardrails—prevent team resource overcommit and auto-inject defaults for Pods with missing configurations
VPA is a good tool but requires caution—start with Off to observe, then switch to Auto; avoid managing the same resource as HPA
There’s no silver bullet for resource sizing—baseline measurement → formula estimation → production observation → continuous iteration is the correct methodology

Poor resource management will cause even the best architecture to collapse under traffic spikes. Treating requests and limits as the seatbelt for every Pod—this is the SRE baseline.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

official Kubernetes resource management documentation — Kubernetes Official, referenced for official Kubernetes resource management documentation

Semantics of Requests and Limits#

Core Differences#

CPU vs Memory: A Fundamental Difference#

QoS Classes#

Three-Level QoS Determination Rules#

Guaranteed#

Burstable#

BestEffort#

Checking QoS#

OOMKilled Diagnosis and Troubleshooting#

Two Types of OOMKilled#

Diagnostic Steps#

Troubleshooting Approach#

Common Root Causes#

ResourceQuota and LimitRange#

ResourceQuota: Namespace-Level Quotas#

LimitRange: Pod/Container-Level Defaults and Constraints#

Vertical Pod Autoscaler (VPA)#

Installing VPA#

VPA Three Modes#

Understanding VPA Recommendations#

VPA Considerations#

Production Practices: Resource Recommendation Methodology#

Rules of Thumb#

Recommendations by Application Type#

Set Core Services to Guaranteed#

Monitoring Resource Utilization#

Summary#

References & Acknowledgments#