One Namespace Ate 80% of Cluster Memory: 4 Production-Grade Decisions for K8s ResourceQuota Multi-Tenant Governance

Overview

The night before a major promotional event, all Pods in the test environment were stuck in Pending. kubectl top nodes showed node CPU usage at 35% and memory under 50%. The Deployment YAML had reasonable requests and limits. The problem? Not a resource shortage — the “ledger” was maxed out. A namespace’s ResourceQuota had hit its ceiling, and the admission controller was rejecting all new Pod creation requests.

This incident forced me to re-examine the entire chain of K8s namespace-level resource governance. Many people think setting LimitRange is sufficient, or that ResourceQuota is just “setting a budget for the dev team.” In reality, the interplay between ResourceQuota, LimitRange, and QoS directly determines resource fairness and stability in multi-tenant clusters.

This article covers 4 production-grade decisions: LimitRange and ResourceQuota must always appear as a pair, quota dimensions must cover compute+object+storage, use ScopeSelector for fine-grained traffic splitting, and quota exhaustion needs alerting and automated remediation. Each decision comes with copy-paste-ready YAML templates and real production incident data.

What ResourceQuota and LimitRange Actually Manage

Let’s explain in plain terms. K8s resource management has two layers:

• Pod-level: requests and limits control how much resource a single container can use. The scheduler uses requests to decide which node to place the Pod on, and the runtime uses limits for throttling and OOM kills. This is covered in detail in Related: Kubernetes Resource Management: Requests and Limits.
• Namespace-level: ResourceQuota and LimitRange control how much cluster resource an entire namespace can consume.

Think of it this way. LimitRange is the “procurement approver” — when each container submits a resource request, it checks whether the requests/limits are within reasonable bounds and rejects anything out of range. ResourceQuota is the “financial auditor” — it watches the total resource consumption of the entire namespace and blocks new Pod creation once the ceiling is exceeded.

Neither can be missing. With only LimitRange and no ResourceQuota, users can create unlimited Pods that comply with LimitRange, eventually exhausting node resources. With only ResourceQuota and no LimitRange, users can submit a Pod with request=10m and limit=64Gi — the scheduler sees 10m as negligible and places it anywhere, but at runtime this Pod can consume 64Gi of memory, making other Pods’ resource guarantees meaningless.

What ResourceQuota Can Limit

ResourceQuota operates at the namespace level and can restrict three categories of resources:

A critical detail: if you restrict cpu or memory requests/limits in a ResourceQuota, then every new Pod in that namespace must explicitly set requests and limits for those resources, or the admission controller will reject the creation. Many people hit this pitfall — after adding ResourceQuota, Pods that previously didn’t set resource limits suddenly fail to create.

What LimitRange Does

LimitRange is the “default value injector” and “range validator” within a namespace:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: production
spec:
  limits:
  # Default values: auto-injected when Pod doesn't set resources
  - type: Container
    default:        # Default limits
      cpu: "500m"
      memory: "512Mi"
    defaultRequest: # Default requests
      cpu: "100m"
      memory: "128Mi"
    # Range validation: requests/limits outside range are rejected
    max:
      cpu: "4"
      memory: "8Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  # Pod-level total limits
  - type: Pod
    max:
      cpu: "8"
      memory: "16Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
  # Storage range
  - type: PersistentVolumeClaim
    max:
      storage: "100Gi"
    min:
      storage: "1Gi"

This configuration does three things:

1. Inject defaults: When developers submit a Pod without resources, the LimitRanger admission controller automatically fills in default and defaultRequest. The Pod won’t be rejected for missing resources.
2. Validate ranges: Any container’s cpu/memory exceeding max or below min will fail creation. Prevents submitting extreme configs like request=10m, limit=64Gi.
3. Limit PVC size: Prevents single PVCs from requesting excessive storage.

Incident Retrospective: A “Quota Miscalculation” Production Outage

Let me break down this incident thoroughly — it explains why the four decisions below are all essential.

An e-commerce platform (anonymized), cluster with 6 nodes, each 16C/64G. Total 96C/384G. Four namespaces divided by team: order, payment, logistics, common. Each namespace had ResourceQuota with requests.cpu=24, requests.memory=96Gi. Seemed balanced — each team got 1/4 of resources.

The problem was in the common namespace. It hosted utility applications: config center, task scheduler, message queue consumers. Developers consistently set only requests without limits for these (“utility apps don’t spike”), and no LimitRange with max constraints was configured.

On the day of the promotion, the message queue backed up, and consumer Pods started consuming memory aggressively. Without limits, each Pod could use all available memory on the node. Three of six nodes were overwhelmed by common namespace Pods, pushing order and payment Pods onto the remaining three nodes. CPU scheduling latency went from 5ms to 80ms, and payment API P99 jumped from 200ms to 1200ms.

The root cause? Not that common namespace exceeded ResourceQuota — its requests.cpu was indeed under 24 cores. But requests only affect scheduling, not runtime. Without limits, there was no runtime cap. ResourceQuota manages the declared total of requests/limits, not actual runtime consumption.

This is the core of the first decision: ResourceQuota manages totals, LimitRange manages per-unit, both are indispensable.

Four Production-Grade Governance Decisions

Decision 1: LimitRange and ResourceQuota Must Always Appear as a Pair

This is the most fundamental and most easily overlooked rule. Many people configure ResourceQuota and think they’re safe, but without LimitRange as a backstop, users can submit extreme configurations:

Extreme config A is the most dangerous. The scheduler sees requests=10m, thinks this Pod barely uses resources, and places it anywhere. But without memory limits, once memory spikes, it eats the entire node.

My recommendation: Every business namespace must have both ResourceQuota and LimitRange configured. Set LimitRange’s max values to 1/3 to 1/2 of the namespace’s allocated resource total. For example, if the namespace has 24 cores of CPU, set max.cpu to 8 (1/3) and max.memory to 1/4 of allocated total.

# Production-grade LimitRange template
apiVersion: v1
kind: LimitRange
metadata:
  name: production-limits
  namespace: order-service
spec:
  limits:
  - type: Container
    default:
      cpu: "1000m"
      memory: "1Gi"
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"
    max:
      cpu: "8"
      memory: "16Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  - type: Pod
    max:
      cpu: "16"
      memory: "32Gi"

Key points of this template:

• default and defaultRequest ensure Pods missing resources still get reasonable defaults
• max limits single-container ceiling, preventing one container from eating the entire namespace quota
• min prevents excessively small requests that mislead the scheduler
• Pod-level max limits total resources across all containers in a Pod

Decision 2: Quota Dimensions Must Cover Compute + Object + Storage

Most teams only configure CPU and memory in ResourceQuota. That’s far from enough. Production quotas should cover three dimensions:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: comprehensive-quota
  namespace: order-service
spec:
  hard:
    # ===== Compute dimension =====
    requests.cpu: "24"
    requests.memory: "96Gi"
    limits.cpu: "48"
    limits.memory: "192Gi"
    
    # ===== Storage dimension =====
    requests.storage: "500Gi"
    persistentvolumeclaims: "20"
    requests.ephemeral-storage: "200Gi"
    
    # ===== Object count dimension =====
    pods: "100"
    services: "30"
    services.loadbalancers: "2"
    services.nodeports: "5"
    secrets: "50"
    configmaps: "50"
    replicationcontrollers: "0"      # Forbid RC, use Deployment uniformly

Why cover all three dimensions? Here are real production incidents:

Incident 1: ConfigMap explosion. A team stored config files in ConfigMaps — one per microservice, each containing a full YAML config. After running 50+ microservices for a year, ConfigMap count exceeded 200. The etcd size limit per ConfigMap is 1MB, and 200 ConfigMaps increased etcd memory usage by 30%. Worse, kubectl get cm -n order took 15 seconds to return.

Incident 2: Service NodePort exhaustion. Developers created NodePort-type Services for every service. The node pool’s available ports (default 30000-32767) ran out quickly. New Service creation failed with provided port is already allocated.

Incident 3: Unlimited PVC growth. A log analysis service requested a 50Gi PVC for each new index. In 3 months, it consumed 2TB of storage. The cluster’s storage backend was cloud disks, and costs jumped 5x.

So quotas can’t just look at CPU and memory. Object count and storage quotas are equally critical.

Here’s the ratio I recommend in production:

Decision 3: Use ScopeSelector for Fine-Grained Quota Splitting

This is the most easily overlooked advanced feature of ResourceQuota. In the same namespace, long-running services and CI/CD short-lived Jobs have completely different resource needs. If you use a single ResourceQuota for both, CI Jobs can eat up large amounts of quota, preventing normal services from scaling.

ResourceQuota supports 4 scopes:

My recommendation: Create multiple ResourceQuotas within the same namespace, using scopes to split traffic. This way CI Job resource consumption doesn’t squeeze normal service quotas:

# Production service quota: only manages non-terminating Pods
apiVersion: v1
kind: ResourceQuota
metadata:
  name: long-running-quota
  namespace: order-service
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "80Gi"
    limits.cpu: "40"
    limits.memory: "160Gi"
    pods: "60"
  scopeSelector:
    matchExpressions:
    - scopeName: NotTerminating
---
# CI/CD Job quota: only manages terminating Pods, separate allocation
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ci-job-quota
  namespace: order-service
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "16Gi"
    limits.cpu: "8"
    limits.memory: "32Gi"
    pods: "20"
  scopeSelector:
    matchExpressions:
    - scopeName: Terminating

With this configuration, even if CI starts 20 Jobs at once, they consume at most 4 cores of CPU and 16Gi of memory. The normal service quota of 20 cores is completely unaffected.

Multiple ResourceQuotas in the same namespace are additive. The K8s admission controller checks all matching ResourceQuotas — if any one hits its ceiling, new Pod creation is rejected. This mechanism needs consideration during design — don’t let scope overlaps cause quota conflicts.

Decision 4: Quota Exhaustion Alerting and Automated Remediation

Quota exhaustion isn’t a “set it and forget it” problem — it requires continuous monitoring and governance. I hit this pitfall at a mobility project: quotas were set but no alerts were configured. We didn’t discover ResourceQuota had been maxed out for days until developers reported “Pods won’t start.”

Alert Configuration

ResourceQuota usage is available via kubectl describe quota or Prometheus kube_resourcequota metrics. Key metrics:

Recommended alert rules (PromQL):

# Alert when quota usage exceeds 80%
100 * (
  kube_resourcequota{type="used"} 
  / kube_resourcequota{type="hard"}
) > 80

# Alert when Pod count nears limit (less than 5 remaining)
kube_resourcequota{resource="pods", type="hard"} 
- kube_resourcequota{resource="pods", type="used"} < 5

For the alert system design, refer to the layered approach in Related: Alerting Strategy Design: From Noise to Signal.

Automated Remediation: Namespace Inspection

Quota governance isn’t just about setting values — it includes cleaning up invalid namespaces. I hit a pitfall before a major promotion — CI/CD pipeline created temporary namespaces (like ci-build-20261015-001234) for each build but forgot to clean up. After two weeks, 300+ empty namespaces piled up, each with ResourceQuota and LimitRange objects. Though no Pods, 300+ ResourceQuota objects caused controller-manager sync loop CPU to spike, and APIServer response latency went from 100ms to 2s.

The fix was a CronJob that scans and cleans up namespaces with no Pods older than 7 days:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: namespace-cleaner
  namespace: kube-system
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: namespace-cleaner-sa
          containers:
          - name: kubectl
            image: bitnami/kubectl:1.31
            command:
            - /bin/sh
            - -c
            - |
              kubectl get namespaces -o json | jq -r '
                .items[] 
                | select(.metadata.name | startswith("ci-build-")) 
                | select(.metadata.creationTimestamp 
                    | . < (now - 604800 | todate))
                | .metadata.name
              ' | while read ns; do
                pod_count=$(kubectl get pods -n "$ns" --no-headers 2>/dev/null | wc -l)
                if [ "$pod_count" -eq 0 ]; then
                  echo "Deleting empty namespace: $ns"
                  kubectl delete namespace "$ns" --ignore-not-found
                fi
              done              
          restartPolicy: OnFailure

Quota and Scheduler Interaction: Easily Overlooked Pitfalls

ResourceQuota quota checking happens at the admission stage — before the API request is written to etcd. But the scheduler’s scheduling decision happens after Pod creation. This means a Pod can pass quota checks (be allowed to create) but fail to schedule.

This time gap creates a phenomenon: quota shows “used 24/24 cores” but only 20 Pods may be Running, with 4 stuck in Pending. Quota was “pre-deducted” but not actually used.

The interaction implications covered in Related: Kubernetes Scheduler Principles and Tuning and Related: Kubernetes Autoscaling: HPA/VPA/CA Deep Dive:

1. Inflated quota: If a namespace has many Pending Pods, quota shows full but resources aren’t actually consumed. Other Pod creation is rejected even though nodes have spare capacity. Solution: monitor Pending Pod count and alert above threshold for manual intervention.

2. Deleting Pod doesn’t immediately release quota: After deleting a Pod, the used value doesn’t instantly decrease by 1. The ResourceQuota controller has a sync period (default 5 minutes), during which quota data is “stale.” If you create a new Pod immediately after deletion, it may be rejected. Wait a few minutes and retry.

3. HPA scaling blocked by quota: HPA scales replicas based on CPU utilization, but if namespace quota is full, new Pods fail to create, making HPA’s scaling action pointless. Namespaces using HPA should reserve 30%+ buffer in quota for scaling.

QoS Classes and Quota Coordination

QoS (Quality of Service) is K8s’ three-tier Pod classification that determines eviction order when node resources are tight:

The coordination point between ResourceQuota and QoS: if ResourceQuota restricts requests for cpu/memory, then Pods in that namespace must set requests. This means BestEffort Pods (without resources) cannot be created in that namespace. This is actually good — production environments shouldn’t have BestEffort Pods anyway.

But there’s a pitfall: if you use a BestEffort scope ResourceQuota to limit BestEffort Pod count, while the main ResourceQuota restricts requests.cpu, BestEffort Pods can’t be created at all (because not setting requests is rejected). These two configs contradict each other. Solution: don’t configure a BestEffort scope ResourceQuota in production namespaces — use NotBestEffort scope to block BestEffort Pods directly.

Production Quota Template

Here’s the complete template I’ve verified in production, suitable for a mid-size team (30-50 microservices):

# ============ LimitRange ============
apiVersion: v1
kind: LimitRange
metadata:
  name: production-limits
  namespace: team-order
spec:
  limits:
  - type: Container
    default:
      cpu: "1000m"
      memory: "1Gi"
    defaultRequest:
      cpu: "200m"
      memory: "256Mi"
    max:
      cpu: "8"
      memory: "16Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  - type: Pod
    max:
      cpu: "16"
      memory: "32Gi"
    min:
      cpu: "100m"
      memory: "128Mi"
  - type: PersistentVolumeClaim
    max:
      storage: "100Gi"
    min:
      storage: "5Gi"
---
# ============ Long-running service quota ============
apiVersion: v1
kind: ResourceQuota
metadata:
  name: service-quota
  namespace: team-order
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "80Gi"
    limits.cpu: "40"
    limits.memory: "160Gi"
    requests.storage: "500Gi"
    persistentvolumeclaims: "20"
    requests.ephemeral-storage: "100Gi"
    pods: "60"
    services: "30"
    services.loadbalancers: "2"
    services.nodeports: "3"
    secrets: "40"
    configmaps: "40"
  scopeSelector:
    matchExpressions:
    - scopeName: NotTerminating
---
# ============ CI/CD Job quota ============
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ci-job-quota
  namespace: team-order
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "16Gi"
    limits.cpu: "8"
    limits.memory: "32Gi"
    pods: "20"
  scopeSelector:
    matchExpressions:
    - scopeName: Terminating
---
# ============ Block BestEffort Pods ============
apiVersion: v1
kind: ResourceQuota
metadata:
  name: block-besteffort
  namespace: team-order
spec:
  hard:
    pods: "0"
  scopeSelector:
    matchExpressions:
    - scopeName: BestEffort

The last block-besteffort is clever: setting BestEffort scope pods quota to 0 means Pods without resources can’t be created at all — more direct than LimitRange alone.

Quota Initialization Automation

Manually applying these YAMLs for every new namespace isn’t realistic. Use K8s namespace initialization via Admission Webhook or Kyverno policy:

# Kyverno policy: auto-inject LimitRange on namespace creation
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-namespace-quota
spec:
  rules:
  - name: inject-limits
    match:
      resources:
        kinds:
        - Namespace
    generate:
      apiVersion: v1
      kind: LimitRange
      name: default-limits
      namespace: "{{request.object.metadata.name}}"
      data:
        spec:
          limits:
          - type: Container
            default:
              cpu: "1000m"
              memory: "1Gi"
            defaultRequest:
              cpu: "200m"
              memory: "256Mi"
            max:
              cpu: "8"
              memory: "16Gi"
            min:
              cpu: "50m"
              memory: "64Mi"

Common Anti-Patterns in Quota Management

Anti-pattern 1: Same quota for all namespaces

Test, staging, and production environments have vastly different resource needs. Giving all namespaces 24 cores means test wastes resources and production doesn’t have enough. Correct approach: tier by environment — test 8 cores, staging 16 cores, production 32 cores.

Anti-pattern 2: Set quota and never adjust

Businesses grow, and quotas need periodic adjustment. I recommend quarterly reviews of quota utilization — below 50% consider reducing, above 80% consider expanding. Quotas aren’t “set once and fixed” — they should track business rhythm.

Anti-pattern 3: Using quotas instead of capacity planning

ResourceQuota is a resource governance tool, not a capacity planning tool. No matter how well quotas are set, if total cluster resources are insufficient, it’s meaningless. Quotas solve “fair distribution” problems; capacity planning solves “is there enough” problems. They can’t replace each other.

Anti-pattern 4: Ignoring requests-limits gap

Some teams set requests == limits (Guaranteed QoS) for “safety.” This is correct for core services. But for batch tasks or CI Jobs, requests=limits means low node utilization — the scheduler reserves resources by limits even if the Pod only uses requests-level amounts. Guaranteed fits core services, Burstable fits general business — don’t apply a blanket rule.

Anti-pattern 5: Not monitoring Pending Pod quota reasons

Pods stuck in Pending have many causes: insufficient node resources, unsatisfiable scheduling constraints, image pull failures. One cause is quota exhaustion — often overlooked because the Pod Events show Forbidden: exceeded quota but many people don’t check Events. Add an alert: when Pending Pod count exceeds 5, automatically check if quota is the cause.

Summary

Back to the opening incident. The fix was: add LimitRange to the common namespace (max.memory=4Gi), forcing every Pod to set limits. Split ResourceQuota using scopes to isolate CI Job quota. After restarting the next day, no more resource contention-induced service degradation occurred.

The core logic of the four decisions:

1. LimitRange + ResourceQuota as a pair: LimitRange governs per-Pod “extreme configs,” ResourceQuota governs namespace “total ceiling.” Missing either creates a vulnerability.
2. Three-layer quota dimensions: Compute (CPU/memory), storage (PVC/ephemeral), object count (pods/services/secrets/configmaps) — all essential. Object count explosion and storage runaway can take down a cluster just as effectively.
3. ScopeSelector fine-grained splitting: Long-running services and CI Jobs use separate quota pools, preventing CI from eating service resources. Multiple ResourceQuotas are additive with scopes for isolation.
4. Quota alerting + automated remediation: 80% warning, 95% urgent, Pending Pod quota cause checking, CronJob auto-cleaning invalid namespaces. Quotas need continuous ops, not “set and forget.”

K8s resource governance is essentially “ledger management” — ResourceQuota is the general ledger, LimitRange is the approval rule, QoS is the priority. All three must work together for multi-tenant clusters to run stably. Don’t wait until Pods are mass-Pending to remember configuring quotas.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Resource Quotas — Kubernetes Official Documentation, complete concept definition of ResourceQuota, quota types, and scope mechanism
2. Limit Ranges — Kubernetes Official Documentation, LimitRange default value injection and range validation mechanism
3. Pod Quality of Service Classes — Kubernetes Official Documentation, QoS class classification standards and eviction priority
4. Configure Memory and CPU Quotas for a Namespace — Kubernetes Official Tutorial, namespace quota configuration hands-on
5. Kubernetes Resource Limit Management — Tencent Cloud Developer Community, illustrated explanation of ResourceQuota, LimitRange, and Resource Limits coordination
6. 11 Kubernetes Pitfalls You Can’t Avoid — Baidu Baijiahao, BestEffort Pod risks without resource requests and QoS optimization recommendations