cgroup v2 Complete Guide: From Architecture Principles to Production Practices

Overview

In today’s era of cloud-native and containerized technologies, Linux cgroups (control groups) serve as the kernel-level foundation for resource isolation and limiting. From Docker container memory limits to Kubernetes Pod CPU Requests/Limits, the underlying mechanism relies on cgroups. However, cgroup v1’s multi-hierarchy architecture, inconsistent controller behavior, and confusing thread model have caused numerous operational headaches in production.

cgroup v2, as a complete reimagining of v1, adopts a unified hierarchy architecture that fundamentally addresses v1’s design flaws. Introduced in Linux 4.5 and matured through multiple kernel iterations, cgroup v2 is now stable on 5.x kernels. Major distributions including Ubuntu 22.04+, RHEL 9+, and Debian 12+ default to cgroup v2, with Docker and Kubernetes offering full support.

This article starts from cgroup v2’s architectural principles, dives deep into core controller mechanisms, covers practical configurations with systemd integration, Docker container limits, and Kubernetes scenarios, and finally addresses v1-to-v2 migration strategies to help you master this critical technology in production.

cgroup v1 vs v2: Why a Rewrite Was Needed

Core Pain Points of v1

Cgroup v1 was designed so that each controller could be mounted on independent hierarchy trees. This provided flexibility but introduced significant issues:

Problem Area	v1 Behavior	Impact
Multi-hierarchy	Each controller on separate tree	Process in different cgroups per controller, fragmented management view
Thread model	Threads spread across cgroups	Confusing resource accounting, difficult attribution
Controller coordination	Controllers operate independently	No cross-resource unified policies (e.g., CPU + memory linkage)
Delegation security	Coarse-grained delegation	Container escape risk, unclear security boundaries
Interface consistency	Different naming/semantics across controllers	High cognitive load, difficult script maintenance

v2 Design Philosophy

The core design principle of cgroup v2 is the unified hierarchy: the entire system has a single cgroup tree, and all controllers are mounted on the same tree. A process belongs to exactly one cgroup, which can have multiple controllers (CPU, memory, IO, etc.) enabled simultaneously.

/sys/fs/cgroup/          ← Unified mount point (cgroup2 filesystem)
├── cgroup.controllers   ← Global available controller list
├── cgroup.subtree_control ← Subtree enabled controllers
├── cpu.weight            ← CPU weight
├── memory.max            ← Memory limit
├── io.max                ← IO limit
├── system.slice/         ← systemd system services
│   ├── nginx.service/
│   │   ├── cpu.max       ← Nginx CPU limit
│   │   └── memory.max    ← Nginx memory limit
│   └── docker.service/
├── user.slice/           ← User sessions
└── myapp/                ← Custom cgroup
    ├── cpu.max
    ├── memory.max
    └── cgroup.procs      ← Process list in this cgroup

Key Differences: v1 vs v2

Feature	cgroup v1	cgroup v2
Hierarchy	Multi-hierarchy, per-controller trees	Single unified hierarchy tree
Mount	Each controller mounted separately	Unified `cgroup2` filesystem mount
Thread model	Threads across cgroups	Threaded controllers, same-process threads default to same cgroup
Controller coordination	Independent controllers	Unified resource policy, cross-controller coordination
Delegation security	Coarse-grained	Subtree delegation, `nsdelegate` mount option
Memory accounting	cgroup-level only	Recursive accounting (`memory_recursiveprot`)
Process placement	Process in multiple cgroups	Process in one cgroup only
Kernel version	2.6.24+	4.5+ (full features 5.2+)

Environment Check and Enablement

Confirming Current cgroup Version

# Check cgroup filesystem mount
mount | grep cgroup

# cgroup v2 output example:
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

# cgroup v1 output example (multiple lines):
# tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
# cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
# cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
# cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)

If the output shows only cgroup2, the system is using cgroup v2. Multiple cgroup mount entries (per-controller) indicate v1 or v1/v2 hybrid mode.

# View available controllers
cat /sys/fs/cgroup/cgroup.controllers

# Output example:
# cpuset cpu io memory hugetlb pids rdma misc

Note: /proc/cgroups is only compatible with cgroup v1, not v2. Use cgroup.controllers to check v2 controllers.

Checking Kernel Version Support

cgroup v2 controllers were introduced across different kernel versions:

Controller	Function	Minimum Kernel
`cpu`	CPU bandwidth limiting	4.15
`cpuset`	CPU affinity and NUMA nodes	5.0
`memory`	Memory limiting and accounting	4.5
`io`	IO bandwidth allocation	4.5
`pids`	Process count limiting	4.5
`devices`	Device file access control (BPF)	4.15
`rdma`	RDMA resource allocation	4.11
`hugetlb`	Huge page usage limiting	5.6
`misc`	Miscellaneous resource control	5.13

Enabling cgroup v2 on Older Systems

For systems that support but don’t default to v2, switch via kernel boot parameters:

# GRUB configuration (edit /etc/default/grub)
# Add to GRUB_CMDLINE_LINUX:
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"

# Update GRUB and reboot
sudo update-grub    # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL/CentOS
sudo reboot

# Verify after reboot
mount | grep cgroup2

For hybrid v1/v2 support (legacy container runtime compatibility):

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=yes"

Core Controllers Explained

CPU Controller

The CPU controller provides two resource control modes: weight-based and max-based (bandwidth limiting).

Weight Mode (cpu.weight)

Weight mode distributes CPU time slices proportionally, similar to nice values but more precise. Range 1-10000, default 100.

# Create cgroup and set CPU weight
sudo mkdir /sys/fs/cgroup/myapp
echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control
echo 500 > /sys/fs/cgroup/myapp/cpu.weight   # Weight 500 (5x default of 100)

# Comparison: v1's cpu.shares
# v1: echo 512 > /sys/fs/cgroup/cpu/myapp/cpu.shares
# v2: echo 500 > /sys/fs/cgroup/myapp/cpu.weight  (different semantics, similar effect)

Weight distribution logic:

# Two cgroups with weights 100 and 300
cgroup_A: cpu.weight = 100  → gets 100/(100+300) = 25% of CPU
cgroup_B: cpu.weight = 300  → gets 300/(100+300) = 75% of CPU

# When CPU is idle, both can use more than their allocated share
# Weight only takes effect during CPU contention

Bandwidth Limit Mode (cpu.max)

Bandwidth limiting sets a hard cap, formatted as $MAX $PERIOD:

# Limit to 50% of a single core (max 50ms per 100ms period)
echo "50000 100000" > /sys/fs/cgroup/myapp/cpu.max

# Explanation: max=50000us, period=100000us → 50000/100000 = 50% of one core

# Limit to 2 full cores
echo "200000 100000" > /sys/fs/cgroup/myapp/cpu.max

# No limit (default)
echo "max" > /sys/fs/cgroup/myapp/cpu.max

CPU Affinity (cpuset)

The cpuset controller for cgroup v2 is available since kernel 5.0:

# Enable cpuset controller
echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control

# Restrict processes to CPUs 0-3
echo "0-3" > /sys/fs/cgroup/myapp/cpuset.cpus

# Restrict memory nodes (NUMA scenarios)
echo "0" > /sys/fs/cgroup/myapp/cpuset.mems

Memory Controller

The memory controller is one of the most important controllers in cgroup v2, providing memory limiting, accounting, and OOM control.

Basic Memory Limits

# Set max memory to 1GB
echo 1073741824 > /sys/fs/cgroup/myapp/memory.max

# Set max memory to 1GB (human-readable format, kernel 5.9+)
echo "1G" > /sys/fs/cgroup/myapp/memory.max

# Limit swap usage (kernel 5.8+)
echo 536870912 > /sys/fs/cgroup/myapp/memory.swap.max   # 512MB swap

# Check current memory usage
cat /sys/fs/cgroup/myapp/memory.current
# Output: 536870912

# Check memory peak
cat /sys/fs/cgroup/myapp/memory.peak
# Output: 805306368

# View detailed memory statistics
cat /sys/fs/cgroup/myapp/memory.stat

Key memory.stat fields explained:

anon 536870912          # Anonymous memory (heap, stack, etc.)
file 268435456          # File cache
kernel 67108864         # Kernel memory
sock 33554432           # Socket buffers
shmem 16777216          # Shared memory
slab_reclaimable 8388608    # Reclaimable slab
slab_unreclaimable 4194304  # Unreclaimable slab
pgfault 1234567         # Page fault count
pgmajfault 1234         # Major page fault count
oom_kill 0               # OOM kill count

Memory Recursive Protection

cgroup v2 introduces the memory_recursiveprot mount option, where child cgroup memory protection works recursively. Combined with memory.low for graceful degradation:

# Set memory floor protection (protected from reclaim below this)
echo 536870912 > /sys/fs/cgroup/myapp/memory.low   # 512MB protected

# Set memory ceiling
echo 1073741824 > /sys/fs/cgroup/myapp/memory.max  # 1GB ceiling

# Under memory pressure:
# - Below memory.low: protected, not reclaimed first
# - Between memory.low and memory.max: can be reclaimed
# - Above memory.max: triggers OOM or kill

OOM Control

# Prevent OOM killer from killing processes in this cgroup
echo 1 > /sys/fs/cgroup/myapp/memory.oom.group

# When memory.oom.group=1, any process triggering OOM
# causes ALL processes in the cgroup to be killed

# View OOM events
cat /sys/fs/cgroup/myapp/memory.events
# Output:
# oom 0
# oom_kill 0
# oom_group_kill 0

IO Controller

The IO controller allows limiting read/write bandwidth and IOPS for block devices.

# Enable IO controller
echo "+io" > /sys/fs/cgroup/cgroup.subtree_control

# View available block devices
cat /sys/fs/cgroup/myapp/io.stat
# Output example:
# 8:0 rbytes=1234567 wbytes=2345678 rios=100 wios=200
# 8:16 rbytes=345678 wbytes=456789 rios=50 wios=60

# Limit /dev/sda (8:0) write bandwidth to 10MB/s
echo "8:0 rbps=max wbps=10485760" > /sys/fs/cgroup/myapp/io.max

# Limit read/write IOPS
echo "8:0 riops=max wiops=1000" > /sys/fs/cgroup/myapp/io.max

# Combined bandwidth and IOPS limits
echo "8:0 rbps=10485760 wbps=10485760 riops=1000 wiops=1000" > /sys/fs/cgroup/myapp/io.max

Getting device numbers:

# Get major:minor for /dev/sda
lsblk -o NAME,MAJ:MIN /dev/sda
# Output: sda 8:0

# Or using stat
stat -c '%t:%T' /dev/sda
# Output: 8:0 (hexadecimal, needs decimal conversion)

PID Controller

The PID controller limits the number of processes/threads in a cgroup, preventing fork bombs and resource leaks:

# Limit to 500 processes
echo 500 > /sys/fs/cgroup/myapp/pids.max

# Check current process count
cat /sys/fs/cgroup/myapp/pids.current
# Output: 42

# When pids.current reaches pids.max, fork() fails with EAGAIN

systemd Integration

On modern Linux systems, systemd is the primary manager of cgroup v2. systemd auto-mounts the cgroup2 filesystem at boot and creates corresponding cgroups for each service unit.

systemd Resource Limit Configuration

# Create a systemd service with resource limits
sudo tee /etc/systemd/system/heavy-app.service << 'EOF'
[Unit]
Description=Heavy Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/heavy-app
# CPU limit: max 2 cores
CPUQuota=200%
# CPU weight (relative priority)
CPUWeight=500
# Memory limits
MemoryMax=2G
MemoryLow=512M
# IO weight
IOWeight=500
# Process count limit
TasksMax=300
# Restart policy
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start heavy-app

Verifying systemd cgroup Configuration

# View service cgroup resource usage
systemctl status heavy-app

# Output includes CGroup info:
# CGroup: /system.slice/heavy-app.service
#         ├─1234 /usr/bin/heavy-app

# View detailed cgroup controller settings
systemctl show heavy-app | grep -E "CPUQuota|MemoryMax|MemoryLow|IOWeight|TasksMax"
# Output:
# CPUQuotaPerSecUSec=2s
# MemoryMax=2147483648
# MemoryLow=536870912
# IOWeight=500
# TasksMax=300

# Real-time cgroup monitoring
systemd-cgtop
# Output example:
# Control Group           Tasks   %CPU   Memory  Input/s Output/s
# /system.slice/heavy-app  1      45.0   1.2G    0B/s    10MB/s
# /system.slice/nginx      4       2.1   256M    0B/s    0B/s

Dynamic Resource Adjustment with systemctl

systemd allows dynamically modifying cgroup limits without restarting services:

# Temporarily increase memory limit
systemctl set-property heavy-app MemoryMax=4G

# Temporarily adjust CPU quota
systemctl set-property heavy-app CPUQuota=300%

# Set IO weight
systemctl set-property heavy-app IOWeight=800

# To make permanent (write to config file)
systemctl set-property --runtime=false heavy-app MemoryMax=4G

Threaded Cgroups

For scenarios requiring different CPU policies for threads within the same process, cgroup v2 provides threaded cgroups:

# Create threaded cgroup
mkdir /sys/fs/cgroup/myapp/worker-threads
echo threaded > /sys/fs/cgroup/myapp/worker-threads/cgroup.type

# Add threads to threaded cgroup
echo $TID > /sys/fs/cgroup/myapp/worker-threads/cgroup.threads

# Set different CPU weights for different thread groups
echo 200 > /sys/fs/cgroup/myapp/worker-threads/cpu.weight
echo 800 > /sys/fs/cgroup/myapp/main-threads/cpu.weight

Manual Cgroup Management in Practice

Creating and Managing Custom Cgroups

Here is a complete practical example demonstrating manual cgroup creation, resource configuration, and process management:

#!/bin/bash
# cgroup-v2-manage.sh — Manual cgroup v2 management example

set -euo pipefail

CGROOT="/sys/fs/cgroup"
CGROUP_NAME="batch-job"

# Ensure running as root
if [ "$(id -u)" -ne 0 ]; then
    echo "Please run as root" >&2
    exit 1
fi

# Create cgroup
echo ">>> Creating cgroup: $CGROUP_NAME"
mkdir -p "$CGROOT/$CGROUP_NAME"

# Enable controllers (in parent's subtree_control)
echo "+cpu +memory +io +pids" > "$CGROOT/cgroup.subtree_control"

# Set CPU limit: 30% of single core
echo "30000 100000" > "$CGROOT/$CGROUP_NAME/cpu.max"

# Set memory limit: 512MB
echo 536870912 > "$CGROOT/$CGROUP_NAME/memory.max"

# Set swap limit: 128MB
echo 134217728 > "$CGROOT/$CGROUP_NAME/memory.swap.max"

# Set process count limit: 100
echo 100 > "$CGROOT/$CGROUP_NAME/pids.max"

# Set IO limit: 5MB/s write
DEVICE=$(stat -c '%t:%T' /dev/sda)
MAJOR=$((0x$(stat -c '%t' /dev/sda)))
MINOR=$((0x$(stat -c '%T' /dev/sda)))
echo "$MAJOR:$MINOR wbps=5242880" > "$CGROOT/$CGROUP_NAME/io.max"

echo ">>> Resource limits set:"
echo "    CPU:   30% single core"
echo "    Memory: 512MB"
echo "    Swap:  128MB"
echo "    Procs: 100"
echo "    IO:    5MB/s write"

# Start process and add to cgroup
echo ">>> Starting process..."
python3 /opt/batch-job/main.py &
PID=$!

# Add process to cgroup
echo $PID > "$CGROOT/$CGROUP_NAME/cgroup.procs"

echo ">>> Process PID=$PID added to cgroup $CGROUP_NAME"
echo ">>> Monitoring resource usage (Ctrl+C to stop)..."

# Real-time monitoring
while kill -0 "$PID" 2>/dev/null; do
    echo "--- $(date '+%H:%M:%S') ---"
    echo "CPU usage: $(cat $CGROOT/$CGROUP_NAME/cpu.stat | grep 'usage_usec' | head -1)"
    echo "Memory:   $(cat $CGROOT/$CGROUP_NAME/memory.current) bytes"
    echo "Procs:    $(cat $CGROOT/$CGROUP_NAME/pids.current)"
    sleep 5
done

echo ">>> Process exited"
echo ">>> Memory peak: $(cat $CGROOT/$CGROUP_NAME/memory.peak) bytes"
echo ">>> OOM events: $(cat $CGROOT/$CGROUP_NAME/memory.events | grep oom)"

# Clean up cgroup
rmdir "$CGROOT/$CGROUP_NAME"
echo ">>> cgroup cleaned up"

CRIU and Cgroup Checkpointing

In container migration and hot upgrade scenarios, CRIU (Checkpoint/Restore In Userspace) relies on cgroups to restore process resource limits:

# Checkpoint a running process
criu dump --tree $PID --images-dir /tmp/checkpoint

# Restore on another machine (cgroup config auto-rebuilt)
criu restore --images-dir /tmp/checkpoint --cgroup-root /sys/fs/cgroup/myapp

Docker and Kubernetes Scenarios

Docker with cgroup v2

Docker supports cgroup v2 since version 20.10. Verify Docker’s cgroup mode:

# Check Docker's cgroup driver
docker info | grep -i cgroup

# cgroup v2 output:
#  Cgroup Driver: cgroupfs
#  Cgroup Version: 2

# Or:
#  Cgroup Driver: systemd
#  Cgroup Version: 2

Running containers with resource limits:

# Start container with CPU and memory limits
docker run -d \
    --name myapp \
    --cpus="1.5" \
    --cpu-shares=512 \
    --memory="1g" \
    --memory-swap="1.5g" \
    --pids-limit=200 \
    --device-write-bps /dev/sda:10mb \
    myapp:latest

# Verify container's cgroup configuration
# Docker's cgroup path under v2:
# /sys/fs/cgroup/system.slice/docker-<container-id>.scope/

CONTAINER_ID=$(docker inspect -f '{{.Id}}' myapp)
CG_PATH="/sys/fs/cgroup/system.slice/docker-${CONTAINER_ID}.scope"

echo "CPU limit: $(cat $CG_PATH/cpu.max)"
echo "Memory limit: $(cat $CG_PATH/memory.max)"
echo "Memory usage: $(cat $CG_PATH/memory.current)"
echo "Process count: $(cat $CG_PATH/pids.current)/$(cat $CG_PATH/pids.max)"

Docker parameter to cgroup v2 file mapping:

Docker Parameter	cgroup v2 File	Description
`--cpus=1.5`	`cpu.max` = “150000 100000”	1.5 core bandwidth limit
`--cpu-shares=512`	`cpu.weight` ≈ 50 (approximate mapping)	Relative weight
`--memory=1g`	`memory.max` = 1073741824	Hard memory limit
`--memory-swap=1.5g`	`memory.swap.max` = 536870912	Swap limit
`--pids-limit=200`	`pids.max` = 200	Process count limit
`--device-write-bps /dev/sda:10mb`	`io.max` = “8:0 wbps=10485760”	IO bandwidth limit

Kubernetes and cgroup v2

Kubernetes supports cgroup v2 as a stable feature since v1.25. kubelet’s cgroup driver configuration:

# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1
kind: KubeletConfiguration
cgroupDriver: systemd   # Recommended (best with cgroup v2)
# If using cgroupfs, kubelet directly manipulates cgroup filesystem

Kubernetes Pod resource requests and limits mapped to cgroup v2:

# Pod definition
apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        cpu: "500m"     # 0.5 core request
        memory: "512Mi"
      limits:
        cpu: "1000m"    # 1 core limit
        memory: "1Gi"

Corresponding cgroup v2 paths and files:

# Pod cgroup path (using systemd cgroup driver)
# /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/
#   kubepods-pod<uid>.slice/
#     cri-containerd-<container-id>.scope/

# CPU requests → cpu.weight
# 500m request → cpu.weight ≈ 50 (based on conversion formula)

# CPU limits → cpu.max
# 1000m limit → cpu.max = "100000 100000" (1 core)

# Memory limits → memory.max
# 1Gi limit → memory.max = 1073741824

# Memory requests → memory.low
# 512Mi request → memory.low = 536870912

CPU requests to cpu.weight conversion formula:

# Kubernetes converts CPU request to cgroup v2 cpu.weight
def cpu_request_to_weight(cpu_millicores):
    # Convert to cores
    cpu_cores = cpu_millicores / 1000.0
    # Weight formula: weight = 1 + (cpu_cores - 1) * 99 (approximate)
    # Actual implementation uses logarithmic mapping for reasonable weights
    if cpu_cores <= 0:
        return 1
    weight = min(10000, max(1, int(100 * (cpu_cores))))
    return weight

# Examples
print(cpu_request_to_weight(100))   # 100m → 10
print(cpu_request_to_weight(500))    # 500m → 50
print(cpu_request_to_weight(1000))  # 1 core → 100
print(cpu_request_to_weight(4000))  # 4 cores → 400

Viewing Cgroup from Inside Containers

Under cgroup v2, the cgroup view inside containers is more unified:

# Enter container
docker exec -it myapp bash

# Check cgroup inside container
cat /sys/fs/cgroup/cgroup.controllers
# Output: cpu memory io pids

cat /sys/fs/cgroup/memory.max
# Output: 1073741824 (1GB limit set by host)

cat /sys/fs/cgroup/cpu.max
# Output: 150000 100000 (1.5 core limit)

# cgroup v2's unified view lets container processes
# access their own resource limits through standard interfaces

Advanced Tuning and Production Practices

Mixed Workload Resource Isolation Strategy

In production, different workload types require different resource isolation strategies:

#!/bin/bash
# setup-mixed-workloads.sh — Mixed workload cgroup configuration

CGROOT="/sys/fs/cgroup"

# Create workload groups
mkdir -p "$CGROOT"/{latency-sensitive,best-effort,batch}

# Enable all needed controllers
echo "+cpu +memory +io +pids" > "$CGROOT/cgroup.subtree_control"

# 1. Latency-sensitive applications (e.g., Web API)
#    High CPU weight, memory protection, low IO limit
echo 10000 > "$CGROOT/latency-sensitive/cpu.weight"     # Highest weight
echo 4294967296 > "$CGROOT/latency-sensitive/memory.max"  # 4GB ceiling
echo 2147483648 > "$CGROOT/latency-sensitive/memory.low"   # 2GB protected
echo 800 > "$CGROOT/latency-sensitive/io.weight"         # High IO priority

# 2. Best-effort applications (e.g., background tasks)
#    Medium CPU weight, moderate memory
echo 200 > "$CGROOT/best-effort/cpu.weight"
echo 2147483648 > "$CGROOT/best-effort/memory.max"  # 2GB
echo 536870912 > "$CGROOT/best-effort/memory.low"   # 512MB protected
echo 200 > "$CGROOT/best-effort/io.weight"

# 3. Batch processing (e.g., data analytics)
#    Low CPU weight, large memory, IO limited
echo 50 > "$CGROOT/batch/cpu.weight"              # Lowest weight
echo 8589934592 > "$CGROOT/batch/memory.max"       # 8GB
echo 1073741824 > "$CGROOT/batch/memory.low"       # 1GB protected
echo 50 > "$CGROOT/batch/io.weight"               # Low IO priority

# Limit batch IO bandwidth (avoid impacting database)
DEV=$(stat -c '%t:%T' /dev/sda)
MAJOR=$((0x$(stat -c '%t' /dev/sda)))
MINOR=$((0x$(stat -c '%T' /dev/sda)))
echo "$MAJOR:$MINOR rbps=52428800 wbps=52428800" > "$CGROOT/batch/io.max"
# Limit read/write to 50MB/s each

echo ">>> Mixed workload cgroup configuration complete"
echo ">>> Latency-sensitive: cpu.weight=10000, mem=4G(2G protected), io=800"
echo ">>> Best-effort:       cpu.weight=200,  mem=2G(512M protected), io=200"
echo ">>> Batch:             cpu.weight=50,   mem=8G(1G protected),  io=50(50MB/s)"

cgroup v2 Monitoring Script

The following script continuously monitors cgroup resource usage, suitable for integration with Prometheus exporters or alerting systems:

#!/usr/bin/env python3
"""cgroup v2 resource monitoring script — collects resource data for specified cgroups"""

import os
import time
import json
from pathlib import Path

class CgroupV2Monitor:
    """Monitor cgroup v2 resource usage"""

    CGROOT = Path("/sys/fs/cgroup")

    def __init__(self, cgroup_path: str):
        self.cgpath = self.CGROOT / cgroup_path.lstrip("/")
        if not self.cgpath.exists():
            raise FileNotFoundError(f"cgroup path not found: {self.cgpath}")

    def read_file(self, name: str) -> str:
        filepath = self.cgpath / name
        if not filepath.exists():
            return ""
        return filepath.read_text().strip()

    def get_cpu_stats(self) -> dict:
        """Get CPU usage statistics"""
        stats = {}
        cpu_max = self.read_file("cpu.max")
        if cpu_max and cpu_max != "max":
            parts = cpu_max.split()
            stats["cpu_quota_us"] = int(parts[0])
            stats["cpu_period_us"] = int(parts[1])
            stats["cpu_limit_cores"] = int(parts[0]) / int(parts[1])
        else:
            stats["cpu_limit_cores"] = -1  # Unlimited

        cpu_stat = self.read_file("cpu.stat")
        for line in cpu_stat.split("\n"):
            if line:
                key, val = line.split()
                stats[f"cpu_{key}"] = int(val)
        return stats

    def get_memory_stats(self) -> dict:
        """Get memory usage statistics"""
        stats = {}
        stats["memory_current"] = int(self.read_file("memory.current") or 0)
        stats["memory_max"] = int(self.read_file("memory.max") or 0)
        stats["memory_peak"] = int(self.read_file("memory.peak") or 0)
        stats["memory_low"] = int(self.read_file("memory.low") or 0)
        stats["memory_swap_current"] = int(self.read_file("memory.swap.current") or 0)
        stats["memory_swap_max"] = int(self.read_file("memory.swap.max") or 0)

        mem_stat = self.read_file("memory.stat")
        for line in mem_stat.split("\n"):
            if line:
                key, val = line.split()
                stats[f"mem_{key}"] = int(val)

        events = self.read_file("memory.events")
        for line in events.split("\n"):
            if line:
                key, val = line.split()
                stats[f"mem_event_{key}"] = int(val)
        return stats

    def get_io_stats(self) -> dict:
        """Get IO usage statistics"""
        stats = {}
        io_stat = self.read_file("io.stat")
        for line in io_stat.split("\n"):
            if not line:
                continue
            parts = line.split()
            dev = parts[0]  # major:minor
            for field in parts[1:]:
                key, val = field.split("=")
                stats[f"io_{dev}_{key}"] = int(val)
        return stats

    def get_pids_stats(self) -> dict:
        """Get process count statistics"""
        return {
            "pids_current": int(self.read_file("pids.current") or 0),
            "pids_max": int(self.read_file("pids.max") or 0),
        }

    def collect_all(self) -> dict:
        """Collect all resource data"""
        return {
            "timestamp": int(time.time()),
            "cgroup": str(self.cgpath),
            "cpu": self.get_cpu_stats(),
            "memory": self.get_memory_stats(),
            "io": self.get_io_stats(),
            "pids": self.get_pids_stats(),
        }

if __name__ == "__main__":
    # Example: monitor nginx service under system.slice
    monitor = CgroupV2Monitor("system.slice/nginx.service")

    while True:
        data = monitor.collect_all()
        print(json.dumps(data, indent=2))

        # Calculate memory usage percentage
        mem = data["memory"]
        if mem["memory_max"] > 0:
            usage_pct = mem["memory_current"] / mem["memory_max"] * 100
            print(f"\nMemory usage: {usage_pct:.1f}%")

        time.sleep(10)

Memory Pressure Awareness with PSI

cgroup v2 integrates PSI (Pressure Stall Information) for resource pressure awareness:

# View cgroup PSI pressure metrics
cat /sys/fs/cgroup/myapp/psi/cpu.pressure
# Output:
# some avg10=12.50 avg60=5.00 avg300=2.00 total=12345678
# full avg10=8.30 avg60=3.00 avg300=1.00 total=8765432

# some: at least one task waiting for CPU
# full: all tasks waiting for CPU
# avg10/60/300: 10s/60s/300s average pressure

cat /sys/fs/cgroup/myapp/psi/memory.pressure
cat /sys/fs/cgroup/myapp/psi/io.pressure

PSI-based autoscaling logic:

#!/usr/bin/env python3
"""PSI-based autoscaling decision engine"""

import re
from pathlib import Path

class PSIMonitor:
    """Read cgroup PSI pressure data"""

    def __init__(self, cgroup_path: str):
        self.cgroot = Path("/sys/fs/cgroup") / cgroup_path.lstrip("/")

    def read_psi(self, resource: str) -> dict:
        """Read PSI data for specified resource"""
        filepath = self.cgroot / f"psi/{resource}.pressure"
        if not filepath.exists():
            return {}

        content = filepath.read_text()
        result = {}
        for line in content.strip().split("\n"):
            match = re.match(
                r'(\w+)\s+avg10=(\S+)\s+avg60=(\S+)\s+avg300=(\S+)\s+total=(\d+)',
                line
            )
            if match:
                result[match.group(1)] = {
                    "avg10": float(match.group(2)),
                    "avg60": float(match.group(3)),
                    "avg300": float(match.group(4)),
                    "total": int(match.group(5)),
                }
        return result

    def should_scale_up(self) -> tuple:
        """Determine if scaling up is needed"""
        cpu_psi = self.read_psi("cpu")
        mem_psi = self.read_psi("memory")

        cpu_pressure = cpu_psi.get("some", {}).get("avg10", 0)
        mem_pressure = mem_psi.get("full", {}).get("avg10", 0)

        if cpu_pressure > 30:
            return True, f"CPU pressure too high ({cpu_pressure:.1f}%), recommend scaling up"
        if mem_pressure > 20:
            return True, f"Memory pressure too high ({mem_pressure:.1f}%), recommend scaling up"

        return False, "Resource pressure normal"

if __name__ == "__main__":
    monitor = PSIMonitor("myapp")

    scale_up, reason = monitor.should_scale_up()
    if scale_up:
        print(f"[ALERT] {reason}")
        # Trigger Kubernetes HPA or autoscaling logic here
    else:
        print(f"[OK] {reason}")

v1 to v2 Migration Guide

Migration Assessment Checklist

Check Item	Method	Criteria
Kernel version	`uname -r`	>= 5.2 (full features)
systemd version	`systemctl --version`	>= 239
Docker version	`docker --version`	>= 20.10
Kubernetes version	`kubectl version`	>= 1.25 (v2 GA)
Custom cgroup scripts	Code audit	No v1 multi-hierarchy dependencies
Monitoring tools	Check docs	Support cgroup v2 paths
Container runtime	`docker info \| grep cgroup`	Supports v2 driver
Legacy applications	Apps reading v1 cgroup paths	Updated or compatible with v2

Migration Steps

#!/bin/bash
# migrate-to-cgroup-v2.sh — cgroup v1 to v2 migration script

set -euo pipefail

echo "=== cgroup v1 → v2 Migration Tool ==="

# 1. Pre-checks
echo ""
echo "[1/5] Pre-checks..."

# Check kernel version
KERNEL_VERSION=$(uname -r | cut -d. -f1-2)
KERNEL_MAJOR=$(echo $KERNEL_VERSION | cut -d. -f1)
KERNEL_MINOR=$(echo $KERNEL_VERSION | cut -d. -f2)
if [ "$KERNEL_MAJOR" -lt 5 ] || { [ "$KERNEL_MAJOR" -eq 5 ] && [ "$KERNEL_MINOR" -lt 2 ]; }; then
    echo "  [FAIL] Kernel version $KERNEL_VERSION too low, need >= 5.2"
    exit 1
fi
echo "  [OK] Kernel version: $KERNEL_VERSION"

# Check systemd version
SYSTEMD_VERSION=$(systemctl --version | head -1 | awk '{print $2}')
if [ "$SYSTEMD_VERSION" -lt 239 ]; then
    echo "  [FAIL] systemd version $SYSTEMD_VERSION too low, need >= 239"
    exit 1
fi
echo "  [OK] systemd version: $SYSTEMD_VERSION"

# Check current cgroup mode
CURRENT_MODE=$(mount | grep -c "^cgroup ")
if [ "$CURRENT_MODE" -eq 0 ]; then
    echo "  [INFO] System may already be using cgroup v2"
    echo "  [OK] No migration needed"
    exit 0
fi
echo "  [OK] Currently on cgroup v1, starting migration preparation"

# 2. Check container runtime compatibility
echo ""
echo "[2/5] Checking container runtime..."
if command -v docker &>/dev/null; then
    DOCKER_VERSION=$(docker version --format '{{.Server.Version}}' 2>/dev/null || echo "0")
    DOCKER_MAJOR=$(echo $DOCKER_VERSION | cut -d. -f1)
    if [ "$DOCKER_MAJOR" -lt 20 ]; then
        echo "  [WARN] Docker $DOCKER_VERSION needs upgrade to >= 20.10"
        echo "  [INFO] Upgrade: curl -fsSL https://get.docker.com | sh"
    else
        echo "  [OK] Docker version: $DOCKER_VERSION"
    fi
fi

# 3. Check Kubernetes compatibility
echo ""
echo "[3/5] Checking Kubernetes..."
if command -v kubectl &>/dev/null; then
    K8S_VERSION=$(kubectl version --short 2>/dev/null | grep Server | awk '{print $3}' | cut -d. -f1-2 || echo "0.0")
    K8S_MINOR=$(echo $K8S_VERSION | cut -d. -f2)
    if [ "$K8S_MINOR" -lt 25 ]; then
        echo "  [WARN] Kubernetes $K8S_VERSION recommended upgrade to >= 1.25"
    else
        echo "  [OK] Kubernetes version: $K8S_VERSION"
    fi
fi

# 4. Check custom scripts
echo ""
echo "[4/5] Checking custom cgroup scripts..."
echo "  [INFO] Searching for scripts depending on cgroup v1 paths..."
V1_SCRIPTS=$(grep -rl "/sys/fs/cgroup/cpu" /etc/init.d/ /usr/local/bin/ /opt/ 2>/dev/null || true)
if [ -n "$V1_SCRIPTS" ]; then
    echo "  [WARN] Found scripts referencing cgroup v1 paths:"
    echo "$V1_SCRIPTS" | while read script; do
        echo "    - $script"
    done
    echo "  [INFO] Need to update v1 paths (e.g., /sys/fs/cgroup/cpu/xxx/cpu.cfs_quota_us)"
    echo "         to v2 paths (e.g., /sys/fs/cgroup/xxx/cpu.max)"
else
    echo "  [OK] No scripts referencing v1 paths found"
fi

# 5. Generate migration instructions
echo ""
echo "[5/5] Migration instructions..."
echo "  1. Edit GRUB config:"
echo "     sudo sed -i 's/GRUB_CMDLINE_LINUX=\"/GRUB_CMDLINE_LINUX=\"systemd.unified_cgroup_hierarchy=1 /' /etc/default/grub"
echo ""
echo "  2. Update GRUB:"
echo "     sudo update-grub    # Debian/Ubuntu"
echo "     sudo grub2-mkconfig -o /boot/grub2/grub.cfg   # RHEL/CentOS"
echo ""
echo "  3. Reboot system:"
echo "     sudo reboot"
echo ""
echo "  4. Verify after reboot:"
echo "     mount | grep cgroup2"
echo "     cat /sys/fs/cgroup/cgroup.controllers"
echo ""
echo "  5. Update kubelet config (if using K8s):"
echo "     Ensure cgroupDriver: systemd"
echo ""
echo "=== Migration preparation complete ==="

cgroup v1 to v2 Interface Mapping

During migration, v1 interface files need to be replaced with v2 equivalents:

v1 File	v2 File	Notes
`cpu.cfs_quota_us`	`cpu.max`	v2 format is “$max $period”
`cpu.cfs_period_us`	`cpu.max`	Same as above
`cpu.shares`	`cpu.weight`	Different ranges (v1: 2-262144, v2: 1-10000)
`memory.limit_in_bytes`	`memory.max`	Direct use
`memory.memsw.limit_in_bytes`	`memory.swap.max`	v2 controls swap independently
`memory.soft_limit_in_bytes`	`memory.low`	Similar semantics
`blkio.throttle.write_bps_device`	`io.max`	Different format
`pids.max`	`pids.max`	Same
`cgroup.procs`	`cgroup.procs`	Same
`tasks` (thread-level)	`cgroup.threads`	v2 threaded cgroup

CPU shares to weight conversion:

def cpu_shares_to_weight(shares: int) -> int:
    """Convert cgroup v1 cpu.shares to v2 cpu.weight"""
    # v1 range: 2-262144, default 1024
    # v2 range: 1-10000, default 100
    # Approximate linear mapping
    weight = max(1, min(10000, int(shares / 1024 * 100)))
    return weight

# Examples
print(cpu_shares_to_weight(1024))  # Default → 100
print(cpu_shares_to_weight(512))   # Low priority → 50
print(cpu_shares_to_weight(2048))  # High priority → 200
print(cpu_shares_to_weight(8192))  # Highest → 800

Common Issues and Troubleshooting

Issue 1: Controller Unavailable

# Symptom: Error when writing to cgroup.subtree_control
echo "+cpu" > /sys/fs/cgroup/cgroup.subtree_control
# bash: echo: write error: Invalid argument

# Troubleshooting 1: Check if controller is in global available list
cat /sys/fs/cgroup/cgroup.controllers
# If cpu is not listed, the kernel doesn't have the controller enabled

# Troubleshooting 2: Check for processes in root cgroup
cat /sys/fs/cgroup/cgroup.procs
# Some controllers can't be enabled when root cgroup has processes
# Need to move processes to child cgroups

# Troubleshooting 3: Check kernel boot parameters
cat /proc/cmdline | grep cgroup
# Confirm no cgroup_no_v1 or cgroup.disable excluding the controller

Issue 2: Cannot Remove cgroup Directory

# Symptom: rmdir reports "Device or resource busy"
rmdir /sys/fs/cgroup/myapp
# rmdir: failed to remove '/sys/fs/cgroup/myapp': Device or resource busy

# Check for active processes
cat /sys/fs/cgroup/myapp/cgroup.procs
# If PIDs exist, move them out first

# Move processes out
for pid in $(cat /sys/fs/cgroup/myapp/cgroup.procs); do
    echo $pid > /sys/fs/cgroup/cgroup.procs  # Move to root cgroup
done

# Check for child cgroups
ls /sys/fs/cgroup/myapp/
# Need to delete all child cgroups first

# Check if threaded type needs to be reverted
cat /sys/fs/cgroup/myapp/cgroup.type
# If threaded, change back to domain
echo "domain" > /sys/fs/cgroup/myapp/cgroup.type

Issue 3: Docker Container Memory Limit Not Effective

# Symptom: docker run --memory=1g but container has much more available memory

# Check 1: Confirm Docker uses cgroup v2
docker info | grep "Cgroup Version"
# Should output 2

# Check 2: Swap limit behavior
# In cgroup v2, memory.swap.max controls swap usage only
# Not memory+swap total (as in v1)
docker run --memory=1g --memory-swap=1g myapp
# This sets swap limit to 0 (swap = memory-swap - memory = 0)

# Check 3: Kernel parameter
sysctl vm.swappiness
# If swappiness=0, system may not use swap

# Check 4: Verify correct limit visible inside container
docker exec myapp cat /sys/fs/cgroup/memory.max
# Should output 1073741824 (1GB)

Issue 4: Kubernetes Pod CPU Limit Appears Inaccurate

# Symptom: Set limits.cpu=1000m but container uses more than 1 core

# Check 1: kubelet cgroup driver
cat /var/lib/kubelet/config.yaml | grep cgroupDriver
# systemd recommended

# Check 2: Pod's cgroup
# Get Pod UID
POD_UID=$(kubectl get pod mypod -o jsonpath='{.metadata.uid}')
# View cgroup
cat /sys/fs/cgroup/kubepods.slice/kubepods-pod${POD_UID//-/_}.slice/cpu.max

# Check 3: Multi-core CPU limiting behavior
# cpu.max = "100000 100000" means 100ms usable per 100ms period
# On multi-core machines, container can run 100ms total across
# multiple cores within one period
# Instantaneous high usage across cores is normal:
# cpu.max limits total, not concurrent cores

Performance Benchmark Comparison

cgroup v1 vs v2 Overhead

Metric	v1	v2	Difference
Process creation overhead	Baseline	+2-3%	v2 unified hierarchy adds minor overhead
Memory accounting precision	cgroup-level	Recursive	v2 more precise
IO limiting accuracy	Fair	Better	v2 io.max more precise
cgroup operation latency	Baseline	-10-15%	v2 single hierarchy reduces lock contention
Multi-container scalability	Linear degradation	Better	v2 reduces hierarchy depth

In most production scenarios, cgroup v2’s performance overhead is negligible, and its advantages in management convenience, security, and accounting precision far outweigh the minimal performance difference.

Summary

cgroup v2 is not merely a version upgrade of v1, but a paradigm shift in resource management. The unified hierarchy architecture fundamentally solves v1’s multi-hierarchy confusion, making resource limiting, accounting, and management clear and predictable.

Key takeaways:

Architecture Understanding: v2’s unified hierarchy ensures a process belongs to exactly one cgroup, with all controllers collaborating on the same tree, eliminating v1’s fragmented views.
Controller Usage: cpu.max (bandwidth limiting), cpu.weight (weight distribution), memory.max (memory ceiling), io.max (IO limiting), and pids.max (process count limiting) are the five most commonly used control files in production.
systemd Integration: In modern Linux, systemd is the primary cgroup manager. Use systemctl set-property for dynamic resource adjustment without service restarts.
Container Scenarios: Docker and Kubernetes fully support cgroup v2. CPU requests map to cpu.weight, limits map to cpu.max, and memory requests/limits map to memory.low/memory.max.
Migration Strategy: Complete the assessment checklist (kernel version, systemd version, container runtime, custom scripts) before switching via GRUB parameters, and update all v1 path references.
Production Monitoring: Build a comprehensive resource monitoring system using memory.events, cpu.stat, io.stat, and PSI pressure metrics, with PSI-based autoscaling decisions.

For new deployments, use cgroup v2 directly. For existing systems, migrate after compatibility assessment. cgroup v2 is a critical foundation of cloud-native infrastructure, and mastering it is essential for managing modern containerized workloads.

Overview#

cgroup v1 vs v2: Why a Rewrite Was Needed#

Core Pain Points of v1#

v2 Design Philosophy#

Key Differences: v1 vs v2#

Environment Check and Enablement#

Confirming Current cgroup Version#

Checking Kernel Version Support#

Enabling cgroup v2 on Older Systems#

Core Controllers Explained#

CPU Controller#

Weight Mode (cpu.weight)#

Bandwidth Limit Mode (cpu.max)#

CPU Affinity (cpuset)#

Memory Controller#

Basic Memory Limits#

Memory Recursive Protection#

OOM Control#

IO Controller#

PID Controller#

systemd Integration#

systemd Resource Limit Configuration#

Verifying systemd cgroup Configuration#

Dynamic Resource Adjustment with systemctl#

Threaded Cgroups#

Manual Cgroup Management in Practice#

Creating and Managing Custom Cgroups#

CRIU and Cgroup Checkpointing#

Docker and Kubernetes Scenarios#

Docker with cgroup v2#

Kubernetes and cgroup v2#

Viewing Cgroup from Inside Containers#

Advanced Tuning and Production Practices#

Mixed Workload Resource Isolation Strategy#

cgroup v2 Monitoring Script#

Memory Pressure Awareness with PSI#

v1 to v2 Migration Guide#

Migration Assessment Checklist#

Migration Steps#

cgroup v1 to v2 Interface Mapping#

Common Issues and Troubleshooting#

Issue 1: Controller Unavailable#

Issue 2: Cannot Remove cgroup Directory#

Issue 3: Docker Container Memory Limit Not Effective#

Issue 4: Kubernetes Pod CPU Limit Appears Inaccurate#

Performance Benchmark Comparison#

cgroup v1 vs v2 Overhead#

Summary#