Linux Process Scheduler: CFS Principles and Tuning

Overview

The process scheduler is a core component of the operating system kernel — it determines which process runs on which CPU and for how long. Linux has used CFS (Completely Fair Scheduler) as its default scheduler since 2.6.23, and after years of evolution, introduced EEVDF (Earliest Eligible Virtual Deadline First) to replace CFS in the 6.6 kernel. This article provides an in-depth analysis of CFS principles, nice/cgroup CPU control, real-time scheduling, CPU affinity, and other core topics, with production tuning experience.

CFS Scheduler Principles

Design Philosophy

The core goal of CFS is “complete fairness” — each process receives CPU time proportional to its weight. Unlike traditional schedulers based on time slices, CFS uses “virtual runtime” (vruntime) to track each process’s CPU consumption:

vruntime = actual_runtime × (NICE_0_LOAD / process_weight)

A process with nice value 0 has a weight of 1024 (NICE_0_LOAD)
Lower nice values mean higher weights, slower vruntime growth, and more CPU time
CFS always selects the process with the smallest vruntime to run

Red-Black Tree and Run Queue

CFS uses a red-black tree to maintain the run queue, sorted by vruntime:

              [vruntime=50]
             /              \
      [vruntime=20]      [vruntime=80]
       /         \
[vruntime=10]  [vruntime=35]

The leftmost node (smallest vruntime) is the next process to be scheduled
After a process runs, its vruntime increases and it is reinserted into the red-black tree
Lookup, insertion, and deletion are all O(log N)

Scheduling Period and Minimum Granularity

# Scheduling period (target latency): all processes run at least once within this period
$ sysctl kernel.sched_latency_ns
kernel.sched_latency_ns = 6000000  # 6ms (6.0×10⁶ ns)

# Minimum granularity: the minimum time a single process runs per turn
$ sysctl kernel.sched_min_granularity_ns
kernel.sched_min_granularity_ns = 1000000  # 1ms

# Wakeup preemption granularity
$ sysctl kernel.sched_wakeup_granularity_ns
kernel.sched_wakeup_granularity_ns = 1000000  # 1ms

Allocation logic within the scheduling period:

Process count <= sched_latency_ns / sched_min_granularity_ns:
    Time slice per process = sched_latency_ns / process_count

Process count > threshold:
    Time slice per process = sched_min_granularity_ns
    Scheduling period auto-expands = sched_min_granularity_ns × process_count

Nice Values and Weights

Nice values range from -20 to 19, default 0. Each level difference means approximately 10% CPU difference:

Nice Value	Weight	Relative CPU Share	Typical Use
-20	88761	88x	Real-time critical tasks
-10	9548	9.3x	High-priority services
0	1024	1x	Default
10	110	0.107x	Background tasks
19	15	0.015x	Batch processing

# View process nice value
$ ps -o pid,ni,comm -p $PID

# Modify process nice value
$ renice -n -5 -p $PID

# Start with a specific nice value
$ nice -n -5 /path/to/program

Note: Nice values are relative. Two processes with nice values 0 and 1 have a CPU ratio of approximately 1.25:1, not 2:1. The weight formula is 1024 × 1.25^(-nice).

cgroup v2 CPU Control

CPU Weight (cpu.weight)

# Create cgroups
$ mkdir /sys/fs/cgroup/app_a
$ mkdir /sys/fs/cgroup/app_b

# Set CPU weight (default 100, range 1-10000)
$ echo 200 > /sys/fs/cgroup/app_a/cpu.weight
$ echo 100 > /sys/fs/cgroup/app_b/cpu.weight
# app_a : app_b = 2:1

# Add processes to cgroups
$ echo $PID_A > /sys/fs/cgroup/app_a/cgroup.procs
$ echo $PID_B > /sys/fs/cgroup/app_b/cgroup.procs

CPU Bandwidth Limit (cpu.max)

# Format: $MAX $PERIOD
# Limit to 1.5 CPU cores
$ echo "150000 100000" > /sys/fs/cgroup/app_a/cpu.max
# Meaning: at most 150ms CPU time per 100ms period

# View current configuration
$ cat /sys/fs/cgroup/app_a/cpu.max
150000 100000

# View CPU usage statistics
$ cat /sys/fs/cgroup/app_a/cpu.stat
usage_usec 12345678
user_usec 9876543
system_usec 2469135
nr_periods 12345
nr_throttled 0       # Number of times throttled
throttled_usec 0     # Total time throttled

cpu.weight vs cpu.max Comparison

Feature	cpu.weight	cpu.max
Type	Relative weight	Absolute limit
When CPU is idle	Can exceed limit and use all CPU	Strictly does not exceed limit
Use case	Priority control	Hard quota
Analogy	Nice value	CPU quota
Docker equivalent	`--cpu-shares`	`--cpus`

Production CPU Limit Strategy

# Database service: high priority + hard limit to prevent overselling
$ echo 500 > /sys/fs/cgroup/postgres/cpu.weight
$ echo "800000 100000" > /sys/fs/cgroup/postgres/cpu.max  # 8 cores

# Web service: medium priority
$ echo 200 > /sys/fs/cgroup/nginx/cpu.weight
$ echo "400000 100000" > /sys/fs/cgroup/nginx/cpu.max     # 4 cores

# Log collector: low priority
$ echo 50 > /sys/fs/cgroup/filebeat/cpu.weight
$ echo "50000 100000" > /sys/fs/cgroup/filebeat/cpu.max   # 0.5 cores

Real-Time Processes and RT Scheduling

Scheduling Policies

Policy	Description	Use Case
SCHED_NORMAL (0)	CFS scheduling	Normal processes
SCHED_BATCH (3)	CFS batch mode	CPU-intensive batch processing
SCHED_IDLE (5)	Very low priority	Background tasks
SCHED_FIFO (1)	Real-time FIFO	Real-time tasks
SCHED_RR (2)	Real-time round-robin	Real-time tasks
SCHED_DEADLINE (6)	Deadline-based	Strict real-time requirements

SCHED_FIFO and SCHED_RR

# View process scheduling policy
$ chrt -p $PID

# Set to SCHED_FIFO, priority 80
$ chrt -f -p 80 $PID

# Set to SCHED_RR, priority 80
$ chrt -r -p 80 $PID

SCHED_FIFO rules:

Same-priority processes run in FIFO order
Higher-priority processes always preempt lower-priority ones
A process yields the CPU only when it voluntarily does so (no time slice limit)

SCHED_RR rules:

Same as SCHED_FIFO, but each process has a time slice (default 100ms)
When the time slice is exhausted, it rotates to the next process at the same priority

RT Scheduling Parameter Tuning

# CPU bandwidth ratio available to RT processes (default 95%)
$ sysctl kernel.sched_rt_runtime_us
kernel.sched_rt_runtime_us = 950000

$ sysctl kernel.sched_rt_period_us
kernel.sched_rt_period_us = 1000000
# Meaning: RT processes can use at most 950ms per 1 second, leaving 50ms for normal processes

# RT throttling statistics
$ grep -H . /proc/sys/kernel/sched_rt_*

Warning: Do not set sched_rt_runtime_us to -1 (disable throttling) in production, or a runaway RT process could lock up the entire system.

DEADLINE Scheduler

struct sched_attr attr = {
    .size = sizeof(attr),
    .sched_policy = SCHED_DEADLINE,
    .sched_runtime  = 10 * 1000 * 1000,  // 10ms runtime
    .sched_deadline = 30 * 1000 * 1000,  // 30ms deadline
    .sched_period   = 30 * 1000 * 1000,  // 30ms period
};
syscall(SYS_sched_setattr, pid, &attr, 0);

CPU Affinity

Affinity Principles

CPU affinity controls which CPU cores a process can run on. Binding a process to specific CPUs can:

Reduce cache misses (maintain CPU cache warmth)
Reduce cross-NUMA memory access latency
Isolate critical services

# View process affinity
$ taskset -p $PID
pid 12345's current affinity mask: ff  # CPU 0-7

# Set affinity (bind to CPU 0 and 1)
$ taskset -cp 0,1 $PID

# Start with specific affinity
$ taskset -c 0,1 /path/to/program

# View all CPU affinity masks
$ taskset -cp $PID

NUMA Affinity

# Bind process to NUMA node with numactl
$ numactl --cpunodebind=0 --membind=0 /path/to/program

# Interleave mode (distribute memory evenly across all NUMA nodes)
$ numactl --interleave=all /path/to/program

# View process NUMA memory distribution
$ numastat -p $PID

isolcpus: Isolating CPU Cores

# GRUB config: isolate CPU 4-7
# /etc/default/grub
GRUB_CMDLINE_LINUX="isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"

# Update GRUB
$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot

# After isolation, manually bind processes to isolated cores
$ taskset -c 4-7 /path/to/critical_service

Parameter	Description
`isolcpus=4-7`	Remove CPU 4-7 from the scheduler, exclude from load balancing
`nohz_full=4-7`	Disable periodic ticks on these CPUs (reduce interrupts)
`rcu_nocbs=4-7`	Migrate RCU callbacks to other CPUs

Suitable for low-latency trading systems, telecom-grade VoIP, real-time control systems, etc.

IRQ Affinity

# View IRQ assignments
$ cat /proc/interrupts | head -30

# Bind NIC interrupt to specific CPU
# Assume NIC interrupt number is 32
$ echo 0f > /proc/irq/32/smp_affinity  # Bind to CPU 0-3

# Hex mask reference:
# CPU 0-3:  0x0f (00001111)
# CPU 4-7:  0xf0 (11110000)
# CPU 0-7:  0xff (11111111)

# Using irqbalance service (enabled by default)
$ systemctl status irqbalance

# Production recommendation: disable irqbalance and manually bind
$ systemctl stop irqbalance
$ systemctl disable irqbalance

Scheduling Latency Troubleshooting

Latency Source Analysis

Process wakeup → [scheduling latency] → process starts running → [execution latency] → complete
                    ↑
         Possible causes:
         1. CPU occupied by higher-priority process
         2. CPU saturated by real-time process
         3. cgroup CPU throttling
         4. NUMA cross-node access
         5. Interrupt storm

Diagnostic Tools

1. schedstat

# View process scheduling statistics
$ cat /proc/$PID/schedstat
12345678 9876543 5678
# Fields: wait_for_CPU_time(ns)  run_time(ns)  time_slice_switches

# Calculate average scheduling latency
$ awk '{printf "avg schedule delay: %.2f ms\n", $1/$3/1000000}' /proc/$PID/schedstat

2. perf sched

# Record scheduling events (10 seconds)
$ perf sched record -- sleep 10

# Analyze scheduling latency
$ perf sched latency --sort max

# Example output:
#   Task                  |   Runtime ms  | Switches | Average delay ms | Maximum delay ms |
#   nginx:worker          |      120.345  |      234 |          0.456   |          5.678   |

3. eBPF Tracing

# Trace process wakeup latency
$ /usr/share/bcc/tools/runqlat -p $PID 5

# Trace run queue length
$ /usr/share/bcc/tools/runqlen 5

# Trace CPU off-CPU time (time waiting to be scheduled)
$ /usr/share/bcc/tools/offcputime -p $PID 5

# Trace scheduler performance
$ /usr/share/bcc/tools/schedstats 5

4. Flamegraph

# Generate off-CPU flamegraph
$ /usr/share/bcc/tools/offcputime -p $PID -f 10 > offcpu.svg

Common Latency Issues and Solutions

Issue	Symptom	Solution
CPU throttling	cgroup `nr_throttled` keeps growing	Increase cpu.max or optimize code
RT process preemption	Normal process latency spikes	Limit RT bandwidth or switch to SCHED_DEADLINE
NUMA cross-node	Periodic latency	Use numactl binding
Interrupt storm	Specific CPU 100% sys	Distribute IRQs across cores
Overload	Run queue length > 4	Scale out or reduce load
Excessive context switches	cs/sec > 50000	Reduce thread count, use I/O multiplexing

Case: API Service P99 Latency Optimization

Symptom: Go API service P99 latency increased from 50ms to 200ms, CPU usage at 60%.

Investigation:

# 1. Check CPU throttling
$ cat /sys/fs/cgroup/app/cpu.stat
nr_throttled 8765     # Throttled 8765 times
throttled_usec 4567890  # Cumulative throttle time 4.5 seconds

# 2. Check cpu.max configuration
$ cat /sys/fs/cgroup/app/cpu.max
200000 100000  # Limited to 2 cores

# 3. Confirm with runqlat
$ /usr/share/bcc/tools/runqlat -p $PID 10
# Shows significant scheduling latency above 10ms

Root cause: cgroup CPU limit is 2 cores, but the service has bursty CPU demand, frequently triggering throttling.

Solution:

# Option 1: Increase limit (if resources allow)
$ echo "400000 100000" > /sys/fs/cgroup/app/cpu.max

# Option 2: Use cpu.weight instead of hard limit (cluster environment)
$ echo 500 > /sys/fs/cgroup/app/cpu.weight
$ echo "max" > /sys/fs/cgroup/app/cpu.max  # No limit

# Option 3: Optimize GOMAXPROCS (Go 1.19+ auto-detects cgroup)
$ GOMAXPROCS=2 ./app  # Explicitly set to match cpu.max

EEVDF Scheduler (Kernel 6.6+)

Linux 6.6 introduced EEVDF (Earliest Eligible Virtual Deadline First) to replace CFS:

Key EEVDF Improvements

Feature	CFS	EEVDF
Scheduling basis	Minimum vruntime	Eligibility + virtual deadline
Latency guarantee	No precise guarantee	Precise deadline-based latency
Nice value	Affects vruntime growth rate	Affects weight and lag
Fairness	Progressive fairness	Precise fairness (lag-based)
Real-time	No guarantee	Deadline guarantee

Key Concepts

Eligibility: Whether a process is “eligible” to be scheduled (based on lag calculation)
Virtual Deadline: Each process has a virtual deadline; the earliest deadline is scheduled first
Lag: The difference between actual CPU time received and entitled CPU time, used for precise compensation

# Check kernel version
$ uname -r
6.6.0-1234-generic

# EEVDF-related sysctl (6.6+)
$ sysctl kernel.sched_base_slice
kernel.sched_base_slice = 750000  # Base time slice 0.75ms (replaces sched_min_granularity_ns)

EEVDF provides significant improvements for latency-sensitive applications, especially audio/video processing, game servers, and similar scenarios.

Multi-Core Load Balancing

Load Balancing Hierarchy

                        [ sched domain hierarchy ]
                              ┌─────────┐
                              |  NUMA   |  (cross-node)
                              | domain  |
                              └────┬────┘
                          ┌────────┼────────┐
                     ┌────┴────┐   ...  ┌────┴────┐
                     |  CPU    |        |  CPU    |  (physical socket)
                     | socket  |        | socket  |
                     └────┬────┘        └─────────┘
                   ┌──────┼──────┐
              ┌────┴───┐    ...    ┌────┴───┐
              |  LLC   |           |  LLC   |  (shared L3 cache)
              | domain |           | domain |
              └────┬───┘           └────────┘
              ┌────┼────┐
         ┌────┴──┐  ... ┌──┴────┐
         |  SMT  |      |  SMT  |  (hyperthreading)
         | domain|      | domain|
         └───────┘      └───────┘

Load Balancing Parameters

# Load balancing interval (busy/idle)
$ sysctl kernel.sched_migration_cost_ns
kernel.sched_migration_cost_ns = 500000  # 0.5ms
# Minimum time a task must be idle before migration

# Enable/disable SMT scheduling
$ sysctl kernel.sched_smt_sibling_cost

Manual Load Balancing Control

# Create independent scheduling domain for specific CPU set (CPU isolation)
# See isolcpus section

# Use cgroup v2 cpuset
$ mkdir /sys/fs/cgroup/dedicated
$ echo "4-7" > /sys/fs/cgroup/dedicated/cpuset.cpus
$ echo "0" > /sys/fs/cgroup/dedicated/cpuset.mems

Summary

The Linux process scheduler is a key determinant of system performance. Understanding its principles helps make correct decisions in the following areas:

CFS/EEVDF fairness is weight-based, not time-slice-based: Each nice level difference means ~10% CPU difference; cgroup cpu.weight follows the same mechanism.
cgroup v2 cpu.max is a hard limit: Suitable for container quota management, but watch out for throttling impact on latency; cpu.weight is relative priority and can exceed when CPU is idle.
Real-time scheduling requires caution: SCHED_FIFO/RR preempt normal processes; always configure sched_rt_runtime_us to leave a safety margin.
CPU affinity is a powerful latency optimization tool: isolcpus + nohz_full + IRQ binding can achieve microsecond-level latency guarantees.
Scheduling latency troubleshooting requires a toolchain: From schedstat/runqlat for latency distribution, to offcputime for root cause tracing, to perf sched for global scheduling behavior analysis.
EEVDF is the future of the scheduler: Enabled by default in kernel 6.6+, with significant improvements for latency-sensitive applications.

Golden rule of scheduler tuning: the default configuration is already good enough. Only modify scheduling parameters when there are clear latency or throughput issues backed by baseline data.

Overview#

CFS Scheduler Principles#

Design Philosophy#

Red-Black Tree and Run Queue#

Scheduling Period and Minimum Granularity#

Nice Values and Weights#

cgroup v2 CPU Control#

CPU Weight (cpu.weight)#

CPU Bandwidth Limit (cpu.max)#

cpu.weight vs cpu.max Comparison#

Production CPU Limit Strategy#

Real-Time Processes and RT Scheduling#

Scheduling Policies#

SCHED_FIFO and SCHED_RR#

RT Scheduling Parameter Tuning#

DEADLINE Scheduler#

CPU Affinity#

Affinity Principles#

NUMA Affinity#

isolcpus: Isolating CPU Cores#

IRQ Affinity#

Scheduling Latency Troubleshooting#

Latency Source Analysis#

Diagnostic Tools#

1. schedstat#

2. perf sched#

3. eBPF Tracing#

4. Flamegraph#

Common Latency Issues and Solutions#

Case: API Service P99 Latency Optimization#

EEVDF Scheduler (Kernel 6.6+)#

Key EEVDF Improvements#

Key Concepts#

Multi-Core Load Balancing#

Load Balancing Hierarchy#

Load Balancing Parameters#

Manual Load Balancing Control#

Summary#