Overview
You’re running a high-frequency trading system online. P99 latency sits at 2ms normally, but occasionally spikes to 20ms. CPU usage isn’t high, memory is sufficient, network is fine. After investigation, you discover the CPU scheduler migrated a critical thread to another core, L3 cache missed entirely, and latency jumped 10x.
This kind of problem can’t be solved by adding resources. The issue is “sharing” — all processes share CPU cores, the scheduler distributes freely, and nobody knows which threads are latency-sensitive.
The solution is CPU isolation: pin your critical workloads to dedicated cores that no other process can touch. Combined with NUMA tuning to keep CPU and memory physically “close,” avoiding the latency doubling that comes from cross-node access.
This note covers the complete chain from basic concepts to production implementation: isolcpus kernel parameters, cpuset cgroups, taskset CPU pinning, NUMA affinity, IRQ binding, and best practices for combining them. All commands tested on Ubuntu 22.04 (kernel 5.15) and CentOS 8.
Fundamentals: Why Isolate CPUs
How the CPU Scheduler Works
Linux defaults to CFS (Completely Fair Scheduler) for CPU time allocation. CFS is designed for “fairness” — each process gets CPU time slices based on weight, and the scheduler freely migrates processes across all available cores.
Sounds fine, but for latency-sensitive scenarios it’s a disaster:
- Context switch overhead: When a thread migrates from CPU A to CPU B, L1/L2 cache is entirely invalidated, requiring reload from memory. A single migration costs microsecond-level latency jitter.
- Cache pollution: Other processes running on your target core push out your previously cached data. When your thread returns, it’s all cache misses.
- Interrupt interference: Network card interrupts and timer interrupts can interrupt your thread at any time. For systems requiring microsecond-level response, each interrupt is a latency spike.
What Is NUMA Architecture
NUMA (Non-Uniform Memory Access) is standard in multi-socket servers. Simply put: each CPU socket has its own local memory. Accessing local memory is fast; accessing another CPU’s memory requires crossing the QPI/UPI bus, roughly doubling the latency.
Think of it this way: NUMA is like two office buildings. You work in Building 1, your filing cabinet is in Building 1, grabbing things is quick. But if you need files from Building 2, you have to walk across — noticeably slower.
A typical dual-socket server:
NUMA Node 0 (CPU 0-31, Memory 0-128GB)
└── Local memory access latency: ~80ns
NUMA Node 1 (CPU 32-63, Memory 128-256GB)
└── Local memory access latency: ~80ns
Cross-node access latency: ~140ns ← nearly double
Check your NUMA topology with numactl --hardware:
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 128768 MB
node 0 free: 89012 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 129020 MB
node 1 free: 91045 MB
node distances:
node 0 1
0: 10 21
1: 21 10
Note node distances: local distance is 10, cross-node is 21. The distance value is relative, representing the access latency ratio. 21 is roughly double 10 — cross-node access is indeed much slower.
Check memory allocation per node with numastat:
$ numastat
node 0 node 1
numa_hit 89342156 67823451
numa_miss 234561 123456
numa_foreign 12345 23456
interleave_hit 456789 345678
local_node 89012345 67012345
other_node 345611 823456
numa_hit: local node allocation hits — higher is betternuma_miss: local node allocation misses (should have been on another node) — lower is betterother_node: cross-node access count — if this keeps growing, your NUMA policy has issues
Method 1: isolcpus Kernel Parameter (Boot-Level Isolation)
What Is isolcpus
isolcpus is a Linux kernel boot parameter that isolates specified CPU cores from the global scheduler at boot time. Isolated cores don’t participate in normal process scheduling — only manually pinned processes can run on them.
This is the strongest isolation method — it takes effect from the moment the kernel boots, independent of runtime configuration.
Configuration
Edit GRUB configuration:
# Edit /etc/default/grub
sudo vim /etc/default/grub
# Add isolcpus parameter to GRUB_CMDLINE_LINUX
# For example, isolate CPU 4-7 (4 cores)
GRUB_CMDLINE_LINUX="isolcpus=4-7"
# Update GRUB
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# On Ubuntu:
# sudo update-grub
# Reboot to apply
sudo reboot
Verify after reboot:
$ cat /proc/cmdline | grep isolcpus
BOOT_IMAGE=/boot/vmlinuz-5.15.0-91-generic root=... isolcpus=4-7
Advanced isolcpus Options
isolcpus doesn’t just simply isolate CPUs — it has sub-options that work together:
# Complete configuration example
GRUB_CMDLINE_LINUX="isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"
| Parameter | Purpose | Use Case |
|---|---|---|
isolcpus=4-7 | Remove CPU 4-7 from global scheduler | Basic isolation |
nohz_full=4-7 | Disable timer interrupts on isolated cores | Low-latency scenarios, reduce interrupt jitter |
rcu_nocbs=4-7 | RCU callbacks not executed on isolated cores | Further reduce kernel interference |
isolcpus=4-7,domain | Exclude isolated cores from load balancing domains | Enabled by default, explicit declaration |
isolcpus=4-7,managed_irq | Restrict interrupt affinity | Reduce hardware interrupt interference |
Using all three together produces the best results. nohz_full reduces timer interrupts, rcu_nocbs moves RCU callbacks away, isolcpus prevents normal process scheduling — three layers of isolation stacked together, leaving essentially only your manually pinned processes on the isolated cores.
Limitations of isolcpus
isolcpus isn’t a silver bullet:
- Requires reboot: Must reboot after changes, not suitable for dynamic adjustment
- Not fully isolated: Kernel threads and certain interrupts can still run on isolated cores, requiring manual migration
- Priority conflict with cgroup cpuset: If both isolcpus and cpuset cgroup are configured, the kernel prioritizes isolcpus restrictions. (Detailed analysis available here)
Method 2: cpuset cgroup (Runtime Isolation)
Why cpuset Instead of isolcpus
isolcpus requires a reboot and isn’t flexible enough. cpuset is a cgroup subsystem that can be dynamically created and modified at runtime without rebooting. Suitable for scenarios requiring frequent adjustments.
cgroup v1 Configuration
# Mount cpuset subsystem (if not already mounted)
sudo mount -t cgroup -o cpuset cpuset /sys/fs/cgroup/cpuset
# Create an isolation group
sudo mkdir /sys/fs/cgroup/cpuset/critical_app
# Assign CPU cores 4-5 to this group
sudo echo "4-5" > /sys/fs/cgroup/cpuset/critical_app/cpuset.cpus
# Assign NUMA node 0 memory
sudo echo "0" > /sys/fs/cgroup/cpuset/critical_app/cpuset.mems
# Enable exclusive mode (key! Other cpusets cannot use these CPUs)
sudo echo "1" > /sys/fs/cgroup/cpuset/critical_app/cpuset.cpu_exclusive
# Add process to this cpuset
sudo echo <PID> > /sys/fs/cgroup/cpuset/critical_app/tasks
The cpu_exclusive option is critical. When set to 1, these CPU cores become this cpuset group’s “private property” — no other cpuset group can use them. True exclusivity.
cgroup v2 Configuration
Modern Linux distributions (kernel 5.15+) default to cgroup v2. Configuration differs slightly:
# Enable cpuset controller
sudo echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control
# Create a sub-group
sudo mkdir /sys/fs/cgroup/critical_app
# Assign CPU cores
sudo echo "4-5" > /sys/fs/cgroup/critical_app/cpuset.cpus
# Assign NUMA node
sudo echo "0" > /sys/fs/cgroup/critical_app/cpuset.mems
# Add process
sudo echo <PID> > /sys/fs/cgroup/critical_app/cgroup.procs
cgroup v2 syntax is cleaner, but note that v2 doesn’t have the cpu_exclusive option — exclusive isolation requires isolcpus as a companion.
Managing Service-Level cpuset with systemd
For production, managing cpuset through systemd is more reliable than manually manipulating cgroup files:
# /etc/systemd/system/critical-app.service
[Unit]
Description=Critical Low-Latency Application
After=network.target
[Service]
ExecStart=/opt/critical-app/bin/server
# Pin to CPU 4-5
CPUAffinity=4,5
# Set NUMA affinity
NUMAPolicy=bind
NUMAMask=0
# Limit memory node
AllowedCPUs=4-5
# Restart policy
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start critical-app
systemd’s advantage: cpuset configuration is automatically restored after service restart, no manual maintenance needed.
Method 3: taskset (Quick CPU Pinning)
Using taskset
taskset is the simplest CPU pinning tool, suitable for temporary validation or quick deployment:
# Pin to CPU 4 at startup
taskset -c 4 ./my_application
# Pin to CPU 4 and 5
taskset -c 4,5 ./my_application
# Check running process CPU affinity
taskset -cp <PID>
# Modify running process CPU affinity
taskset -cp 4 <PID>
taskset’s limitation: it only manages process affinity and doesn’t prevent other processes from using the same CPU cores. If you use taskset without isolcpus, other processes can still be scheduled on the same core — your process is just “suggested” to run on these cores.
Programmatic CPU Pinning
Pin CPU directly in C/C++ programs:
#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <pthread.h>
int main() {
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(4, &cpuset); // Pin to CPU 4
// Pin current thread
if (sched_setaffinity(0, sizeof(cpu_set_t), &cpuset) != 0) {
perror("sched_setaffinity failed");
return 1;
}
printf("Pinned to CPU 4\n");
// Your business logic...
while (1) { /* work */ }
return 0;
}
Pin specific threads in multi-threaded programs:
void* worker_thread(void* arg) {
cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(5, &cpuset); // Pin this thread to CPU 5
pthread_t current = pthread_self();
if (pthread_setaffinity_np(current, sizeof(cpu_set_t), &cpuset) != 0) {
perror("pthread_setaffinity_np failed");
}
// Thread business logic...
while (1) { /* work */ }
return NULL;
}
Verify which CPU a thread is actually running on:
# Check which CPU all threads of a process are currently running on
ps -eLo pid,tid,psr,comm | grep my_application
# The psr column shows the current processor number
NUMA Affinity Tuning
numactl Tool
numactl is the core tool for controlling NUMA policy:
# View NUMA topology
numactl --hardware
# View NUMA statistics
numastat
# Run program on NUMA node 0 (both CPU and memory bound to node 0)
numactl --cpunodebind=0 --membind=0 ./my_application
# Shorthand
numactl -N 0 -m 0 ./my_application
# Only bind CPU node, memory allocated freely
numactl --cpunodebind=0 ./my_application
# Interleave memory across all nodes (balanced)
numactl --interleave=all ./my_application
NUMA Policy Selection
| Policy | Command | Use Case | Effect |
|---|---|---|---|
| Bind | numactl -N 0 -m 0 ./app | CPU-intensive, latency-sensitive | CPU and memory on same node, optimal latency |
| Interleave | numactl --interleave=all ./app | Memory-intensive, large memory apps | Memory bandwidth doubled, but latency varies |
| Preferred | numactl --preferred=0 ./app | Prefer local, borrow if insufficient | Allocate locally when possible, cross-node when needed |
Database NUMA Configuration
Databases are the hardest hit by NUMA issues. MySQL and PostgreSQL don’t natively understand NUMA by default, making “cross-node memory access” a common cause of performance degradation.
MySQL NUMA Configuration:
# Start MySQL with interleave across all NUMA nodes
numactl --interleave=all mysqld_safe &
# Or bind to a specific node
numactl --cpunodebind=0 --membind=0 mysqld_safe &
PostgreSQL NUMA Configuration:
# In postgresql.conf
# Use huge pages to reduce TLB misses
huge_pages = on
# Bind at startup
numactl --interleave=all pg_ctl start
Benchmark data (PostgreSQL 15, 64GB memory, dual E5-2680 v4):
| Configuration | QPS | P99 Latency | Notes |
|---|---|---|---|
| Default (no NUMA config) | 12,000 | 8ms | Severe cross-node memory access |
| Interleave mode | 18,000 | 5ms | Memory balanced, bandwidth improved |
| Bind mode (single node) | 15,000 | 3ms | Optimal latency, but bandwidth limited |
The conclusion is clear: for bandwidth use interleave, for latency use bind. Choose based on your business scenario.
IRQ Pinning
Why Pin Interrupts
Network card interrupts are shared across all CPUs by default (via the irqbalance daemon). For systems with CPU isolation, isolated cores shouldn’t have any interrupt interference.
Disable irqbalance
# Stop irqbalance service
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance
Manual Interrupt Pinning
# View network card interrupt numbers
$ cat /proc/interrupts | grep eth0
28: 123456 234567 345678 456789 0 0 0 0 PCI-MSI 1572864-edge eth0-TxRx-0
29: 234567 345678 456789 567890 0 0 0 0 PCI-MSI 1572865-edge eth0-TxRx-1
30: 345678 456789 567890 678901 0 0 0 0 PCI-MSI 1572866-edge eth0-TxRx-2
31: 456789 567890 678901 789012 0 0 0 0 PCI-MSI 1572867-edge eth0-TxRx-3
# Check current interrupt affinity (bitmask)
$ cat /proc/irq/28/smp_affinity
00000000,00000000,00000000,000000ff
# ff means CPU 0-7 handle this interrupt
# Pin interrupt 28 to CPU 0 (mask 01 = CPU 0)
$ sudo echo 1 > /proc/irq/28/smp_affinity
# Pin interrupts 28-31 to CPU 0-3 respectively
for i in 28 29 30 31; do
cpu=$(($i - 28))
mask=$(printf "%x" $((1 << $cpu)))
echo $mask | sudo tee /proc/irq/$i/smp_affinity
done
Interrupt masks are hexadecimal bitmasks. 01 = CPU 0, 02 = CPU 1, 04 = CPU 2, 08 = CPU 3.
Using a set_irq_affinity Script
Manually calculating bitmasks is error-prone. Use a community script to simplify:
#!/bin/bash
# set_irq_affinity.sh
# Usage: ./set_irq_affinity.sh eth0 "0-3"
IFACE=$1
CPUS=$2
# Get NIC IRQ list
IRQS=$(grep "$IFACE" /proc/interrupts | cut -d: -f1 | tr -d ' ')
i=0
for IRQ in $IRQS; do
# Assign one CPU per queue
CPU=$(echo $CPUS | awk -F'[-,]' "{print \$$((i % $(echo $CPUS | tr ',' '\n' | wc -l) + 1))}")
printf "%x\n" $((1 << $CPU)) > /proc/irq/$IRQ/smp_affinity
echo "IRQ $IRQ -> CPU $CPU"
i=$((i + 1))
done
Production Implementation: Complete Configuration
Scenario
A dual-socket E5-2680 v4 server (56 cores, 112 threads, 256GB memory) running three types of workloads:
- Low-latency trading engine (needs exclusive CPU, microsecond-level response)
- MySQL database (needs NUMA affinity, high throughput)
- System services and monitoring (normal priority)
CPU Allocation Plan
NUMA Node 0 (CPU 0-27, physical cores 0-13, hyperthreads 28-41)
├── CPU 0-1: System services + interrupt handling
├── CPU 2-13, 28-39: Trading engine (isolated)
└── CPU 40-41: Spare
NUMA Node 1 (CPU 14-27, physical cores 56-83, hyperthreads 84-111)
├── CPU 56-69, 84-97: MySQL (NUMA bind)
└── CPU 70-71, 98-111: System services and monitoring
Complete Configuration Script
#!/bin/bash
# cpu-isolation-setup.sh
# Production CPU isolation and NUMA tuning script
# Tested on Ubuntu 22.04 (kernel 5.15)
set -euo pipefail
echo "=== 1. Configure GRUB boot parameters ==="
# Check if already configured
if grep -q "isolcpus" /proc/cmdline; then
echo "isolcpus already in boot parameters, skipping GRUB configuration"
else
echo "Please manually edit /etc/default/grub and add:"
echo 'GRUB_CMDLINE_LINUX="... isolcpus=2-13,28-39 nohz_full=2-13,28-39 rcu_nocbs=2-13,28-39"'
echo "Then run: sudo update-grub && sudo reboot"
exit 1
fi
echo "=== 2. Disable irqbalance ==="
sudo systemctl stop irqbalance 2>/dev/null || true
sudo systemctl disable irqbalance 2>/dev/null || true
echo "irqbalance disabled"
echo "=== 3. Pin network interrupts to CPU 0-1 ==="
# Get NIC interrupt numbers
NIC_IRQS=$(grep -E "eth0|ens192|enp" /proc/interrupts | awk -F: '{print $1}' | tr -d ' ')
i=0
for IRQ in $NIC_IRQS; do
# Alternate between CPU 0 and 1
CPU=$((i % 2))
MASK=$(printf "%x" $((1 << CPU)))
echo $MASK > /proc/irq/$IRQ/smp_affinity 2>/dev/null || true
echo "IRQ $IRQ -> CPU $CPU"
i=$((i + 1))
done
echo "=== 4. Migrate kernel threads off isolated cores ==="
ISOLATED_CPUS="2-13,28-39"
for pid in $(pgrep -f "rcu\|kworker\|ksoftirqd\|migration\|watchdog"); do
CURRENT_AFFINITY=$(taskset -pc $pid 2>/dev/null | grep -oP 'list: \K.*' || echo "")
if [ -n "$CURRENT_AFFINITY" ]; then
if taskset -pc $pid 2>/dev/null | grep -qP "[2-9]|1[0-3]|2[8-9]|3[0-9]"; then
taskset -pc 0 $pid 2>/dev/null || true
echo "Migrated kernel thread $pid to CPU 0"
fi
fi
done
echo "=== 5. Configure MySQL NUMA affinity ==="
if systemctl list-unit-files | grep -q mysql; then
echo "Add to MySQL systemd service:"
echo " NUMAPolicy=bind"
echo " NUMAMask=1"
echo " CPUAffinity=56-69,84-97"
fi
echo "=== 6. Verify isolation ==="
echo "--- Processes on isolated cores (should only be pinned ones) ---"
for cpu in 2 3 4 5; do
PROCS=$(ps -eLo pid,psr,comm | awk -v c=$cpu '$2==c {print $1, $3}')
if [ -n "$PROCS" ]; then
echo "CPU $cpu: $PROCS"
fi
done
echo "--- NUMA statistics ---"
numastat
echo "=== Configuration complete ==="
Verifying Isolation Effects
After configuration, verify with these methods:
# 1. Confirm isolcpus is active
cat /proc/cmdline | grep isolcpus
# 2. Check for unexpected processes on isolated cores
# Isolated cores should be idle or only have your pinned processes
mpstat -P 2-5 1 5
# Watch the %idle column — should be near 100% if no process is pinned
# 3. Check NUMA memory allocation
numastat -p <PID>
# Watch the Total column to confirm memory is on the correct node
# 4. Measure latency improvement
# Use cyclictest to measure scheduling latency
cyclictest -p 80 -t 1 -a 4 -d 0 -i 1000 -l 100000
# -a 4: pin to CPU 4
# Compare max latency before and after isolation
CPU Isolation in Container Environments
Kubernetes CPU Isolation
In Kubernetes, regular CPU request/limit uses CFS bandwidth control, which doesn’t provide exclusive isolation. For true CPU exclusivity, you need the static CPU Manager policy.
# kubelet configuration (/var/lib/kubelet/config.yaml)
kind: KubeletConfiguration
cpuManagerPolicy: static
reservedSystemCPUs: "0,1"
Then declare integer CPU requests in the Pod:
apiVersion: v1
kind: Pod
metadata:
name: critical-app
spec:
containers:
- name: app
image: critical-app:latest
resources:
requests:
cpu: "4" # Must be integer to trigger exclusive allocation
memory: "8Gi"
limits:
cpu: "4"
memory: "8Gi"
# Specify NUMA affinity (requires Topology Manager)
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
Note: CPU Manager’s static policy requires integer CPU requests. If you use 500m (0.5 core), it falls back to regular CFS throttling — no exclusivity.
Docker CPU Isolation
# Use --cpuset-cpus to pin container to specific CPUs
docker run -d \
--name critical-app \
--cpuset-cpus="4-7" \
--memory="16g" \
--memory-swappiness=0 \
critical-app:latest
Docker’s --cpuset-cpus only restricts which CPUs the container can use, but doesn’t prevent other containers from using the same CPUs. For true exclusivity, combine with host-level isolcpus.
Common Issues and Troubleshooting
Issue 1: isolcpus Configured but Processes Still on Isolated Cores
isolcpus only prevents normal process scheduling — kernel threads are unaffected. Common “escapees”:
# Check kernel threads on isolated cores
ps -eLo pid,psr,comm | awk '$2 >= 2 && $2 <= 13 {print}'
# May see kworker, ksoftirqd, migration, watchdog, etc.
# Manually migrate these threads to non-isolated cores
for pid in $(pgrep kworker); do
taskset -pc 0 $pid 2>/dev/null
done
for pid in $(pgrep ksoftirqd); do
taskset -pc 0 $pid 2>/dev/null
done
Note: some kernel threads (like migration) cannot be migrated and will error if you try. These have minimal performance impact and can be ignored.
Issue 2: OOM After numactl Binding
# Wrong: only bound memory node, but not enough memory on that node
$ numactl --membind=1 ./app
# Result: Node 1 memory exhausted, app OOM-killed
# Correct: check available memory first
$ numactl --hardware | grep "node 1 free"
node 1 free: 89012 MB
# If insufficient, use interleave mode
$ numactl --interleave=all ./app
Issue 3: cpuset Not Working in cgroup v2
cgroup v2 doesn’t enable the cpuset controller by default — you need to enable it manually:
# Check if cpuset is available
cat /sys/fs/cgroup/cgroup.controllers
# If cpuset isn't in the output, enable it at the root cgroup
# Enable cpuset controller
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
Issue 4: Performance Dropped After Isolation
This is usually caused by over-isolation. Common reasons:
- Too many CPU cores isolated, system services squeezed onto few cores, becoming a bottleneck
- NUMA binding not configured alongside CPU isolation, causing cross-node memory access
- Network interrupts all pinned to few cores, interrupt processing can’t keep up
Recommendation: isolate 2-4 cores first for testing, observe effects, then gradually expand.
Issue 5: Container –cpuset-cpus Conflict with Host isolcpus
# Host configured isolcpus=4-7
# Container configured --cpuset-cpus="4-7"
# In this case, the container can use CPU 4-7 normally
# Because isolcpus prevents "normal scheduling", while taskset/cpuset is "explicit allocation"
# The two don't conflict
But if the host has isolcpus=4-7 and the container wants CPU 0-3 (non-isolated cores), that’s also fine. The conflict scenario is: the container wants 4-7 but another cpuset on the host has already allocated those cores to a different container. In this case, cgroup v2’s cpuset.cpus takes precedence.
Performance Benchmarks
Test Environment
- Server: Dual Intel Xeon E5-2680 v4 (28 cores, 56 threads total)
- Memory: 256GB DDR4 ECC
- OS: Ubuntu 22.04 LTS (kernel 5.15.0-91)
- Test tools: cyclictest (scheduling latency), sysbench (comprehensive performance), iperf3 (network performance)
Scheduling Latency Comparison
# Without isolation
$ cyclictest -p 80 -t 1 -a 0 -i 1000 -l 100000
T:0 ( 1234) P:80 I:1000 C:100000 Min: 3 Act: 5 Avg: 8 Max: 47
# With isolcpus + nohz_full
$ cyclictest -p 80 -t 1 -a 4 -i 1000 -l 100000
T:0 ( 1234) P:80 I:1000 C:100000 Min: 2 Act: 3 Avg: 4 Max: 12
| Configuration | Min Latency | Avg Latency | Max Latency | Improvement |
|---|---|---|---|---|
| No isolation | 3μs | 8μs | 47μs | Baseline |
| isolcpus | 2μs | 4μs | 12μs | Max latency reduced 74% |
| isolcpus + nohz_full + rcu_nocbs | 2μs | 3μs | 9μs | Max latency reduced 81% |
NUMA Affinity Comparison (MySQL sysbench)
| Configuration | QPS | P95 Latency | CPU Usage |
|---|---|---|---|
| Default (no NUMA config) | 8,200 | 12ms | 65% |
| Interleave | 11,500 | 8ms | 70% |
| Bind (single node) | 9,800 | 5ms | 60% |
| Bind + cpuset isolation | 12,100 | 4ms | 72% |
Bind + cpuset isolation produces the best combined effect — NUMA binding reduces memory latency, cpuset isolation reduces CPU contention, and the two optimizations stack better than either alone.
Summary
CPU isolation and NUMA tuning are the “last mile” of performance optimization. Adding CPUs and memory is horizontal scaling; CPU isolation is vertical optimization — squeezing more performance from the same hardware by eliminating interference and cross-node access.
Key takeaways:
Layered isolation. isolcpus for boot-level isolation (strongest), cpuset for runtime isolation (most flexible), taskset for quick pinning (simplest). Use them together: isolcpus as the foundation + cpuset for service management + taskset for temporary debugging.
NUMA must be tuned alongside CPU binding. Binding CPU without binding memory is doing half the job — CPU runs on Node 0 while data sits on Node 1, and every memory access crosses nodes. Use numactl --cpunodebind=0 --membind=0 to bind both to the same node.
Interrupts must be managed. Hardware interrupts on isolated cores will interrupt your real-time threads. Disable irqbalance, manually pin network card interrupts to non-isolated cores.
Special handling for containers. Kubernetes’ CPU Manager static policy can give containers exclusive CPUs, but requires integer CPU requests. Docker’s –cpuset-cpus needs host-level isolcpus for true exclusivity.
Test before deploying. Isolation isn’t “more is better.” Over-isolation squeezes system services and causes new problems. Benchmark with cyclictest and sysbench first, confirm improvements, then expand scope.
One final note: this approach applies to physical machines or dedicated VMs. Cloud servers (AWS, Alibaba Cloud, etc.) have virtualization layers that interfere with NUMA topology exposure — numactl --hardware may not show the real NUMA topology. CPU isolation on cloud environments isn’t as effective as on bare metal, but that’s not a reason to skip it — even in virtualized environments, taskset and cpuset still reduce context switching and provide meaningful latency optimization.
References & Acknowledgments
This article referenced the following materials during writing. Thanks to the original authors for their contributions:
- 基于CPU隔离技术提升关键业务性能,最大化硬件利用率 — CSDN blogger, isolcpus and cpuset configuration methods and test comparisons
- 别再乱配isolcpus了!深入Linux内核cmdline解析,避开CPU隔离的5个常见配置误区 — CSDN blogger, isolcpus kernel parsing and configuration pitfall analysis
- Linux cgroup v2 资源控制实战 — CSDN blogger, cgroup v2 cpuset configuration methods
- Linux 组调度与容器编排:Kubernetes 的 CPU 资源分配底层 — CSDN blogger, Kubernetes CPU management and CFS group scheduling principles
- 【Linux性能调优核心技巧】:CPU亲和性绑定的5种高阶用法 — CSDN blogger, multiple approaches to CPU affinity binding
- Red Hat Enterprise Linux Performance Tuning Guide - CPU Scheduling — Red Hat, CPU scheduling policies and NUMA topology
- linux下CPU绑定、任务绑核、IRQ绑核具体怎么操作? — CSDN blogger, taskset, sched_setaffinity, and IRQ pinning operations