Overview

You’re running a high-frequency trading system online. P99 latency sits at 2ms normally, but occasionally spikes to 20ms. CPU usage isn’t high, memory is sufficient, network is fine. After investigation, you discover the CPU scheduler migrated a critical thread to another core, L3 cache missed entirely, and latency jumped 10x.

This kind of problem can’t be solved by adding resources. The issue is “sharing” — all processes share CPU cores, the scheduler distributes freely, and nobody knows which threads are latency-sensitive.

The solution is CPU isolation: pin your critical workloads to dedicated cores that no other process can touch. Combined with NUMA tuning to keep CPU and memory physically “close,” avoiding the latency doubling that comes from cross-node access.

This note covers the complete chain from basic concepts to production implementation: isolcpus kernel parameters, cpuset cgroups, taskset CPU pinning, NUMA affinity, IRQ binding, and best practices for combining them. All commands tested on Ubuntu 22.04 (kernel 5.15) and CentOS 8.

Fundamentals: Why Isolate CPUs

How the CPU Scheduler Works

Linux defaults to CFS (Completely Fair Scheduler) for CPU time allocation. CFS is designed for “fairness” — each process gets CPU time slices based on weight, and the scheduler freely migrates processes across all available cores.

Sounds fine, but for latency-sensitive scenarios it’s a disaster:

  1. Context switch overhead: When a thread migrates from CPU A to CPU B, L1/L2 cache is entirely invalidated, requiring reload from memory. A single migration costs microsecond-level latency jitter.
  2. Cache pollution: Other processes running on your target core push out your previously cached data. When your thread returns, it’s all cache misses.
  3. Interrupt interference: Network card interrupts and timer interrupts can interrupt your thread at any time. For systems requiring microsecond-level response, each interrupt is a latency spike.

What Is NUMA Architecture

NUMA (Non-Uniform Memory Access) is standard in multi-socket servers. Simply put: each CPU socket has its own local memory. Accessing local memory is fast; accessing another CPU’s memory requires crossing the QPI/UPI bus, roughly doubling the latency.

Think of it this way: NUMA is like two office buildings. You work in Building 1, your filing cabinet is in Building 1, grabbing things is quick. But if you need files from Building 2, you have to walk across — noticeably slower.

A typical dual-socket server:

NUMA Node 0 (CPU 0-31, Memory 0-128GB)
  └── Local memory access latency: ~80ns

NUMA Node 1 (CPU 32-63, Memory 128-256GB)
  └── Local memory access latency: ~80ns

Cross-node access latency: ~140ns  ← nearly double

Check your NUMA topology with numactl --hardware:

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 128768 MB
node 0 free: 89012 MB
node 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 129020 MB
node 1 free: 91045 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Note node distances: local distance is 10, cross-node is 21. The distance value is relative, representing the access latency ratio. 21 is roughly double 10 — cross-node access is indeed much slower.

Check memory allocation per node with numastat:

$ numastat
                           node 0           node 1
numa_hit              89342156         67823451
numa_miss               234561          123456
numa_foreign             12345           23456
interleave_hit          456789          345678
local_node            89012345         67012345
other_node             345611           823456
  • numa_hit: local node allocation hits — higher is better
  • numa_miss: local node allocation misses (should have been on another node) — lower is better
  • other_node: cross-node access count — if this keeps growing, your NUMA policy has issues

Method 1: isolcpus Kernel Parameter (Boot-Level Isolation)

What Is isolcpus

isolcpus is a Linux kernel boot parameter that isolates specified CPU cores from the global scheduler at boot time. Isolated cores don’t participate in normal process scheduling — only manually pinned processes can run on them.

This is the strongest isolation method — it takes effect from the moment the kernel boots, independent of runtime configuration.

Configuration

Edit GRUB configuration:

# Edit /etc/default/grub
sudo vim /etc/default/grub

# Add isolcpus parameter to GRUB_CMDLINE_LINUX
# For example, isolate CPU 4-7 (4 cores)
GRUB_CMDLINE_LINUX="isolcpus=4-7"

# Update GRUB
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
# On Ubuntu:
# sudo update-grub

# Reboot to apply
sudo reboot

Verify after reboot:

$ cat /proc/cmdline | grep isolcpus
BOOT_IMAGE=/boot/vmlinuz-5.15.0-91-generic root=... isolcpus=4-7

Advanced isolcpus Options

isolcpus doesn’t just simply isolate CPUs — it has sub-options that work together:

# Complete configuration example
GRUB_CMDLINE_LINUX="isolcpus=4-7 nohz_full=4-7 rcu_nocbs=4-7"
ParameterPurposeUse Case
isolcpus=4-7Remove CPU 4-7 from global schedulerBasic isolation
nohz_full=4-7Disable timer interrupts on isolated coresLow-latency scenarios, reduce interrupt jitter
rcu_nocbs=4-7RCU callbacks not executed on isolated coresFurther reduce kernel interference
isolcpus=4-7,domainExclude isolated cores from load balancing domainsEnabled by default, explicit declaration
isolcpus=4-7,managed_irqRestrict interrupt affinityReduce hardware interrupt interference

Using all three together produces the best results. nohz_full reduces timer interrupts, rcu_nocbs moves RCU callbacks away, isolcpus prevents normal process scheduling — three layers of isolation stacked together, leaving essentially only your manually pinned processes on the isolated cores.

Limitations of isolcpus

isolcpus isn’t a silver bullet:

  1. Requires reboot: Must reboot after changes, not suitable for dynamic adjustment
  2. Not fully isolated: Kernel threads and certain interrupts can still run on isolated cores, requiring manual migration
  3. Priority conflict with cgroup cpuset: If both isolcpus and cpuset cgroup are configured, the kernel prioritizes isolcpus restrictions. (Detailed analysis available here)

Method 2: cpuset cgroup (Runtime Isolation)

Why cpuset Instead of isolcpus

isolcpus requires a reboot and isn’t flexible enough. cpuset is a cgroup subsystem that can be dynamically created and modified at runtime without rebooting. Suitable for scenarios requiring frequent adjustments.

cgroup v1 Configuration

# Mount cpuset subsystem (if not already mounted)
sudo mount -t cgroup -o cpuset cpuset /sys/fs/cgroup/cpuset

# Create an isolation group
sudo mkdir /sys/fs/cgroup/cpuset/critical_app

# Assign CPU cores 4-5 to this group
sudo echo "4-5" > /sys/fs/cgroup/cpuset/critical_app/cpuset.cpus

# Assign NUMA node 0 memory
sudo echo "0" > /sys/fs/cgroup/cpuset/critical_app/cpuset.mems

# Enable exclusive mode (key! Other cpusets cannot use these CPUs)
sudo echo "1" > /sys/fs/cgroup/cpuset/critical_app/cpuset.cpu_exclusive

# Add process to this cpuset
sudo echo <PID> > /sys/fs/cgroup/cpuset/critical_app/tasks

The cpu_exclusive option is critical. When set to 1, these CPU cores become this cpuset group’s “private property” — no other cpuset group can use them. True exclusivity.

cgroup v2 Configuration

Modern Linux distributions (kernel 5.15+) default to cgroup v2. Configuration differs slightly:

# Enable cpuset controller
sudo echo "+cpuset" > /sys/fs/cgroup/cgroup.subtree_control

# Create a sub-group
sudo mkdir /sys/fs/cgroup/critical_app

# Assign CPU cores
sudo echo "4-5" > /sys/fs/cgroup/critical_app/cpuset.cpus

# Assign NUMA node
sudo echo "0" > /sys/fs/cgroup/critical_app/cpuset.mems

# Add process
sudo echo <PID> > /sys/fs/cgroup/critical_app/cgroup.procs

cgroup v2 syntax is cleaner, but note that v2 doesn’t have the cpu_exclusive option — exclusive isolation requires isolcpus as a companion.

Managing Service-Level cpuset with systemd

For production, managing cpuset through systemd is more reliable than manually manipulating cgroup files:

# /etc/systemd/system/critical-app.service
[Unit]
Description=Critical Low-Latency Application
After=network.target

[Service]
ExecStart=/opt/critical-app/bin/server
# Pin to CPU 4-5
CPUAffinity=4,5
# Set NUMA affinity
NUMAPolicy=bind
NUMAMask=0
# Limit memory node
AllowedCPUs=4-5
# Restart policy
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl start critical-app

systemd’s advantage: cpuset configuration is automatically restored after service restart, no manual maintenance needed.

Method 3: taskset (Quick CPU Pinning)

Using taskset

taskset is the simplest CPU pinning tool, suitable for temporary validation or quick deployment:

# Pin to CPU 4 at startup
taskset -c 4 ./my_application

# Pin to CPU 4 and 5
taskset -c 4,5 ./my_application

# Check running process CPU affinity
taskset -cp <PID>

# Modify running process CPU affinity
taskset -cp 4 <PID>

taskset’s limitation: it only manages process affinity and doesn’t prevent other processes from using the same CPU cores. If you use taskset without isolcpus, other processes can still be scheduled on the same core — your process is just “suggested” to run on these cores.

Programmatic CPU Pinning

Pin CPU directly in C/C++ programs:

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <pthread.h>

int main() {
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(4, &cpuset);  // Pin to CPU 4

    // Pin current thread
    if (sched_setaffinity(0, sizeof(cpu_set_t), &cpuset) != 0) {
        perror("sched_setaffinity failed");
        return 1;
    }

    printf("Pinned to CPU 4\n");
    // Your business logic...
    while (1) { /* work */ }
    return 0;
}

Pin specific threads in multi-threaded programs:

void* worker_thread(void* arg) {
    cpu_set_t cpuset;
    CPU_ZERO(&cpuset);
    CPU_SET(5, &cpuset);  // Pin this thread to CPU 5

    pthread_t current = pthread_self();
    if (pthread_setaffinity_np(current, sizeof(cpu_set_t), &cpuset) != 0) {
        perror("pthread_setaffinity_np failed");
    }

    // Thread business logic...
    while (1) { /* work */ }
    return NULL;
}

Verify which CPU a thread is actually running on:

# Check which CPU all threads of a process are currently running on
ps -eLo pid,tid,psr,comm | grep my_application

# The psr column shows the current processor number

NUMA Affinity Tuning

numactl Tool

numactl is the core tool for controlling NUMA policy:

# View NUMA topology
numactl --hardware

# View NUMA statistics
numastat

# Run program on NUMA node 0 (both CPU and memory bound to node 0)
numactl --cpunodebind=0 --membind=0 ./my_application

# Shorthand
numactl -N 0 -m 0 ./my_application

# Only bind CPU node, memory allocated freely
numactl --cpunodebind=0 ./my_application

# Interleave memory across all nodes (balanced)
numactl --interleave=all ./my_application

NUMA Policy Selection

PolicyCommandUse CaseEffect
Bindnumactl -N 0 -m 0 ./appCPU-intensive, latency-sensitiveCPU and memory on same node, optimal latency
Interleavenumactl --interleave=all ./appMemory-intensive, large memory appsMemory bandwidth doubled, but latency varies
Preferrednumactl --preferred=0 ./appPrefer local, borrow if insufficientAllocate locally when possible, cross-node when needed

Database NUMA Configuration

Databases are the hardest hit by NUMA issues. MySQL and PostgreSQL don’t natively understand NUMA by default, making “cross-node memory access” a common cause of performance degradation.

MySQL NUMA Configuration:

# Start MySQL with interleave across all NUMA nodes
numactl --interleave=all mysqld_safe &

# Or bind to a specific node
numactl --cpunodebind=0 --membind=0 mysqld_safe &

PostgreSQL NUMA Configuration:

# In postgresql.conf
# Use huge pages to reduce TLB misses
huge_pages = on

# Bind at startup
numactl --interleave=all pg_ctl start

Benchmark data (PostgreSQL 15, 64GB memory, dual E5-2680 v4):

ConfigurationQPSP99 LatencyNotes
Default (no NUMA config)12,0008msSevere cross-node memory access
Interleave mode18,0005msMemory balanced, bandwidth improved
Bind mode (single node)15,0003msOptimal latency, but bandwidth limited

The conclusion is clear: for bandwidth use interleave, for latency use bind. Choose based on your business scenario.

IRQ Pinning

Why Pin Interrupts

Network card interrupts are shared across all CPUs by default (via the irqbalance daemon). For systems with CPU isolation, isolated cores shouldn’t have any interrupt interference.

Disable irqbalance

# Stop irqbalance service
sudo systemctl stop irqbalance
sudo systemctl disable irqbalance

Manual Interrupt Pinning

# View network card interrupt numbers
$ cat /proc/interrupts | grep eth0
  28:   123456   234567   345678   456789   0   0   0   0   PCI-MSI 1572864-edge   eth0-TxRx-0
  29:   234567   345678   456789   567890   0   0   0   0   PCI-MSI 1572865-edge   eth0-TxRx-1
  30:   345678   456789   567890   678901   0   0   0   0   PCI-MSI 1572866-edge   eth0-TxRx-2
  31:   456789   567890   678901   789012   0   0   0   0   PCI-MSI 1572867-edge   eth0-TxRx-3

# Check current interrupt affinity (bitmask)
$ cat /proc/irq/28/smp_affinity
00000000,00000000,00000000,000000ff
# ff means CPU 0-7 handle this interrupt

# Pin interrupt 28 to CPU 0 (mask 01 = CPU 0)
$ sudo echo 1 > /proc/irq/28/smp_affinity

# Pin interrupts 28-31 to CPU 0-3 respectively
for i in 28 29 30 31; do
    cpu=$(($i - 28))
    mask=$(printf "%x" $((1 << $cpu)))
    echo $mask | sudo tee /proc/irq/$i/smp_affinity
done

Interrupt masks are hexadecimal bitmasks. 01 = CPU 0, 02 = CPU 1, 04 = CPU 2, 08 = CPU 3.

Using a set_irq_affinity Script

Manually calculating bitmasks is error-prone. Use a community script to simplify:

#!/bin/bash
# set_irq_affinity.sh
# Usage: ./set_irq_affinity.sh eth0 "0-3"

IFACE=$1
CPUS=$2

# Get NIC IRQ list
IRQS=$(grep "$IFACE" /proc/interrupts | cut -d: -f1 | tr -d ' ')

i=0
for IRQ in $IRQS; do
    # Assign one CPU per queue
    CPU=$(echo $CPUS | awk -F'[-,]' "{print \$$((i % $(echo $CPUS | tr ',' '\n' | wc -l) + 1))}")
    printf "%x\n" $((1 << $CPU)) > /proc/irq/$IRQ/smp_affinity
    echo "IRQ $IRQ -> CPU $CPU"
    i=$((i + 1))
done

Production Implementation: Complete Configuration

Scenario

A dual-socket E5-2680 v4 server (56 cores, 112 threads, 256GB memory) running three types of workloads:

  1. Low-latency trading engine (needs exclusive CPU, microsecond-level response)
  2. MySQL database (needs NUMA affinity, high throughput)
  3. System services and monitoring (normal priority)

CPU Allocation Plan

NUMA Node 0 (CPU 0-27, physical cores 0-13, hyperthreads 28-41)
  ├── CPU 0-1: System services + interrupt handling
  ├── CPU 2-13, 28-39: Trading engine (isolated)
  └── CPU 40-41: Spare

NUMA Node 1 (CPU 14-27, physical cores 56-83, hyperthreads 84-111)
  ├── CPU 56-69, 84-97: MySQL (NUMA bind)
  └── CPU 70-71, 98-111: System services and monitoring

Complete Configuration Script

#!/bin/bash
# cpu-isolation-setup.sh
# Production CPU isolation and NUMA tuning script
# Tested on Ubuntu 22.04 (kernel 5.15)

set -euo pipefail

echo "=== 1. Configure GRUB boot parameters ==="

# Check if already configured
if grep -q "isolcpus" /proc/cmdline; then
    echo "isolcpus already in boot parameters, skipping GRUB configuration"
else
    echo "Please manually edit /etc/default/grub and add:"
    echo 'GRUB_CMDLINE_LINUX="... isolcpus=2-13,28-39 nohz_full=2-13,28-39 rcu_nocbs=2-13,28-39"'
    echo "Then run: sudo update-grub && sudo reboot"
    exit 1
fi

echo "=== 2. Disable irqbalance ==="
sudo systemctl stop irqbalance 2>/dev/null || true
sudo systemctl disable irqbalance 2>/dev/null || true
echo "irqbalance disabled"

echo "=== 3. Pin network interrupts to CPU 0-1 ==="
# Get NIC interrupt numbers
NIC_IRQS=$(grep -E "eth0|ens192|enp" /proc/interrupts | awk -F: '{print $1}' | tr -d ' ')
i=0
for IRQ in $NIC_IRQS; do
    # Alternate between CPU 0 and 1
    CPU=$((i % 2))
    MASK=$(printf "%x" $((1 << CPU)))
    echo $MASK > /proc/irq/$IRQ/smp_affinity 2>/dev/null || true
    echo "IRQ $IRQ -> CPU $CPU"
    i=$((i + 1))
done

echo "=== 4. Migrate kernel threads off isolated cores ==="
ISOLATED_CPUS="2-13,28-39"
for pid in $(pgrep -f "rcu\|kworker\|ksoftirqd\|migration\|watchdog"); do
    CURRENT_AFFINITY=$(taskset -pc $pid 2>/dev/null | grep -oP 'list: \K.*' || echo "")
    if [ -n "$CURRENT_AFFINITY" ]; then
        if taskset -pc $pid 2>/dev/null | grep -qP "[2-9]|1[0-3]|2[8-9]|3[0-9]"; then
            taskset -pc 0 $pid 2>/dev/null || true
            echo "Migrated kernel thread $pid to CPU 0"
        fi
    fi
done

echo "=== 5. Configure MySQL NUMA affinity ==="
if systemctl list-unit-files | grep -q mysql; then
    echo "Add to MySQL systemd service:"
    echo "  NUMAPolicy=bind"
    echo "  NUMAMask=1"
    echo "  CPUAffinity=56-69,84-97"
fi

echo "=== 6. Verify isolation ==="
echo "--- Processes on isolated cores (should only be pinned ones) ---"
for cpu in 2 3 4 5; do
    PROCS=$(ps -eLo pid,psr,comm | awk -v c=$cpu '$2==c {print $1, $3}')
    if [ -n "$PROCS" ]; then
        echo "CPU $cpu: $PROCS"
    fi
done

echo "--- NUMA statistics ---"
numastat

echo "=== Configuration complete ==="

Verifying Isolation Effects

After configuration, verify with these methods:

# 1. Confirm isolcpus is active
cat /proc/cmdline | grep isolcpus

# 2. Check for unexpected processes on isolated cores
# Isolated cores should be idle or only have your pinned processes
mpstat -P 2-5 1 5
# Watch the %idle column — should be near 100% if no process is pinned

# 3. Check NUMA memory allocation
numastat -p <PID>
# Watch the Total column to confirm memory is on the correct node

# 4. Measure latency improvement
# Use cyclictest to measure scheduling latency
cyclictest -p 80 -t 1 -a 4 -d 0 -i 1000 -l 100000
# -a 4: pin to CPU 4
# Compare max latency before and after isolation

CPU Isolation in Container Environments

Kubernetes CPU Isolation

In Kubernetes, regular CPU request/limit uses CFS bandwidth control, which doesn’t provide exclusive isolation. For true CPU exclusivity, you need the static CPU Manager policy.

# kubelet configuration (/var/lib/kubelet/config.yaml)
kind: KubeletConfiguration
cpuManagerPolicy: static
reservedSystemCPUs: "0,1"

Then declare integer CPU requests in the Pod:

apiVersion: v1
kind: Pod
metadata:
  name: critical-app
spec:
  containers:
  - name: app
    image: critical-app:latest
    resources:
      requests:
        cpu: "4"       # Must be integer to trigger exclusive allocation
        memory: "8Gi"
      limits:
        cpu: "4"
        memory: "8Gi"
  # Specify NUMA affinity (requires Topology Manager)
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname

Note: CPU Manager’s static policy requires integer CPU requests. If you use 500m (0.5 core), it falls back to regular CFS throttling — no exclusivity.

Docker CPU Isolation

# Use --cpuset-cpus to pin container to specific CPUs
docker run -d \
    --name critical-app \
    --cpuset-cpus="4-7" \
    --memory="16g" \
    --memory-swappiness=0 \
    critical-app:latest

Docker’s --cpuset-cpus only restricts which CPUs the container can use, but doesn’t prevent other containers from using the same CPUs. For true exclusivity, combine with host-level isolcpus.

Common Issues and Troubleshooting

Issue 1: isolcpus Configured but Processes Still on Isolated Cores

isolcpus only prevents normal process scheduling — kernel threads are unaffected. Common “escapees”:

# Check kernel threads on isolated cores
ps -eLo pid,psr,comm | awk '$2 >= 2 && $2 <= 13 {print}'
# May see kworker, ksoftirqd, migration, watchdog, etc.

# Manually migrate these threads to non-isolated cores
for pid in $(pgrep kworker); do
    taskset -pc 0 $pid 2>/dev/null
done
for pid in $(pgrep ksoftirqd); do
    taskset -pc 0 $pid 2>/dev/null
done

Note: some kernel threads (like migration) cannot be migrated and will error if you try. These have minimal performance impact and can be ignored.

Issue 2: OOM After numactl Binding

# Wrong: only bound memory node, but not enough memory on that node
$ numactl --membind=1 ./app
# Result: Node 1 memory exhausted, app OOM-killed

# Correct: check available memory first
$ numactl --hardware | grep "node 1 free"
node 1 free: 89012 MB

# If insufficient, use interleave mode
$ numactl --interleave=all ./app

Issue 3: cpuset Not Working in cgroup v2

cgroup v2 doesn’t enable the cpuset controller by default — you need to enable it manually:

# Check if cpuset is available
cat /sys/fs/cgroup/cgroup.controllers
# If cpuset isn't in the output, enable it at the root cgroup

# Enable cpuset controller
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control

Issue 4: Performance Dropped After Isolation

This is usually caused by over-isolation. Common reasons:

  1. Too many CPU cores isolated, system services squeezed onto few cores, becoming a bottleneck
  2. NUMA binding not configured alongside CPU isolation, causing cross-node memory access
  3. Network interrupts all pinned to few cores, interrupt processing can’t keep up

Recommendation: isolate 2-4 cores first for testing, observe effects, then gradually expand.

Issue 5: Container –cpuset-cpus Conflict with Host isolcpus

# Host configured isolcpus=4-7
# Container configured --cpuset-cpus="4-7"

# In this case, the container can use CPU 4-7 normally
# Because isolcpus prevents "normal scheduling", while taskset/cpuset is "explicit allocation"
# The two don't conflict

But if the host has isolcpus=4-7 and the container wants CPU 0-3 (non-isolated cores), that’s also fine. The conflict scenario is: the container wants 4-7 but another cpuset on the host has already allocated those cores to a different container. In this case, cgroup v2’s cpuset.cpus takes precedence.

Performance Benchmarks

Test Environment

  • Server: Dual Intel Xeon E5-2680 v4 (28 cores, 56 threads total)
  • Memory: 256GB DDR4 ECC
  • OS: Ubuntu 22.04 LTS (kernel 5.15.0-91)
  • Test tools: cyclictest (scheduling latency), sysbench (comprehensive performance), iperf3 (network performance)

Scheduling Latency Comparison

# Without isolation
$ cyclictest -p 80 -t 1 -a 0 -i 1000 -l 100000
T:0 ( 1234) P:80 I:1000 C:100000 Min: 3 Act: 5 Avg: 8 Max: 47

# With isolcpus + nohz_full
$ cyclictest -p 80 -t 1 -a 4 -i 1000 -l 100000
T:0 ( 1234) P:80 I:1000 C:100000 Min: 2 Act: 3 Avg: 4 Max: 12
ConfigurationMin LatencyAvg LatencyMax LatencyImprovement
No isolation3μs8μs47μsBaseline
isolcpus2μs4μs12μsMax latency reduced 74%
isolcpus + nohz_full + rcu_nocbs2μs3μs9μsMax latency reduced 81%

NUMA Affinity Comparison (MySQL sysbench)

ConfigurationQPSP95 LatencyCPU Usage
Default (no NUMA config)8,20012ms65%
Interleave11,5008ms70%
Bind (single node)9,8005ms60%
Bind + cpuset isolation12,1004ms72%

Bind + cpuset isolation produces the best combined effect — NUMA binding reduces memory latency, cpuset isolation reduces CPU contention, and the two optimizations stack better than either alone.

Summary

CPU isolation and NUMA tuning are the “last mile” of performance optimization. Adding CPUs and memory is horizontal scaling; CPU isolation is vertical optimization — squeezing more performance from the same hardware by eliminating interference and cross-node access.

Key takeaways:

Layered isolation. isolcpus for boot-level isolation (strongest), cpuset for runtime isolation (most flexible), taskset for quick pinning (simplest). Use them together: isolcpus as the foundation + cpuset for service management + taskset for temporary debugging.

NUMA must be tuned alongside CPU binding. Binding CPU without binding memory is doing half the job — CPU runs on Node 0 while data sits on Node 1, and every memory access crosses nodes. Use numactl --cpunodebind=0 --membind=0 to bind both to the same node.

Interrupts must be managed. Hardware interrupts on isolated cores will interrupt your real-time threads. Disable irqbalance, manually pin network card interrupts to non-isolated cores.

Special handling for containers. Kubernetes’ CPU Manager static policy can give containers exclusive CPUs, but requires integer CPU requests. Docker’s –cpuset-cpus needs host-level isolcpus for true exclusivity.

Test before deploying. Isolation isn’t “more is better.” Over-isolation squeezes system services and causes new problems. Benchmark with cyclictest and sysbench first, confirm improvements, then expand scope.

One final note: this approach applies to physical machines or dedicated VMs. Cloud servers (AWS, Alibaba Cloud, etc.) have virtualization layers that interfere with NUMA topology exposure — numactl --hardware may not show the real NUMA topology. CPU isolation on cloud environments isn’t as effective as on bare metal, but that’s not a reason to skip it — even in virtualized environments, taskset and cpuset still reduce context switching and provide meaningful latency optimization.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

  1. 基于CPU隔离技术提升关键业务性能,最大化硬件利用率 — CSDN blogger, isolcpus and cpuset configuration methods and test comparisons
  2. 别再乱配isolcpus了!深入Linux内核cmdline解析,避开CPU隔离的5个常见配置误区 — CSDN blogger, isolcpus kernel parsing and configuration pitfall analysis
  3. Linux cgroup v2 资源控制实战 — CSDN blogger, cgroup v2 cpuset configuration methods
  4. Linux 组调度与容器编排:Kubernetes 的 CPU 资源分配底层 — CSDN blogger, Kubernetes CPU management and CFS group scheduling principles
  5. 【Linux性能调优核心技巧】:CPU亲和性绑定的5种高阶用法 — CSDN blogger, multiple approaches to CPU affinity binding
  6. Red Hat Enterprise Linux Performance Tuning Guide - CPU Scheduling — Red Hat, CPU scheduling policies and NUMA topology
  7. linux下CPU绑定、任务绑核、IRQ绑核具体怎么操作? — CSDN blogger, taskset, sched_setaffinity, and IRQ pinning operations