Linux Performance Analysis Powerhouse: BPF and the bcc Toolset

eBPF: The Kernel Programmability Revolution

Traditional performance analysis tools fall into two categories: “overview” tools like top, vmstat, and iostat that tell you what’s happening at the system level but lack detail; and “tracing” tools like strace and gdb that show detail but incur massive overhead, making them impractical for production.

eBPF (Extended Berkeley Packet Filter) changed everything. It allows you to run sandboxed programs in the kernel without modifying kernel source code or loading kernel modules. These programs attach to kernel hook points (kprobes, tracepoints, perf events, etc.) and are triggered when events occur, collecting data and passing it to userspace via ring buffer.

Why is this revolutionary? Three reasons:

Safe: eBPF programs are verified by the verifier at load time, ensuring no infinite loops and no illegal memory access — without the risks of loading kernel modules as root.
Low overhead: eBPF programs execute directly in kernel space, with only collected data copied to userspace via ring buffer, keeping hot-path overhead minimal.
Programmable: You can write precise probe programs tailored to your specific problem, rather than making do with generic tools.

For complete eBPF technical documentation, refer to the BPF Compiler Collection (BCC) official repository. All tools in this article come from that project.

bcc Toolset Installation

bcc (BPF Compiler Collection) is an eBPF-based performance analysis toolset maintained by the iovisor project, providing dozens of ready-to-use tracing tools.

Ubuntu / Debian

# Ubuntu 20.04+ recommended method
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# Verify installation
/usr/share/bcc/tools/biolatency --help

CentOS / RHEL

# CentOS 8 / RHEL 8+
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

# CentOS 7 requires ELRepo
sudo yum install -y epel-release
sudo yum install -y bcc-tools kernel-devel-$(uname -r)

# Tools are installed in /usr/share/bcc/tools/ by default
ls /usr/share/bcc/tools/

After installation, add the tools path to PATH:

echo 'export PATH=$PATH:/usr/share/bcc/tools' >> ~/.bashrc
source ~/.bashrc

Kernel Version Requirements

bcc depends on eBPF features that have evolved with kernel versions. Key features and their minimum kernel versions:

Feature	Minimum Kernel Version
Basic kprobe tracing	4.1+
perf event tracing	4.3+
Ring buffer	4.6+
BTF (BPF Type Format) support	5.0+
CO-RE (Compile Once - Run Everywhere)	5.0+

Core Tools in Practice

biolatency: Disk I/O Latency Distribution

biolatency traces block device I/O latency distribution and outputs a histogram. It is more valuable than iostat — iostat gives averages, while biolatency gives the distribution, helping you spot tail latency issues.

# Trace all block devices for 10 seconds
sudo biolatency 10

# Example output
#     usecs           : count     distribution
#        0 -> 1       : 0        |                                        |
#        2 -> 3       : 0        |                                        |
#        4 -> 7       : 0        |                                        |
#        8 -> 15      : 12       |                                        |
#       16 -> 31      : 45       |*                                       |
#       32 -> 63      : 128      |****                                    |
#       64 -> 127     : 340      |***********                             |
#      128 -> 255     : 890      |*****************************           |
#      256 -> 511     : 567      |******************                      |
#      512 -> 1023    : 234      |*******                                 |
#     1024 -> 2047    : 89       |**                                      |
#     2048 -> 4095    : 23       |                                        |
#     4096 -> 8191    : 5        |                                        |

From the output above, most I/O latency falls in the 128-255 microsecond range, but a small number of I/Os exceed 2ms — these tail latencies are what need attention.

To break down by disk, add the -d flag:

# Statistics per disk
sudo biolatency -d 10

# To trace I/O latency for a specific process
sudo biolatency -p $(pidof mysqld) 10

execsnoop: Process Execution Tracing

execsnoop traces the exec() system call of new processes, outputting each new process’s PID, PPID, and command line in real time. It is extremely useful for tracking down “short-lived processes” (like cron jobs, script call chains).

sudo execsnoop

# Example output
# PCOMM            PID    PPID   RET ARGS
# bash              4567   4566     0 /bin/bash
# ls                4568   4567     0 /bin/ls --color=auto -la
# grep              4569   4567     0 /bin/grep --color=auto foo
# python3           4570   1        0 /usr/bin/python3 /opt/app/main.py

A typical scenario: CPU spikes on a production server but top shows no corresponding process — likely a short-lived process repeatedly starting. execsnoop catches them instantly:

# Count the most frequently executed programs in 10 seconds
sudo execsnoop 2>&1 | tail -n +2 | awk '{print $1}' | sort | uniq -c | sort -rn | head

tcplife: TCP Connection Lifecycle

tcplife traces the lifecycle of TCP connections, outputting source/destination addresses, ports, transferred bytes, and duration for each connection. Very effective for troubleshooting “where exactly is the connection time spent.”

sudo tcplife

# Example output
# PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
# 4567  curl       10.0.1.5        43822 93.184.216.34   443   2     12    145
# 4568  nginx      10.0.1.5        80    192.168.1.100   54321 0     45    2300
# 4569  mysqld     10.0.1.5        3306  192.168.1.200   48932 1     8     56

Watch the MS (milliseconds) column to quickly spot slow connections. For example, a MySQL connection lasting tens of seconds with low transfer volume may indicate lock waiting.

opensnoop: File Open Tracing

opensnoop traces the open() system call across all processes, outputting file paths being opened and return values. A go-to tool for troubleshooting “which config file is the program reading” or “which process is frantically opening files.”

sudo opensnoop

# Trace only a specific process
sudo opensnoop -p $(pidof nginx)

# Trace only failed opens (for permission issues)
sudo opensnoop -e

# Example output
# PID    COMM     FD ERR PATH
# 4567   nginx    12   0 /etc/nginx/nginx.conf
# 4567   nginx    13   0 /etc/nginx/conf.d/default.conf
# 4567   nginx    14  -1 /etc/nginx/ssl/private.key
# 4568   mysqld   15   0 /var/lib/mysql/ibdata1

Above, ERR of -1 indicates a failed open, and PATH shows the file — that’s a permission issue caught in the act.

Custom BPF Trace Scripts

When built-in tools don’t meet your needs, you can write custom BPF scripts. bcc supports writing userspace logic in Python with embedded C code for kernel-space probes.

Below is a custom script that counts read() system calls and bytes read per process:

#!/usr/bin/env python3
"""
readsize.py - Count read() calls and bytes read per process
Usage: sudo python3 readsize.py [duration in seconds]
"""

from bcc import BPF
from time import sleep

# BPF C code: attach to read syscall entry and exit
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

// Pass fd argument between entry and exit
struct val_t {
    u64 fd;
};

// Map for passing fd between entry and exit
BPF_HASH(args, u32, struct val_t);

// Statistics result: PID -> [call count, bytes read]
struct info_t {
    u64 count;
    u64 bytes;
};
BPF_HASH(info, u32, struct info_t);

// read entry probe: record fd
TRACEPOINT_PROBE(syscalls, sys_enter_read) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    struct val_t val = {.fd = args->fd};
    args.update(&pid, &val);
    return 0;
}

// read exit probe: count return value (bytes read)
TRACEPOINT_PROBE(syscalls, sys_exit_read) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    struct val_t *valp = args.lookup(&pid);
    if (valp == 0) {
        return 0;  // No matching entry record, skip
    }
    args.delete(&pid);

    // Get return value (bytes read)
    int ret = args->ret;
    if (ret < 0) {
        return 0;  // read failed, don't count
    }

    // Update statistics
    struct info_t zero = {0, 0};
    struct info_t *infop = info.lookup_or_init(&pid, &zero);
    infop->count += 1;
    infop->bytes += ret;

    return 0;
}
"""

b = BPF(text=bpf_text)

# Frontend: periodically output statistics
print("Tracing read() syscall... Hit Ctrl-C to end.")

try:
    sleep(99999)
except KeyboardInterrupt:
    print("\n%-16s %-8s %12s %12s" % ("COMM", "PID", "CALLS", "BYTES"))
    print("-" * 52)

    info = b.get_table("info")
    for k, v in sorted(info.items(), key=lambda x: x[1].bytes, reverse=True):
        comm = b.get_kprobe_functions  # placeholder
        # Get process name
        try:
            comm_str = b.ksymname(k.value) or b"<unknown>"
        except Exception:
            comm_str = b"<unknown>"
        print("%-16s %-8d %12d %12d" % (comm_str[:16], k.value, v.count, v.bytes))

The script above uses tracepoint probes (syscalls/sys_enter_read and syscalls/sys_exit_read), which are more stable than kprobes — kprobes bind to kernel function symbols that may change across kernel versions, while tracepoints bind to agreed-upon event names with better cross-version compatibility.

Script usage:

sudo python3 readsize.py

# Example output
# Tracing read() syscall... Hit Ctrl-C to end.
# ^C
# COMM             PID            CALLS        BYTES
# ----------------------------------------------------
# mysqld           4567            89023    234567890
# nginx            4589            23456     89012345
# python3          4590             1234      5678901

A more concise approach can use bcc’s BPF.trace_print() or the @ syntax (simplified map notation in BPF C), but the full version above is better for understanding the underlying principles.

Production Environment Considerations

Kernel Version Compatibility

bcc tool compatibility is the biggest production pitfall. Key points:

Kernel >= 5.0: Enable BTF support. BTF provides kernel data structure type information, enabling eBPF programs to adapt to structural layout differences across kernel versions (CO-RE technology).
CentOS 7 kernel 3.10: Many bcc tools are unavailable or have limited functionality. Recommend upgrading the kernel (via ELRepo installing kernel-lt or kernel-ml) before use.
Reinstall kernel-headers after kernel upgrades, otherwise bcc will fail at runtime due to missing header files:

# Run after each kernel upgrade
sudo apt install -y linux-headers-$(uname -r)
# or
sudo dnf install -y kernel-devel-$(uname -r)

Performance Overhead Control

eBPF itself has minimal overhead, but that doesn’t mean it can be used without limits. Production usage guidelines:

Avoid high-frequency event probes: Attaching to probes like sched_switch (context switches) that fire tens of thousands of times per second — even at 1 microsecond per invocation, the cumulative impact is significant. On high-load systems, prefer statistical aggregation (histograms) over per-event output.
Use ring buffer instead of perf buffer: Kernel 4.6+ supports ring buffer, which batches data transfer and reduces user/kernel space switches.
Set timeouts: Always add a timeout parameter when using bcc tools in production (e.g., biolatency 10) to avoid leaving them running indefinitely.
Avoid -p tracing all events of high-concurrency processes: For high-concurrency processes like Nginx and MySQL, -p tracing every connection generates massive event volumes. Narrow the scope, e.g., tracing only specific ports.

Security and Permissions

All bcc tools require CAP_SYS_ADMIN or root privileges to load BPF programs. In production environments:

Do not grant CAP_BPF or CAP_SYS_ADMIN to application containers.
In Kubernetes clusters, run bcc tools via a dedicated DaemonSet, mounting kernel symbol tables and debugfs via hostPath.
For long-running eBPF observability programs, consider mature platform-level solutions like Polycube or Cilium, rather than running bcc scripts directly on production nodes.

Summary

eBPF + bcc is the Swiss Army knife of Linux performance analysis. Compared to traditional tools, it can dive into kernel internals at extremely low overhead — from disk latency distribution to TCP connection lifecycle, from process execution tracing to file open monitoring — covering every aspect of performance troubleshooting.

Mastering eBPF is not just about learning a few tools; it’s about understanding the programmability it provides. When built-in tools can’t answer your question, you can write custom probes to precisely ask the question you want answered. That’s why eBPF is called “the kernel programmability revolution.”

Further reading: Brendan Gregg’s BPF Performance Tools is the most authoritative eBPF performance analysis reference, covering 100+ tools with usage scenarios and output interpretation.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

BPF Compiler Collection (BCC) official repository — GitHub, referenced for BPF Compiler Collection (BCC) official repository
Brendan Gregg’s BPF Performance Tools — Brendan Gregg, referenced for Brendan Gregg’s BPF Performance Tools

eBPF: The Kernel Programmability Revolution#

bcc Toolset Installation#

Ubuntu / Debian#

CentOS / RHEL#

Kernel Version Requirements#

Core Tools in Practice#

biolatency: Disk I/O Latency Distribution#

execsnoop: Process Execution Tracing#

tcplife: TCP Connection Lifecycle#

opensnoop: File Open Tracing#

Custom BPF Trace Scripts#

Production Environment Considerations#

Kernel Version Compatibility#

Performance Overhead Control#

Security and Permissions#

Summary#

References & Acknowledgments#