Overview#
Performance profiling is a core SRE skill. Linux provides a rich set of performance analysis tools, from the simple top to the powerful perf/eBPF — each with its applicable scenario. Knowing when and how to use these tools is key to quickly identifying performance bottlenecks. This article systematically covers the Linux performance profiling toolkit across five dimensions — CPU, memory, IO, network, and comprehensive tools — and provides use-case quick reference tables and real-world examples.
top: The Classic Process Monitor#
$ top
# Top summary
top - 15:30:00 up 30 days, 3:21, 3 users, load average: 1.23, 0.98, 0.85
Tasks: 234 total, 1 running, 233 sleeping, 0 stopped, 0 zombie
%Cpu(s): 12.3 us, 3.4 sy, 0.0 ni, 83.3 id, 0.5 wa, 0.0 hi, 0.5 si, 0.0 st
MiB Mem : 32000.0 total, 5000.0 free, 15000.0 used, 12000.0 buff/cache
MiB Swap: 4096.0 total, 3900.0 free, 196.0 used. 20000.0 avail Mem
# Process list
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12345 root 20 0 12.5g 6.2g 12345 S 150 19.2 120:34 java
12346 www-data 20 0 500.0m 100.0m 20000 S 15 0.3 5:23 nginx
top Interactive Commands#
| Key | Function |
|---|
P | Sort by CPU usage |
M | Sort by memory usage |
N | Sort by PID |
T | Sort by running time |
1 | Expand individual CPU core details |
H | Show threads instead of processes |
c | Show full command line |
f | Select display fields |
W | Save configuration |
k | Kill process |
r | Renice process |
z | Color display |
top Key Field Interpretation#
load average: 1.23, 0.98, 0.85
# 1-minute/5-minute/15-minute average load
# Load ≠ CPU usage; includes processes waiting for CPU, IO, and locks
%Cpu(s): 12.3 us, 3.4 sy, 0.0 ni, 83.3 id, 0.5 wa, 0.0 hi, 0.5 si, 0.0 st
# us: User-space CPU
# sy: Kernel-space CPU
# ni: CPU for processes with nice > 0
# id: Idle
# wa: IO wait
# hi: Hardware interrupts
# si: Software interrupts
# st: Time stolen by hypervisor
htop: Interactive Process Monitor#
$ apt install htop # Debian/Ubuntu
$ dnf install htop # RHEL/CentOS
$ htop
# More user-friendly interface than top:
# - Color CPU core visualization
# - Process tree view
# - Mouse support
# - Keyboard shortcuts
htop Common Operations#
| Key | Function |
|---|
F5 | Tree view |
F6 | Sort |
F7/F8 | Decrease/increase nice value |
F9 | Send signal |
F10 | Quit |
t | Tree display |
H | Show/hide threads |
Tab | Switch process owner |
htop Configuration#
# ~/.config/htop/htoprc
# Or configure via F2:
# - Display fields
# - Color scheme
# - Update frequency
# - CPU average display mode
$ apt install atop
$ dnf install atop
$ atop 1 # Refresh every second
atop’s advantage is seeing process-level resource consumption, including disk I/O and network:
ATOP - server 2026/07/10 15:30:00 -------------------
1m 5m 15m
cpu sys 30% user 50% sys 25% user 45% sys 20% user 40%
cpu irq 2% idle 18% irq 1% idle 29% irq 1% idle 39%
cpu wait 0% wait 0% wait 0%
DISK busy read write | busy read write | busy read write
sda 45% 12K 345K | 30% 10K 300K | 25% 8K 250K
NET transport receive send | transport receive send
eth0 1234/s 100KB 50KB | 1000/s 80KB 40KB
PID USER CPU% MEM% DSK% NET% COMMAND
12345 root 150 19 5 1 java
12346 www 15 0 2 0 nginx
Common top Tips#
# Batch mode (for scripts)
$ top -b -n 1 -c | head -20
# Monitor specific processes
$ top -p 12345,12346
# Specify refresh count
$ top -n 5
# Output to file
$ top -b -n 10 -d 5 > top-output.txt
# Top 10 by CPU usage
$ top -b -n 1 -o %CPU | head -17
vmstat: System-Level CPU/Memory Overview#
$ vmstat 1 5
procs -----------memory---------- ---cpu----
r b swpd free buff cache si so us sy id wa st
2 0 100000 5G 2G 10G 0 0 30 10 55 5 0
1 0 100000 5G 2G 10G 0 0 35 12 50 3 0
| Field | Description | What to Watch |
|---|
r | Run queue length | > CPU core count indicates CPU bottleneck |
b | Blocked processes | > 0 indicates IO wait |
us | User-space CPU | High means app is CPU-intensive |
sy | Kernel-space CPU | > 20% means frequent syscalls |
id | Idle CPU | Low indicates CPU bottleneck |
wa | IO wait | > 5% indicates IO bottleneck |
si/so | Swap in/out | > 0 indicates memory shortage |
mpstat: Multi-Core CPU Statistics#
$ apt install sysstat
$ mpstat -P ALL 1 5
# -P ALL shows all CPU cores
10:00:01 AM CPU %usr %nice %sys %iowait %irq %soft %steal %idle %gnice
10:00:02 AM all 30.5 0.0 8.2 0.5 0.0 0.3 0.0 60.5 0.0
10:00:02 AM 0 55.0 0.0 10.0 0.0 0.0 0.0 0.0 35.0 0.0
10:00:02 AM 1 5.0 0.0 3.0 0.0 0.0 0.0 0.0 92.0 0.0
# CPU 0 high load, CPU 1 idle → uneven load distribution
pidstat: Process-Level CPU Statistics#
# View all process CPU usage
$ pidstat 1 5
# View specific process
$ pidstat -p 12345 1 5
# View thread-level CPU
$ pidstat -p 12345 -t 1 5
# View process context switches
$ pidstat -w 1 5
# View process CPU usage distribution
$ pidstat -p 12345 -u -t 1
# Install
$ apt install linux-tools-common linux-tools-$(uname -r)
$ dnf install perf
# View top CPU-consuming functions
$ perf top
# Record 10 seconds of CPU events
$ perf record -a -g -- sleep 10
# View report
$ perf report
# Count CPU cycles
$ perf stat -a -- sleep 10
# Trace specific process
$ perf record -p 12345 -g -- sleep 10
$ perf report
perf Common Events#
# List available events
$ perf list
# Common hardware events
$ perf stat -e cycles,instructions,cache-misses,branch-misses -- sleep 10
# Common software events
$ perf stat -e context-switches,cpu-migrations,page-faults -- sleep 10
# Trace syscalls
$ perf trace -p 12345
# Trace specific syscalls
$ perf trace -e read,write -p 12345
perf record Detailed Usage#
# Record call stacks
$ perf record -g -p 12345 -- sleep 30
# Record specific event
$ perf record -e cpu-clock -g -p 12345 -- sleep 30
# Record all CPUs
$ perf record -a -g -- sleep 10
# Specify sampling frequency
$ perf record -F 99 -g -p 12345 -- sleep 30
# -F 99: Sample 99 times per second
# Record to specific file
$ perf record -o perf.data -g -p 12345 -- sleep 30
# Analyze
$ perf report --sort overhead,symbol
Flame Graph#
# Install flamegraph tools
$ git clone https://github.com/brendangregg/FlameGraph.git
$ cd FlameGraph
# 1. Collect data with perf
$ perf record -F 99 -a -g -- sleep 30
$ perf script > out.perf
# 2. Generate flame graph
$ ./stackcollapse-perf.pl out.perf > out.folded
$ ./flamegraph.pl out.folded > flamegraph.svg
# 3. Open flamegraph.svg in a browser
Flame Graph Interpretation#
┌───[func_c]──────┐
│ │
[func_b] [func_d]
│
[func_a]
│
[main]
- Width = CPU time proportion consumed by the function
- Height = Call stack depth
- Color = Random (no specific meaning; On-CPU graphs typically use warm colors)
- Flat top = CPU hotspot function
Different Types of Flame Graphs#
| Type | Collection Method | Use Case |
|---|
| On-CPU | perf record | CPU hotspot analysis |
| Off-CPU | eBPF/offcputime | IO/lock wait analysis |
| Memory | perf record -e minor-faults | Memory allocation analysis |
| System Call | perf trace | Syscall analysis |
# View process on-CPU time
$ /usr/share/bcc/tools/profile.py -p 12345 -F 99 --duration 10
# View process off-CPU time
$ /usr/share/bcc/tools/offcputime -p 12345 10
# View syscalls
$ /usr/share/bcc/tools/trace.py 'SyS_*'
# View CPU hotspots
$ /usr/share/bcc/tools/profile.py -af
# View context switches
$ /usr/share/bcc/tools/stackcount.py t:sched:sched_switch
free: Memory Overview#
$ free -h
total used free shared buff/cache available
Mem: 32Gi 15Gi 5Gi 200Mi 12Gi 16Gi
Swap: 4Gi 196Mi 3.9Gi
# Key fields:
# total: Total memory
# used: Used (excluding buff/cache)
# free: Completely free
# buff/cache: Kernel cache (reclaimable)
# available: Memory available for applications (free + reclaimable buff/cache)
Important: To determine if memory is insufficient, check available, not free. Low free with sufficient available is normal.
$ cat /proc/meminfo
MemTotal: 33554432 kB
MemFree: 5242880 kB
MemAvailable: 16777216 kB # Available for applications
Buffers: 2097152 kB # Block device buffers
Cached: 10485760 kB # Page Cache
SwapCached: 200704 kB # Cached swap entries
Active: 12582912 kB # Active memory
Inactive: 4194304 kB # Inactive memory
SwapTotal: 4194304 kB
SwapFree: 3997696 kB
Dirty: 1024 kB # Dirty pages
AnonPages: 8388608 kB # Anonymous pages (application memory)
Slab: 2097152 kB # Slab cache
SReclaimable: 1572864 kB # Reclaimable Slab
SUnreclaim: 524288 kB # Unreclaimable Slab
vmstat: Memory and Swap#
$ vmstat 1 5
procs -----------memory---------- ---cpu----
r b swpd free buff cache si so us sy id wa st
1 0 200000 5G 2G 10G 0 0 30 10 55 5 0
2 1 200000 4.5G 2G 10G 5 0 40 15 40 5 0
# si > 0: Swapping in → memory shortage
# so > 0: Swapping out → memory shortage
# b > 0: Processes waiting on IO → possibly swap IO
pmap: Process Memory Mapping#
# View process memory mapping
$ pmap -x 12345 | tail -5
# Output: Address Size RSS Dirty Permissions Description
# View memory mapping summary
$ pmap -x 12345 | tail -1
total 12345678 6789012 123456
# Sort by largest mapping
$ pmap -x 12345 | sort -k3 -n -r | head -10
# View shared libraries
$ pmap 12345 | grep ".so" | sort -k2 -n -r
smaps: Detailed Memory Mapping#
# View detailed memory mapping of a process
$ cat /proc/12345/smaps_rollup
Rss: 6553600 kB # RSS
Pss: 6291456 kB # Proportional set size
Private_Clean: 10240 kB
Private_Dirty: 5242880 kB # Private dirty pages (watch for continuous growth)
Shared_Clean: 51200 kB
Shared_Dirty: 20480 kB
Anonymous: 5242880 kB # Anonymous memory
# View specific mapping region
$ cat /proc/12345/smaps | grep -A 15 "heap"
sar: Historical Memory Data#
# View memory usage trend
$ sar -r 1 5
# KBMEMFREE KBMEMUSED %MEMUSED KBBUFFERS KBCACHED KBSSWAPUSED
# 5242880 28311552 84.38 2097152 10485760 200704
# View swap activity
$ sar -W 1 5
# pswpin/s pswpout/s
# 0 0 # No swap activity
# View paging activity
$ sar -B 1 5
# pgpgin/s pgpgout/s fault/s majflt/s pgfree/s
# 0 0 1234.56 0.00 5678.90
# View historical data
$ sar -r -f /var/log/sa/sa10 # Data from the 10th
$ sar -r -s 09:00:00 -e 12:00:00 # Specify time range
iostat: Disk IO Statistics#
$ iostat -xdm 1 5
# -x: Extended statistics
# -d: Device only
# -m: In MB
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await %util
sda 0.5 2.0 50.0 100.0 2.5 5.0 50.0 1.5 10.0 75.0
nvme0n1 0.0 0.5 200.0 50.0 1.0 0.25 5.0 0.1 0.5 12.0
| Field | Description | What to Watch |
|---|
r/s w/s | Read/write IOPS per second | Evaluate load |
rMB/s wMB/s | Read/write throughput per second | Evaluate bandwidth |
avgrq-sz | Average request size (sectors) | Small = random IO, large = sequential IO |
avgqu-sz | Average queue length | > 1 indicates backlog |
await | Average I/O latency (ms) | SSD < 5ms, HDD < 20ms |
%util | Device utilization | > 80% indicates near bottleneck |
Note: %util can be misleading on multi-queue devices (NVMe). When I/O rate is high but await is low, even %util=100% doesn’t necessarily indicate a bottleneck.
iotop: Process-Level IO Monitor#
$ apt install iotop
$ iotop -o # Only show processes with IO
$ iotop -oP # Only show processes (not threads)
Total DISK READ: 50.0 M/s | Total DISK WRITE: 20.0 M/s
PID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
12345 be4 root 45.0 M/s 5.0 M/s 0.00 % 90.0 % dd if=/dev/zero of=/tmp/test
12346 be4 www 2.0 M/s 10.0 M/s 0.00 % 15.0 % nginx: worker
pidstat: Process-Level IO Statistics#
# View IO statistics
$ pidstat -d 1 5
# PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
# 12345 46080.00 5120.00 0.00 0 dd
# Combined CPU + IO + Memory
$ pidstat -urd 1 5
biolatency/biotop (eBPF)#
# View block IO latency distribution
$ /usr/share/bcc/tools/biolatency 10
# View process IO throughput ranking
$ /usr/share/bcc/tools/biotop 10 1
# View which processes are waiting on IO
$ /usr/share/bcc/tools/biosnoop 10
# View file IO latency
$ /usr/share/bcc/tools/filetop 10 1
# iftop: Network traffic monitoring
$ iftop -i eth0 -n
# nethogs: Process-level network traffic
$ nethogs eth0
# ss: Connection state statistics
$ ss -s # Summary
$ ss -tn state established | wc -l # Active connections
$ ss -tn state time-wait | wc -l # TIME_WAIT count
# sar: Network statistics
$ sar -n DEV 1 5 # NIC traffic
$ sar -n TCP 1 5 # TCP statistics
$ sar -n SOCK 1 5 # Socket statistics
# tcpdump: Packet capture
$ tcpdump -i eth0 -nn port 443 -c 1000 -w capture.pcap
$ tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-syn != 0' # Only SYN packets
# iperf3: Network throughput testing
$ iperf3 -c target -P 10 -t 60 # 10 concurrent connections for 60 seconds
sar: System Activity Reporter#
sar is the most comprehensive system performance historical data tool, suitable for trend analysis and post-incident investigation.
# Install
$ apt install sysstat
$ dnf install sysstat
# Enable data collection
# /etc/default/sysstat
ENABLED="true"
$ systemctl enable --now sysstat
sar Common Options#
# CPU usage
$ sar -u 1 5 # Real-time
$ sar -u -f /var/log/sa/sa10 # Data from the 10th
# Per-core CPU usage
$ sar -P ALL 1 5
# Memory usage
$ sar -r 1 5
# Swap activity
$ sar -W 1 5
# Disk IO
$ sar -d 1 5 # Device level
$ sar -p -d 1 5 # Show device names
# Network traffic
$ sar -n DEV 1 5
$ sar -n EDEV 1 5 # NIC errors
# TCP statistics
$ sar -n TCP 1 5
$ sar -n ETCP 1 5 # TCP errors
# Context switches
$ sar -w 1 5
# Interrupts
$ sar -I SUM 1 5
# File operations
$ sar -v 1 5
# Historical data query
$ sar -u -f /var/log/sa/sa10 -s 09:00:00 -e 12:00:00
sar Data File Management#
# Data file location
$ ls /var/log/sa/
sa01 sa02 ... sa10 sa11 ... sa31
# sa10: Binary data from the 10th
# sar10: Text data from the 10th (if sa2 script is configured)
# Configure data retention
# /etc/sysconfig/sysstat or /etc/default/sysstat
HISTORY=7 # Keep 7 days
COMPRESSAFTER=15 # Compress after 15 days
dstat: Multi-Dimensional Real-Time Monitoring#
$ apt install dstat
$ dstat -tcdnym --top-cpu --top-mem --top-io
# Simultaneously displays: Time CPU Disk Network Memory + top CPU/Memory/IO processes
# Common combinations
$ dstat -cdnm # CPU Disk Network Memory
$ dstat --disk-util # Disk utilization
$ dstat --tcp # TCP states
$ dstat --fs # File system
$ dstat --unix # Unix Socket
$ dstat --output stats.csv # Output to CSV
# pidstat: Process statistics
$ pidstat -u 1 # CPU
$ pidstat -r 1 # Memory
$ pidstat -d 1 # IO
$ pidstat -w 1 # Context switches
# cifsiostat: CIFS statistics
$ cifsiostat 1
# nfsiostat: NFS statistics
$ nfsiostat 1
Use-Case Quick Reference#
| Symptom | Primary Tool | Secondary Tool | Key Metric |
|---|
| High CPU usage | top/htop | perf top | %CPU > 80% |
| High load but low CPU usage | vmstat | pidstat -w | r > cores, wa high |
| Uneven CPU usage | mpstat -P ALL | top + press 1 | Large inter-core variance |
| Find CPU hotspot functions | perf record | flamegraph | overhead% |
| Abnormally high process CPU | pidstat -u | perf record -p | %CPU trend |
| Excessive context switches | pidstat -w | vmstat | cswch/s > 50000 |
| Scheduling latency | perf sched | runqlat (bcc) | latency > 10ms |
| Symptom | Primary Tool | Secondary Tool | Key Metric |
|---|
| Insufficient memory | free -h | vmstat | available low |
| OOM Killer | journalctl -k | dmesg | “Out of memory” |
| Memory leak | pmap -x | smaps_rollup | RSS continuous growth |
| Frequent swap | vmstat | sar -W | si/so > 0 |
| Process memory detail | pmap -x | /proc/PID/smaps | Private_Dirty |
| High Page Cache usage | free -h | /proc/meminfo | Cached + SReclaimable |
| Memory allocation tracing | perf record -e kmem | memleak (bcc) | Allocation call stack |
Disk IO Issues#
| Symptom | Primary Tool | Secondary Tool | Key Metric |
|---|
| High IO latency | iostat -x | biolatency (bcc) | await > 20ms |
| Low IO throughput | iostat -x | biotop (bcc) | rMB/s + wMB/s |
| Find IO-heavy process | iotop | pidstat -d | kB_rd/s + kB_wr/s |
| High IO wait | top/vmstat | iostat | wa > 5% |
| File-level IO | filetop (bcc) | opensnoop (bcc) | Read/write count |
| IO queue backlog | iostat -x | vmstat | avgqu-sz > 1 |
Network Issues#
| Symptom | Primary Tool | Secondary Tool | Key Metric |
|---|
| High network latency | ping | tcpdump | RTT |
| Low throughput | iperf3 | sar -n DEV | Bandwidth |
| High connection count | ss -s | sar -n SOCK | Established count |
| Many TIME_WAIT | ss | netstat | time-wait count |
| Packet loss | ifconfig | sar -n EDEV | drops/errors |
| Find traffic-heavy process | nethogs | iftop | Process-level traffic |
| TCP retransmission | ss -ti | nstat | retransRate |
| Packet analysis | tcpdump | wireshark | Packet content |
Real-World Examples#
Example 1: CPU Usage Spike Investigation#
# 1. Detect CPU 100%
$ top
# PID 12345 (java) CPU 200%
# 2. Check user-space vs kernel-space
$ pidstat -p 12345 -u 1 5
# %usr 150% %sys 50% → primarily user-space
# 3. View thread-level CPU
$ top -H -p 12345
# PID 12350 (java) CPU 100%
# 4. Collect flame graph
$ perf record -F 99 -p 12350 -g -- sleep 30
$ perf script > out.perf
$ ./flamegraph.pl out.folded > flamegraph.svg
# 5. Analyze flame graph for hotspot functions
# Example: HashMap.resize() consuming 80%
# Root cause: HashMap concurrent resize causing infinite loop
Example 2: Database IO Latency Investigation#
# 1. Check IO latency
$ iostat -xdm 1 5
# sda: await=35ms, %util=95% # HDD await > 20ms is abnormal
# 2. Find IO-heavy process
$ iotop -oP
# PID 54321 (mysqld) READ 45MB/s
# 3. Check MySQL
$ mysql -e "SHOW PROCESSLIST"
# Found many full table scan queries
# 4. Use biolatency to confirm latency distribution
$ /usr/share/bcc/tools/biolatency 10
# Latency distribution: 50% at 16-32ms, 10% at 64-128ms
# 5. Fix: Add indexes, optimize queries
Example 3: Memory Leak Investigation#
# 1. Monitor RSS changes
$ while true; do
cat /proc/12345/status | grep VmRSS
sleep 60
done
# VmRSS continuously growing, 100MB increase per hour
# 2. Analyze memory distribution
$ cat /proc/12345/smaps_rollup
# Private_Dirty continuously growing → anonymous memory leak
# 3. Use eBPF to trace memory allocation
$ /usr/share/bcc/tools/memleak -p 12345
# Shows leak call stack
# 4. For Java applications
$ jmap -dump:format=b,file=/tmp/heap.hprof 12345
# Analyze heap dump with MAT
Example 4: Network Latency Investigation#
# 1. Check network latency
$ ping target
# RTT = 50ms (normal < 1ms)
# 2. Check TCP retransmission
$ nstat -az | grep -i retrans
# TcpRetransSegs value is high
# 3. Capture packets for analysis
$ tcpdump -i eth0 -nn host target -w capture.pcap -c 10000
$ tcpdump -r capture.pcap -nn | grep -c "retransmission"
# 4. Check NIC errors
$ sar -n EDEV 1 5
# rxerr/s or txerr/s > 0
# 5. Check connection queue
$ ss -lnt
# Recv-Q > 0 indicates accept queue backlog
# Use dstat for multi-dimensional monitoring
$ dstat -tcdnym --top-cpu --top-mem --top-io --output perf.csv 1 300
# Analyze CSV data
$ python3 -c "
import csv
with open('perf.csv') as f:
reader = csv.reader(f)
for row in reader:
if 'cpu' in str(row).lower():
print(row)
"
# Use sar for post-incident historical data analysis
$ sar -u -f /var/log/sa/sa10 -s 14:00:00 -e 15:00:00
# Analyze CPU changes during the incident period
Performance Issue Investigation
├── CPU Issues
│ ├── High overall CPU → top → vmstat
│ ├── Find hotspot process → top/htop → pidstat
│ ├── Find hotspot function → perf record → flamegraph
│ └── Uneven CPU → mpstat -P ALL → taskset
│
├── Memory Issues
│ ├── Insufficient memory → free → vmstat → /proc/meminfo
│ ├── OOM → journalctl -k → /var/log/messages
│ ├── Memory leak → pmap → smaps → memleak (bcc)
│ └── Frequent swap → vmstat → sar -W
│
├── IO Issues
│ ├── High IO latency → iostat -x → biolatency (bcc)
│ ├── Find IO-heavy process → iotop → pidstat -d
│ ├── High IO wait → top → vmstat
│ └── File-level IO → filetop (bcc) → opensnoop (bcc)
│
├── Network Issues
│ ├── High latency → ping → tcpdump
│ ├── Low throughput → iperf3 → sar -n DEV
│ ├── High connection count → ss → sar -n SOCK
│ └── Packet loss → ifconfig → sar -n EDEV
│
└── Comprehensive Analysis
├── Real-time monitoring → dstat → atop
├── Historical trends → sar
└── Deep analysis → perf → eBPF/bcc
Summary#
The Linux performance profiling toolkit is extensive, but each tool has its applicable scenario. Key takeaways:
- top/htop are entry-level tools: Quickly understand overall system status and identify abnormal processes.
- vmstat is the versatile tool: One command shows CPU, memory, and IO overview simultaneously.
- iostat is the core of IO diagnosis: The
-x option provides key metrics like await and %util. - perf + flamegraph is the deep analysis powerhouse: Pinpoints CPU hotspots at the function level.
- eBPF/bcc is the next-generation toolkit: Can trace any kernel and user-space event with more precise latency analysis.
- sar is the only choice for historical data analysis: sysstat data collection must be enabled in production.
- pidstat is the handy helper for process-level analysis: Covers CPU, memory, IO, and context switches.
- Tool selection follows the “broad to narrow” principle: Start with top/sar for the big picture, use pidstat/iostat to pinpoint process/device, then use perf/eBPF to analyze root cause.
The golden rule of performance profiling: describe the problem first, then choose the tool. Clarify the symptom (high CPU? slow IO? network jitter?) → determine the scope (process-level? system-level?) → select the tool → collect data → analyze root cause — rather than randomly running commands.