Disk I/O Performance Diagnosis and Optimization

Introduction

Disk I/O is often the slowest link in the system performance chain. A single mechanical disk seek takes about 10ms, while memory access takes only about 100ns — a 100,000x difference. When business applications experience latency jitter or slow response times, the investigation inevitably points to the I/O subsystem. This article starts from the metrics system, combined with hands-on tool usage and production case studies, to build a reusable I/O diagnosis methodology.

I/O Performance Metrics

Before diving in, you must understand the four core metrics and their interrelationships.

These four metrics have key constraint relationships:

• In small-block random read/write scenarios, the bottleneck is IOPS (e.g., database OLTP 4KB random writes)
• In large-block sequential read/write scenarios, the bottleneck is throughput (e.g., log appending, video streaming)
• Latency is the end-user perceived metric; even with sufficient IOPS and throughput, high per-request latency causes stuttering
• Increasing queue depth boosts concurrency but also means longer per-request wait times

An important insight: IOPS × block size ≈ throughput. For example, 4KB blocks at 100 IOPS yields ~0.4 MB/s throughput; 1MB blocks at 100 IOPS yields ~100 MB/s. Understanding this formula helps identify the bottleneck type.

Diagnostic Tools in Practice

iostat: Macro I/O Overview

iostat from the sysstat package is the first stop for I/O diagnosis:

# Install
yum install -y sysstat        # RHEL/CentOS
apt install -y sysstat        # Debian/Ubuntu

# View extended statistics for all devices, refresh every 1 second, 5 times
iostat -dxm 1 5

Key output field interpretation:

Device  r/s    w/s   rkB/s  wkB/s  rrqm/s  wrqm/s  %util  aqu-sz  await  r_await  w_await
sda     125.30 38.20 5012.0 1528.0  8.50     2.10    98.70   15.32   45.20   32.10    88.40

• r/s w/s: Read/write operations per second (after merging), reflects IOPS
• rkB/s wkB/s: Read/write throughput per second
• %util: Device utilization, sustained >80% is the alert threshold
• aqu-sz: Average queue depth, indicates backlog level
• await: Average I/O latency (ms), includes queue wait + device service time
• r_await / w_await: Separate read/write latency statistics, helps identify bottleneck direction

Note: %util can be misleading on multi-queue devices like NVMe. An NVMe SSD may show %util at 100% but still have headroom. In such cases, refer to await and aqu-sz instead.

iotop: Identifying I/O Hotspot Processes

iostat tells you which disk is busy; iotop tells you who is reading/writing:

# Show only processes with actual I/O, refresh every 2 seconds
iotop -o -d 2

# Non-interactive mode, output 3 times then exit
iotop -b -o -n 3

Watch the DISK READ and DISK WRITE columns to quickly identify processes generating heavy I/O. If iotop is unavailable, you can extract directly from /proc:

# View I/O statistics per process (unit: bytes)
cat /proc/diskstats | head
for pid in $(ls /proc | grep -E '^[0-9]+$'); do
    io=$(cat /proc/$pid/io 2>/dev/null | grep "write_bytes" | awk '{print $2}')
    [ -n "$io" ] && [ "$io" -gt 0 ] && echo "PID=$pid WRITE_BYTES=$io CMD=$(cat /proc/$pid/cmdline 2>/dev/null | tr '\0' ' ')"
done | sort -t= -k3 -rn | head -10

fio: Benchmark Testing

fio (Flexible I/O Tester) is the industry standard for I/O performance testing. Below is a test configuration covering common scenarios:

# fio_test.fio - Disk benchmark configuration
[global]
ioengine=libaio
direct=1
runtime=60
time_based=1
group_reporting=1
directory=/mnt/testdir
filename=fio_testfile

# Test 1: 4KB random read - simulates database OLTP
[randread-4k]
bs=4k
rw=randread
iodepth=32
numjobs=4
name=randread-4k

# Test 2: 4KB random write - simulates database writes
[randwrite-4k]
bs=4k
rw=randwrite
iodepth=32
numjobs=4
name=randwrite-4k

# Test 3: 1MB sequential read - simulates large file reads
[seqread-1m]
bs=1m
rw=read
iodepth=8
numjobs=1
name=seqread-1m

# Test 4: Mixed read/write 70/30 - simulates real workloads
[mixed-rw]
bs=4k
rw=randrw
rwmixread=70
iodepth=32
numjobs=4
name=mixed-rw

Running the test:

# Create test directory
mkdir -p /mnt/testdir

# Run the test
fio fio_test.fio

# Quick single test (command-line mode)
fio --name=randread --ioengine=libaio --direct=1 --bs=4k --rw=randread --iodepth=32 --runtime=30 --time_based --filename=/dev/sdb

# Key output metrics to focus on:
#   IOPS    - I/O operations per second
#   BW      - Bandwidth (throughput)
#   lat     - Latency distribution (avg/min/max/p99)

Warning: direct=1 bypasses the page cache to test raw device performance. In production environments, always use a test partition or temporary file to avoid data corruption on production data disks.

I/O Scheduler Comparison and Selection

The I/O scheduler is responsible for sorting and merging I/O requests submitted to the block layer. Different schedulers are optimized for different scenarios.

Viewing and switching schedulers:

# View the current scheduler for a device
cat /sys/block/sda/queue/scheduler
# Example output: [mq-deadline] kyber bfq none

# Switch scheduler (takes effect immediately, lost on reboot)
echo bfq > /sys/block/sda/queue/scheduler

# Permanent setting via udev rules
cat > /etc/udev/rules.d/60-io-scheduler.rules << 'EOF'
# NVMe SSD uses none
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
# SATA SSD uses mq-deadline
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="mq-deadline"
# HDD uses bfq
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="bfq"
EOF

Selection strategy summary:

• NVMe SSD → none: The device has its own hardware queues; software-layer sorting is unnecessary overhead
• SATA SSD → mq-deadline: Balances latency control and simplicity
• HDD → bfq or mq-deadline: Needs to reduce seeking and prevent starvation
• Virtual machine disks → none: The host already handles scheduling; re-scheduling in the guest is redundant

SSD vs HDD Optimization Differences

SSD-Specific Optimization

1. Enable TRIM

TRIM informs the SSD which data blocks have been deleted, making its internal garbage collection more efficient, directly impacting long-term write performance stability.

# Check TRIM support
lsblk -D
# If DISC-GRAN and DISC-MAX are non-zero, TRIM is supported

# Manually execute TRIM (one-time reclaim of all unused blocks)
fstrim -v /
# Example output: /: 1234567890 bytes were trimmed

# Configure periodic TRIM (systemd timer, recommended)
systemctl enable --now fstrim.timer
# Runs weekly by default, view the configuration
cat /usr/lib/systemd/system/fstrim.timer

Do not run fstrim directly on individual member disks of a RAID array. Use the RAID controller’s discard capability or run it on the logical volume instead.

2. Mount Parameter Optimization

# /etc/fstab
UUID=xxx  /data  ext4  defaults,noatime,discard  0 2

# noatime: Do not update file access time, reduces write amplification
# discard: Enable online TRIM (continuous mode, minor performance impact)
#   - NVMe: recommend periodic fstrim over discard
#   - SATA SSD: either is fine

HDD-Specific Optimization

# Confirm rotational flag (1=HDD, 0=SSD)
cat /sys/block/sda/queue/rotational

# Disabling read-ahead may harm random I/O, but can boost sequential reads
# View current read-ahead value
blockdev --getra /dev/sda
# Set read-ahead to 8MB (improves large file sequential reads)
blockdev --setra 8192 /dev/sda

Production Case: High I/O Wait Troubleshooting Walkthrough

Symptom Discovery

A MySQL replica reported replication lag. SSH login felt noticeably sluggish. Running top showed %wa at 60-80%:

top - 14:32:01 up 45 days
%Cpu(s):  5.2 us,  3.1 sy,  0.0 ni, 18.5 id, 72.8 wa,  0.0 hi,  0.4 si

%wa (iowait) represents the percentage of CPU time spent waiting for I/O completion. High iowait does not necessarily mean the disk is slow — if the CPU is idle and waiting for I/O, iowait also rises. You need to confirm whether it’s slow I/O or an idle CPU.

Identifying the Bottleneck

Step 1: iostat to confirm device-level issues

iostat -dxm 1

Key findings:

Device  r/s     w/s    rkB/s  wkB/s   %util  aqu-sz  await
sda     5230.0  1850.0 20920  7400    100.00  88.45   13.28

%util=100%, aqu-sz=88 (extremely high queue depth), await=13ms — the disk is fully loaded and severely backlogged. IOPS reached 7000+, which is near the limit for a SATA SSD.

Step 2: iotop to identify the process

iotop -o -d 2

Found a mysqld process with DISK READ consistently at 200+ MB/s, far exceeding normal business traffic.

Step 3: MySQL-level analysis

-- View currently executing queries
SHOW FULL PROCESSLIST;

-- View long-running queries
SELECT id, user, host, time, state, LEFT(info, 200) AS query
FROM information_schema.processlist
WHERE time > 10 AND info IS NOT NULL
ORDER BY time DESC;

Found a full table scan query running:

SELECT COUNT(*) FROM order_log WHERE remark LIKE '%keyword%';

The order_log table is 200GB, the remark column has no index, and LIKE '%keyword%' cannot use an index, triggering a full table scan.

Resolution

Immediate mitigation:

-- Kill the problematic query
KILL QUERY 12345;

I/O wait dropped from 72% to under 5%, and MySQL replica lag began catching up.

Root cause fix:

1. Add a full-text index on the remark column or change fuzzy queries to exact match + prefix match
2. Migrate large table COUNT operations to offline analytics
3. Adjust InnoDB I/O-related parameters to reduce per-I/O pressure:

# my.cnf - InnoDB I/O optimization
[mysqld]
# Concurrent dirty page flushing threads, SSD can be set to 4-8
innodb_io_capacity = 2000
innodb_io_capacity_max = 4000
# Read-ahead window, can be reduced for SSD
innodb_read_io_threads = 8
innodb_write_io_threads = 8

Troubleshooting Methodology Summary

top (%wa high)
  → iostat (confirm %util / await / aqu-sz)
    → iotop (identify process)
      → process-level analysis (MySQL/app logs)
        → root cause: SQL/code/config
          → mitigation + root fix

Core principle: Don’t jump to tuning disk parameters when you see high iowait. First confirm whether the I/O pressure is reasonable — if it’s an unreasonable full table scan, no amount of I/O bandwidth will be enough.

References

• Linux kernel documentation - Block I/O layer
• fio - Flexible I/O Tester official documentation
• sysstat / iostat documentation
• Red Hat: Storage administration guide - I/O scheduling