Disk Space and inode Management: A Complete Troubleshooting Guide from df to lsof

Overview

3 AM, you get paged — the production database can’t write, Nginx won’t start, SSH is sluggish. You log in and df -h shows the root partition at 100%. Every ops engineer has been there. Disk full is one of the most frequent production incidents — based on real-world operations statistics, disk-related alerts account for roughly 15%-20% of all alerts.

But the 100% you see in df -h may not be the full picture:

• inode exhaustion: There’s still disk space available, but the filesystem has run out of inodes. You still get No space left on device. This is more insidious and harder to diagnose.
• Deleted but unreleased files: You rm a large file, but df shows no space freed. A process is still holding the file descriptor, so the data blocks aren’t actually released.
• Reserved space: ext4 reserves 5% of space for root by default. On a 400GB data disk, that’s 20GB hidden away.

This article covers the full troubleshooting workflow for disk space and inode management — diagnosis, cleanup strategies, and prevention. All commands have been tested on Ubuntu 22.04 and CentOS 7.9 and are ready to use in production.

Two Dimensions of Disk Space: blocks and inodes

Many people understand disk space only as “how many GB are left.” But Linux filesystems manage two independent resources: data blocks and inodes.

Think of it this way: the filesystem is like a parking lot. Blocks are parking spots. inodes are parking records. The lot might have empty spots (blocks aren’t full), but if the record book is full (inodes exhausted), new cars can’t park.

What is an inode

An inode (Index Node) is the data structure that describes a file’s metadata in the filesystem. Each file or directory corresponds to one inode, which stores:

A key point: inodes don’t store filenames. Filenames are stored in directory entries (dentries), which map names to inode numbers. This is why hard links work — multiple filenames can point to the same inode.

The total inode count is fixed when the filesystem is created. ext4 defaults to one inode per 16KB of space, meaning a 100GB partition has approximately 6.55 million inodes. That’s plenty for large files, but if your system accumulates millions of small files (mail queues, log fragments, session files), inodes can run out before blocks do.

inode Strategy Differences: xfs vs ext4

ext4’s inode count is fixed at mkfs time and cannot be adjusted later. If you didn’t estimate well at creation time, you’re stuck with reformatting.

xfs takes a different approach — it uses dynamic inode allocation. When new inodes are needed, xfs automatically allocates them from free space. This means xfs theoretically doesn’t suffer from “inodes exhausted but space available” problems. The tradeoff is that xfs inodes don’t have a fixed size like ext4, which can be limiting in scenarios requiring large inodes (e.g., storing extensive ACLs).

# Check filesystem type
df -T

# Check ext4 inode configuration
tune2fs -l /dev/sda1 | grep -i "inode\|block count\|block size"

# Check xfs inode information
xfs_info /dev/sda1 | grep -i "imax\|inodes"

Real Case: inode Exhaustion with Space Available

One day, an application reports No space left on device. You log in:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   30G  40% /

$ df -i /
Filesystem     Inodes  IUsed IFree IUse% Mounted on
/dev/sda1     3276800 3276800    0  100% /

Disk space shows 30GB free, but inodes are 100% exhausted. This contradiction is not uncommon in Linux systems — the root cause is typically a massive accumulation of small files (Ref: “No space left on device” with disk space available).

Step 1: Quickly Assess Disk Status

When you receive a disk alert, the first thing to do is not delete files — it’s to figure out what’s actually full.

Check blocks and inodes Together

# Check disk space usage
df -h

# Check inode usage (critical!)
df -ih

Adding -h to df -i displays inode usage in human-readable format. If you only check df -h and skip df -i, you’ll easily miss inode exhaustion.

Typical output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   95G  5.0G  95% /
/dev/sdb1       500G  120G  380G  24% /data
tmpfs           7.8G  1.2M  7.8G   1% /run

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1      6.5M   6.5M     0  100% /
/dev/sdb1      3.2M   125K  3.1M    1% /data
tmpfs          512K   105   512K    1% /run

This output shows the root partition has both a space shortage and inode exhaustion simultaneously.

Find the Largest Space-Consuming Directories

# Drill down from root level by level
du -h --max-depth=1 / 2>/dev/null | sort -rh | head -20

# Drill into a specific directory
du -h --max-depth=1 /var 2>/dev/null | sort -rh | head -20

How this works: du calculates the size of each first-level subdirectory, sort -rh sorts in reverse human-readable order, and head -20 shows only the top 20. By drilling down level by level, you can usually pinpoint the culprit in 3-4 iterations.

Find Directories with the Most inodes

# Count files per top-level directory
find / -xdev -type f 2>/dev/null | \
  awk -F/ '{print "/"$2}' | \
  sort | uniq -c | sort -rn | head -20

Or use a more intuitive approach, drilling down level by level:

# Count files in each first-level directory under root
for d in /*; do
  [ -d "$d" ] && echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"
done | sort -rn | head -10

In production, the usual inode killers are:

Deleted but Unreleased Files: The Most Hidden Trap

This is one of the most classic pitfalls in disk troubleshooting. You rm a large file, but df shows no space freed. The reason: a process is still holding the file’s file descriptor (Ref: Comprehensive Disk Space Troubleshooting Guide).

How It Works

In Linux filesystems, a file’s data blocks are only truly released after all file descriptors holding it are closed. rm only removes the directory entry mapping the filename to the inode — if a process is still reading or writing the file through its file descriptor, the inode and data blocks won’t be reclaimed.

Think of it like removing a book’s label from the shelf (rm), but the book is still sitting there (data blocks remain) until someone takes it away (file descriptor closes).

Find Deleted but Still-Open Files

# List deleted files still held by processes
lsof +L1 2>/dev/null

# Or use a more precise approach
lsof -nP | grep -i "(deleted)"

Example output:

COMMAND   PID  USER   FD   TYPE  DEVICE  SIZE/OFF  NODE NAME
nginx    1234  root   8w   REG   8,1     10G      12345 /var/log/nginx/access.log (deleted)
java     5678  app   42w   REG   8,1     5G       12346 /opt/app/logs/gc.log (deleted)

The output shows that the nginx process (PID 1234) is still holding a write handle to /var/log/nginx/access.log. The file has been deleted, but 10GB of space hasn’t been released. The java process is similar, holding 5GB of GC logs.

Three Ways to Release Space

Option 1: Restart the Process (Recommended)

# Graceful nginx reload
nginx -s reload

# Restart the java application
systemctl restart app

After restart, the process releases the file descriptor and space is immediately reclaimed. This is the safest approach.

Option 2: Truncate File Content Without Restarting

If the process can’t be restarted (e.g., a production database), you can truncate the file through /proc:

# Truncate via /proc filesystem
cat /dev/null > /proc/1234/fd/8

Where 1234 is the PID and 8 is the file descriptor number (from the FD column in lsof output, strip the letter suffix).

This has the same effect as > /var/log/nginx/access.log, but doesn’t need the filename — it operates directly through the fd. The file content is emptied, space is released, but the file descriptor stays open, so the process won’t error.

Option 3: Use the truncate Command

# Truncate via /proc
truncate -s 0 /proc/1234/fd/8

Same effect as cat /dev/null >, but with clearer syntax.

Warning: Never directly kill -9 a process to release file descriptors. In production, force-killing a process can cause data corruption or service outages. Always prefer graceful restart or the /proc approach.

ext4 Reserved Space: The Hidden 5%

ext4 filesystems reserve 5% of space for the root user by default. This design prevents situations where regular users fill the disk completely and root can’t even log in — because login requires writing to files like /var/log/wtmp. This makes sense for the root partition, but on a pure data disk, it’s wasted space.

Check Reserved Space

# Check reserved block count
tune2fs -l /dev/sda1 | grep "Reserved block count"

# Example output:
# Reserved block count:     2621440
# Block size:               4096
# Reserved space = 2621440 * 4096 / 1024 / 1024 / 1024 ≈ 10GB

Adjust Reserved Space

# Set reserved ratio to 1% (recommended for data disks)
tune2fs -m 1 /dev/sdb1

# Remove reservation entirely (only for temp or test disks)
tune2fs -m 0 /dev/sdb1

# Specify exact reserved block count (precise control)
tune2fs -r 0 /dev/sdb1

Adjusting a 400GB data disk from 5% to 1% frees up 16GB of usable space. For pure log or backup disks, setting it to 0% is fine — no user processes run there, so there’s no risk of root being unable to log in.

xfs filesystems don’t have this reservation mechanism, so no adjustment is needed.

Log Cleanup: The #1 Culprit of Disk Full

In production environments, when disk fills up, the root cause is most likely logs. Nginx access logs, Java GC logs, application business logs — these files grow continuously and will eventually cause problems if left unmanaged.

Manual Cleanup Strategies

# Truncate a log file (preserve the file itself, only clear content)
cat /dev/null > /var/log/nginx/access.log

# Delete log files older than 7 days
find /var/log -name "*.log" -mtime +7 -delete

# Delete log files larger than 100MB
find /var/log -name "*.log" -size +100M -delete

# Compress logs older than 3 days
find /var/log -name "*.log" -mtime +3 -exec gzip {} \;

The difference between cat /dev/null > and rm: the former clears the file content but keeps the file itself, so processes currently writing won’t error; the latter deletes the file, and the process’s next write will either trigger recreation (if configured) or error. For logs of running services, prefer truncation.

logrotate: Production-Grade Log Management

Manual cleanup is a temporary fix. The long-term solution is logrotate — Linux’s built-in log rotation tool that automatically rotates, compresses, and deletes old logs by size or time.

# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily              # Rotate daily
    rotate 14          # Keep 14 days
    compress           # Compress old logs
    delaycompress      # Delay compression by one day
    missingok          # Don't error if file is missing
    notifempty         # Don't rotate empty files
    create 644 nginx adm  # Create new file after rotation
    sharedscripts      # Run scripts only once for multiple matches
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

Key parameter explanations:

The choice between copytruncate and create is critical. create mode creates a new file and requires notifying the process to reopen the log via postrotate (e.g., nginx’s kill -USR1). copytruncate mode copies the current log and then truncates the original file — the process doesn’t need a restart or signal — but the tradeoff is a tiny time window between copy and truncate where log entries might be lost.

For services that can’t receive signals (certain Java applications), use copytruncate:

# /etc/logrotate.d/myapp
/opt/app/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    size 500M
}

Test logrotate Configuration

# Debug mode (doesn't actually execute, just shows what it would do)
logrotate -d /etc/logrotate.d/nginx

# Force rotation
logrotate -f /etc/logrotate.d/nginx

Docker Disk Cleanup

Docker is another major disk space consumer. Image layers, container writable layers, logs, and volumes — each layer can accumulate GBs of data.

Quick Diagnosis

# View Docker overall disk usage
docker system df

# View detailed information
docker system df -v

Example output:

TYPE            TOTAL   ACTIVE  SIZE      RECLAIMABLE
Images          45      12      52.3GB    38.1GB (72%)
Containers      23      8       1.2GB     800MB (66%)
Local Volumes   15      10      8.5GB     2.3GB (27%)
Build Cache     0       0       0B        0B

The RECLAIMABLE column shows reclaimable space. In this example, there are 38GB of images and 800MB of container data that can be cleaned up.

Cleanup Commands

# One-click cleanup: stopped containers + unused networks + dangling images + build cache
docker system prune

# More thorough: include images not referenced by containers
docker system prune -a

# Clean unused volumes (use caution! Confirm no data needs to be preserved)
docker volume prune

# Clean build cache
docker builder prune

# Only clean dangling images (<none> tagged)
docker image prune

Clean Container Logs

Docker container logs are stored by default at /var/lib/docker/containers/<id>/<id>-json.log. Without size limits, a high-traffic container’s logs can consume tens of GB.

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  }
}

This configuration limits each container’s log file to a maximum of 100MB, with 3 files retained. After modification, restart Docker:

systemctl restart docker

Note: The log limits in daemon.json only apply to newly created containers. Existing containers need to be recreated to apply the new configuration. If you urgently need to clean a specific container’s logs, you can truncate the file directly:

cat /dev/null > /var/lib/docker/containers/<container-id>/<container-id>-json.log

Automated Disk Inspection Script

Manual troubleshooting is a fundamental skill, but production environments can’t rely on manual inspection. Here’s a ready-to-use disk inspection script that works with cron for periodic execution and alerts when thresholds are exceeded.

#!/bin/bash
# disk-check.sh - Disk space and inode inspection script
# Usage: ./disk-check.sh [threshold percentage, default 85]

THRESHOLD=${1:-85}
HOSTNAME=$(hostname)
ALERT_LIST=""

# Check disk space usage
while read -r line; do
    USAGE=$(echo "$line" | awk '{print $5}' | tr -d '%')
    MOUNT=$(echo "$line" | awk '{print $6}')
    if [ "$USAGE" -ge "$THRESHOLD" ] 2>/dev/null; then
        ALERT_LIST="${ALERT_LIST}Disk space: ${MOUNT} usage ${USAGE}%\n"
    fi
done < <(df -h | grep -v "^Filesystem" | grep -v "tmpfs")

# Check inode usage
while read -r line; do
    USAGE=$(echo "$line" | awk '{print $5}' | tr -d '%')
    MOUNT=$(echo "$line" | awk '{print $6}')
    if [ "$USAGE" -ge "$THRESHOLD" ] 2>/dev/null; then
        ALERT_LIST="${ALERT_LIST}inode: ${MOUNT} usage ${USAGE}%\n"
    fi
done < <(df -ih | grep -v "^Filesystem" | grep -v "tmpfs")

# Check for large deleted-but-open files
DELETED_FILES=$(lsof +L1 2>/dev/null | awk 'NR>1 && $7+0 > 104857600 {print $1, $2, $7/1024/1024 "MB", $NF}')
if [ -n "$DELETED_FILES" ]; then
    ALERT_LIST="${ALERT_LIST}Deleted unreleased files:\n${DELETED_FILES}\n"
fi

# Output alerts
if [ -n "$ALERT_LIST" ]; then
    echo -e "[Disk Inspection Alert] ${HOSTNAME}\n"
    echo -e "$ALERT_LIST"
    # Plug in your alert channel here (WeChat Work / DingTalk / Email, etc.)
    # send_alert "$ALERT_LIST"
    exit 1
else
    echo "[Disk Inspection] ${HOSTNAME} OK"
    exit 0
fi

Use with cron:

# Daily inspection at 9 AM
0 9 * * * /opt/scripts/disk-check.sh 85 >> /var/log/disk-check.log 2>&1

# Hourly check (for environments with fast disk growth)
0 * * * * /opt/scripts/disk-check.sh 90 >> /var/log/disk-check.log 2>&1

inode Recovery: What to Do When inodes Run Out

inode exhaustion is trickier than disk space full, because you can’t simply delete files — you might not even be able to touch a temporary file.

Emergency Recovery Steps

# Step 1: Find the directory with the most files
for d in /*; do
  [ -d "$d" ] && echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"
done | sort -rn | head -10

# Step 2: Drill into subdirectories, level by level
# Assuming /var has the most files, continue drilling down
for d in /var/*; do
  [ -d "$d" ] && echo "$(find "$d" -xdev 2>/dev/null | wc -l) $d"
done | sort -rn | head -10

# Step 3: Batch delete small files
# First confirm file count and type
find /var/log/app -type f -name "*.tmp" | wc -l

# Delete (use -print to confirm first, then switch to -delete)
find /var/log/app -type f -name "*.tmp" -delete

Efficient Batch Deletion of Large Numbers of Files

When file counts reach millions, rm -rf may fail due to argument list length limits or take extremely long. More efficient approaches:

# Method 1: find -delete (recommended, fastest)
find /path/to/dir -type f -delete

# Method 2: find + xargs (for when additional processing is needed)
find /path/to/dir -type f -print0 | xargs -0 rm -f

# Method 3: rsync to empty a directory (excellent performance)
mkdir /tmp/empty_dir
rsync -a --delete /tmp/empty_dir/ /path/to/dir/

In testing with 5 million files, find -delete is 3-5x faster than rm -rf, and rsync --delete is even faster because it bypasses the unlink syscall and operates at the filesystem level.

Long-Term Solution: Adjusting inode Count

If your workload genuinely requires storing large numbers of small files, you can adjust the inode ratio at filesystem creation time:

# ext4: one inode per 4KB (default is 16KB)
mkfs.ext4 -i 4096 /dev/sdb1

# Or one inode per 2KB (more aggressive)
mkfs.ext4 -i 2048 /dev/sdb1

# Verify the adjustment
tune2fs -l /dev/sdb1 | grep "Inode count"

The tradeoff is that each inode occupies 256 bytes. Doubling the inode count means doubling metadata overhead. For large-file storage scenarios, this isn’t cost-effective — you need to balance based on your actual file size distribution.

Note: ext4’s inode count is fixed at mkfs time and cannot be changed afterward. If an already-formatted partition doesn’t have enough inodes, your only options are reformatting or migrating data. xfs supports dynamic inode allocation and doesn’t have this problem.

Monitoring System: Prometheus + node_exporter

Inspection scripts alone aren’t enough — you need a continuous monitoring system. Prometheus + node_exporter is the de facto standard solution.

Key Monitoring Metrics

Alert Rules

# disk-alerts.yml
groups:
  - name: disk_alerts
    rules:
      # Disk space usage > 85%
      - alert: DiskSpaceHigh
        expr: |
          100 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} 
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} * 100) > 85          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High disk space usage: {{ $labels.mountpoint }}"
          description: "{{ $labels.instance }} {{ $labels.mountpoint }} usage {{ $value }}%"

      # Disk space usage > 95% (critical)
      - alert: DiskSpaceCritical
        expr: |
          100 - (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} 
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} * 100) > 95          
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Disk space critically low: {{ $labels.mountpoint }}"
          description: "{{ $labels.instance }} {{ $labels.mountpoint }} usage {{ $value }}%"

      # inode usage > 85%
      - alert: InodeUsageHigh
        expr: |
          100 - (node_filesystem_files_free{fstype!~"tmpfs|overlay"} 
          / node_filesystem_files{fstype!~"tmpfs|overlay"} * 100) > 85          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High inode usage: {{ $labels.mountpoint }}"
          description: "{{ $labels.instance }} {{ $labels.mountpoint }} inode usage {{ $value }}%"

In practice, inode usage alerts are more easily overlooked than disk space alerts. Many teams only configure disk space alerts without inode alerts, only to discover the monitoring blind spot when an incident occurs. Don’t fall into this trap — monitor both dimensions.

Quick Reference: Common Issues

Here’s a quick reference table of high-frequency production issues for fast troubleshooting:

Production Disk Management Checklist

Based on real-world operations experience, here’s a production disk management checklist. Implement each item:

Monitoring:

• Monitor both blocks and inode usage simultaneously — set thresholds at 85% warning, 95% critical
• Monitor sudden disk write rate spikes (may indicate log explosion or anomalies)
• Set up periodic checks for deleted-but-open files

Log Management:

• Configure logrotate for all logging services — rotate by day or by size
• Configure Docker daemon.json log limits (max-size + max-file)
• Rotate application logs daily, compress and retain for 7-30 days

Cleanup Strategies:

• Run docker system prune periodically to clean unused images
• Configure tmpwatch or systemd-tmpfiles for automatic cleanup of temp directories
• Set expiration for database binlogs (e.g., MySQL expire_logs_days = 7)

Capacity Planning:

• When using ext4 for data disks, adjust inode ratio based on file size distribution
• For large numbers of small files, prefer xfs (dynamic inode allocation)
• Reserve 15%-20% space buffer — don’t wait until it’s full to act

Summary

Disk space management seems simple but has plenty of traps. Here are the key takeaways from production experience:

Be thorough in diagnosis. df -h is only the first step — you must also check df -ih. inode exhaustion presents the same symptoms as disk full, but the troubleshooting approach is completely different. Don’t draw conclusions from just one dimension.

Deleting a file doesn’t mean freeing space. Files held by processes don’t release space even after deletion. Anyone who’s been bitten by this remembers it vividly. lsof +L1 should be a standard command in your disk troubleshooting toolkit.

Prevention beats firefighting. logrotate, Docker log limits, Prometheus monitoring — spending ten minutes configuring these before an incident can prevent a 3 AM page. The biggest risk in production isn’t technical difficulty — it’s “knew what to do but didn’t do it.”

Plan inodes ahead. ext4’s inode count is immutable after creation. Estimate before formatting. For large numbers of small files, xfs is more worry-free. Getting this decision wrong is costly to fix later.

One final tip: don’t panic when disk fills up. Run df -h for space, df -ih for inodes, lsof +L1 for ghost files, du to locate large directories — four steps and the problem is basically identified.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Linux 磁盘空间满了怎么办？超详细排查指南 — Tencent Cloud Developer Community, provided a systematic disk space troubleshooting workflow and analysis of deleted-but-unreleased file behavior
2. Linux 磁盘明明有空间，却报 No space left on device — CSDN, detailed analysis of inode exhaustion diagnosis methods and inode mechanism principles
3. Linux 文件系统深度解析：ext4、XFS、inode、硬链接 vs 软链接 — CSDN, provided in-depth analysis of ext4 block group layout and xfs dynamic inode allocation mechanism
4. 记录一次 inode 耗尽造成的生产事故 — CSDN, a real production incident postmortem demonstrating inode exhaustion diagnosis and logrotate remediation