Click the blue text to follow us // A journey of a thousand miles begins with a single step
【Problem Description】
After logging into the Linux system, the cloud host disk shows available space with the command df -h, but it reports a read-only error: No space left on device.
【Problem Analysis】
This is a typical symptom of running out of inodes; the file system cannot create new files.
【Solution】
1. Emergency Diagnosis: Confirm inode exhaustion
# Check inode usage details (key column is IUse%)
df -i
# Check only the root partition
df -i /2. Localization: Identify which directory is consuming a large number of inodes
# Method A: Quick localization (recommended)
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n | tail -20
# Method B: If you know which partition (e.g., /var), scan only that partition
sudo find /var -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n | tail -20
# Method C: Directly count the number of inodes in each top-level subdirectory
for dir in /*; do [ -d "$dir" ] && echo "$dir: $(sudo find "$dir" -xdev -type f | wc -l)"; done | sort -n -k2Output example: The directory with the largest number is the problem area
15420 /var/spool/clientmqueue
8920 /var/log3. Cleanup: Perform corresponding cleanup based on the localization results
ScenarioA: Too many log files (most common)
# Clean systemd logs (keep the last 100MB)
sudo journalctl --vacuum-size=100M
# Delete old compressed logs
sudo rm -f /var/log/*.gz /var/log/*.[0-9] /var/log/*.old
# Manually truncate current logs (use with caution)
sudo truncate -s 0 /var/log/syslogScenarioB: Too many temporary files
sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*ScenarioC: Mail queue accumulation
# Sendmail queue
sudo rm -f /var/spool/clientmqueue/*
# Postfix queue
sudo postsuper -d ALLScenarioD: Package manager cache
# CentOS/RHEL
sudo yum clean all
# or
sudo dnf clean all
# Ubuntu/Debian
sudo apt cleanScenarioE: Docker/container images
docker system prune -a -fdocker volume prune -fScenarioF: User session files
sudo rm -rf /var/lib/php/sessions/*4. Verification: Check the effect after cleanup
# Check inode usage again
df -i
# Confirm the number of files in the directory has decreased
sudo find /problem_directory -xdev -type f | wc -l5. Ultimate Solution: Rebuild the file system (if cleanup is ineffective)
If you cannot free up enough inodes, you can only back up and rebuild:
# 1. Backup data (to a new hard drive or remote)
sudo rsync -av /original_mount_point/ /backup_location/
# 2. Unmount the partition
sudo umount /dev/sdXN
# 3. Reformat (increase inode density)
# Method 1: Specify the number of inodes (e.g., 5 million)
sudo mkfs.ext4 -N 5000000 /dev/sdXN
# Method 2: Specify bytes/inode ratio (the smaller, the more inodes)
sudo mkfs.ext4 -i 4096 /dev/sdXN # 1 inode per 4KB, default is 16KB
# 4. Remount
sudo mount /dev/sdXN /mount_point
# 5. Restore data
sudo rsync -av /backup_location/ /mount_point/6. Preventive Measures: Avoid Recurrence
# Create a monitoring script (alert when inode>80%)
cat > /usr/local/bin/check_inode.sh << 'EOF'
#!/bin/bash
USE=$(df -i / | awk 'NR==2 {print $5}' | sed 's/%//')
[ $USE -gt 80 ] && echo "ALERT: Root inode usage at ${USE}% on $(hostname)" | mail -s "Inode Alert" [email protected]
EOF
# Execute every 30 minutes
chmod +x /usr/local/bin/check_inode.sh
echo "*/30 * * * * root /usr/local/bin/check_inode.sh" | sudo tee /etc/cron.d/inode_checkThe core principle: Quick localization → Precise cleanup → Monitoring prevention. The problem 99% is caused by a large number of small files (logs, caches).
Final Thoughts
Preparation is the key to success. If you plan ahead, you won’t be caught off guard; if you set things in order beforehand, you won’t be troubled; if you act with intention, you won’t feel regret; if you follow the right path, you won’t be lost. A healthy body and the tranquility and joy that come with it, along with an active, clear, and profound understanding of things, and a gentle, moderate desire that leads to a clear conscience—these benefits cannot be replaced by wealth or status. (Arthur Schopenhauer)
