At three in the morning, my phone alarm shattered the night. The screen glaringly displayed: “Production environment server disk usage at 95%”. My heart skipped a beat as I trembled to connect to the VPN and typed df -h, confirming that terrifying red warning. My instinctive reaction was to delete some seemingly unimportant log files with rm -rf, praying it would hold off the crisis temporarily. We have all been that “firefighter” battling disk space in the dead of night. rm -rf is a blunt instrument that seems to solve the problem but often buries deeper issues: accidental deletion of critical data, service anomalies, and problems resurfacing weeks later. Today, we will abandon this barbaric approach and transform into “disk space detectives”, systematically learning ten more advanced, safer, and fundamental strategies to handle disk alerts, allowing you to respond calmly and precisely next time. First thing: Precise positioning – Who stole my disk space? Before deleting any files, a precise diagnosis must be conducted. The du command is your first surgical tool.
1. Quickly locate large directories:
# Check the size of each subdirectory in the current directory and sort it
du -sh /* | sort -hrThis command starts from the root directory, listing the sizes of all top-level directories, allowing you to quickly identify the “culprit” in which partition.
2. Deep dive to find specific files: After pinpointing a large directory (like /var), continue to drill down:
# Find the largest 10 files or directories in /var
du -ah /var | sort -rh | head -n 10Parameter explanation:
· -a: Show files, not just directories.
· -h: Human-readable format (GB, MB).
· sort -rh: Sort in reverse order by human-readable numbers.
· head -n 10: Show only the top 10 results.
3. A more intuitive alternative tool: ncdu If you dislike command-line sorting, you can install ncdu, which provides an interactive interface allowing you to browse with the keyboard arrows and directly see which directory is the largest.
# Install ncdu (CentOS/RHEL)
yum install ncdu -y
# Install ncdu (Ubuntu/Debian)
apt-get install ncdu -y
# Use ncdu /
At this point, you have identified the “big consumers” of disk space, usually logs, cache files, or user-uploaded content. But don’t rush to delete; let’s look at a more peculiar situation.
Second thing: The mystery of ghost space – Files deleted, space not released. Have you ever encountered a situation where du shows a directory is not large, but df still shows high disk usage? This is often due to files being deleted but still being held by processes. Use the lsof command to catch the “ghosts”:
# Find files that have been deleted but are still held by processes
lsof +L1Alternatively, check directly in the /proc filesystem:
# Find large deleted files
lsof | grep deletedThe command output will show the PID of the process and the size of the deleted file it occupies. The solution is to restart the process; only then will the kernel truly release the space occupied by that file.
# Gracefully restart the process (e.g., nginx)
kill -TERM
# Or, if the process is critical, reload it with the HUP signal without interrupting the service
kill -HUP This situation is extremely common in long-running services (like Java applications) where log files are not configured for rotation.
Third thing: The art of log management – From chaotic accumulation to scientific governance. Logs are the primary killer of disk space. We cannot simply delete; we must learn to manage scientifically.
1. Use logrotate for automatic log rotation. logrotate is a built-in Linux log management tool that can automatically cut, compress, and delete old logs based on time or size. Check your application configuration files in the /etc/logrotate.d/ directory; a standard configuration example is as follows:
# /etc/logrotate.d/your-app/
/var/log/your-app/*.log {
daily # Rotate daily
missingok # Ignore if the log does not exist
rotate 30 # Keep 30 copies
compress # Compress old logs
delaycompress # Delay compression by one cycle
notifempty # Do not rotate empty files
copytruncate # Copy current log and clear it instead of moving, suitable for programs that do not support reloading
}Manually execute a rotation test immediately:
logrotate -vf /etc/logrotate.d/your-app2. Clean up historical log files. For historical log files not managed by logrotate, you can use the find command to clean them up:
# Delete *.log files older than 7 days
find /var/log -name "*.log" -type f -mtime +7 -exec rm -f {} \;
# A safer approach is to confirm the found files before deletion
find /var/log -name "*.log" -type f -mtime +7 -exec echo {} \;
# After confirmation, execute deletion
find /var/log -name "*.log" -type f -mtime +7 -exec rm -f {} \;Fourth thing: Clean up the “legacy” of crashes – Core Dump files. When a program encounters a segmentation fault or other serious crashes, the system may generate a core or core.PID file for subsequent debugging. These files can be huge (several GB is common). Find and clean them up:
# Search for core files throughout the system
find / -name "core" -o -name "core.*" 2>/dev/null
# Delete found core files
find / -name "core" -o -name "core.*" -exec rm -f {} \; 2>/dev/nullPrevention: For production environments, if debugging is not intended, you can disable core dump generation with the command ulimit -c 0.
Fifth thing: Clean up package manager caches. Package managers leave behind a lot of cache files, and cleaning them is usually harmless.
For APT (Ubuntu/Debian):
apt-get autoremove # Remove automatically installed, no longer needed packages
apt-get clean # Clean all downloaded .deb package caches
# Or
apt-get autoclean # Only clean outdated .deb package cachesFor YUM (CentOS/RHEL):
yum autoremove # Remove no longer needed dependencies
yum clean all # Clean all cachesSixth thing: Examine Docker’s disk usage. If your server is running Docker, it may be a hidden “disk devourer”.
1. View overall Docker disk usage:
docker system dfThis command will detail how much space is occupied by images, containers, data volumes, and build caches.
2. Clean up unused resources:
# Delete all dangling images, containers, networks, and build caches (dangerous operation, please confirm first!)
docker system prune -a -f
# A safer approach is to clean up step by step
docker container prune -f # Delete stopped containers
docker image prune -f # Delete dangling images
docker volume prune -f # Delete unused data volumesNote: docker system prune -a will delete all unused images (including those that may be needed later), so use with caution in production environments.
Seventh thing: Handle temporary files and caches. The system’s /tmp directory and various application cache directories are also targets for cleanup.
# Clean /tmp directory (usually automatically cleaned after reboot)
rm -rf /tmp/*
# Clean user caches (like pip cache)
rm -rf ~/.cache/pip/*Eighth thing: Handle mail queues (rare but possible). If the server is configured with local mail services (like Postfix), a backlog of mail queues can also take up significant space.
# View mail queue
mailq
# Clear mail queue (caution! Confirm that the emails are not important before executing)
postsuper -d ALLNinth thing: The ultimate weapon – One-click disk space analysis script. Automate the diagnostic process by writing a script that runs quickly when an alert occurs. Create a script disk_doctor.sh:
#!/bin/bash
echo "=== Disk Space Overview ==="
df -h
echo -e "\n=== Top 10 Largest Directories ==="
du -sh /* 2>/dev/null | sort -hr | head -n 10
echo -e "\n=== Deleted but Unreleased Space Files ==="
lsof +L1 | awk '$5 == "REG" && $9 ~ /deleted/ {print $2, $9, $7}'
echo -e "\n=== Docker Disk Usage ==="
command -v docker &>/dev/null && docker system df || echo "Docker not installed"
echo -e "\n=== Check Large Log Files ==="
find /var/log -type f -size +100M -exec ls -lh {} \; 2>/dev/null
echo -e "\n=== Check Core Files ==="
find / -name "core" -o -name "core.*" 2>/dev/null | head -n 5
Grant execute permissions and run:
chmod +x disk_doctor.sh && ./disk_doctor.shTenth thing: Preventive measures – Establish a monitoring and alert system. The above nine items are “curative”, but the most astute approach is “preventive”.
1. Configure a simple disk monitoring script and add it to crontab:
#!/bin/bash
CURRENT=$(df / | grep / | awk '{ print $5}' | sed 's/%//g')
THRESHOLD=85
if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
echo "Warning: Root partition disk usage at $CURRENT%" | mail -s "Disk Space Alert" [email protected] # Or send to your alert platform (like DingTalk, WeChat Work, Prometheus)
fi2. Use mature monitoring systems:
· Prometheus + Alertmanager: Easily monitor disk usage with node_exporter and set rich alert rules.
· Zabbix/Grafana: Also provides powerful visualization and alerting capabilities.
3. Architectural optimizations:
· Collect logs into a centralized logging system (like ELK/Loki), keeping only the most recent logs locally.
· Store large static files (user uploads) in object storage (like AWS S3, Alibaba Cloud OSS).
· Implement dynamic log level adjustments at the application level to avoid excessive DEBUG logs under normal circumstances.
Conclusion: From “firefighter” to “architect” mindset. In the face of full disks, we have mastered a complete methodology from diagnosis, cleaning to prevention. Let’s review the core process:
1. Diagnosis first: Use du/ncdu to locate problem directories.
2. Investigate ghosts: Use lsof to check unreleased space.
3. Focused cleanup: Logs (logrotate), caches (package manager/Docker), core files.
4. Automation: Use scripts to improve diagnostic efficiency.
5. Monitoring prevention: Establish alert mechanisms to avoid recurrence from an architectural perspective.
From now on, when the disk alert sounds again, you will no longer be just a “firefighter” who knows only rm -rf, but a “system doctor” armed with precision instruments and a systematic battle plan. This is the hallmark of a seasoned Linux professional’s expertise.
Interactive topic: What is the strangest “disk space disappearance case” you have encountered in your career? How did you solve it in the end? Feel free to share your stories in the comments, and let’s exchange and progress together!