1. Why Diagnose in 60 Seconds?
Scenario: Woken up by an alarm in the middle of the night, the boss is demanding answers, and users are complaining that the website is down.
Goal: Identify in the shortest time (60 seconds) which hardware resource (CPU/memory/disk/network) is causing the issue, and then immediately **”stem the bleeding”**.
Principles:
- Save lives first, then treat the illness — Ensure the system does not crash first, then gradually find the root cause.
- Use fewer commands, but with strong effects — Only use 10 commands, each with a clear purpose.
2. 10 Commands in 60 Seconds (Memorize to Save Lives)
| Time | Command | Quick Check | If Exceeded, Immediate Action |
|---|---|---|---|
| 0-5 seconds | <span>uptime</span> |
Load Value (System Load) | load>CPU cores×2 → System is too slow, restart the most resource-intensive service first |
| 5-10 seconds | <span>dmesg -T tail</span> |
Check for OOM (Out of Memory) or hardware errors | OOM → Clear cache + restart process |
| 10-15 seconds | <span>vmstat 1 3</span> |
r (Running Processes), b (Blocked Processes) | r>CPU cores×2 or b>10 → System is congested |
| 15-20 seconds | <span>iostat -xzm 1 3</span> |
%util (Disk Utilization), await (Wait Time) | %util>90% and await>100ms → Disk bottleneck |
| 20-25 seconds | <span>free -m</span> |
available (Available Memory) | available<10% → Clear cache or add memory |
| 25-30 seconds | <span>sar -n DEV 1 3 | grep eth0</span> |
Network Card Traffic | Traffic>70% of network card limit → Throttle traffic |
| 30-35 seconds | <span>ss -s</span> |
TIME_WAIT Connection Count | Too many TIME_WAIT → Enable reuse |
| 35-40 seconds | <span>lsof | wc -l</span> |
File Handle Count | Handles>300,000 → Restart service or impose limits |
| 40-45 seconds | <span>ps --sort=-%cpu | head</span> |
Processes Consuming Most CPU | CPU>80% → Kill or limit |
| 45-50 seconds | <span>ps --sort=-%mem | head</span> |
Processes Consuming Most Memory | Memory>30% → Restart or limit |
| 50-55 seconds | <span>df -h</span> |
Disk Usage | Usage>90% → Delete large files |
| 55-60 seconds | <span>journalctl -k -p err -n 10</span> |
Kernel Errors | Hardware errors → Prepare to switch to backup machine |
3. Don’t Understand the Commands? Look Here!
1. <span>uptime</span>
03:12:45 up 45 days, load average: 48.2, 46.8, 32.0
- Load average indicates how busy the system is; the higher the number, the slower it is.
- If load>CPU cores×2, the system is very slow.
2. <span>vmstat 1 3</span>
r b swpd free buff cache si so bi bo in cs us sy id wa st
48 2 0 200M 1.9G 6.3G 0 0 120 240 103k 142k 95 5 0 0 0
- r (running): Number of processes waiting for CPU; >CPU cores×2 indicates slowness.
- b (blocked): Number of processes waiting for disk/IO; >10 indicates congestion.
3. <span>iostat -xzm 1 3</span>
Device rMB/s wMB/s %util await
sda 0.2 450 98.5 210
- %util: How busy the disk is; >90% indicates a disk bottleneck.
- await: Time waiting for disk response; >100ms is slow.
4. <span>free -m</span>
total used free shared buff/cache available
128G 125G 1.2G 0.5G 1.9G 600M
- available: Truly available memory; <10% is dangerous.
4. Three Emergency Measures After 60 Seconds
| Problem | Emergency Command | Description |
|---|---|---|
| CPU Overload | <span>cpulimit -p <PID> -l 50</span> |
Limit process CPU to 50% |
| Disk IO Overload | <span>ionice -c3 -p <PID></span> |
Lower process IO priority |
| Memory + Swap Overload | <span>echo 3 > /proc/sys/vm/drop_caches</span> |
Clear cache, free memory |
5. One-Click 60-Second Inspection Script
#!/bin/bash
# 60-second inspection script, results saved in /tmp/pt60.log
{
echo "=== $(date) ==="
uptime
dmesg -T | tail -n 5
vmstat 1 3 | tail -n 1
iostat -xzm 1 2 | tail -n +4 | awk 'NR%3==0'
free -m
ss -s
sar -n DEV 1 1 | tail -n +3 | awk 'NR%2==0'
ps -eo pid,%cpu,%mem,comm --sort=-%cpu | head -6
ps -eo pid,%cpu,%mem,comm --sort=-%mem | head -6
df -h | awk 'int($5)>90'
} | tee /tmp/pt60.log
Usage:
chmod +x pt60.sh
./pt60.sh
6. Summary (Memorize)
- In 60 seconds, do one thing: Identify the most congested hardware resource and immediately stem the bleeding.
- 10 commands, each useful; memorize them, you can type them even in the dark.
- Save lives first, then treat the illness; find the root cause slowly tomorrow.
Mnemonic:
“If load is high, check CPU first; if CPU is fine, check disk; if memory is full, clear cache; if network is full, throttle; if disk is full, delete large files; if kernel errors, prepare to switch out.”
Wishing you success in resolving alarms in 60 seconds, so you can continue to sleep well!