At 3 AM, the alert group went off. The boss asked, “Why is the server down?” The developer said, “The code hasn’t changed.” The operations team said, “The resources look fine.”
As a technical person, the most taboo thing when encountering performance issues is <span>random guessing</span>.
🚑 01. Comprehensive Diagnosis: System Load (<span>top</span>)
First, take a look at the “vital signs.” Enter <span>top</span>, focusing on the first five lines.
💻 Real Terminal Echo
top - 14:28:14 up 100 days, 3:30, 2 users, load average: 5.15, 4.05, 2.50
Tasks: 201 total, 2 running, 199 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.2 us, 5.1 sy, 0.0 ni, 45.5 id, 33.9 wa, 0.0 hi, 0.3 si
MiB Mem : 15886.0 total, 2510.5 free, 8213.2 used, 5162.3 buff/cache
MiB Swap: 2048.0 total, 2048.0 free, 0.0 used. 7125.0 avail Mem
🧐 Core Metric Interpretation
- **Load Average (Average Load)**：
- Where to look: The first line
<span>5.15, 4.05, 2.50</span>(representing 1, 5, and 15 minutes respectively). - Diagnosis: Assuming this is a 4-core server, the current load is 5.15, indicating that processes are queuing, and the system is overloaded.
- **us (user)**: 15.2% —— User processes consumption, not high.
- wa (wait): 33.9% —— Alert! This value is high, indicating that the CPU is spending a lot of time waiting for disk I/O.
- Conclusion: It is likely not that the CPU cannot compute, but that the slow disk is dragging down the CPU.
🧠 02. In-Depth Analysis: CPU Troubleshooting (<span>vmstat</span>)
If <span>top</span> does not reveal the issue, or if you want to know deeper bottlenecks, use <span>vmstat</span>.
💻 Real Terminal Echo
Enter <span>vmstat 1</span> (refresh every second):
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 0 251014 120560 516230 0 0 0 40 1205 3452 85 10 5 0 0
7 0 0 250980 120560 516230 0 0 0 0 1340 5102 90 9 1 0 0
🧐 Core Metric Interpretation
- **r (running)**：
- Value: 6, 7.
- Diagnosis: The number of processes running or waiting for CPU. If you only have 4 cores, but this shows >4 for a long time, it indicates that the CPU power is severely insufficient.
- **b (blocked)**：
- Value: 0.
- Diagnosis: Processes in uninterruptible sleep (usually waiting for I/O). If there is a value here, it usually accompanies disk stuttering.
- **us (user)**：
- Value: 85%, 90%.
- Diagnosis: User space usage is extremely high, indicating that applications (like Java, Python) are doing heavy calculations.
💾 03. Avoid Misjudgment: Memory Troubleshooting (<span>free</span>)
Newcomers are most easily frightened by “memory usage rate”.
💻 Real Terminal Echo
Enter <span>free -h</span>:
total used free shared buff/cache available
Mem: 15Gi 4.2Gi 2.5Gi 1.0Gi 8.8Gi 10Gi
Swap: 2.0Gi 0B 2.0Gi
🧐 Core Metric Interpretation
- **used (used)**: 4.2Gi.
- **buff/cache (cache)**: 8.8Gi —— Note! This is Linux caching files in memory to speed up reads, not a memory leak.
- available (available):10Gi —— Only look at this!
- Diagnosis: As long as
<span>available</span>is still large, it indicates that the memory is very healthy. Do not restart services just because<span>used</span>plus<span>buff/cache</span>is close to full.
💿 04. Capture the Evidence: Disk I/O (<span>iostat</span>)
Did you see a high <span>wa</span> in <span>top</span>? Let’s confirm it.
💻 Real Terminal Echo
Enter <span>iostat -xz 1</span>:
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm %util await svctm
vda 0.00 450.00 0.00 12500.00 0.00 0.00 0.00 0.00 99.5 12.5 2.10
scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
🧐 Core Metric Interpretation
- %util：99.5.
- Diagnosis: The disk is already running at full load (close to 100%). At this point, any new read/write requests will queue, causing system stuttering.
- await：12.5 (milliseconds).
- Diagnosis: This is the average wait time for I/O requests. If it’s an SSD, this value should be within 1-2ms; if it’s a mechanical disk, over 10ms also indicates it’s slow.
Who is writing to the disk? At this point, you can use the
<span>iotop -o</span>command, which lists which process (PID) is doing heavy read/write operations, similar to<span>top</span>.
📡 05. Modern Networking: Ports and Connections (<span>ss</span>)
Check port usage, don’t use the slow <span>netstat</span> anymore, modern Linux recommends using <span>ss</span>.
💻 Real Terminal Echo
Enter <span>ss -lntp</span> (to view listening ports):
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1234,fd=3))
LISTEN 0 100 *:8080 *:* users:(("java",pid=2561,fd=10))
LISTEN 0 50 *:3306 *:* users:(("mysqld",pid=1890,fd=15))
🧐 Core Metric Interpretation
- Recv-Q / Send-Q：
- Diagnosis: In
<span>LISTEN</span>state, if<span>Recv-Q</span>(current full connection queue length) exceeds<span>Send-Q</span>(maximum queue length), it indicates that the connection queue is full, and new requests will be rejected. This is usually due to high concurrency, and the service cannot handle it. - Process：
- You can directly see whether it’s
<span>java</span>or<span>mysqld</span>occupying the port, along with their PID.
📝 Summary: First Aid Cheat Sheet
When encountering issues, don’t panic, follow this order to diagnose:
- top 👉 Is the load high? Is the CPU busy (
<span>us</span>high) or is the disk busy (<span>wa</span>high)? - vmstat 1 👉 Is the running queue
<span>r</span>blocked? - free -h 👉 Don’t care about
<span>used</span>, just check if<span>available</span>is enough. - iostat -xz 1 👉 Is
<span>%util</span>spiking to 100%?
💡 Pitfall Guide：
- Do not randomly execute
<span>echo 3 > /proc/sys/vm/drop_caches</span>to clear caches, as this will cause the system to trigger an I/O storm to rebuild the cache, exacerbating the stuttering.
The content of this article has been validated in practice, and it is recommended to forward and save it for future reference!