来自：马哥Linux运维
Introduction: Starting with a Painful Downtime Incident
It was a Friday afternoon, just as I was about to leave work happily, when the monitoring system suddenly went into a frenzy of alarms. The response time of our core online business system skyrocketed from the usual 200ms to 8 seconds, and user complaint calls flooded in. Our team urgently investigated and found that the system resource usage seemed “normal”—CPU usage at 65%, 30% memory free, and disk I/O not at its peak.
This situation of “appearing normal but actually crashing” is something many operations colleagues have encountered. The root of the problem lies in: we often judge system health by a single metric, ignoring that Linux system performance is a complex symphony.
Today, I want to share the pitfalls I’ve encountered over the years and the methodologies I’ve summarized—how to systematically locate and resolve Linux performance issues.
Background: Why is Performance Tuning So Important?
The Real Cost of Performance Issues
In today’s internet era, system performance directly affects user experience and business revenue. According to research data from Google:
- • For every additional second of page load time, the conversion rate drops by 7%
- • 53% of mobile users abandon pages that take more than 3 seconds to load
- • A serious performance failure can lead to millions in business losses
Typical Performance Bottleneck Scenarios
Throughout my operations career, I’ve encountered the most performance issues concentrated in the following scenarios:
- 1. Traffic Surges During E-commerce Promotions: Events like Double 11 and 618 can see instantaneous traffic spikes of 10-20 times the usual
- 2. Database Slow Queries Triggering Avalanches: An unoptimized SQL query can bring down the entire system
- 3. Memory Leaks as a Chronic Poison: Full GC and memory overflow issues in Java applications
- 4. I/O Bottlenecks as Invisible Killers: Performance drops sharply during log writing and data backups
Core Methodology: Three-Step Localization Method
After years of practical experience, I’ve summarized a “three-step localization method” that can quickly pinpoint over 90% of performance issues.
Step 1: Global Scan (10-Second Quick Diagnosis)
Just like a doctor checks temperature and blood pressure first, we need to quickly understand the overall state of the system.
# My Golden Three Commands
uptime # Check load trends
dmesg | tail # Check system logs
vmstat 1 # Check overall resource usage
Practical Tip: I like to write these three commands as an alias:
alias health='uptime; echo "---"; dmesg | tail -5; echo "---"; vmstat 1 5'
By looking at the three values of load average, you can quickly determine:
- • If the 1-minute load > 5-minute load > 15-minute load: the problem is worsening
- • If the 15-minute load > 5-minute load > 1-minute load: the problem is alleviating
Step 2: Layered Deep Dive (Precise Bottleneck Localization)
◆ CPU Bottleneck Localization
CPU issues are like traffic jams; we need to distinguish whether it’s “too many cars” or “too narrow a road”.
# Three Pronged CPU Analysis
top -H # View CPU usage at the thread level
mpstat -P ALL 1 # View usage of each CPU core
pidstat -u 1 # View detailed CPU usage by process
Real Case: On one occasion, we found an 8-core server with a total CPU usage of only 12.5%, but the system was responding very slowly. Using <span>mpstat -P ALL</span>, we discovered that one core was at 100% usage while the other 7 cores were almost idle. It turned out to be a bottleneck from a single-threaded program!
Solution:
# Use taskset to bind CPU cores and fully utilize multi-core
taskset -c 0-3 ./your-application # Bind to cores 0-3
# Or modify process affinity
echo "2" > /proc/irq/24/smp_affinity # Bind interrupt handling to a specific CPU
◆ Memory Bottleneck Localization
Memory issues are like a room cluttered with junk; we need to distinguish whether it’s “really full” or “poorly organized”.
# Memory Analysis Combo
free -h # Check memory overview
cat /proc/meminfo # Detailed memory information
slabtop # Kernel object cache
Pitfalls: Many people panic when they see low free memory from the <span>free</span> command, but actually, Linux uses free memory for caching, which is a good thing!
The correct judgment method:
# Truly available memory = free + buffers + cached
sar -r 1 # View memory usage trends
# If swap occurs frequently, then it's truly insufficient memory
sar -W 1 # View swap activity
Optimization Tips:
# Adjust swappiness (recommended server setting below 10)
echo 10 > /proc/sys/vm/swappiness
# Clear cache (use with caution, generally not needed)
sync && echo 3 > /proc/sys/vm/drop_caches
# Huge page memory optimization (suitable for databases and similar scenarios)
echo 2048 > /proc/sys/vm/nr_hugepages
◆ I/O Bottleneck Localization
I/O issues are like toll booths on a highway, easily causing congestion.
# I/O Analysis Tools
iostat -x 1 # Disk I/O statistics
iotop # Real-time I/O monitoring
blktrace # I/O tracing tool
Key Metric Interpretation:
- • %util: Disk utilization; sustained 100% indicates disk saturation
- • await: Average wait time; over 10ms requires attention
- • r_await/w_await: Read/write latency, helps determine whether the issue is read or write related
Real Case: On a MySQL server, iostat showed disk utilization at only 50%, but await was as high as 200ms. A deeper analysis revealed that a large number of random small I/O operations were the cause, and the solution was to adjust innodb_flush_method and increase SSD cache.
Step 3: Comprehensive Tuning (Systematic Resolution)
Performance tuning is not about treating symptoms but requires systematic thinking.
◆ Kernel Parameter Optimization Checklist
# Network Optimization (suitable for high concurrency scenarios)
cat >> /etc/sysctl.conf << EOF
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 32768
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10000 65535
EOF
# Filesystem Optimization
echo "* soft nofile 655350" >> /etc/security/limits.conf
echo "* hard nofile 655350" >> /etc/security/limits.conf
Experience Sharing: My Tuning Toolbox
1. Establishing Performance Baselines
Never wait until problems arise to start collecting data! My approach is:
# Use sar to establish a 24/7 performance baseline
/usr/lib64/sa/sa1 1 1 # Collect once per minute
/usr/lib64/sa/sa2 -A # Generate daily report
2. Automated Alert Scripts
#!/bin/bash
# Simple performance alert script
LOAD=$(uptime | awk -F'load average:''{print $2}' | awk '{print $1}' | cut -d, -f1)
THRESHOLD=5
if (( $(echo "$LOAD > $THRESHOLD" | bc -l) )); then
echo "Warning: System load too high Load: $LOAD" | mail -s "Performance Alert" [email protected]
# Automatically collect diagnostic information
top -bn1 > /tmp/high_load_$(date +%Y%m%d_%H%M%S).txt
iostat -x 1 10 >> /tmp/high_load_$(date +%Y%m%d_%H%M%S).txt
fi
3. Load Testing and Validation
After tuning, it’s essential to validate the results:
# CPU Load Testing
stress --cpu 8 --timeout 60s
# Memory Load Testing
stress --vm 2 --vm-bytes 1G --timeout 60s
# I/O Load Testing
fio --name=randwrite --ioengine=libaio --iodepth=64 --rw=randwrite --bs=4k --direct=1 --size=1G --numjobs=8
Trends and Extensions: The Future of Performance Tuning
1. eBPF: A Revolution in Performance Analysis
eBPF is changing the game for performance analysis, allowing code to run safely in kernel mode for zero-overhead performance monitoring.
# Use bpftrace to monitor system call latency
bpftrace -e 'tracepoint:syscalls:sys_enter_* { @start[tid] = nsecs; }
tracepoint:syscalls:sys_exit_* /@start[tid]/ {
@latency = hist((nsecs - @start[tid]) / 1000);
delete(@start[tid]);
}'
2. Intelligent Operations
Combining machine learning for performance prediction and automated tuning is becoming a reality:
- • Performance prediction based on historical data
- • Automated parameter tuning
- • Anomaly detection and root cause analysis
3. New Challenges in Cloud-Native Environments
Containers and Kubernetes environments bring new dimensions to performance tuning:
- • cgroup resource limitations
- • Container network performance
- • Pod scheduling optimization
Conclusion: Continuous Learning, Never Stop
Performance tuning is an art of practice; there is no silver bullet, only the experience accumulated over time. The methodologies and tools shared in this article are the result of countless sleepless nights.
Remember: Performance optimization is not a one-time task but a continuous process. Establish monitoring, set baselines, continuously optimize, and validate results to form a closed loop.
If you found this article helpful, feel free to share it with more operations colleagues. Any questions or your tuning experiences are also welcome for discussion in the comments section.
Follow me, and in the next issue, we will discuss “Practical Performance Tuning for Kubernetes Clusters,” including how to optimize etcd performance, scheduler optimization, network plugin selection, and other hardcore content.
-End-
Reading this means you enjoy the articles from this public account. Feel free to pin (star) this public account Linux Technology Enthusiast, so you can receive notifications immediately!
In this public account Linux Technology Enthusiast, reply: Linux, to receive 2TB of learning materials!
Recommended Reading
1. Linux Learning Route Planning Map (with commonly used command quick reference)
2. Essential "Network Port Collection" for operations, this one is enough
3. Linux Learning Guide (Collection Edition)
4. A 20,000-word systematic summary to help you achieve freedom with Linux commands