This article from the technical community, “Linux Server Performance Parameter Indicators“, explains the reference for Linux server performance parameter indicators. When troubleshooting performance issues, you can make some preliminary judgments based on the data obtained from these commands, which is worth learning and understanding.
These are just some simple tools to view system-related parameters. Many tools also work by analyzing and processing data under /proc and /sys. More detailed and professional performance monitoring and tuning may require more specialized tools (such as perf, systemtap, etc.) and techniques to complete. After all, system performance monitoring itself is a vast subject.
1. CPU and Memory
1.1 top
➜ ~ top
The three values after the first line represent the system’s average load over the previous 1, 5, and 15 minutes. It can also indicate whether the system load is rising, stable, or declining. When this value exceeds the number of CPU executable units, it indicates that the CPU’s performance has become saturated and is a bottleneck.
The second line records the system’s task status information. “running” is self-explanatory, including tasks currently running on the CPU and those scheduled to run; “sleeping” typically refers to tasks waiting for an event (such as an IO operation) to complete, which can be further divided into interruptible and uninterruptible types; “stopped” refers to tasks that have been paused, usually by sending SIGSTOP or by operating a foreground task with Ctrl-Z; “zombie” tasks, although the resources of terminated processes are automatically reclaimed, the task descriptor of the exited task needs to be accessed by the parent process to be released. This kind of process shows as a defunct state, and whether due to the parent process exiting early or not calling wait, such processes should be given special attention to whether the program is designed incorrectly.
The third line indicates CPU usage rates based on type:
√ (us) user: CPU time occupied in user mode with low nice value (high priority) (nice<=0). Under normal circumstances, as long as the server is not too idle, most of the CPU time should be spent executing this type of program.
√ (sy) system: CPU time occupied in kernel mode, where the operating system transitions from user mode to kernel mode through system calls to execute specific services; usually, this value is relatively small, but when the server is performing IO-intensive operations, this value can be larger.
√ (ni) nice: CPU time occupied in user mode with high nice value (low priority) (nice>0). By default, newly started processes have nice=0, which will not be counted here unless the program’s nice value is modified manually through renice or setpriority().
√ (id) idle: CPU time occupied while idle (executing kernel idle handler).
√ (wa) iowait: time spent waiting for IO to complete.
√ (hi) irq: time consumed by the system handling hardware interrupts.
√ (si) softirq: time consumed by the system handling soft interrupts. Remember that soft interrupts are divided into softirqs, tasklets (which are actually a special case of the former), and work queues. It is unclear which of these times are being counted, as the execution of work queues is no longer in the interrupt context.
√ (st) steal: meaningful only in the case of virtual machines, as the CPU in a virtual machine also shares the physical CPU. This time indicates the time the virtual machine waits for the hypervisor to schedule the CPU, which also means that during this time the hypervisor schedules the CPU to execute on other CPUs, and the CPU resources during this period are “stolen”. This value is not zero on my KVM VPS machine, but it is only at the 0.1 magnitude level. Can this be used to determine VPS overselling?
A high CPU usage rate often indicates something, and this also points to relevant troubleshooting ideas when the server’s CPU usage is too high:
√ When the user usage rate is too high, it usually means that certain individual processes are consuming a large amount of CPU, which can easily be found using top. If the program is suspected to be abnormal, you can use perf and other methods to identify hotspot calling functions for further investigation.
√ When the system usage rate is too high, if IO operations (including terminal IO) are quite frequent, this may cause a high CPU usage rate for this part. For example, on file servers and database servers, otherwise (for example, >20%) there may be issues with some parts of the kernel or driver modules.
√ When the nice usage rate is too high, it is usually intentional behavior. When the initiator of the process knows that certain processes consume a higher CPU, they will set their nice value to ensure that they do not overwhelm other processes’ requests for CPU usage.
√ When the iowait usage rate is too high, it usually means that certain programs’ IO operation efficiency is very low, or the performance of the IO corresponding device is so low that read and write operations take a long time to complete.
√ When the irq/softirq usage rate is too high, it is likely that some peripherals are experiencing issues, leading to a large number of irq requests. At this point, you should check the /proc/interrupts file to investigate the issue.
√ When the steal usage rate is too high, the virtual machine provider may have oversold!
The fourth and fifth lines provide information about physical memory and virtual memory (swap space):
total = free + used + buff/cache. Now buffers and cached Mem information are combined, but the relationship between buffers and cached Mem is not clearly stated in many places. In fact, by comparing the data, these two values correspond to the Buffers and Cached fields in /proc/meminfo: Buffers refer to the block cache for raw disks, mainly caching file system metadata (such as superblock information, etc.) in raw block form, and this value is generally small (around 20M); Cached refers to read caching for specific files to increase file access efficiency and can be said to be used for file caching in the file system.
The avail Mem is a new parameter value used to indicate how much memory space can be allocated to newly started programs without swapping, which is roughly equivalent to free + buff/cached. This also confirms the previous statement that free + buffers + cached Mem is the truly available physical memory. Moreover, using swap space is not necessarily a bad thing; thus, the swap usage rate is not a serious parameter. However, frequent swap in/out is not good, and this situation needs attention, usually indicating a shortage of physical memory.
Finally, there is a resource usage list for each program, where the CPU usage rate is the sum of the usage rates of all CPU cores. Typically, when executing top, the program itself will read /proc extensively, so this top program will usually rank high as well.
Although top is very powerful, it is generally used for real-time monitoring of system information on the console and is not suitable for long-term (days, months) monitoring of system load information. It also misses short-lived processes and cannot provide statistical information.
1.2 vmstat
vmstat is another commonly used system detection tool aside from top. The screenshot below shows the system load while compiling boost with -j4.
r indicates the number of runnable processes, and the data roughly matches; b indicates the number of uninterruptible sleeping processes; swpd indicates the amount of virtual memory used, which has the same meaning as top-Swap-used; as the manual states, usually the number of buffers should be much smaller than the cached Mem, with buffers generally around 20M; the io domain’s bi and bo indicate the number of blocks received and sent to the disk per second (blocks/s); the system domain’s in indicates the number of system interrupts per second (including clock interrupts), and cs indicates the number of context switches caused by process switching.
Speaking of this, many people used to struggle with whether the -j parameter when compiling the Linux kernel refers to CPU Core or CPU Core + 1? By modifying the -j parameter value while compiling boost and the Linux kernel and monitoring with vmstat, I found that the context switch did not change significantly in both cases, and only when the -j value was significantly increased did the context switch increase significantly. So, it seems unnecessary to be overly concerned about this parameter, although I have not tested the specific compilation duration. It is said that if not in system startup or benchmark mode, a context switch >100000 indicates that the program definitely has issues.
1.3 pidstat
If you want to comprehensively track a certain process, nothing is more suitable than pidstat—stack space, page faults, active and passive context switches, and other information are all at a glance. The most useful parameter of this command is -t, which can list detailed information about each thread in the process.
-r: Displays page fault errors and memory usage status. A page fault error occurs when a program needs to access a page mapped in virtual memory that has not yet been loaded into physical memory. There are two main types of page fault errors:
√ minflt/s refers to minor faults, which occur when the required physical page is already in physical memory for some reason (such as shared pages, caching mechanisms, etc.) but is not referenced in the current process’s page table. The MMU only needs to set the corresponding entry, which is relatively inexpensive.
√ majflt/s refers to major faults, where the MMU needs to allocate a free physical page from the currently available physical memory (if no free pages are available, it needs to swap some physical pages to the swap space to free up a page) and load data from external sources into that physical page, then set the corresponding entry. This cost is quite high, with several data levels of difference from the former.
-s: Stack usage status, including StkSize for the stack space reserved for threads and StkRef for the actual stack space used. Using ulimit -s, I found that the default stack space on CentOS 6.x is 10240K, while on CentOS 7.x and Ubuntu series, the default stack space size is 8196K.
-u: CPU usage status, with parameters similar to those above.
-w: Number of thread context switches, further divided into cswch/s due to waiting for resources and nvcswch/s due to CPU time causing passive switches.
It can be cumbersome to first obtain the program’s PID with ps and then operate pidstat, so this killer feature -C allows you to specify a string, and if the Command contains this string, the program’s information will be printed and counted. -l can show the complete program name and parameters.
➜ ~ pidstat -w -t -C “ailaw” -l
It seems that when checking a single task, especially a multi-threaded one, pidstat is more effective than the commonly used ps!
1.4 Others
When you need to monitor a single CPU condition, in addition to htop, you can also use mpstat to check whether the workload of each core on an SMP processor is balanced and whether there are any hotspot threads occupying a core.
➜ ~ mpstat -P ALL 1
If you want to directly monitor the resources occupied by a certain process, you can either use top -u taozj to filter out unrelated processes or use the following method to select, where the ps command can customize the items to be printed:
while :; do ps -eo user,pid,ni,pri,pcpu,psr,comm | grep ‘ailawd’; sleep 1; done
If you want to clarify the inheritance relationship, a commonly used parameter can be used to display the process tree structure, which is much more detailed and visually appealing than pstree.
➜ ~ ps axjf
2. Disk IO
iotop can intuitively display the real-time disk read rates of various processes and threads; lsof can not only show the open information of ordinary files (users) but also the open information of device files like /dev/sda1. For instance, when a partition cannot be unmounted, you can use lsof to find out the usage status of the disk partition, and adding the +fg parameter can also display the file open flag mark.
2.1 iostat
➜ ~ iostat -xz 1
Whether using iostat -xz 1 or sar -d 1, the important parameters for disks are:
√ avgqu-s: The average length of the wait queue for I/O requests sent to the device. For a single disk, if the value >1 indicates that the device is saturated, excluding the logical disk situation of multiple disk arrays.
√ await (r_await, w_await): The average wait time (ms) for each device I/O request operation, including the time spent in the queue and the time being served.
√ svctm: The average service time (ms) for I/O requests sent to the device. If svctm is close to await, it indicates that there is almost no I/O waiting, and the disk performance is good; otherwise, the disk queue waiting time is long, and the disk response is poor.
√ %util: The utilization rate of the device, indicating the proportion of time used for I/O work per second. For a single disk, when %util >60%, performance will decline (reflected in an increase in await); when close to 100%, the device is saturated, but this excludes the logical disk situation of multiple disk arrays.
Moreover, although the monitored disk performance is poor, it does not necessarily affect the application’s response. The kernel usually uses asynchronous I/O technology and read/write caching technology to improve performance, but this is constrained by the limitations of physical memory mentioned above.
These parameters are also applicable to network file systems.
3. Network
The importance of network performance for servers is self-evident. The tool iptraf can intuitively display the send and receive speed information of network cards, and you can also obtain similar throughput information conveniently through sar -n DEV 1. Network cards are typically equipped with maximum rate information, such as 100M or 1G network cards, making it easy to check the utilization rate of devices.
Usually, the transmission rate of network cards is not the most concerning aspect in network development; rather, it is the packet loss rate, retransmission rate, and network latency for specific UDP and TCP connections.
3.1 netstat
➜ ~ netstat -s
This displays the overall data information of various protocols since the system was started. Although the parameter information is relatively rich and useful, it is cumulative, and unless you run it twice to take the difference, you cannot determine the current network status of the system, or you can use watch to intuitively observe the trend of its value changes. Therefore, netstat is usually used to detect port and connection information:
netstat –all(a) –numeric(n) –tcp(t) –udp(u) –timers(o) –listening(l) –program(p)
–timers can cancel reverse domain name queries to speed up display; commonly used commands include:
➜ ~ netstat -antp # List all TCP connections ➜ ~ netstat -nltp # List all local TCP listening sockets, do not add the -a parameter
3.2 sar
The sar tool is incredibly powerful, managing everything from CPU, disks, to page swapping. Here, -n is mainly used to analyze network activity. While it further breaks down data into various layers and protocols such as NFS, IP, ICMP, SOCK, we only care about TCP and UDP. The following command shows the regular segment and datagram send/receive situation, as well as:
TCP
➜ ~ sudo sar -n TCP,ETCP 1
√ active/s: TCP connections initiated locally, such as through connect(), with the TCP state transitioning from CLOSED -> SYN-SENT.
√ passive/s: TCP connections initiated remotely, such as through accept(), with the TCP state transitioning from LISTEN -> SYN-RCVD.
√ retrans/s (tcpRetransSegs): The number of TCP retransmissions per second, usually occurring in cases of poor network quality or server overload, where packet loss occurs based on TCP’s acknowledgment retransmission mechanism.
√ isegerr/s (tcpInErrs): The number of erroneous packets received per second (e.g., checksum failures).
UDP
➜ ~ sudo sar -n UDP 1
√ noport/s (udpNoPorts): The number of datagrams received per second that have no application program at the specified destination port.
√ idgmerr/s (udpInErrors): The number of datagrams received by the local host that cannot be dispatched for reasons other than the above.
Of course, these data can indicate network reliability to some extent, but they only hold significance when combined with specific business requirements.
3.3 tcpdump
tcpdump is undoubtedly a great tool. Everyone knows that when debugging locally, they like to use Wireshark, but what to do when issues occur on the online server?
The appendix of the reference materials provides an idea: restore the environment, use tcpdump to capture packets, and when the issue reproduces (such as when logs show or a certain state appears), you can stop the capture. Tcpdump itself has -C/-W parameters to limit the size of the stored packet capture files. When this limit is reached, the saved packet data rotates automatically, so the overall number of captured packets remains controllable. After that, you can take the data packets offline and analyze them with Wireshark however you want, isn’t that great? Although tcpdump does not have a GUI interface, its packet capture functionality is not weak at all; you can specify network cards, hosts, ports, protocols, and various filtering parameters. The captured packets are complete and timestamped, making online program data packet analysis simple.
The following is a small test; it can be seen that Chrome initiated three connections to the Web server when starting up. Since the dst port parameter was limited here, the server’s response packets were filtered out, but the process of establishing connections (SYNC, ACK) is still very clear! When using tcpdump, it is essential to configure the capture filtering conditions as much as possible, both for ease of subsequent analysis and because tcpdump can impact the performance of the network card and system, which in turn can affect the performance of online services.
If you find this article helpful, please don’t hesitate to click the “Like” and “See” buttons at the end of the article, or share it directly in your circle of friends.
