The previous chapter mainly discussed the relevant work of the CPU, the differences between processes and threads, and the types of CPU contexts. Friends who have not read it can click the link below.
Linux Performance Optimization: Principles of CPU Process Handling (Part 1)
1. CPU Context Switching
When does context switching occur?
By combining the types of context switching and the lifecycle of processes, we can summarize the following situations:
- The CPU should be fully utilized under normal circumstances, so the time the CPU is executed is divided into average time segments allocated to each process. When the time allocated to a certain process is completed, it will be suspended, and the next process in the time segment will be executed. This process will cause a context switch, which is the most common without event intervention.
- We know that when writing programs, we can add a sleep function, which is also a context switch triggered by the process actively ending its execution.
- There is a priority in the process information. When the system assigns a high priority to it, it will be executed first, preempting the CPU from other processes, thus causing a context switch.
- When resources are insufficient, the process cannot execute and is suspended, which is also a type of context switch.
- When hardware events occur, such as interrupts and exception handlers being executed, context switching will also occur.
Understanding these common situations can help us better understand the factors affecting CPU performance.
2. Observing CPU Context Switching
As mentioned in the previous article, context switching has a significant impact on CPU performance. How can we observe context switching information? This can be done using the vmstat tool.
# Execute every 3 seconds
root@node:~# vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 477767936 883712 9970296 0 0 0 2 0 0 0 0 99 0 0
0 0 0 477766688 883716 9970120 0 0 0 12 380 754 0 0 100 0 0
0 0 0 477770016 883716 9970292 0 0 0 0 1956 4706 1 1 98 0 0From the output information, we can observe the following important information:
r：Number of processes in the run and ready queue.b：Number of processes in an uninterruptible state.in：Number of system interrupts.cs：Number of context switches.
The vmstat command provides an overview of the system’s context switching situation, while the pidstat tool can be used to view specific processes.
# Output every 3 seconds
root@node:~# pidstat -w 3
Linux 4.15.0-58-generic (zw2ahyper02n06) 11/26/2025 _x86_64_ (64 CPU)
09:20:52 PM UID PID cswch/s nvcswch/s Command
09:20:55 PM 0 8 0.33 0.00 ksoftirqd/0
09:20:55 PM 0 9 12.83 0.00 rcu_sched
09:20:55 PM 0 12 0.33 0.00 watchdog/0Here, two parameters are very important: cswch and nvcswch.
- cswch indicates the number of voluntary context switches per second, which occur when a process voluntarily gives up execution, typically while waiting for I/O completion, resource allocation, or during sleep calls. In other words, voluntary context switches occur more frequently when system resources are insufficient.
- nvcswch indicates the number of involuntary context switches per second, which, as the name implies, are forced switches that generally occur when a higher-priority process needs to execute or during forced scheduling. This indicates that the current system load is high and the demand for CPU is elevated.
3. Analyzing CPU Performance Issues
With the CPU metrics provided above and the context switching information we can observe, how do we analyze them?
We start with the most direct commands: uptime and top. Generally, when CPU performance issues arise, the direct reflection is often an abnormal increase in load average and %CPU (CPU usage). When these indicators appear, we will further analyze.
-
In the output of the vmstat command, the value of r represents the number of processes in the run or ready queue. When compared to the number of CPU cores, if it exceeds the number of cores, it indicates that there is contention for CPU resources, leading to high CPU pressure. This often results in an increase in the subsequent parameter cs (context switches), causing a significant rise in us (user mode) and sy (kernel mode) usage rates.
-
As mentioned earlier, vmstat only provides an overall situation. We need to know which specific process is causing the anomaly. The output of the pidstat command will show processes with abnormal cswch and nvcswch values. If the differences in process values are not clear, we may also need to observe the thread values of these two parameters, as the unit of CPU processing is threads. Use the command pidstat -wt 3 (with the -t parameter).
root@cs1ahyper01n07:~# pidstat -wt 3
Linux 4.15.0-58-generic (cs1ahyper01n07) 11/27/2025 _x86_64_ (64 CPU)
05:17:45 PM UID TGID TID cswch/s nvcswch/s Command
05:17:48 PM 0 8 - 10.79 0.00 ksoftirqd/0
05:17:48 PM 0 - 8 10.79 0.00 |__ksoftirqd/0
05:17:48 PM 0 9 - 107.62 0.00 rcu_sched
05:17:48 PM 0 - 9 107.62 0.00 |__rcu_sched
05:17:48 PM 109 4695 - 1.27 0.00 ntpd
05:17:48 PM 109 - 4695 1.27 0.00 |__ntpd
05:17:48 PM 0 - 4832 3.81 0.00 |__basereport
05:17:48 PM 0 - 58955 1.90 0.00 |__basereport
05:17:48 PM 0 - 9412 1.59 0.00 |__basereport
05:17:48 PM 0 - 4863 48.25 0.00 |__bkunifylogbeatTID is the thread ID, and TGID is the main thread ID, which is the same as the process ID.
- In addition to these context parameters and usage rate parameters, we should also pay attention to the interrupt parameter in. Next, we will look at how to troubleshoot interrupt exceptions.
4. Common Interrupt Exception Troubleshooting
In the vmstat output, the in parameter represents the number of CPU interrupts. This value generally changes when CPU performance issues occur. This value can be checked in the virtual file system /proc/interrupts.
Typically, use the command watch -d /proc/interrupts to observe changes every 2 seconds.
60: 0 0 0 256 0 0 0 0 PCI-MSI 131075-edge virtio4-request
61: 0 0 0 0 0 0 0 0 PCI-MSI 196608-edge virtio7-config
62: 0 0 227 0 0 0 0 0 PCI-MSI 196609-edge virtio7-req.0
NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts
LOC: 19978406 21499538 19037427 19051650 24423046 16673118 16385116 16393098 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 0 0 0 0 Performance monitoring interrupts
IWI: 0 0 0 0 0 0 0 0 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 6886121 6006679 7293859 5923199 5446282 7366956 7156611 6750252 Rescheduling interrupts
CAL: 492454 478584 493309 495546 405377 394787 400837 410006 Function call interrupts
TLB: 585098 564485 581136 594965 564140 550393 550720 550566 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 5006 5006 5006 5006 5006 5006 5006 5006 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 0 0 0 0 Posted-interrupt wakeup eventBased on the output content, we can analyze potential issues.
- NMI：This value represents non-maskable interrupts. A value of 0 is normal; a non-zero value indicates a hardware fault. You can use the dmesg command to check for hardware fault logs.
- LOC：This value represents CPU core scheduling. It indicates whether the load is evenly distributed across all cores. If there are significant differences between cores, it suggests a CPU fault or timer issue. You can use the top/htop commands to observe this.
- RES：This represents rescheduling interrupts. When the load is too high, CPU core balancing scheduling occurs. If there are abnormal processes, this value will increase sharply, leading to frequent process scheduling (observed with pidstat) and increased context switching.
- TLB：This is the process page table refresh to prevent cache inconsistency across different CPU cores. If this value increases abnormally, you can use ps aux –sort=-%mem to check if there are processes frequently requesting/releasing memory.
Other indicators are mostly related to hardware. If they show non-zero values, you can check the dmesg logs for any hardware faults.
In the future, we will continue to discuss CPU-related performance issues. Feel free to leave comments and follow us!
Linux Performance Tuning Series
– CPU Section
Linux Performance Optimization: Principles of CPU Process Handling (Part 1)
– Memory Section
Linux Performance Tuning: About Memory
Linux Performance Tuning: Why Has Swap Increased?
Linux Performance Tuning: Understanding Cache in Memory
Linux Performance Tuning: How to Quickly Locate Memory Leaks?
Linux Performance Tuning: Detailed Usage of Memory Analysis Tools memleak-bpfcc and valgrind
Linux Performance Tuning: A Comprehensive Guide to Troubleshooting Memory Issues
– Disk Section
Linux Performance Tuning: In-Depth Understanding of File Systems and Disks
Linux Performance Tuning: Detailed Explanation of Disk Workflows and Performance Metrics
Linux Performance Tuning: Further Discussion on Disk Performance Metrics and Process-Level I/O
Linux Performance Tuning: Detailed Explanation of Common Scenarios for FIO Performance Testing
Linux System Tuning: In-Depth Analysis of Increased Disk Latency Issues
Linux Performance Tuning: Using strace to Analyze File System Performance Issues
Linux Performance Tuning: Common Disk I/O Performance Optimization Methods