As a blogger with years of experience in the Linux field, I have seen too many people stumble over performance issues in Linux systems. Have you ever encountered a scenario where you set up a Linux server with great enthusiasm, only to find that the website responds so slowly that it makes you question your life choices? Or when using a Linux desktop system, opening a file manager takes forever, making the operation extremely frustrating? These issues are actually Linux’s way of “complaining”; it needs your optimization and tuning.
Performance optimization is crucial for Linux systems. It’s like a car; although it performs well when it leaves the factory, if it is not regularly maintained and adjusted, it will gradually lose power and increase fuel consumption over time. The same goes for Linux systems; over long periods of operation, performance can decline due to various factors such as hardware aging, software vulnerabilities, configuration errors, and load changes.
1. Linux Performance Issues
1.1 Key Performance Metrics
Before delving into optimization strategies, we must first clarify the key metrics for measuring Linux system performance, much like how a doctor needs to know whether various physiological indicators are normal before diagnosing a patient.
Throughput: Simply put, this is the amount of work the system processes in a unit of time. For example, how many HTTP requests your server can handle per second, or how much data the disk can read and write per second. The higher the throughput, the stronger the system’s processing capability. Imagine a highway; throughput is like the number of vehicles passing through in a unit of time. The greater the throughput, the busier and more efficient the road becomes. In a web server scenario, low throughput means that during high concurrent access, the server cannot process a large number of requests in time, leading to slow page loading or even timeouts.
Latency: This refers to the time taken from when a request is made to when a response is received. It reflects the system’s response speed; the lower the latency, the better the user experience. Just like when you order something online, you hope the delivery arrives quickly; this waiting time is similar to system latency. In database queries, if latency is too high, it may be due to unreasonable indexing, slow disk I/O, etc., causing query results to be returned slowly.
CPU Utilization: This indicates the proportion of time the CPU spends processing tasks over a period. For example, if the CPU utilization is 80%, it means the CPU is busy working 80% of the time during that period. Excessively high CPU utilization can lead to slower system responses because CPU resources are heavily occupied and cannot process new tasks in time. When you run multiple large programs simultaneously, CPU utilization may spike, causing the computer to lag.
Memory Utilization: This is the ratio of used memory to total memory. Memory acts like a temporary warehouse for the computer; programs need to load data into memory to run. If memory utilization is too high, the system may frequently swap memory (swap), moving temporarily unused data to disk, which significantly reduces system performance because disk read/write speeds are much slower than memory. When you open too many applications and memory is full, the system will start using virtual memory (disk space simulating memory), causing the computer to run slowly.
1.2 Where to Find Performance Issues
Having understood the performance metrics, the next step is to learn how to identify performance issues within the system. Remember these common performance problem scenarios and their manifestations.
(1) CPU Bottleneck
High Load: When the system’s average load (Load Average) consistently exceeds the number of CPU cores, it indicates that the system is overloaded, and the CPU may be overwhelmed. Average load refers to the average number of processes in a runnable and uninterruptible state over a unit of time. For example, if a 4-core CPU has an average load consistently above 4, there is a problem. During large-scale data calculations, such as big data analysis tasks, the CPU may remain in a high-load state for extended periods.
Process Contention: Multiple processes competing for CPU resources can lead to frequent context switching. Context switching refers to the need for the CPU to save and restore process state information when switching from one process to another. Excessive context switching consumes CPU time and reduces system performance. You can use the vmstat command to check the value of cs (context switch count); if this value is high, there may be a process contention issue. When multiple processes simultaneously require CPU computing resources, contention occurs, such as when multiple video transcoding tasks run at the same time.
(2) Memory Bottleneck
Memory Leak: When a program allocates memory but does not release it in a timely manner, it leads to continuous memory occupation, eventually exhausting system memory. Memory leaks are often caused by programming errors and are more common in languages like C/C++. For example, in C, if memory is allocated using malloc but not released with free, it will cause a memory leak. As memory leaks worsen, the system will slow down and may even crash.
Frequent Swapping: When physical memory is insufficient, the system will swap data from memory to the disk’s swap space. Frequent memory swapping can lead to a sharp decline in system performance because disk I/O speeds are much slower than memory. You can use the free command to check swap usage; if swap space is heavily used, it indicates a frequent swapping problem. When you run a large game while also opening many other applications, leading to insufficient physical memory, the system will frequently use swap space.
(3) Disk I/O Bottleneck
Slow Read/Write: If the disk read/write speed is too slow, it causes applications to wait too long for data read/write operations. This may be due to disk aging, file system fragmentation, unreasonable I/O scheduling strategies, etc. For example, if a mechanical hard drive has been used for a long time, the read/write speed will decline as the read/write head ages. When copying large files, if the speed is very slow, there may be a disk I/O problem.
Disk Busy: If the disk is busy for extended periods, it cannot respond to new I/O requests in time. You can use the iostat command to check disk usage; if disk usage is consistently above 80%, it indicates that the disk is quite busy. When many processes simultaneously perform read/write operations on the disk, such as frequent reads/writes in a database, it can lead to a busy disk.
(4) Network Bottleneck
Insufficient Bandwidth: If the network bandwidth cannot meet the data transmission needs, it leads to slow data transfer speeds. For example, if your server has a bandwidth of 10Mbps but needs to transmit a large amount of data during peak business hours, it will result in insufficient bandwidth. During video live streaming, if there are too many viewers and the server’s bandwidth is limited, it will cause the stream to lag.
High Latency: If the time taken for data to be transmitted over the network is too long, it may be due to network congestion, routing issues, or network device failures. You can use the ping command to test network latency; if the latency is too high, such as exceeding 100ms, it may affect normal business operations. When you access foreign websites, due to the long distance and many routing nodes, high latency issues may arise.
2. Hardware Optimization: Equipping Linux with a “Powerful Engine”
Hardware is the foundation of Linux system operation, just like the engine and chassis of a car; the quality of hardware performance directly affects the overall performance of the system. Next, I will detail how to optimize key hardware aspects such as CPU, memory, storage, and network.
2.1 CPU: The Core Driving Force of Performance
The CPU, as the core component of the computer, is like the brain of a person, bearing the responsibility for various computational tasks within the system, and its performance plays a decisive role in the operating speed of Linux systems.
When selecting a CPU, do not blindly pursue high-end configurations; instead, choose based on the type of business. For scenarios that require processing a large number of parallel tasks, such as big data analysis and distributed computing, a multi-core CPU is the best choice. Multi-core CPUs can handle multiple tasks simultaneously, greatly improving parallel processing capability, just like multiple workers can speed up project progress. Multi-core CPUs from Intel’s Xeon series are widely used in data center servers and can efficiently process massive amounts of data.
For scenarios that require extremely high computational speed, such as high-frequency algorithm trading in financial transactions or complex simulations in scientific computing, high-frequency CPUs can perform better. High frequency means the CPU can execute more instructions in a unit of time, just like a sprinter can run faster and complete a sprint in a shorter time. For example, AMD’s Ryzen series excels in single-core performance, providing a smoother experience for applications that require high single-core performance, such as certain games and design software.
CPU cache is key to improving CPU processing speed; it acts like a small high-speed warehouse close to the CPU, storing data and instructions that the CPU may use soon. To optimize program code, make data and instruction access align better with the CPU cache’s working mechanism, thus improving cache hit rates. For example, when accessing an array, accessing array elements in order can fully utilize the cache prefetch mechanism, improving cache hit rates. If you have a two-dimensional array array[N][N], accessing it in the order of array[i][j] (where i is the outer loop and j is the inner loop) aligns with the memory layout, allowing the cache to load subsequent adjacent elements when accessing array[0][0]. Subsequent accesses to elements like array[0][1] can be read directly from the cache, greatly improving access speed. However, if accessed in the order of array[j][i], memory access becomes jumpy, and the cache cannot effectively prefetch data, leading to reduced cache hit rates and slower access speeds.
Using the cpufreq tool can dynamically adjust CPU frequency, setting it reasonably based on system load to achieve a balance between performance and energy consumption. The cpufreq tool provides various frequency adjustment strategies, such as the ondemand strategy, which immediately increases CPU frequency to meet computational demands when system load increases; when the load decreases, it lowers the frequency to save energy. This is like a car automatically adjusting its speed based on road conditions, speeding up on the highway and slowing down in congested urban areas, ensuring both driving efficiency and fuel savings. In Linux systems, you can set the frequency adjustment strategy by modifying the /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor file; for example, to set the strategy to ondemand, you can use the command echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor.
2.2 Memory: The High-Speed Channel for Data
Memory is the temporary storage area for data and program execution, acting like a transfer station, and its performance directly affects the system’s response speed. Increasing physical memory according to system load and business needs is crucial. For example, in a large database server, when it stores massive amounts of data and requires frequent data queries and updates, if memory is insufficient, the database will have to read data from the disk frequently, and the disk I/O speed is far lower than that of memory, leading to extremely slow system response speeds. Sufficient memory can load frequently used data and indexes into memory, reducing disk I/O operations and significantly improving system response speed. For instance, a database server originally with 8GB of memory often experiences slow response times during peak business periods; when the memory is expanded to 16GB, system performance is significantly improved, and query response times are greatly shortened.
Memory compression technologies such as KSM (Kernel Samepage Merging) and zRAM can effectively improve memory utilization. KSM reduces memory usage by merging identical pages in memory. For example, in cloud servers running multiple virtual machines, several virtual machines may load the same operating system kernel and base library files; KSM can identify and merge these identical pages, freeing up more memory space. zRAM compresses data in memory before storing it, further reducing memory usage. It acts like a compressed warehouse, allowing more items to be stored in limited space. For cloud servers with limited memory resources, enabling zRAM can compress data storage when memory is tight, preventing system performance degradation due to insufficient memory.
Regularly cleaning memory pages and organizing memory fragmentation can ensure efficient memory allocation. Memory fragmentation is like clutter in a room, leading to wasted space and affecting memory allocation efficiency. You can use the sysctl command to adjust memory parameters, such as setting the vm.drop_caches parameter to clean caches and free memory. Executing echo 3 > /proc/sys/vm/drop_caches can clean system caches, including page caches, dentries caches, and inode caches, allowing for more effective memory utilization and avoiding performance degradation due to memory fragmentation.
2.3 Storage: The Solid Foundation for Data
Storage devices are the carriers of data, and their performance significantly impacts system stability and data read/write speeds. Replacing system disks and data disks with SSDs can significantly enhance I/O performance. SSDs use flash memory chips as storage media, and compared to traditional mechanical hard drives, they have no moving parts, resulting in faster data read/write speeds. For example, in a video website, a large number of user requests for video resources occur daily; if video files are stored on traditional mechanical hard drives, the read/write speed of the mechanical hard drive will struggle to meet the demand during high concurrent access, leading to slow video loading and stuttering. However, with SSDs, their fast data read/write capabilities can quickly respond to a large number of user requests, ensuring smooth video playback and greatly enhancing user experience.
Common RAID technologies such as RAID 5 and RAID 10 can enhance data reliability and read/write performance in different business scenarios. RAID 5 uses parity to distribute data and parity information across multiple disks, allowing one disk to fail without losing data while also providing a certain performance boost. It is suitable for scenarios that require a certain level of data reliability while also needing some read/write performance, such as general enterprise file servers. RAID 10 combines the mirroring of RAID 1 and the striping of RAID 0, providing 100% data redundancy while also achieving high read/write speeds, making it suitable for scenarios with high data security and performance requirements, such as financial trading systems, ensuring absolute data security while quickly processing large amounts of transaction data.
2.4 Network: The Link Connecting the World
The network is the bridge for Linux systems to communicate with the outside world, and its performance affects data transmission speed and stability. Choosing gigabit or 10-gigabit network cards is crucial for improving network bandwidth and throughput. For cloud service providers, handling a large number of user cloud storage and computing requests involves massive data transmission; if using a 100-megabit network card, network bandwidth can easily become a bottleneck, leading to slow user data uploads and downloads. Upgrading to gigabit or 10-gigabit network cards can significantly increase network bandwidth and throughput, meeting the demands of large data transmissions and improving service quality. Similarly, for servers of large gaming companies, during high concurrency periods, a large number of players interact with game data simultaneously; high-performance network cards can ensure fast transmission of game data, reducing game latency and providing players with a smoother gaming experience.
Configuring network QoS (Quality of Service) policies can allocate different bandwidth priorities for different applications. In enterprise networks, applications such as video conferencing and critical business systems require high network quality, needing stable bandwidth and low latency. By configuring QoS policies, we can set high priorities for these critical applications, ensuring that during network congestion, critical applications still receive sufficient bandwidth to operate normally and stably. For example, setting the traffic priority for video conferencing to the highest ensures that during insufficient network bandwidth, video conferencing can proceed smoothly, avoiding issues like lag and disconnection, while for non-critical applications, such as employee web browsing and file downloads, lower priorities can be set, allowing data transmission when the network is idle.
3. Software Optimization: Injecting “Intelligent Soul” into Linux
Software is the soul of the Linux system; reasonable software configuration and optimization can fully leverage hardware performance, making the system run more efficiently and stably. Next, I will detail the strategies and methods for Linux software optimization from key aspects such as kernel parameter tuning, process and resource management, memory management optimization, and network optimization.
3.1 Kernel Parameter Tuning: Deep Customization of the System
The kernel is the core of the Linux system; it acts like a behind-the-scenes commander, responsible for managing various resources and functions of the system. By adjusting kernel parameters, we can deeply customize the system to better adapt to different application scenarios and business needs.
(1) Key Parameter Interpretation
File system buffer size: Taking vm.dirty_background_ratio and vm.dirty_ratio as examples, vm.dirty_background_ratio indicates the threshold at which the system starts flushing dirty pages (modified but not yet written back to disk data) to the disk, with a default value typically around 10%. When the proportion of dirty pages in the system reaches this threshold, the kernel starts a background thread to gradually write dirty pages back to the disk, just like a warehouse manager arranging workers to move temporary goods (dirty pages) back to the official warehouse (disk) when the proportion reaches a certain level. vm.dirty_ratio is the upper limit for write operations, generally set between 20% – 40%; when the proportion of dirty pages reaches this value, all write operations will be blocked until the dirty pages are written back to the disk, similar to a temporary storage area being full, where new goods (write operations) cannot come in until some goods are moved out (dirty pages written back to disk). These two parameters significantly impact system I/O performance; if set improperly, they may lead to frequent disk I/O, affecting system performance.
Kernel shared memory: kernel.shmmax and kernel.shmall control the maximum size of shared memory segments allowed by the system and the total number of shared memory pages, respectively. Tuning these two parameters is particularly important when handling databases and large-scale data processing. For example, in a large database server, a significant amount of shared memory is needed to store database indexes and data caches to improve query and write speeds. If kernel.shmmax is set too low, it may prevent the database from allocating sufficient shared memory, thus affecting performance. kernel.shmmax is typically set to 50% – 75% of physical memory, and the specific value needs to be adjusted based on application requirements and server memory size.
(2) Parameter Adjustment Practice
For applications such as log servers and file storage servers that frequently read and write files, we can adjust kernel parameters in the /sys and /proc directories based on system load and application characteristics to achieve performance improvements. Suppose we have a log server that generates a large number of log files daily; to improve log writing efficiency, we can appropriately increase the values of vm.dirty_background_ratio and vm.dirty_ratio, for example, setting vm.dirty_background_ratio to 15 and vm.dirty_ratio to 30. We can add or modify the following two lines in the /etc/sysctl.conf file:
vm.dirty_background_ratio = 15
vm.dirty_ratio = 30Then execute sudo sysctl -p to make the changes effective. This way, when processing log writes, the system will start flushing dirty pages to the disk when the dirty page ratio reaches 15% and will allow the dirty page ratio to reach 30% before blocking write operations, thus reducing the frequency of disk I/O and improving log writing efficiency.
(3) Disabling Unused Modules
In Linux systems, some kernel modules are loaded by default, but some may not be needed in actual use. For example, when a server acts as a web server, Bluetooth and infrared-related modules are considered unnecessary. These unused modules not only consume system resources but may also affect system startup speed and stability. We can use the lsmod command to view all currently loaded kernel modules and then use the rmmod command to unload unnecessary modules. For example, to unload the Bluetooth module, we can execute sudo rmmod bluetooth. To prevent these modules from being loaded automatically during the next system startup, we can create a custom configuration file in the /etc/modprobe.d/ directory, such as disable_unused_modules.conf, and add blacklist bluetooth and blacklist infrared to it, so that the system will not load Bluetooth and infrared modules at startup, thus reducing kernel size, speeding up startup, and lowering memory usage.
3.2 Process and Resource Management: Reasonable Allocation of System Resources
Processes are running instances of programs in the Linux system; they are like busy workers consuming various system resources. Properly managing processes and allocating resources can ensure efficient system operation, avoiding resource waste and conflicts.
(1) Priority Adjustment
In Linux systems, each process has a priority that determines the order in which processes receive CPU resources. We can use the nice and renice commands to adjust process priorities. The nice command is used to set the priority when starting a process, while the renice command is used to change the priority of a running process. Process priority is typically represented by a nice value, which ranges from -20 to 19, where -20 indicates the highest priority and 19 indicates the lowest priority; by default, a process’s nice value is 0.
For example, if we have a video transcoding task that requires significant CPU resources and takes a long time, running it at the default priority may affect the operation of other critical processes, such as the web server’s response speed. In this case, we can lower its priority when starting the video transcoding task. Assuming the command for the video transcoding task is ffmpeg -i input.mp4 output.mp4, we can use the following command to start it with a nice value set to 10, lowering its priority:
nice -n 10 ffmpeg -i input.mp4 output.mp4If the video transcoding task is already running, we can use the renice command to adjust its priority. First, we need to find the process ID of the video transcoding task using the ps -ef | grep ffmpeg command; assuming the process ID is 1234, we can then use the following command to set its nice value to 15, further lowering its priority:
sudo renice -n 15 1234It is important to note that ordinary users can only increase the nice value (i.e., lower the priority), while only superusers (root) can decrease the nice value (i.e., raise the priority).
(2) Resource Limitation
Cgroups (control groups) are a resource management mechanism provided by the Linux kernel that allows for fine-grained management of CPU, memory, disk I/O, and other resources, acting like a resource allocator that can allocate reasonable resources to different process groups based on our needs, preventing any one process group from over-consuming resources and causing system lag.
For example, to limit the memory usage of a certain process group, we can use cgroups. First, create a cgroups directory; suppose the process group we want to limit is video_process, we can create a subdirectory named video_process in the /sys/fs/cgroup/memory/ directory:
sudo mkdir /sys/fs/cgroup/memory/video_processThen, set memory limit parameters in this directory. For example, to limit the memory usage of this process group to 1GB, we can edit the memory.limit_in_bytes file and write 1073741824 (1GB = 1024 * 1024 * 1024 = 1073741824 bytes):
sudo sh -c 'echo 1073741824 > /sys/fs/cgroup/memory/video_process/memory.limit_in_bytes'Next, add the processes that need memory limitation to this cgroups group. Assuming the video transcoding process ID is 1234, we can write it to the tasks file:
sudo sh -c 'echo 1234 > /sys/fs/cgroup/memory/video_process/tasks'This way, the memory usage of the video transcoding process is limited to 1GB. If this process tries to use more than 1GB of memory, the system will handle it according to the cgroups configuration, such as issuing a warning or terminating the process, thus ensuring system stability and the normal operation of other processes.
3.3 Memory Management Optimization: Efficient Use of Memory Resources
Memory is a very important resource in Linux systems; it acts like a temporary warehouse where the data and code needed for program execution are stored. Properly managing memory can improve system efficiency and avoid performance degradation due to memory leaks and overuse.
Swappiness Parameter Adjustment:
Swappiness is an important parameter of the Linux kernel that controls the system’s tendency to use swap space under memory pressure, with a range of 0 to 100, expressed as a percentage (%). When the system’s physical memory is insufficient, it will transfer temporarily unused data from memory to the disk’s swap space; this process is called memory swapping. A higher swappiness value (e.g., 60 – 100) means the kernel will use swap space earlier and more aggressively; a lower value (e.g., 0 – 30) means the kernel will try to avoid using swap space and use physical memory as much as possible; a value of 0 means that swap space will not be used at all unless memory is exhausted (OOM, Out of Memory).
For different application scenarios, we need to set the swappiness value reasonably. In desktop environments, since users typically run multiple applications simultaneously and require high system response speeds, to avoid system lag due to frequent memory swapping, the swappiness value can be set lower, such as 10. In high-performance computing scenarios, where memory read/write speeds are critical and disk I/O speeds are relatively slow, frequent memory swapping can severely impact computational efficiency, so the swappiness value should also be set low, even to 0. For database servers, since databases have their own optimized memory management mechanisms and require high data read/write performance, it is generally recommended to set the swappiness value to 1 – 10.
We can use the following command to check the current swappiness value:
cat /proc/sys/vm/swappinessOr use the sysctl command:
sysctl vm.swappinessIf we want to temporarily change the swappiness value, for example, to set it to 10, we can use the following command:
sudo sysctl vm.swappiness=10To permanently change the swappiness value, we need to edit the /etc/sysctl.conf file and add or modify the following line:
vm.swappiness = 10Then execute sudo sysctl -p to make the changes effective.
Memory Monitoring and Analysis:
To promptly detect memory issues, we need to use some memory monitoring and analysis tools. top and htop are tools that display real-time system resource usage, showing CPU, memory, disk I/O, and other resource usage, including used memory, free memory, shared memory, buffered memory, and swap space usage. In the output of the top command, the Mem line shows the usage of physical memory, while the Swap line shows the usage of swap space; in the output of the htop command, similar memory usage information is displayed in a more intuitive way, such as using different colored progress bars to represent the usage ratios of memory and swap.
The free command is used to check the system’s memory usage, displaying total memory, used memory, free memory, shared memory, buffered memory, and the size of swap space. For example, executing free -h will output memory usage information in a human-readable format, with the -h parameter indicating that memory sizes will be displayed in an easily readable way, such as 1.5G, 200M, etc.
The vmstat command reports statistics about processes, memory, paging, block I/O, traps, and CPU activity, allowing us to understand memory paging situations, such as the number of pages swapped in (si) and swapped out (so) per second. If the values of si and so are high, it indicates that the system is frequently swapping memory, which may indicate insufficient memory.
The pmap command is used to view the memory mapping of processes, showing the memory address range occupied by each process, memory types (such as code segment, data segment, heap, stack, etc.), and corresponding file mappings. By using pmap, we can analyze whether a process’s memory usage is reasonable and whether there are signs of memory leaks.
In addition to using these tools to monitor memory metrics, we can also combine log analysis tools (such as dmesg or journalctl) to quickly locate the root causes of memory leaks or excessive usage issues. The dmesg command is used to view messages in the kernel ring buffer, which contains various information from the system startup process and important events during kernel operation; when memory issues arise, relevant error messages or warnings may be found in the output of dmesg. journalctl is a tool for querying and displaying systemd logs, allowing us to view system logs, service logs, etc.; by analyzing these logs, we can understand the system’s operation regarding memory and find clues about memory issues.
Enabling and Configuring Transparent Huge Pages:
Transparent Huge Pages (THP) is a memory management mechanism in the Linux kernel that allows the operating system to use larger memory pages, thereby reducing the number of page table entries, lowering TLB (Translation Lookaside Buffer) miss rates, and improving memory access efficiency. In traditional memory management, the memory page size is typically 4KB, while with transparent huge pages, the page size can reach 2MB or even larger. This is particularly useful for databases and large applications, as they often require large contiguous memory spaces; using large pages can reduce memory fragmentation and improve memory utilization.
We can use the following command to check the current status of THP:
cat /sys/kernel/mm/transparent_hugepage/enabledIf the output is [always] madvise never, it indicates that THP is enabled and always uses large pages; if the output is madvise [never], it indicates that THP is disabled.
In different application scenarios, we need to configure THP reasonably. In database servers, since databases have high memory access performance requirements, it is generally recommended to enable THP. This can be done by editing the /etc/default/grub file, adding transparent_hugepage=always to the GRUB_CMDLINE_LINUX parameter, and then executing sudo update-grub to update the GRUB configuration. After rebooting the system, THP will be enabled permanently.
In virtualized environments, THP configuration needs to be more cautious. While enabling THP can improve memory access efficiency, excessive enabling of THP may lead to performance issues in some cases. For example, in situations where multiple virtual machines share physical memory, if each virtual machine enables THP, it may lead to increased memory contention, thereby reducing system performance. Therefore, in virtualized environments, testing and adjustments should be made based on actual conditions to determine whether to enable THP and how to configure it.
3.4 Network Optimization: Ensuring High-Speed Data Transmission
The network is the bridge for Linux systems to communicate with the outside world, acting like a high-speed information highway; the quality of network performance directly affects data transmission speed and stability. Next, I will introduce how to enhance Linux system network performance from aspects such as application optimization, socket optimization, transport layer optimization, and network layer optimization.
Application Optimization:
At the application level, we can adopt some optimization measures to enhance network performance. I/O multiplexing technology is an efficient I/O processing method; epoll is a multiplexing mechanism provided by the Linux kernel that offers higher performance and scalability compared to traditional select and poll. Select and poll require traversing all file descriptors to check for events, while epoll uses an event-driven approach, where the kernel actively notifies the application when an event occurs, significantly reducing system overhead. In a high-concurrency web server, using epoll can handle a large number of client connections simultaneously, improving the server’s concurrent processing capability.
Asynchronous I/O (AIO) is also a technology for enhancing I/O performance, allowing applications to continue executing other tasks without waiting for I/O operations to complete, thus improving system concurrency. AIO is suitable for application scenarios that require high I/O response times, such as real-time data processing and multimedia applications.
In addition to these technologies, we can also implement other optimization measures to enhance network performance. Using long connections instead of short connections can reduce the overhead of establishing and tearing down connections, improving data transmission efficiency. In an online gaming server, if each player uses short connections with the server, establishing and tearing down connections for each data send and receive consumes a lot of system resources and network bandwidth. However, by using long connections, players can maintain a continuous connection with the server, allowing data to be transmitted at any time, greatly enhancing the game’s fluidity.
Caching infrequently changing data can reduce the number of network transmissions and improve response speed. For example, in a news website, for some news content that does not update frequently, it can be cached locally; when users request it again, it can be read directly from the local cache instead of fetching it from the server again, thus reducing server load and improving user access speed.
Compressing network I/O data can reduce the size of data transmissions and improve transmission speed. In web servers, enabling Gzip compression can compress webpage content before transmitting it to clients, significantly reducing data transmission volume, especially for text-heavy webpages, where the compression effect is more pronounced. Clients automatically decompress and display the compressed data upon receipt.
Reducing DNS resolution delays can speed up network requests. DNS resolution is the process of converting domain names into IP addresses; if DNS resolution delays are too high, it can slow down network requests. We can configure a local DNS caching server, such as dnsmasq, to reduce the number of DNS resolutions and improve resolution speed. Dnsmasq caches the recently resolved domain names and their corresponding IP addresses; when the same domain name is requested again, it retrieves the IP address directly from the cache instead of querying external DNS servers.
Socket Optimization:
Sockets are endpoints for network communication; adjusting socket buffer sizes can enhance network performance. In Linux systems, several key socket buffer size parameters require our attention, including net.core.optmem_max, net.core.rmem_max, and net.core.sndbuf.
4. Practical Exercises: Bringing Theory to Life
Having learned the theoretical knowledge of Linux performance optimization, we will now conduct practical exercises to apply these theories to solve actual performance issues.
4.1 Simulating Performance Problem Scenarios
Before officially performing performance optimization, we need to simulate common performance problem scenarios in a test environment so that we can conduct targeted optimization operations.
(1) High Concurrency Access Simulation: Use the Apache JMeter tool to simulate high concurrency access scenarios for web servers. JMeter is a powerful open-source performance testing tool that can simulate multiple users sending HTTP requests to the server simultaneously to test the server’s performance under high concurrency.
Specific operations are as follows:
- Download and install JMeter from the official JMeter website (https://jmeter.apache.org/download_jmeter.cgi); extract the latest version of the JMeter package for use.
- Open JMeter, create a new test plan. In the test plan, add a thread group, set the number of threads to 100 (simulating 100 concurrent users), and set the loop count to 1000 (each user sends 1000 requests).
- Add an HTTP request under the thread group, setting the request URL to the address and port of the web server you want to test, for example, http://192.168.1.100:8080/index.html.
- Run the test plan and observe JMeter’s result tree and aggregate report to check the server’s response time, throughput, error rate, and other performance metrics.
(2) Memory Leak Simulation: In C language, we can write a simple program to simulate a memory leak. Here is a sample code:
#include
#include
int main() {
while (1) {
char *ptr = (char *)malloc(1024 * 1024); // Allocate 1MB of memory each time
if (ptr == NULL) {
perror("malloc failed");
return 1;
}
// Memory is not released here, leading to a memory leak
}
return 0;
}Compile and run this program; over time, you will notice that the system’s memory usage continuously rises, which is a manifestation of a memory leak.
(3) Disk I/O Busy Simulation: Use the dd command to simulate a busy disk I/O scenario. The dd command can be used to copy files and specify read/write block sizes and quantities; by continuously performing read/write operations on large files, the disk can be kept busy.
For example, the following commands can create a 1GB file in the /tmp directory and continuously read and write to it:
# Create a 1GB file
dd if=/dev/zero of=/tmp/bigfile bs=1M count=1024
# Continuously read the file
while true; do dd if=/tmp/bigfile of=/dev/null bs=1M; done
# Continuously write to the file
while true; do dd if=/dev/urandom of=/tmp/bigfile bs=1M; doneAfter running these commands, use the iostat command to check the disk’s I/O status; you will find that the disk usage is very high and in a busy state.
4.2 Performance Analysis and Localization
After simulating performance problem scenarios, the next step is to use various performance analysis tools to analyze the system and identify performance bottlenecks.
Top Tool: top is one of the most commonly used performance monitoring tools in Linux systems; it can display the resource usage of various processes in real-time, including CPU usage, memory usage, and swap space usage.
When simulating high concurrency access scenarios, running the top command will show that CPU usage may spike, and certain processes may have high CPU usage, which may indicate where the performance bottleneck lies. For example, if the CPU usage of the web server process (such as httpd or nginx) consistently exceeds 80%, it indicates that this process may have performance issues that need further analysis.
Htop Tool: htop is an enhanced version of top, providing a more user-friendly interface and more features, supporting mouse operations, and allowing for a more intuitive view of system resource usage.
When simulating memory leak scenarios, using the htop command will show that memory usage continuously rises, and the memory occupied by processes also keeps increasing. Through the htop interface, you can easily find the processes that occupy the most memory and determine whether they are causing memory leaks.
Vmstat Tool: The vmstat command is used to report statistics about virtual memory, processes, CPU activity, etc.; it can display overall performance metrics of the system.
When simulating busy disk I/O scenarios, running vmstat 1 (reporting system status every second) allows you to observe the bi (data read from block devices) and bo (data written to block devices) metrics; if these two values remain high, it indicates that disk I/O is busy, which may indicate a disk performance bottleneck. Additionally, you can observe the wa (I/O wait time percentage) metric; if the wa value is high, it also indicates that the system has I/O performance issues.
Sar Tool: sar (System Activity Reporter) is a system activity reporting tool that can collect, report, and save system activity information, including CPU usage, memory usage, disk I/O, etc.
For example, using sar -u 1 10 can collect CPU usage every second, for a total of 10 collections. By analyzing the output of sar, you can understand CPU usage at different times and identify high CPU usage periods and their causes. Using sar -b 1 can provide real-time views of disk I/O read/write situations, including the number of read and write blocks per second, helping us locate disk I/O performance issues.
4.3 Optimization Implementation and Effect Verification
After identifying performance bottlenecks through performance analysis, we can implement optimization operations according to the strategies introduced earlier and verify the optimization effects.
(1) Optimization Implementation
For high CPU usage: If a certain process occupies too much CPU resource, you can use the nice or renice command to adjust the priority of that process, reducing its CPU usage. For example, if the PID of a certain process is 1234, using renice -n 10 1234 can set its nice value to 10, lowering its priority. If the system load is too high, consider increasing the number of CPU cores or optimizing program code to reduce unnecessary computational operations.
For memory leaks: For the previously simulated C language memory leak program, you need to add memory release statements in the code. The modified code is as follows:
#include
#include
int main() {
while (1) {
char *ptr = (char *)malloc(1024 * 1024); // Allocate 1MB of memory each time
if (ptr == NULL) {
perror("malloc failed");
return 1;
}
// Release memory
free(ptr);
ptr = NULL;
}
return 0;
}Recompile and run the modified program; memory usage will no longer continuously rise, resolving the memory leak issue.
For low disk I/O performance: If the performance issue is due to disk aging, consider replacing it with an SSD. If it is due to file system fragmentation, use tools like e4defrag to defragment the file system. In the simulated busy disk I/O scenario, we can adjust the dd command parameters, such as reducing the block size (bs), to lower the pressure on disk I/O. Additionally, optimizing application I/O operations, such as adopting asynchronous I/O techniques, can improve I/O efficiency.
(2) Effect Verification
After completing the optimization operations, run performance tests again to observe changes in system performance metrics.
- Response time reduction: In the simulated high concurrency access scenario, the average response time of the web server may have been 500ms before optimization, but after optimization, it may be reduced to 100ms, indicating a significant improvement in system response speed.Throughput increase: The server’s throughput may have been 1000 requests/second before optimization, but after optimization, it may increase to 5000 requests/second, indicating enhanced system request handling capability.
- Resource utilization reduction: In the simulated memory leak and busy disk I/O scenarios, memory usage and disk usage have significantly decreased after optimization, leading to more reasonable utilization of system resources.
By comparing performance metrics before and after optimization, we can intuitively see the effectiveness of the optimization strategies, proving that our optimization operations were successful. In practical applications, it may be necessary to repeatedly perform performance analysis and optimization until system performance meets the expected goals.