Linux Kernel Parameter Tuning

In Linux system administration, kernel parameter tuning is a key technique for optimizing performance, enhancing security, and ensuring stability. Kernel parameters in Linux can be adjusted through the /proc/sys and sysctl interfaces, significantly affecting system behavior. According to a report by Red Hat, proper kernel tuning can improve system performance by 20%-50%.

1. Basic Knowledge of Kernel Parameter Tuning

1.1 What are Kernel Parameters?

Kernel parameters are runtime configuration variables of the Linux kernel that control kernel behavior, such as the network stack, memory management, CPU scheduling, etc. These parameters are stored in the /proc/sys directory and can be adjusted using the sysctl command or by editing files. Kernel parameters are divided into static (compile-time) and dynamic (runtime), with tuning primarily focused on dynamic parameters.

Key Features:

• Dynamic Adjustment: No need to reboot the kernel.
• Diverse Categories: Networking, memory, file systems, security, etc.
• Persistence: Saved through /etc/sysctl.conf.
• Scope: Global or specific to processes.

Kernel parameter tuning is central to Linux performance optimization.

1.2 Importance of Tuning

Kernel parameter tuning directly affects system performance:

• Performance Improvement: Optimizing TCP parameters increases network throughput.
• Security: Adjusting security parameters prevents attacks.
• Stability: Configuring memory parameters avoids OOM (Out of Memory) issues.
• Compliance: Meets enterprise requirements.
• Cost Optimization: Reduces resource waste.

For example, Uber reduced latency by 30% through kernel tuning.

1.3 Typical Tuning Scenarios

• Web Servers: Optimize network parameters to support high concurrency.
• Databases: Adjust memory parameters to improve query speed.
• Cloud Environments: Optimize virtualization parameters.
• Security Servers: Strengthen security parameters.
• Embedded Systems: Tune power consumption parameters.

1.4 Challenges of Tuning

• Complexity: Numerous parameters that affect each other.
• Risk: Improper tuning can lead to crashes.
• Compatibility: Differences between kernel versions.
• Monitoring: Requires real-time observation of effects.
• Persistence: Configurations need to be saved.

1.5 Goals of Tuning

• Efficiency: Maximize resource utilization.
• Security: Enhance protection.
• Stability: Reduce failures.
• Maintainability: Easy to audit.
• Automation: Scripted tuning.

2. Principles of Linux Kernel Parameters

2.1 Storage Mechanism of Kernel Parameters

Parameters are stored in the /proc/sys virtual file system:

• Read: cat /proc/sys/net/ipv4/ip_forward
• Write: echo 1 > /proc/sys/net/ipv4/ip_forward

The sysctl command simplifies operations:

• sysctl net.ipv4.ip_forward=1

Persistence: /etc/sysctl.conf or /etc/sysctl.d/.

2.2 Parameter Classification

1. Network Parameters: net.ipv4.*
2. Memory Parameters: vm.*
3. CPU Parameters: kernel.sched.*
4. File System: fs.*
5. Security Parameters: kernel.kptr_restrict

2.3 Adjustment Principles

Adjusting parameters changes kernel behavior, such as enabling ip_forward for forwarding.

Risk: Incorrect values can cause issues; use sysctl -a to view all.

2.4 Sources of Parameters

• Default values: Kernel code.
• Boot parameters: GRUB cmdline.
• Runtime: sysctl.

2.5 Summary of Principles

Kernel parameters are runtime configurations that can be dynamically adjusted through /proc/sys.

3. Common Kernel Parameter Tuning

3.1 Network Parameters

• net.ipv4.ip_forward=1: Enable IP forwarding.
• net.ipv4.tcp_max_syn_backlog=4096: SYN queue size.
• net.core.somaxconn=65535: Connection queue size.
• net.ipv4.tcp_congestion_control=bbr: BBR congestion control.

Example:

sudo sysctl net.ipv4.tcp_congestion_control=bbr

3.2 Memory Parameters

• vm.swappiness=10: Reduce Swap usage.
• vm.dirty_ratio=20: Dirty page ratio.
• vm.overcommit_memory=1: Allow over-allocation.

Example:

sudo sysctl vm.swappiness=10

3.3 CPU Parameters

• kernel.sched_min_granularity_ns=10000000: Minimum time slice.
• kernel.sched_latency_ns=24000000: Scheduling period.

Example:

sudo sysctl kernel.sched_min_granularity_ns=10000000

3.4 File System Parameters

• fs.file-max=2097152: Maximum file descriptors.
• fs.nr_open=2097152: Open file limit.

Example:

sudo sysctl fs.file-max=2097152

3.5 Security Parameters

• kernel.kptr_restrict=1: Restrict kernel pointer leakage.
• kernel.dmesg_restrict=1: Restrict dmesg.

Example:

sudo sysctl kernel.kptr_restrict=1

4. Tools for Kernel Parameter Tuning

4.1 sysctl

Usage:

sysctl -a | grep net.ipv4
sysctl -w net.ipv4.ip_forward=1
sysctl -p /etc/sysctl.conf

4.2 /proc/sys

Usage:

echo 1 > /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward

4.3 tuned

Installation:

sudo dnf install tuned

Usage:

sudo tuned-adm profile virtual-host
sudo tuned-adm active

Configuration File:

sudo nano /etc/tuned/virtual-host/tuned.conf

4.4 ktune

Purpose: Red Hat tool for optimizing parameters.

4.5 sysctl.conf

Persistence:

sudo nano /etc/sysctl.d/99-custom.conf

Add:

net.ipv4.ip_forward=1

5. Practical Steps for Kernel Parameter Tuning

5.1 Network Tuning Practice

1. Enable Forwarding:

   sudo sysctl -w net.ipv4.ip_forward=1
   

2. TCP Optimization:

   sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
   sudo sysctl -w net.core.somaxconn=65535
   sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
   

3. Persistence:

   sudo nano /etc/sysctl.conf
   

   Add the above parameters.

   sudo sysctl -p
   

5.2 Memory Tuning Practice

1. Reduce Swap:

   sudo sysctl -w vm.swappiness=10
   

2. Optimize Dirty Pages:

   sudo sysctl -w vm.dirty_ratio=20
   sudo sysctl -w vm.dirty_background_ratio=10
   

5.3 CPU Tuning Practice

1. Adjust Time Slices:

   sudo sysctl -w kernel.sched_min_granularity_ns=10000000
   sudo sysctl -w kernel.sched_latency_ns=24000000
   

2. Real-time Priority:

   chrt -f 99 ./realtime_app
   

5.4 File System Tuning Practice

1. Increase File Descriptors:

   sudo sysctl -w fs.file-max=2097152
   sudo sysctl -w fs.nr_open=2097152
   

2. Inode Limits:

   sudo sysctl -w fs.inode-nr=100000
   

5.5 Security Tuning Practice

1. Restrict Kernel Pointers:

   sudo sysctl -w kernel.kptr_restrict=1
   

2. Disable dmesg:

   sudo sysctl -w kernel.dmesg_restrict=1
   

6. Case Studies

6.1 Case 1: High Concurrency Network Tuning

Scenario: Web server SYN queue full.

Diagnosis:

ss -s | grep SYN

Tuning:

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sudo sysctl -w net.core.somaxconn=65535

Result: Connection handling capacity increased by 2 times.

6.2 Case 2: Memory Overload Tuning

Scenario: High Swap usage in the database.

Diagnosis:

free -h

Tuning:

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.dirty_ratio=20

Result: Swap usage reduced to 0%.

6.3 Case 3: CPU Scheduling Tuning

Scenario: High latency in real-time applications.

Diagnosis:

chrt -p <pid>

Tuning:

chrt -f 99 ./app

Result: Latency reduced by 50%.

7. Best Practices for Kernel Parameter Tuning

7.1 Backup Configuration

sudo cp /etc/sysctl.conf /backup/sysctl_$(date +%F).conf

7.2 Test Changes

• Temporary Adjustments:

  sudo sysctl -w param=value
  

• Monitor Effects:

  sar -n DEV 1 5
  

7.3 Categorized Management

• Use /etc/sysctl.d/ for file separation:

  sudo nano /etc/sysctl.d/99-network.conf
  

  Add network parameters.

7.4 Monitoring and Alerts

• Use Prometheus to monitor sysctl parameters.

7.5 Avoid Common Mistakes

• Do not blindly copy parameters.
• Test in low-load environments.

8. Conclusion

Linux kernel parameter tuning is a powerful tool for system optimization. Through sysctl and other tools, performance can be significantly enhanced.