Click the blue “Most Coders” to follow me!

Add a “star” to learn technology together every day at 18:03.

Linux Network Performance Tuning: Practical Optimization of TCP and IP Protocol Stack Parameters

🚀 Introduction: The Necessity of Network Performance Optimization

In today’s high-concurrency and high-traffic internet environment, network performance often becomes a bottleneck for systems. As a senior operations engineer, I have encountered countless performance issues in production environments due to improper TCP/IP parameter configurations. Today, I will share a complete Linux network performance tuning plan to help everyone thoroughly resolve network performance bottlenecks.

📊 Common Manifestations of Network Performance Issues

Real Cases from Production Environments

• • High Concurrent Connection Scenarios: During e-commerce promotions, the number of server connections surges, resulting in a large number of TIME_WAIT states.
• • Large File Transfer Scenarios: During data backups, network throughput is severely insufficient, leading to low transfer efficiency.
• • Microservice Call Scenarios: Frequent calls between services result in latency jitter and unstable response times.

The root cause of these issues often lies in the default TCP/IP parameters of the Linux kernel, which cannot meet high-performance demands.

🔧 Core Parameter Optimization of the TCP Protocol Stack

1. TCP Connection Management Optimization

# /etc/sysctl.conf configuration file

# TCP connection queue length optimization
net.core.somaxconn = 65535                    # Increase listening queue length
net.core.netdev_max_backlog = 30000           # Network card receive queue length
net.ipv4.tcp_max_syn_backlog = 65535          # SYN queue length

# TIME_WAIT state optimization
net.ipv4.tcp_tw_reuse = 1                     # Allow reuse of TIME_WAIT sockets
net.ipv4.tcp_fin_timeout = 30                 # Reduce FIN_WAIT_2 state time
net.ipv4.tcp_max_tw_buckets = 10000           # Limit the number of TIME_WAIT sockets

# Connection keepalive mechanism
net.ipv4.tcp_keepalive_time = 600             # Time to start sending keepalive probes
net.ipv4.tcp_keepalive_probes = 3             # Number of keepalive probes
net.ipv4.tcp_keepalive_intvl = 15             # Interval between probes

2. TCP Buffer Optimization

# TCP receive/send buffer optimization
net.core.rmem_default = 262144                # Default receive buffer size
net.core.rmem_max = 16777216                  # Maximum receive buffer size
net.core.wmem_default = 262144                # Default send buffer size
net.core.wmem_max = 16777216                  # Maximum send buffer size

# TCP socket buffer auto-tuning
net.ipv4.tcp_rmem = 4096 87380 16777216       # TCP read buffer min default max
net.ipv4.tcp_wmem = 4096 65536 16777216       # TCP write buffer min default max
net.ipv4.tcp_mem = 94500000 915000000 927000000 # TCP memory allocation low pressure high

# Enable TCP window scaling
net.ipv4.tcp_window_scaling = 1               # Support larger TCP windows

3. TCP Congestion Control Optimization

# Congestion control algorithm selection
net.ipv4.tcp_congestion_control = bbr          # Use BBR algorithm (recommended)
# Other options: cubic, reno, bic

# Fast retransmit and recovery
net.ipv4.tcp_frto = 2                          # F-RTO algorithm detects false timeouts
net.ipv4.tcp_dsack = 1                         # Enable DSACK support
net.ipv4.tcp_fack = 1                          # Enable FACK congestion avoidance

# TCP slow start threshold
net.ipv4.tcp_slow_start_after_idle = 0         # Disable slow start after idle

🌐 IP Protocol Stack Parameter Optimization

1. IP Layer Processing Optimization

# IP forwarding and routing optimization
net.ipv4.ip_forward = 0                        # Disable forwarding on non-router devices
net.ipv4.conf.default.rp_filter = 1           # Enable reverse path filtering
net.ipv4.conf.all.rp_filter = 1

# IP fragmentation handling
net.ipv4.ipfrag_high_thresh = 262144           # High threshold for IP fragmentation
net.ipv4.ipfrag_low_thresh = 196608            # Low threshold for IP fragmentation
net.ipv4.ipfrag_time = 30                     # Fragment reassembly timeout

# ICMP optimization
net.ipv4.icmp_echo_ignore_broadcasts = 1      # Ignore broadcast ICMP
net.ipv4.icmp_ignore_bogus_error_responses = 1 # Ignore erroneous ICMP responses

2. Port Range Optimization

# Local port range expansion
net.ipv4.ip_local_port_range = 1024 65535      # Available port range

# UDP port optimization
net.ipv4.udp_mem = 94500000 915000000 927000000
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192

⚡ Network Queue and Interrupt Optimization

1. Network Device Queue Optimization

# Increase network device processing queue
echo 'echo 4096 > /proc/sys/net/core/netdev_budget' >> /etc/rc.local
echo 'echo 2 > /proc/sys/net/core/netdev_budget_usecs' >> /etc/rc.local

# RPS/RFS optimization (load balancing for multi-core CPUs)
echo 'f' > /sys/class/net/eth0/queues/rx-0/rps_cpus  # Adjust based on CPU core count

2. Interrupt Optimization Script

#!/bin/bash
# network_irq_balance.sh - Network interrupt balancing script

# Get network card interrupt number
IRQ_LIST=$(grep eth0 /proc/interrupts | awk -F: '{print $1}' | xargs)

# Bind interrupts to different CPU cores
CPU_COUNT=$(nproc)
i=0

for irq in$IRQ_LIST; do
    cpu_mask=$((1 << (i % CPU_COUNT)))
    printf"%x"$cpu_mask > /proc/irq/$irq/smp_affinity
    echo"IRQ $irq -> CPU $((i % CPU_COUNT))"
    ((i++))
done

🎯 Special Optimization for High-Concurrency Scenarios

1. Large Connection Count Optimization

# File descriptor limit
echo'* soft nofile 1048576' >> /etc/security/limits.conf
echo'* hard nofile 1048576' >> /etc/security/limits.conf

# Process count limit  
echo'* soft nproc 1048576' >> /etc/security/limits.conf
echo'* hard nproc 1048576' >> /etc/security/limits.conf

# systemd service limits
echo'DefaultLimitNOFILE=1048576' >> /etc/systemd/system.conf
echo'DefaultLimitNPROC=1048576' >> /etc/systemd/system.conf

2. Memory Management Optimization

# Virtual memory management
vm.swappiness = 10                             # Reduce swap usage
vm.dirty_ratio = 15                            # Dirty page writeback ratio
vm.dirty_background_ratio = 5                  # Background writeback ratio
vm.overcommit_memory = 1                       # Allow memory overcommitment

📈 Performance Monitoring and Validation

1. Key Metrics Monitoring Script

#!/bin/bash
# network_monitor.sh - Network performance monitoring

echo"=== Network Connection Status Statistics ==="
ss -s

echo -e "\n=== TCP Connection Status Distribution ==="
ss -tan | awk 'NR>1{state[$1]++} END{for(i in state) print i, state[i]}'

echo -e "\n=== Network Throughput ==="
sar -n DEV 1 1 | grep -E "eth0|Average"

echo -e "\n=== Memory Usage ==="
free -h

echo -e "\n=== System Load ==="
uptime

2. Load Testing Validation Commands

# Use wrk for HTTP load testing
wrk -t12 -c400 -d30s --latency http://your-server-ip/

# Use iperf3 for network bandwidth testing
iperf3 -s  # Server
iperf3 -c server-ip -t 60 -P 10  # Client

# TCP connection load testing
ab -n 100000 -c 1000 http://your-server-ip/

🔥 Practical Case: E-commerce System Optimization

Comparison Data Before and After Optimization

Metric	Before Optimization	After Optimization	Improvement Rate
QPS	15,000	45,000	200%
Average Latency	120ms	35ms	71%
99% Latency	800ms	150ms	81%
Concurrent Connections	10,000	50,000	400%
CPU Usage	85%	45%	-47%

Key Optimization Points

1. 1. BBR Congestion Control: After enabling, network throughput increased by 40%.
2. 2. TCP Buffer Tuning: Significantly reduced network latency jitter.
3. 3. Connection Reuse Optimization: Reduced TIME_WAIT state by 90%.
4. 4. Interrupt Balancing: Significant improvement in multi-core CPU utilization.

💡 Best Practice Recommendations

1. Scene-Specific Tuning Strategies

High-Concurrency Web Servers

# Focus on optimizing connection count and quick release
net.ipv4.tcp_tw_reuse = 1
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Large File Transfer Servers

# Focus on optimizing buffer and window sizes
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_window_scaling = 1

Database Servers

# Focus on optimizing connection keepalive and stability
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_retries2 = 5

2. Production Environment Deployment Process

1. 1. Test Environment Validation: Apply configurations in the test environment first.
2. 2. Gray Release: Deploy on a few selected servers first.
3. 3. Monitoring Observation: Closely monitor key performance indicators.
4. 4. Full Deployment: Fully roll out after confirming no issues.

3. Configuration Persistence

# Apply all sysctl configurations
sysctl -p

# Verify if the configuration takes effect
sysctl net.ipv4.tcp_congestion_control
sysctl net.core.somaxconn

# Set to take effect automatically on boot
echo 'sysctl -p' >> /etc/rc.local
chmod +x /etc/rc.local

⚠️ Precautions and Common Pitfalls

1. Parameter Tuning Misconceptions

• • Blindly Increasing Buffer Sizes: May lead to insufficient memory.
• • Over-Optimizing TIME_WAIT: May cause port exhaustion.
• • Ignoring Business Characteristics: Different businesses require different parameter strategies.

2. Rollback Plan

# Backup current configuration
cp /etc/sysctl.conf /etc/sysctl.conf.backup.$(date +%Y%m%d)

# Quick rollback script
cat > /root/network_rollback.sh << 'EOF'
#!/bin/bash
cp /etc/sysctl.conf.backup.* /etc/sysctl.conf
sysctl -p
echo "Network config rollback completed!"
EOF
chmod +x /root/network_rollback.sh

🎓 Conclusion

Through systematic tuning of TCP/IP protocol stack parameters, we can significantly enhance the network performance of Linux servers. The key points are:

1. 1. Understand Business Characteristics: Choose appropriate optimization strategies based on actual business scenarios.
2. 2. Gradual Tuning: Avoid modifying too many parameters at once for easier problem identification.
3. 3. Continuous Monitoring: Establish a comprehensive monitoring system to promptly detect performance issues.
4. 4. Testing and Validation: Conduct thorough performance testing after each tuning.

I hope this article helps everyone better optimize network performance in production environments. If you encounter issues in practice, feel free to discuss in the comments!