In-Depth Analysis of the Linux Network Stack: From TCP/IP Protocol Stack to Zero-Copy Technology

Source: Linux Technology Enthusiast

What exactly happens to a complete network packet from the network card to the application? Why can zero-copy technology improve network performance by dozens of times?

As operations engineers, we deal with various network issues every day. But do you really understand how, when a user clicks a link, the packet is passed layer by layer in the Linux kernel until it reaches the application? Today, we will delve into the underlying implementation of the Linux network stack, from hardware interrupts to user-space applications, unveiling the mysteries of high-performance network programming.

🔍 Introduction: A seemingly simple question

In production environments, we often encounter scenarios like:

  • • Why does Nginx have several times higher QPS than Apache on the same hardware?
  • • Why does the performance of the file server suddenly skyrocket after enabling zero-copy?
  • • Why is the CPU usage low sometimes, but network latency is high?

The answers to these questions are hidden in the deep mechanisms of the Linux network stack.

📡 Chapter 1: The Fantastic Journey of a Packet

1.1 Network Card Reception: The First Stop of Hardware Interrupts

When a network packet arrives at the network card, the entire processing flow begins:

# Check network card interrupt status
cat /proc/interrupts | grep eth0

# Check network card queue statistics
ethtool -S eth0 | head -20

The complete process of the network card receiving packets:

  1. 1. Hardware Reception: The network card writes the packet into the receive buffer (RX Ring Buffer)
  2. 2. Interrupt Trigger: The network card sends a hardware interrupt (Hard IRQ) to the CPU
  3. 3. Driver Processing: The network card driver quickly processes in the interrupt context
  4. 4. Soft Interrupt: Trigger NET_RX_SOFTIRQ soft interrupt, hand over to the kernel network stack

1.2 Kernel Network Stack: A Seven-Layer Journey

The path of packet transmission in the kernel:

Network Card → Driver → netif_rx() → NET_RX_SOFTIRQ → 
__netif_receive_skb() → Protocol Stack Processing → Socket Buffer

Key data structure sk_buff (socket buffer):

struct sk_buff {
    struct sk_buff *next;
    struct sk_buff *prev;
    struct net_device *dev;
    unsigned char *head;
    unsigned char *data;
    unsigned char *tail;
    unsigned char *end;
    // ... more fields
};

🏗️ Chapter 2: Implementation of the TCP/IP Protocol Stack in Linux

2.1 Layered Processing: Each with Its Own Role

The Linux network stack processes strictly according to the OSI model:

Physical Layer → Data Link Layer (L2)

# Check Ethernet frame processing statistics
cat /proc/net/dev

Network Layer (L3): IP Protocol Processing

// Simplified IP layer processing flow
int ip_rcv(struct sk_buff *skb, struct net_device *dev,
           struct packet_type *pt, struct net_device *orig_dev)
{
    // IP header checksum
    if (!pskb_may_pull(skb, sizeof(struct iphdr)))
        goto inhdr_error;
    
    // Route lookup
    if (ip_route_input(skb, iph->daddr, iph->saddr, iph->tos, dev))
        goto drop;
    
    // Pass to transport layer
    return dst_input(skb);
}

Transport Layer (L4): TCP/UDP ProcessingThe TCP state machine is the core of the entire transport layer:

# Check TCP connection state statistics
ss -s
netstat -s | grep -i tcp

2.2 TCP Connection Management: From Three-Way Handshake to Four-Way Teardown

Management of the TCP connection lifecycle:

// TCP state transitions (simplified)
enum tcp_states {
    TCP_ESTABLISHED = 1,
    TCP_SYN_SENT,
    TCP_SYN_RECV,
    TCP_FIN_WAIT1,
    TCP_FIN_WAIT2,
    TCP_TIME_WAIT,
    TCP_CLOSE,
    // ... more states
};

Key performance parameter tuning:

# TCP window tuning
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' >> /etc/sysctl.conf

# TCP congestion control algorithm
echo 'net.ipv4.tcp_congestion_control = bbr' >> /etc/sysctl.conf

⚡ Chapter 3: In-Depth Analysis of Zero-Copy Technology

3.1 Performance Bottlenecks of Traditional I/O

In traditional network I/O operations, data needs to undergo multiple copies:

Disk → Kernel Buffer → User Space Buffer → Socket Buffer → Network Card

This process involves:

  • • 4 data copies: 2 DMA copies + 2 CPU copies
  • • 4 context switches: User Space ↔ Kernel Space
  • • High CPU consumption: Used for data copy operations

3.2 Detailed Explanation of Zero-Copy Technology

sendfile() System Call

#include <sys/sendfile.h>

// Zero-copy send file
ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);

// Actual application example
int send_file_zero_copy(int socket_fd, int file_fd, size_t file_size) {
    off_t offset = 0;
    ssize_t sent = 0;
    
    while (offset < file_size) {
        sent = sendfile(socket_fd, file_fd, &amp;offset, file_size - offset);
        if (sent <= 0) {
            if (errno == EAGAIN) continue;
            return -1;
        }
        offset += sent;
    }
    return 0;
}

splice() System Call

// Pipe zero-copy
ssize_t splice(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out,
               size_t len, unsigned int flags);

// tee(): Zero-copy data copy
ssize_t tee(int fd_in, int fd_out, size_t len, unsigned int flags);

mmap() Memory Mapping

// Memory mapping for zero-copy
void *mmap(void *addr, size_t length, int prot, int flags,
           int fd, off_t offset);

// Actual application
int send_file_mmap(int socket_fd, const char *filename) {
    int fd = open(filename, O_RDONLY);
    struct stat st;
    fstat(fd, &amp;st);
    
    // Map file to memory
    void *mapped = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
    
    // Directly send mapped memory
    ssize_t sent = send(socket_fd, mapped, st.st_size, 0);
    
    munmap(mapped, st.st_size);
    close(fd);
    return sent;
}

3.3 Modern Zero-Copy Technology: io_uring

io_uring, introduced in Linux 5.1, is a revolutionary breakthrough in asynchronous I/O:

#include <liburing.h>

struct io_uring ring;
struct io_uring_sqe *sqe;
struct io_uring_cqe *cqe;

// Initialize io_uring
io_uring_queue_init(QUEUE_DEPTH, &amp;ring, 0);

// Submit sendfile operation
sqe = io_uring_get_sqe(&amp;ring);
io_uring_prep_sendfile(sqe, socket_fd, file_fd, 0, file_size);
io_uring_submit(&amp;ring);

// Wait for completion
io_uring_wait_cqe(&amp;ring, &amp;cqe);

📊 Chapter 4: Performance Optimization Practices and Monitoring

4.1 Network Stack Performance Monitoring

Real-time monitoring of the performance of each layer of the network stack:

# Network interrupt distribution
cat /proc/interrupts | grep -E "(CPU|eth)"

# Soft interrupt statistics
watch -n 1 'cat /proc/softirqs | head -2 &amp;&amp; cat /proc/softirqs | grep NET'

# Network queue depth
for i in /sys/class/net/*/queues/rx-*/rps_cpus; do
    echo $i: $(cat $i)
done

4.2 Advanced Performance Analysis Tools

Using eBPF for in-depth performance analysis:

// Simplified eBPF program: Monitor TCP latency
SEC("kprobe/tcp_sendmsg")
int trace_tcp_sendmsg(struct pt_regs *ctx) {
    u64 ts = bpf_ktime_get_ns();
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    
    // Record send timestamp
    bpf_map_update_elem(&amp;timestamps, &amp;pid, &amp;ts, BPF_ANY);
    return 0;
}

Using tools for analysis:

# Using bcc toolkit
tcplife          # TCP connection lifecycle
tcpretrans       # TCP retransmission statistics
tcptop           # TCP connection TOP statistics

# Using bpftrace
bpftrace -e 'kprobe:tcp_sendmsg { @send[comm] = count(); }'

4.3 Production Environment Optimization Strategies

CPU Affinity Binding:

# Bind network card interrupts to specific CPU
echo 2 > /proc/irq/24/smp_affinity

# Set RPS (Receive Packet Steering)
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus

Kernel Parameter Optimization:

# Network buffer tuning
net.core.netdev_max_backlog = 5000
net.core.rmem_default = 262144
net.core.wmem_default = 262144

# TCP tuning
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1

🚀 Chapter 5: The Power of Zero-Copy in Practical Applications

5.1 Web Server Performance Comparison

Traditional I/O vs. Zero-Copy Performance Testing:

# Performance testing script
import time
import os

def traditional_copy(src, dst):
    with open(src, 'rb') as f1, open(dst, 'wb') as f2:
        while True:
            data = f1.read(8192)
            if not data:
                break
            f2.write(data)

def zero_copy(src, dst):
    with open(src, 'rb') as f1, open(dst, 'wb') as f2:
        # Use sendfile
        os.sendfile(f2.fileno(), f1.fileno(), 0, os.path.getsize(src))

Measured data comparison:

  • • Traditional I/O: 1GB file transfer took ~2.3 seconds, CPU usage 85%
  • • Zero-Copy: 1GB file transfer took ~0.8 seconds, CPU usage 12%

5.2 High-Performance Server Architecture

Design of a high-performance HTTP server based on zero-copy:

// High-performance server based on epoll + sendfile
int create_high_performance_server() {
    int epfd = epoll_create1(EPOLL_CLOEXEC);
    struct epoll_event events[MAX_EVENTS];
    
    while (1) {
        int nfds = epoll_wait(epfd, events, MAX_EVENTS, -1);
        
        for (int i = 0; i < nfds; ++i) {
            if (events[i].events &amp; EPOLLIN) {
                // Handle request
                int client_fd = events[i].data.fd;
                
                // Zero-copy send file
                sendfile(client_fd, file_fd, NULL, file_size);
            }
        }
    }
}

🎯 Chapter 6: Troubleshooting and Performance Tuning Practices

6.1 Common Network Performance Issue Diagnosis

Issue 1: Network Latency Suddenly Spikes

# Check soft interrupt handling
mpstat -I SUM 1

# Check network queue
ss -i sport :80

# Analyze TCP retransmissions
netstat -s | grep -i retrans

Issue 2: Zero-Copy Not Effective

# Check sendfile support
strace -e sendfile your_application

# Monitor system calls
perf trace -s your_application

6.2 Best Practices in Production Environments

Network Stack Optimization Checklist:

  1. 1. ✅ Enable multi-queue network cards (RSS/RPS)
  2. 2. ✅ Configure CPU affinity
  3. 3. ✅ Tune kernel network parameters
  4. 4. ✅ Use zero-copy technology
  5. 5. ✅ Monitor network stack performance metrics

Zero-Copy Application Scenarios:

  • • Static file serving (Nginx)
  • • Proxy servers (HAProxy)
  • • Message queues (Kafka)
  • • Databases (MySQL, PostgreSQL)

💡 Summary and Outlook

Through this in-depth analysis, we have unveiled the mysteries of the Linux network stack:

  1. 1. Packet Processing Flow: From hardware interrupts to soft interrupts, and then to protocol stack processing
  2. 2. TCP/IP Implementation Mechanism: How the Linux kernel efficiently handles network protocols
  3. 3. Zero-Copy Technology Principles: How to eliminate unnecessary data copies
  4. 4. Performance Optimization Strategies: From kernel parameters to application layer optimizations

Key Technical Benefits:

  • • Zero-copy technology can improve network I/O performance by 50-300%
  • • Reasonable network stack tuning can reduce latency by 30-50%
  • • Modern tools like eBPF make performance analysis more precise

Reprint Statement: All reprinted articles must indicate the original source or reprint source (in cases where the reprint source does not indicate the original source), and if there is any infringement, please contact for deletion.

Related posts

  1. Mastering Linux File Transfer Commands
  2. In-Depth Analysis of Linux SquashFS: From Principles to Practice
  3. Comprehensive Analysis of Strings in C++: From Basics to Advanced
  4. Creating Virtual Environments (Windows + Linux)
  5. Domestic EtherCAT Motion Control Edge Controller (Part 5): IO Configuration and Homing Motion Development with Python+Qt
  6. A Detailed Explanation of Function Pointers in C++
  7. 20 Common String Operations in Python: How Many Do You Master?
  8. Five Important Considerations for Microcontroller Embedded Programming
  9. Uniform Initialization in C++: The Power of Braces
  10. Deep Dive into Python: Tuples, Sets, Object-Oriented Programming, File Operations, Exception Handling, and Generators
  11. Practical Showdown: Performance Testing of Rust vs C++ High-Frequency Trading Engines Reveals Surprising Results
  12. Communication Case Between S7-1200 and Modbus Gateway
  13. A Brief Analysis of the TCP/IP Reference Model
  14. A Brief Overview | File Handles in Linux Systems – Knowledge Points Learning
  15. So Many Vulnerabilities in TCP/IP Protocol?
  16. TCP/IP Protocol Basics: TCP and UDP Explained
  17. Commonly Used Linux Commands Reference
  18. How to Elegantly Write LaTeX with Python?
  19. Modern Operating Systems and TCP/IP

Leave a Comment