Optimizing Linux System Configuration for One Million Concurrent Connections
When building high-performance servers, the ability to handle high concurrency is crucial. To achieve one million concurrent connections on a Linux system, it is essential to adjust the following core parameter configurations.
Shell Resource Configuration
Check and adjust the shell resource limits using <span>ulimit -a</span>, as these limits directly affect the resource usage of the current shell and its child processes:
real-time non-blocking time (-R) unlimited # Real-time non-blocking time
core file size (-c) 0 # Core dump file size
data seg size (-d) unlimited # Data segment memory limit
scheduling priority (-e) 0 # Scheduling priority
file size (-f) unlimited # Single file size limit
pending signals (-i) 26986 # Number of pending signals
max locked memory (-l) 880360 # Locked memory limit
max memory size (-m) unlimited # Physical memory usage limit
open files (-n) 1048576 # ⭐Key Parameter⭐
pipe size (-p) 8 # Pipe buffer size
POSIX message queues (-q) 819200 # Message queue size
real-time priority (-r) 0 # Real-time priority
stack size (-s) 8192 # Thread stack size
cpu time (-t) unlimited # CPU time limit
max user processes (-u) 26986 # Maximum number of user processes
virtual memory (-v) unlimited # Virtual memory limit
file locks (-x) unlimited # Number of file locks
<span>open files (-n)</span>: Must be increased to one million (example: 1,048,576)
- Principle: Each TCP connection requires one file descriptor (socket fd)
- Goal: Support over one million concurrent connections in a single process
Protocol Level
In the TCP/IP protocol stack, the combination of source IP address, destination IP address, source port number, destination port number, and transport layer protocol type (usually TCP or UDP) is referred to as the “five-tuple”. This five-tuple is used to uniquely identify a network connection (or session).
We can trace this from the function calls:
// First Stage: Socket Creation and Protocol Binding
// Set the specified handler for sock based on protocol type
socket(AF_INET, SOCK_STREAM, 0)
--> SYSCALL_DEFINE3(socket, int, family, int, type, int, protocol)
--> __sys_socket(family, type, protocol)
// Specify sock handler based on protocol
/* linux/include/linux/net.h*/
struct net_proto_family {
int family;
int (*create)(struct net *net, struct socket *sock,
int protocol, int kern);
struct module *owner;
};
--> pf->create(net, sock, protocol, kern);
// linux/net/ipv4/af_inet.c
/*
if (protocol == answer->protocol) {
if (protocol != IPPROTO_IP)
break;
} else {
// Check for the two wild cases.
if (IPPROTO_IP == protocol) {
protocol = answer->protocol;
break;
}
if (IPPROTO_IP == answer->protocol)
break;
}
}
*/
// Specify handler based on protocol
--> sock->ops = answer->ops;
// Example of some functions
--> conststruct proto_ops inet_stream_ops = {
.bind = inet_bind,
.connect = inet_stream_connect,
.accept = inet_accept,
.listen = inet_listen,
.sendmsg = inet_sendmsg,
.recvmsg = inet_recvmsg,
}
// Second Stage: Application Layer Receiving Data (User Space -> Kernel Space)
// Application layer receives data: through recv() system call, eventually reaching tcp_recvmsg function, extracting data from the receive queue.
recv(fd, buf, BUFFER_LENGTH, 0)
// linux/net/socket.c
--> SYSCALL_DEFINE3(recvmsg, int, fd, struct user_msghdr __user *, msg, unsignedint, flags)
--> __sys_recvmsg(fd, msg, flags, true)
--> sock = sockfd_lookup_light(fd, &err, &fput_needed)
// After finding the corresponding sock, start receiving data
--> ___sys_recvmsg(sock, msg, &msg_sys, flags, 0);
--> ____sys_recvmsg(sock, msg_sys, msg, uaddr, flags, nosec);
--> sock_recvmsg_nosec(sock, msg, flags)
--> INDIRECT_CALL_INET(sock->ops->recvmsg, inet6_recvmsg, inet_recvmsg, sock, msg, msg_data_left(msg), flags);
// linux/net/ipv4/tcp_ipv4.c
--> struct proto tcp_prot = {
.recvmsg = tcp_recvmsg,
.sendmsg = tcp_sendmsg,
}
// linux/net/ipv4/tcp.c
--> tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int flags, int *addr_len)
--> tcp_recvmsg_locked(sk, msg, len, flags, &tss, &cmsg_flags)
// Extract data from the receive queue.
--> last = skb_peek_tail(&sk->sk_receive_queue);
// Third Stage: Network Layer Receiving Data (NIC Interrupt -> Soft Interrupt -> TCP Processing)
// Network layer receives data: through NIC interrupts, soft interrupt processing, eventually reaching tcp_v4_rcv function, placing packets into the receive queue.
// linux/net/ipv4/tcp_ipv4.c
--> tcp_v4_rcv(struct sk_buff *skb)
--> __inet_lookup_skb(net->ipv4.tcp_death_row.hashinfo,
skb, __tcp_hdrlen(th), th->source,
th->dest, sdif, &refcounted);
// linux/include/net/inet_hashtables.h
--> __inet_lookup(net, hashinfo, skb, doff, saddr, sport, daddr, dport, dif, 0, &refcounted);
// linux/include/net//ipv4/inet_hashtables.c
--> u16 hnum = ntohs(dport);
hash = inet_ehashfn(net, daddr, hnum, saddr, sport);
From the last line, we can see that the connection is ultimately identified by the five-tuple. If the server uses only one port, then the connection with a client can be represented as
(TCP, client_ip, dest_port, src_ip, src_port);
Here, only dest_port is subject to change, but we also note that dest_port is a u16 variable, meaning the maximum value is , indicating that a client can establish a maximum of 65,536 connections with a server that has only one port. If we want to simulate one million requests from a single client, we need to configure the server with 16 ports.
Linux System Settings
The following are Linux kernel parameters primarily used for network performance optimization and resource management, which are crucial in high-concurrency server scenarios:
-
<span>net.ipv4.tcp_mem = 262144 524288 786432</span>
<span>262144</span>(Low): The lower threshold for TCP memory usage. Below this value, TCP uses memory without any pressure.<span>524288</span>(Pressure): The pressure threshold for TCP memory usage. When total memory usage exceeds this value, TCP enters “memory pressure” mode, attempting to limit memory allocation for connections.<span>786432</span>(High): The upper threshold for TCP memory usage. When total memory usage reaches or exceeds this value, TCP will refuse new connection attempts or memory allocation requests, potentially leading to connection failures or errors.
-
Function: Controls the total memory allocated by the kernel for the TCP protocol (measured in system page size
<span>PAGE_SIZE</span>, typically 4KB). -
Three values:
-
Purpose: Prevent TCP from exhausting all system memory, ensuring stable system operation. This needs to be adjusted based on physical memory size during high-concurrency connections.
<span>net.ipv4.tcp_wmem = 1024 1024 2048</span>
<span>1024</span>(Min): The minimum value for each TCP socket’s send buffer. Even if the application requests a smaller size, the kernel will allocate at least this much.<span>1024</span>(Default): The initial size of the send buffer for each TCP socket upon creation.<span>2048</span>(Max): The maximum value to which each TCP socket’s send buffer can automatically grow.
-
Function: Controls the memory size allocated by the kernel for the send buffer of each TCP socket (in bytes).
-
Three values:
-
Purpose: Balance the speed of sending data and memory consumption. A too-small
<span>Max</span>limit may become a bottleneck in high-bandwidth environments; too large<span>Min/Default</span>may waste memory (especially with many connections). Here, for testing high concurrency, smaller settings (1KB-2KB) are used.
<span>net.ipv4.tcp_rmem = 1024 1024 2048</span>
- Function: Sets the maximum number of file descriptors (File Descriptors – FDs) that the kernel allows the system to have open simultaneously.
- Description: File descriptors are used to represent open files, sockets (network connections), pipes, etc. Each active TCP connection occupies at least one file descriptor.
- Purpose: Prevent the system from exhausting kernel resources due to too many open files/connections. For systems like web servers and databases that need to handle a large number of connections or files, this value needs to be set sufficiently high, while ensuring that each user’s
<span>ulimit -n</span>(maximum FDs at the process level) is also adjusted accordingly.
<span>fs.file-max = 1048576</span>
<span>1024</span>(Min): The minimum value for each TCP socket’s receive buffer.<span>1024</span>(Default): The initial size of the receive buffer for each TCP socket upon creation.<span>2048</span>(Max): The maximum value to which each TCP socket’s receive buffer can automatically grow.
-
Function: Controls the memory size allocated by the kernel for the receive buffer of each TCP socket (in bytes). The mechanism corresponds exactly to
<span>tcp_wmem</span>. -
Three values:
-
Purpose: Similar to
<span>tcp_wmem</span>, controls the memory used for receiving data for each connection. Again, the settings here (1KB-2KB) are smaller, suitable for a large number of small connections.
<span>net.nf_conntrack_max = 1048576</span>
-
Function: Sets the maximum number of entries in the kernel’s
<span>Netfilter</span>connection tracking table (<span>conntrack</span>). -
Description: The
<span>nf_conntrack</span>kernel module (usually relied upon by<span>iptables</span>/<span>nftables</span>or<span>firewalld</span>) is responsible for tracking all stateful connections (TCP, UDP, ICMP, etc.). Each active stateful network connection (including established TCP connections) occupies one entry in the connection tracking table. -
Purpose: Prevent new connections from being rejected due to a too-small tracking table.
<span>net.ipv4.ip_local_port_range="1024 65536"</span>
- Function: Sets the kernel port range to 1024 – 65536, allowing clients to use as many ports as possible.
- Purpose: Increase the number of available ports for clients, supporting large-scale concurrent outbound connections.
The Linux network stack is like a precision Swiss watch; every optimization is an exploration of the system’s limits. When we achieve one million concurrent connections on a single machine, we are not only breaking through a numerical barrier but also gaining a profound understanding of the essence of computer systems.