From: CSDN, Author: dog250
Link: https://blog.csdn.net/dog250/article/details/80532754
“Is the TCP/IP protocol stack better in kernel mode or user mode?”
The root of the question lies in why we must deliberately distinguish between kernel mode and user mode.
Introduction
To avoid making this article a dull lecture, I will start with an example analysis.
Recently, I have been comparing the packet processing performance of mtcp and the Linux kernel protocol stack for establishing new TCP connections almost every night. I also looked into Tencent’s F-Stack. It should be noted that my mtcp uses netmap as the underlying support, not DPDK.
During the testing process, I confirmed the scalable issues of the Linux kernel protocol stack and how the user mode protocol stack addresses this problem. However, this did not lead me to a clear conclusion that the user mode protocol stack is necessarily better than the kernel mode protocol stack. So, what do I mean? Let’s look at a diagram that roughly describes my testing conclusions:
It can be seen that the Linux kernel protocol stack has serious scalable issues. Although we see that the performance of the user mode protocol stack does not perfectly scale linearly with the increase in CPU cores, it is significantly better. Upon seeing this conclusion, we can’t help but ask, Why? Two questions:
-
Why does the PPS curve of the kernel protocol stack show severe convexity?
-
Why does the CPS (Connections Per Second) of the kernel protocol stack show almost no change with the increase in CPU cores?
The first question is easy to answer. As stated in “The Mythical Man-Month”, nothing can perfectly scale linearly because communication incurs costs. Well, after I cleverly bypassed the first question, I had to deeply analyze the second question.
We know that the Linux kernel protocol stack links all Listener sockets and established connections to two global hash tables, which means that each CPU core may operate on these two hash tables. As a preemptive SMP kernel, locking is necessary when Linux handles new TCP connections.
Fortunately, the locking granularity in the new kernel has been refined to hash slots, which greatly improves performance. However, when facing TCP SYN requests that hash to the same slot, it still encounters issues!
Particularly serious is, if the user mode server only listens on one Nginx port 80, then this mechanism is equivalent to a global kernel lock! For the slot lock of the Listener, it is optimized for multiple Listeners (up to INET_LHTABLE_SIZE buckets, i.e., 32), and for a new connection with only one Listener, it will have no effect. However, this only occurs during frequent service starts and stops + reuseport, which is unrelated to the scenario we describe.
For testing TCP new connections, it is evident that frequent operations on the established hash table will occur, locking during hash table insertion after the handshake is completed, and locking during hash table deletion when the connection is destroyed!
The problem has been clearly described, and the answer is about to be revealed.
For TCP CPS testing, there will be a frequent process of connection creation and destruction, which maps to the code where the inet_hash and inet_unhash functions will be executed frequently. Let’s take a look at unhash:
void inet_unhash(struct sock *sk){struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo; spinlock_t *lock;int done;if (sk_unhashed(sk))return;if (sk->sk_state == TCP_LISTEN) lock = &hashinfo->listening_hash[inet_sk_listen_hashfn(sk)].lock;else // In a multi-core SMP with high pressure, multiple sockets are likely to hash to the same slotlock = inet_ehash_lockp(hashinfo, sk->sk_hash); spin_lock_bh(lock); // This is the root of the problem done =__sk_nulls_del_node_init_rcu(sk);if (done) sock_prot_inuse_add(sock_net(sk), sk->sk_prot, -1); spin_unlock_bh(lock);}There is no need to elaborate on the hashing process, which also involves a spinlock serialization process.
This seems to explain why the CPS of the kernel protocol stack is so low, but still does not explain why the CPS of the kernel protocol stack is so unscalable. In other words, why does its curve show convexity?
From the curve, the slope of the dashed line decreases with the increase in CPU cores. The slope of the curve is negatively correlated with communication costs. Here, the communication cost refers to the spin after conflicts!
Without seeking complete quantitative analysis, we only need to prove one more thing, that is, as the number of CPU cores increases, slot conflicts will intensify, leading to more frequent spinlocks, thus the frequency of CPU cores and spinlocks is positively correlated!!
This is evident and easy to understand. If our hash function is perfect, then every hash is unbiased, and the final distribution of hash buckets will be probabilistically uniform. The increase in CPU cores does not change this conclusion:
The conclusion is that the increase in CPU cores will only exacerbate conflicts, thus the higher the number of CPU cores, the higher the frequency of spinlocks, which is clearly positively correlated! The frequency of spinlocks increases with the increase in CPU cores, and the benefits of increasing CPU cores are perfectly offset by the increased costs of spinlocks. Therefore, as the number of CPU cores increases, the CPS hardly changes.
Well, this is the flaw of the kernel protocol stack. Whether it can be improved depends on your determination. The above description does not prove that implementing the protocol stack in kernel mode is bad; on the contrary, it merely proves that the implementation method of the Linux kernel has its issues, that’s all.
Before entering the real metaphysical discussion, let’s digress a bit.
First, I recommend Tencent Cloud’s F-Stack, with its GitHub address:
https://github.com/F-Stack/f-stack
We can see its full picture from the following link:
http://www.f-stack.org/
The reason for this recommendation is that it is a user mode protocol stack implementation that we can all see and touch, facilitating deep communication in a Chinese community, and it is open source. There is no ulterior motive, but I genuinely feel it is good.
Secondly, let’s discuss the first generation Ethernet’s CSMA/CD.
This thing is something almost everyone in IT has heard of. Generally, the first or second lesson in learning networking will mention this, but it will only be mentioned in those two lessons and never again. We all know that today’s switched Ethernet no longer uses the CSMA/CD protocol… but in any case, returning to the basics always yields some gains.
CSMA/CD is a passive preemptive protocol based on NAK rather than ACK. It has the typical characteristic of a 90s child: if there’s no contact, nothing is wrong; if something happens, I will contact you. This scenario is commonly seen when parents drop their kids off at the car, saying, “Call me when you arrive”… but generally, kids will say, “I’ll call if something is wrong; if not, I won’t call.” This behavior is NAK. ACK is entirely opposite. CSMA/CD is a NAK protocol, stop and wait if there’s a conflict, continue if there’s no conflict. Any NAK protocol has a fatal issue: the scalability problem, as referenced in my earlier discussion about hash probability uniformity. For the first generation Ethernet, if there are more nodes on the network, there will be more conflicts, leading to a decline in communication quality for single nodes. This is why the user count/performance curve of the first generation CSMA/CD-based Ethernet shows an upward convex trend.
Now, returning to the spinlock issue of the Linux kernel protocol stack CPS, it is also a NAK protocol, waiting until it can proceed if there is a conflict, which is a completely passive response. This is destined to lose scalability!!
The Linux kernel is filled with a lot of this passive logic, but there is no way to optimize them because they have existed from the beginning. A typical scenario is TCP short connections combined with nf_conntrack. Both require operating global spinlocks, alas, how tragic!
Lastly, let’s talk about HTTP. Those working on lower-level protocols generally do not pay much attention to HTTP, but the Connection field in the HTTP request header can have a huge impact on performance. If you set it to close, it means the server will disconnect the TCP connection after completing the task. If you set it to Keep-Alive, it means you can request multiple HTTPs over the same TCP connection. But this doesn’t seem very useful. As a browser client, who cares how many CPS the server can handle! Even if I set the Connection to Keep-Alive, what can I do if others don’t follow suit?
ACK and NAK
It is generally believed that NAK protocols can maximize space savings but waste time. However, in the early days of the TCP protocol, where bandwidth resources were very tight, even an extra byte was considered too much. Why was ACK chosen instead of NAK?
The answer is precisely space for scalability. Later facts proved that TCP’s choice of ACK was the correct choice.
NAK indicates you must passively wait for bad news; no news is good news, but how long must you wait? It’s unknown. In contrast, the proactive reporting of ACK allows you to plan your next actions.
Have you ever experienced parting ways with a friend after a night of drinking? I can hold my liquor, so I generally worry about my friend encountering some trouble on the way home. I usually ask my friend to send me a WeChat message to confirm they arrived home safely. When I receive their message, I can sleep peacefully. What would happen if I used the NAK protocol in this situation? For a channel that is inherently unreliable, even if they call for help, the call may not go through. How long would I have to wait to ensure my friend is home safe?
TCP’s ACK mechanism drives its sending engine to continuously send data, ultimately adapting to various network environments. This is precisely why TCP, over 30 years later, still remains intact (PS: this does not relieve my grievances, as I dislike TCP. I am only stating that TCP’s choice of ACK over NAK is the correct one, that’s all).
Hierarchy of Disdain
A clear distinction between kernel mode and user mode has formed a hierarchy of disdain, or conversely, a hierarchy of worship. Those writing code in kernel mode disdain those writing applications, while those coding in user mode worship those working on the kernel (and then drag Java and C into the mix, disdaining Java developers for working in C?).
Let’s not discuss the hierarchy of disdain or the hierarchy of worship. As long as we distinguish between kernel mode and user mode, when trying to implement a function, we inevitably face a choice of which mode to implement it in. This is quickly followed by a debate, with a few examples:
-
There is a small web server in the Linux 2.4 kernel that was subsequently looked down upon.
-
Nowadays, Facebook has created KTLS, aiming to place the SSL process in the kernel to support high-performance HTTPS.
-
I myself ported the OpenVPN data channel into the kernel and flaunt this to others.
-
Tencent’s F-Stack has integrated the BSD protocol stack in user mode, claiming to be efficient and flexible.
-
It seems mtcp did something similar.
-
Telira’s salespeople tirelessly tell me how efficient user mode DPI is.
-
Many of Microsoft’s network services are implemented in kernel mode.
No matter how it goes, the basic pattern is what was originally implemented in kernel mode is now moved to user mode, and what was originally implemented in user mode is now moved to kernel mode. It’s a back-and-forth process, with some being very successful and others failing. The so-called success is due to the migration solving specific problems in specific scenarios, while many migrations fail, often due to being done for the sake of migration.
Please note, do not be misled by any user mode protocol stack; there is no universal solution!
Main Text
Many people confuse cause and effect, just as it’s common to confuse goals and means.
We all know that the Linux kernel protocol stack has low packet throughput, but when you ask why the Linux kernel protocol stack is like this, most answers are “the switching overhead is high”, “too many memory copies”, “cache misses are too high”, “interrupts are too frequent”… However, note that as a business using the protocol stack, no one cares about these. In fact, these are causes, not results. What the business cares about is the fact that the packet throughput is low. Why is the throughput low is what kernel engineers need to investigate and resolve. The descriptions above about switching, copying, cache misses, interrupts, etc., are actually causes of low packet throughput, not the problem itself. Moreover, optimizing these issues is not the goal; the only goal is to improve packet throughput. Optimizing these issues is merely a means to achieve that goal.
Confusing means and goals is very common, which is also why it is difficult for any phone to sell more than the iPhone. Look at what Apple’s advertisements promote; they focus on the product itself, while many Android phone advertisements boast about data, showcasing advanced technologies in CPU, memory, etc. But do those internet celebrities who buy phones to take videos care about these? Do they pay attention to these?
Returning to the topic of protocol stacks, our optimization goal is singular: to improve packet throughput. Our optimization goal is not to avoid switching, reduce cache misses, or avoid frequent interrupts; these are just means to achieve the goal. If there are better means, we need not focus on those.
Similarly, moving the protocol stack around is not the goal; no company would list “migrating the protocol stack to user mode” or “implementing HTTP in kernel mode” as KPIs. These are merely means. Moving the protocol stack between kernel mode and user mode does not help the overall goal. Clearly, if we can solve the problem on the spot without migration, wouldn’t that be better? If I had the ability to turn stone into gold, why would I need to work hard to make my wife and daughter admire me?
Next, we need to evaluate whether to use migration as a means or to solve the low packet throughput problem of the protocol stack on the spot. Before evaluating, we must first clarify where the root of the problem lies. Well, some have already been listed: high switching overhead, expensive memory copies, high cache misses, too frequent interrupts…
Once we know the causes, we can treat them accordingly. The current issue is to assess the feasibility, cost, and difficulty of resolving these issues in user mode and kernel mode, to decide which mode to use to solve the problem. Ultimately, it will be found that implementing a protocol stack in user mode is easier. In other words, if these issues can be resolved in the kernel mode protocol stack through patches, no one would bother with user mode protocol stacks.
It’s not that implementing the protocol stack in kernel mode is bad, but it’s very difficult to solve the packet scalability issues under multi-core in kernel mode, because the Linux kernel was not designed with multi-core scalability in mind from the beginning. Once faced with multi-core, the following issues become entrenched:
-
Task switching
-
Memory copying
-
CPU jumps
-
Locks and interrupts
The kernel, at least the Linux kernel, does not provide any infrastructure to perfectly solve the aforementioned problems. As the number of CPU cores increases, many tricks will have to be added to the kernel to address these issues, making the kernel heavier.
In summary, while it is possible to implement a stable kernel thread for efficient packet reception in kernel mode, it is not easy. It is precisely because it is simpler to accomplish the same work in user mode that many user mode protocol stacks have emerged.
Interrupts and Polling
Now let’s take a look at the philosophy behind interrupts and polling.
Interrupts are essentially a product influenced by a saving mindset, a typical example of the “Hollywood Principle”. This is a way for the system to passively receive notifications. Although subjectively, this method allows the system to “do something else when no notification is received”, in objective reality, during this time no system can focus entirely on doing what is called “something else”.
Have you ever been interviewed? If you are not particularly outstanding, it is generally difficult to obtain the contact information of the interviewer. After the interview, a typical response is “Just wait for the notification.” How many people can maintain a calm mindset to wait for the notification while continuing their current work? (I couldn’t before, but I can now, except for myself). At this moment, many people will think, if only I could ask, but they cannot.
Additionally, have you had the experience of being focused on doing something, and the most annoying thing is when your phone rings? Many people will silence their phones and throw them far away, checking messages only after completing their tasks.
Yes, that’s the characteristic of interrupts and polling. Interrupts are signals imposed by the outside world that you must respond to passively, while polling is your active handling of matters. The biggest impact of interrupts is that they disrupt the continuity of your current work, while polling does not; it is entirely under your control. For me, my phone is usually on silent, and WeChat is set to “Do Not Disturb” for all messages. I choose to check my phone messages actively when I’m idle, using polling to treat my severe phone anxiety.
So how can we treat the Linux kernel’s packet reception phone anxiety? The answer is that there is currently no way. Because if you want to change interrupts to polling, you must have a kernel thread to actively poll the network card, and this behavior is incompatible with the current Linux kernel protocol stack, meaning you have to implement an entirely new protocol stack in the kernel, facing endless risks of panic and oops. Although Linux already supports a busy polling mode for sockets, it is merely a relief, not a cure! In contrast, implementing polling entirely in user mode is a very clean solution.
Regarding memory copying, task switching, and cache misses, try analyzing them with the above thoughts, and the final conclusion remains that re-implementing a self-managed processing process in user mode will be cleaner than modifying the kernel protocol stack.
Almost all evaluation conclusions indicate that user mode protocol stacks are a clean solution, but they are not the only correct solution, nor is there evidence that user mode protocol stacks are the best solution. Choosing to implement the protocol stack in user mode to solve the low packet throughput problem is entirely because it is easier than solving the same problem in kernel mode, that’s all.
Significance
Here, let’s discuss a philosophical issue.
Some say the kernel should only be responsible for the control plane, while all operations of the data plane should be handed over to user mode.
Well, here’s another Platonic classification, which still holds no meaning. Of course, those who bring up similar concepts will have plenty of reasons to persuade others to believe that their classification is objective and based on evidence, but that remains a subjective assumption. Continuing on, perhaps the ultimate conclusion will be that only microkernel-based operating systems are the perfect operating systems. However, in fact, almost all operating systems we use are not fully microkernel-based; Microsoft Windows is not, and Linux is even a monolithic kernel system, far from a microkernel system.
It seems that only one statement can explain this contradiction: perfect things do not exist. Everything we see in the world is a shadow in Plato’s eyes, and perfection represents abstract essence, which belongs to the realm of the divine, something we are powerless to gaze upon! From a young age, we have been told by teachers that there are no perfect spheres in the world.
Therefore, if microkernels and monolithic kernels do not exist, why must kernel mode only handle the control plane? If implementing the network protocol stack in kernel mode is wrong, why has the UNIX/Linux kernel network protocol stack existed for over 40 years? Well, existence is reasonable.
The world evolves rather than being designed, even if there is a designer, they are a poor designer. Look at our bodies; they have many bugs, right? Therefore, even with God, what exists is merely a shadow in Plato’s eyes.
—END—