Is TCP/IP Protocol Stack Better in Kernel Mode or User Mode?

Original: https://blog.csdn.net/dog250/article/details/80532754

“Is the TCP/IP protocol stack better in kernel mode or user mode?”

The root of the problem lies in why we must deliberately distinguish between kernel mode and user mode.

Introduction

To avoid making this article a dry lecture, I will start with an example analysis.

Recently, I have been comparing the packet processing performance of mtcp and the Linux kernel protocol stack for new TCP connections almost every night, while also learning about Tencent’s F-Stack. I should clarify that my mtcp uses netmap as the underlying support, not DPDK.

During the testing process, I confirmed the scalability issues of the Linux kernel protocol stack and how the user mode protocol stack addresses these issues. However, this did not lead me to a clear conclusion that the user mode protocol stack is necessarily better than the kernel mode protocol stack.

What do I mean? Let’s first look at a diagram that roughly describes my testing conclusions:

It can be seen that the Linux kernel protocol stack has serious scalability issues. Although we see that the performance of the user mode protocol stack does not perfectly scale linearly with the increase in CPU cores, it is already much better.Upon seeing this conclusion, we can’t help but ask, Why? Two questions:

• Why does the PPS curve of the kernel protocol stack show severe convexity?
• Why does the CPS (new TCP connections per second) of the kernel protocol stack hardly change with the increase in CPU cores?

The first question is easy to answer; as stated in “The Mythical Man-Month”, nothing can perfectly scale linearly because communication has costs. Alright, after cleverly bypassing the first question, I must delve into the second question.We know that the Linux kernel protocol stack links all Listener sockets and established connections to two global hash tables, which means that each CPU core may operate on these two hash tables. As a preemptive SMP kernel, Linux must lock when handling new TCP connections.Fortunately, the locking granularity of the new kernel has been refined to hash slots, greatly improving performance; however, for TCP SYN requests hashed to the same slot, it still fails!It is particularly serious that if the user mode server only listens on one Nginx port (port 80), then this mechanism effectively acts as a global kernel lock! For the Listener’s slot lock, it is optimized for multiple Listeners (up to INET_LHTABLE_SIZE buckets, i.e., 32), and for a new connection with only one Listener, it will have no effect. However, this only occurs when frequently starting and stopping services + reuseport, which is unrelated to the scenario we describe.For the TCP new connection test, it is evident that frequent operations on the establish hash table are required, locking to insert into the hash table after the handshake is completed and locking to delete from the hash table when the connection is destroyed!The problem has been clearly described; now to reveal the answer.For the TCP CPS test, there will be frequent connection creation and destruction processes, which map to the code, meaning that the inet_hash and inet_unhash functions will be executed frequently. Let’s take a look at unhash:

void inet_unhash(struct sock *sk){struct inet_hashinfo *hashinfo = sk->sk_prot->h.hashinfo;    spinlock_t *lock;int done;
if (sk_unhashed(sk))return;
if (sk->sk_state == TCP_LISTEN)   lock = &hashinfo->listening_hash[inet_sk_listen_hashfn(sk)].lock;else // Multi-core SMP and under high pressure, it is inevitable that multiple sockets will be hashed to the same slot  lock = inet_ehash_lockp(hashinfo, sk->sk_hash);
   spin_lock_bh(lock); // This is the root of the problem   done =__sk_nulls_del_node_init_rcu(sk);   if (done)       sock_prot_inuse_add(sock_net(sk), sk->sk_prot, -1);    spin_unlock_bh(lock);}

There is no need to elaborate on the hash process, which will similarly involve a spinlock serialization process.

This seems to explain why the CPS of the kernel protocol stack is so low, but still does not explain why the CPS of the kernel protocol stack is so unscalable; in other words, why its curve is convex.From the curve, the slope of the dashed line decreases as the number of CPU cores increases. The slope of the curve is negatively correlated with communication costs. Here, the communication cost is the spinning after a conflict!Without seeking complete quantitative analysis, we only need to prove another thing: as the number of CPU cores increases, slot conflicts will intensify, leading to more frequent spinlocks, meaning the frequency of CPU cores and spinlocks is positively correlated!This is apparent and easy to understand. If our hash function is perfect, then each hash is unbiased, and the final hash bucket distribution will be probabilistically uniform. The increase in CPU cores does not alter this conclusion:

The conclusion is that the increase in CPU cores will only exacerbate conflicts; therefore, the more CPU cores, the higher the frequency of spinlocks, which is clearly positively correlated! The frequency of spinlocks increases with the number of CPU cores, and the benefits of increasing CPU cores are perfectly offset by the increased costs of spinlocks, which is why the CPS hardly changes with the increase in CPU cores.Alright, this is the flaw of the kernel protocol stack. Whether it can be improved depends on your determination. The above description does not prove that implementing the protocol stack in kernel mode is bad; on the contrary, it merely demonstrates that the implementation method of the Linux kernel has inherent problems, and that’s all.Now back to the issue of spinlocks in the Linux kernel protocol stack’s CPS, which is also a NAK protocol; when there is a conflict, you wait until you receive, which is a completely passive response. It is destined to lose scalability!There are numerous instances of this passive logic in the Linux kernel, but there is no way to optimize them because they have existed since the beginning. A typical scenario is TCP short connections combined with nf_conntrack. Both require operations on global spinlocks; alas, what a pity!Let’s talk about HTTP. Those who work on lower-level protocols generally do not pay much attention to HTTP, but the Connection field in the HTTP request header can have a huge impact on performance. If you set it to close, it means the server will close the TCP connection after completing the task. If you set it to Keep-Alive, it means you can make multiple HTTP requests over the same TCP connection.But this doesn’t seem very useful. As a browser client, who cares how many CPS your server can handle! Even if I personally set the Connection to Keep-Alive, what can I do if others do not?

ACK and NAK

It is generally believed that NAK protocols can maximize space savings but waste time. However, at the dawn of the bandwidth-constrained TCP protocol, even a single extra byte was considered too much. Why choose ACK instead of NAK?The answer is precisely to trade space for scalability. Subsequent facts have proven that TCP’s choice of ACK was the right choice.NAK indicates you must passively wait for bad news; no news is good news, but how long must you wait? It is uncertain. The active reporting of ACK allows you to plan your next actions.Have you ever had the experience of parting with a friend after drinking late at night? I can hold my liquor, so I generally worry about my friend encountering some issues on the way home. I usually ask my friend to send a WeChat message once they get home to let me know they arrived safely. Upon receiving their message, I can sleep peacefully.If a NAK protocol were used in this scenario, what would happen? For an inherently unreliable channel, even if they called for help, the call might not go through. How long must I wait to ensure my friend is safely home?The TCP ACK mechanism, driven by a clock, continuously sends data, ultimately adapting to various network environments. This is also why TCP, which was developed over 30 years ago, is still functioning well today (PS: This does not relieve my feelings, as I dislike TCP. I only mention TCP positively here because its choice of ACK over NAK was correct, and that’s all).

The Hierarchy of Disdain

A clear distinction between kernel mode and user mode has created a hierarchy of disdain, or conversely, a hierarchy of worship. Those who write code in kernel mode disdain those who write applications, while those who write user mode code worship those who work on the kernel (and then drag Java and C into the mix, disdaining C for Java?).Let’s not talk about the hierarchy of disdain or the hierarchy of worship. As long as we distinguish between kernel mode and user mode, when implementing a function, we inevitably face a choice of which mode to implement it in. This is followed by a debate; let me just cite a few examples.

• There was a small web server in the Linux 2.4 kernel, which was subsequently looked down upon.

• Nowadays, Facebook has created KTLS, aiming to place the SSL process in the kernel to support high-performance HTTPS.

• I myself have ported the OpenVPN data channel into the kernel and brag about it to everyone.

• Tencent’s F-Stack integrates the BSD protocol stack into user mode, calling it efficient and flexible.

• It seems that mtcp has done the same.

• Telira’s salespeople tirelessly tell me how efficient user mode DPI is.

• Many of Microsoft’s network services are implemented in kernel mode.

Regardless of the approach, the basic routine is to move something that was originally implemented in kernel mode to user mode, and vice versa, and so on, with some being very successful and others failing.所谓的成功是因为这个移植解决了特定场景下的特定问题，比如用户态协议栈的mmap+Polling 模式就解决了协议栈收包PPS吞吐率低的问题，然而很多移植都失败了，这些失败的案例很多都是为了移植而移植，故意捣腾的。请注意，不要被什么用户态协议栈所误导，世界上没有万金油！

Main Text

Many people confuse cause and effect, just as it is common to confuse goals and means.We all know that the Linux kernel protocol stack has low packet throughput, but when you ask why it is so, most answers are “high switching costs,” “too many memory copies,” “high cache misses,” “too frequent interrupts”…However, note that as a business using the protocol stack, no one cares about these; in fact, these are causes rather than results. What businesses care about is the fact of low packet throughput, and why it is low is precisely what kernel engineers need to clarify and resolve. The descriptions above regarding switching, copying, cache misses, interrupts, and so on are actually causes of low packet throughput rather than the problem itself.At the same time, optimizing these issues is not the goal; there is only one goal: to improve packet throughput. Optimizing these issues is merely a means to achieve that goal.Confusing means and goals is very common, which is also why it is difficult for any phone to sell more than an Apple phone. Look at what Apple advertises; it is the product itself, while many Android phone ads boast about data, making a big fuss over what advanced technologies their CPUs and memory use. But do those influencers who buy phones to shoot videos care about this? Do they pay attention to this?Returning to the protocol stack topic. Now, our optimization goal is singular: to improve packet throughput; our optimization goal is not to avoid switching, reduce cache misses, or avoid frequent interrupts; these are merely means to achieve the goal. If there are better means, we can completely disregard these.Similarly, moving the protocol stack back and forth is not the goal; no company would make “porting the protocol stack to user mode” or “implementing HTTP protocol in kernel mode” a KPI; these are just means. Moving the protocol stack between kernel and user mode does not help the overall goal except to showcase one’s superb skills.So, if problems can be solved on the spot without porting, wouldn’t that be better? If I had the Midas touch, why would I still need to work hard to make my wife and daughter admire me?Next, we need to assess whether to use the means of porting or the means of solving issues on the spot to address the low packet throughput of the protocol stack. Before assessing, we must first clarify where the problem lies. Well, some issues have already been listed: high switching costs, expensive memory copies, high cache misses, too frequent interrupts…Now that we know the causes, we can treat them accordingly. The current issue is to evaluate the feasibility, cost, and difficulty of resolving these problems in user mode and kernel mode to determine which mode to use. Ultimately, we will find that implementing a protocol stack in user mode is easier.In other words, if these issues could be resolved in the kernel mode protocol stack via patches, no one would bother with user mode protocol stacks.It’s not that implementing the protocol stack in kernel mode is bad; rather, solving packet scalability issues in kernel mode is very challenging because the Linux kernel was not designed with multi-core scalability in mind from the start. Once faced with multi-core, the following issues become entrenched:

• Task switching
• Memory copying
• CPU hopping
• Locks and interrupts

The kernel, at least the Linux kernel, does not provide any infrastructure to perfectly solve the above problems. As the number of CPU cores increases, the need to address these issues will lead to more and more tricks being added to the kernel, making it heavier.In short, if it were possible to efficiently receive packets with a stable kernel thread in kernel mode, it wouldn’t be impossible; it’s just that implementing it in kernel mode is indeed not easy. The reason why user mode is simpler for completing the same work is precisely why there are so many user mode protocol stacks.Interrupts and PollingNow let’s take a look at the philosophy behind interrupts and polling.Interrupts are essentially a product of a frugal mindset, a typical example of the “Hollywood Principle”. It is a way for the system to passively receive notifications. Although subjectively, this method allows the system to “do something else when no notifications are received,” in objective reality, during this time, no system can fully concentrate on doing what is called “something else”.Have you ever been interviewed? Unless you are someone exceptional, it is generally difficult to obtain the contact information of the interviewer. After the interview, the common phrase is, “Wait for the notification,” which makes it hard for many people to remain calm while waiting for the notification and continue their current work. (I previously couldn’t, but now I can. I exclude myself.) At this time, many people think, “If only I could ask!” But they can’t.Moreover, have you ever had the experience of being focused on doing something and being interrupted by your phone ringing? At that time, many people would silence their phones and throw them away, checking messages only after finishing their tasks.This is the characteristic of interrupts and polling. Interrupts are signals imposed on you from the outside, and you must respond passively, while polling allows you to handle things proactively. For interrupts, the greatest impact is that it disrupts the continuity of your current work, while polling does not; it is entirely under your control.For me personally, my phone is usually on silent, and WeChat is set to “Do Not Disturb” for all messages. I choose to check my phone messages actively when I’m idle, using polling to treat my severe phone anxiety.So how do we treat the Linux kernel’s packet receiving phone anxiety? The answer is that there is currently no way.Because if you want to turn interrupts into polling, there must be a kernel thread to actively poll the network card, and this behavior is incompatible with the current Linux kernel protocol stack, which means you would have to implement an entirely new protocol stack in the kernel, facing endless panic and oops risks.Although Linux already supports a busy polling mode for sockets, this is merely a relief, not a cure! In contrast, implementing polling entirely in user mode is a very clean solution.Regarding memory copying, task switching, and cache misses, try to analyze using the above thought process, and the final conclusion is still that re-implementing a self-managed processing process in user mode will be cleaner than modifying the kernel protocol stack.Almost all evaluation conclusions indicate that user mode protocol stacks are a clean solution, but they are not the only correct solution, nor is there evidence to suggest that user mode protocol stacks are the best solution. The choice to implement a protocol stack in user mode to address low packet throughput issues is entirely because it is easier than solving the same problems in kernel mode, that’s all.

Significance

Here, I’d like to discuss a philosophical question.

Some say that the kernel should only be responsible for the control plane, while all operations of the data plane should be left to user mode.

Well, here’s another Platonic classification, which is still meaningless. Of course, those who introduce similar concepts certainly have ample reasons to persuade others to believe that their classification is objective and based on evidence, but it remains a subjective assumption.

If we continue, perhaps the ultimate conclusion will be that only microkernel-based operating systems are the most perfect operating systems. But in fact, none of the operating systems we use are fully microkernel-based; Microsoft’s Windows is not, and Linux is even more of a monolithic kernel system, not a microkernel system.

It seems that only one statement can explain this contradiction: perfect things do not exist. Everything we see in the world is a shadow in Plato’s eyes, and perfection represents the abstract essence, which is the realm of God, something we are powerless to gaze upon! Since childhood, we have been told that there are no perfect spheres in the world.

Therefore, if microkernels and monolithic kernels do not exist, then why can’t kernel mode only handle the control plane? If it is wrong for kernel mode to handle the network protocol stack, then why has the UNIX/Linux kernel network protocol stack existed for over 40 years? Well, existence is reasonable.

The world evolves rather than is designed, even if there is a designer, they are a poor designer; look, our bodies have many bugs, right? Therefore, even from God’s perspective, what exists is merely a shadow in Plato’s eyes.

Recently, many friends have been asking me for essential materials for programmers, so I dug out my hidden treasures and share them for free!

Scan the QR code on the poster to get it for free.