When I graduated, the country still had a system of job allocation. My best friend, Zhang Dapang, was assigned to a big city called Database, where he could sit in a high-end data center every day, specializing in SQL query optimization. His job was stable and comfortable.
The student next door, Xiaobai, was sent to Compiler Town, where he specifically compiled C source files into EXE programs. Although it was tiring, the technical content was very high, with a good salary and plenty of holidays.
I was not a good student, a typical underachiever, having to retake the CET-4 exam twice before passing. I was assigned to a village whose name I didn’t even know, supposedly to handle some HTTP requests. This village was actually just an old computer, but I was comforted by the fact that I could go online and occasionally communicate with my friends.
However, the counselor said that we all had bright futures.
HTTP Server 1.0
HTTP was a novel concept that sparked a bit of interest in my work, preventing me from sinking into despair.
On my first day, the operating system chief threw a pile of documents at me: “This is the HTTP protocol, read it in two days!”
With my level of English, I wouldn’t be able to finish reading these dozens of pages of the English HTTP protocol even if I didn’t eat or sleep for two days. But I thought, “A dead pig is not afraid of boiling water; I’ll just take my time.”
Two weeks later, I finally had a rough understanding of what HTTP was: it was simply a matter of browsers on some computers sending a predefined text (HTTP Request) to my old computer, which would then process it (usually by retrieving a file with an html extension from the hard drive) and send that file back in text form (HTTP Response). It was that simple.
The only tricky part was that I had to ask the operating system to establish a TCP connection channel under the HTTP layer, because all text data had to be received and sent through these TCP channels, which were established using sockets.
Once I understood the principle, I quickly put together the first version of the program, which looked like this:
(Note: For details, see the article “Zhang Dapang’s Socket”)
Look, these sockets, bind, listen, accept… are all interfaces provided by the operating system chief. All I could do was assemble them: first listen on port 80, then enter an infinite loop. If a connection request came in, I would accept it (accept), create a new socket, and finally use this socket to receive and send HTTP data.
The chief named my program Http Server, version 1.0.
This name sounds quite advanced, and I like it.
I eagerly took it for a test. The program started and was “squatting” on port 80. After a while, there was a connection request, so I quickly accepted it, established a new socket, and succeeded! Next, I needed to read the HTTP Request from the socket.
But the receive call was so slow; I waited a full 100 milliseconds without a response! I was blocked!
The operating system chief said, “Don’t worry, I’m also waiting to read data from the network card. Once it’s done, I’ll copy it to you.”
I was happy to take a break.
But the operating system chief said, “Don’t relax; there are many more browsers trying to connect. You can’t just sit here!”
I asked, “What should I do if I can’t relax? The receive call is blocking on your end; what else can I do but join the blocking queue and let the CPU be used by others?”
The chief said, “Sigh, didn’t you learn about multi-processing in college? You’re clearly single-threaded; once you block, it’s over. Find a way to use multi-processing, with each process handling one request!”
The chief was right; I had forgotten about multi-process concurrent programming.
HTTP 2.0: Multi-Processing
The idea of multi-processing is very simple. After accepting a connection, for this new socket, instead of processing it in the main process, I create a child process to take over. This way, the main process won’t be blocked on receive and can continue to accept new connections.
I rewrote the code and upgraded the HTTP server to V2.0. This time, it ran much smoother and could handle many connections concurrently.
At that time, the web was just emerging, and not many people were accessing my HTTP Server. There were only a few dozen connection requests per minute, which I could easily handle.
Since it was a new thing, I still had the capital to boast to Xiaoming, who was working on databases, and Xiaobai, who was working on compilers, telling them that I was a network expert.
In just a few years, the web rapidly developed, and my old machine could no longer keep up. I replaced it with a powerful server and moved to a data center with a spring-like climate.
Now, there are hundreds of connection requests every second, and some connections last quite a long time. So I often have to create hundreds or thousands of processes to handle them, each process consuming a lot of system resources. Clearly, the operating system chief was overwhelmed.
He said, “We can’t keep doing this. With so many processes, just switching processes is exhausting me.”
“How about I use threads instead of processes for each socket connection?”
“That might be better, but I still have to switch threads. Think of a way to limit the number!”
How can I limit it? I could only say that at any given time, I could only support x connections, and the others would have to queue up and wait.
This is definitely not a good solution.
HTTP Server 3.0: Select Model
The chief said, “Let’s think carefully. For me, a socket connection is just a so-called file descriptor (File Descriptor, abbreviated as fd, which is an integer). Behind this fd is a simple data structure, but we are using a very heavyweight thing, ‘process’, to represent read and write operations on it, which is a bit wasteful.”
I said, “How about we switch back to a single-process model? But then we would go back to the old path, and a blocking receive would mean we couldn’t do anything.”
“Single-process is not impossible, but we need to change the way we work.”
“Change to what?” I couldn’t guess what the chief was hinting at.
“Think about the essence of your blocking; isn’t it because the browser hasn’t sent the data yet? Naturally, I can’t give it to you, and since you’re eager to read, I have to block you. In a single-process situation, once you block, you can’t do anything else.”
“Right, that’s it.”
“So after you accept a client connection, you can’t be so eager to read. Let’s do this: every time you tell me a batch of socket numbers, you can block and rest.”
[Note: In fact, what is passed between the HTTP Server and the operating system is not the socket fd numbers, but a data structure called fd_set]
I asked, “Isn’t this the same as before? Originally, it was blocking when calling receive, and now it’s still blocking.”
“Let me finish. I will check these numbered sockets in the background. If I find that these sockets can be read or written, I will mark the corresponding sockets and wake you up to process the data from these sockets. After you finish, tell me your socket fds again, and go back to blocking. This will repeat in cycles.”
I began to understand: “This is a way for us to communicate. I tell you what I want to wait for, then block. If an event occurs, you wake me up and let me do things.”
“Exactly. The key point is that you wait for my notification. After I wake you from the blocking state, you must traverse all the socket fds (which is actually the fd_set data structure) to see who has a mark and handle those that do. I call this method the select model.“
I rewrote the HTTP server using the select method, abandoning the model of one process per socket request. Now, I could handle all sockets with one process.
HTTP Server 4.0: epoll
This select method ran for a while and worked quite well. I just had to tell the chief the socket fds and wait for his notification.
One day, I casually asked the chief, “How many socket fds can I tell you at most?”
“1024.”
“So that means I can monitor a maximum of 1024 sockets with one process?”
“Yes, you might consider using more processes.”
This was a solution, but after using the select method for a while, I discovered its drawbacks. The biggest problem was that I needed to constantly copy the socket numbers (which are actually the fd_set data structure) to the operating system chief, which was resource-intensive. Additionally, after recovering from blocking, I needed to traverse over a thousand socket fds to see if there were any flags to handle.
In reality, many sockets are not active; during a period, the browser may not send any data, and out of those thousand sockets, only a few dozen may need to be processed. Yet, I had to check all the socket fds, which was quite annoying.
Couldn’t the chief just tell me which sockets had changed?
I shared this idea with the chief, and he said, “Well, now that the traffic is increasing, the select method is no longer sufficient. We need to keep up with the times. I’ve thought of a new method called epoll.”
“You see, using epoll is actually similar to select,” the chief continued, “The difference is that I will only tell you those sockets that can be read or written. You only need to handle these ready sockets.”
“It seems the chief has thought this through. This method is much simpler for me.”
I upgraded the HTTP Server again using epoll. Since I no longer needed to traverse the entire set, only handling those sockets that had changes and were active, the system’s processing capability saw a significant leap.
My HTTP Server became widely popular, with countless users worldwide. Eventually, my friend Xiaoming from the database also found out and asked me, “Everyone says you can easily support tens of thousands of concurrent connections. Is that true?”
I humbly replied, “You’re flattering me; actually, I still need to optimize the system.”
He said, “Impressive! You really got lucky.”
I replied, “Didn’t the counselor say back when we graduated that everyone has a bright future?”
Postscript: Recently, a few people asked me about the differences between select and epoll. In fact, I wrote an article about it a few years ago, but there were some small errors. Today, I整理一下再发一次。
《Explaining Technology Through Stories, Only at Code Farmer’s Comeback》
