Core Principles of Web Crawlers: How an HTTP Request is Completed

Author: Da Mu Jiang

https://my.oschina.net/luozhou/blog/3003053

Overview

In the previous article, “Do You Know What Happens Behind the Scenes When You Ping?” we analyzed the process of a Ping using actual packet capture (a common interview question). We learned that ping relies on the ICMP protocol and also involves ARP requests in a local area network. Today, we will similarly use packet capture analysis tools to examine how the familiar HTTP request works.

Environment Preparation

I originally wanted to find a website for packet capture analysis, but the HTTP requests on a production environment website are too numerous and interfere with the analysis, so I simplified it to a demo that returns a string for HTTP requests.

Environment:

1. Demo service responding to HTTP requests
2. Client IP: 192.168.2.135
3. Server: 45.76.105.92
4. Packet capture tool: Wireshark

After deploying the demo to the server and starting it successfully, access the following:

Open the packet capture tool Wireshark to capture packets, and the results are as follows:

From the image above, we can see that we successfully captured an HTTP request and response, but we also see many TCP requests. Next, let’s analyze what these TCP requests are for.

Packet Capture Analysis

A) Three-Way Handshake

Initially, the local machine sent two requests to the server. We’ll explain why there are two requests later; for now, let’s focus on the HTTP corresponding port request, as follows:

192.168.2.135:60738---->45.76.105.92:8081

From the screenshot above, we know this is the first handshake of the TCP protocol. Those familiar with the TCP protocol know that establishing a connection involves a three-way handshake, and disconnecting involves a four-way handshake.

Let’s look at the first request:

60738 -> 8081 [SYN] Seq=0 Win=64240 Len=0 Mss=1460 Ws=256 SACK_PERM=1

Let’s analyze this packet request information:

60783->8081 Port number: Source port—> Destination port
[SYN]: Synchronization handshake signal
Seq: Message number
Win: TCP window size
Len: Message length
Mss: Maximum segment size
Ws: Window scaling adjustment factor
SACK_PERM: SACK option, where 1 indicates SACK is enabled.

For the concepts above, here’s a brief explanation before we introduce the TCPHeader data structure diagram for a better understanding of the TCP header data structure:

1. Win: TCP Window Size refers to the maximum number of bytes that TCP can accept for transmission, which can be dynamically adjusted. This is the TCP sliding window, which controls the rate of data sent by dynamically adjusting the window size. The image above occupies 2 bytes or 16 bits, so the maximum number supported is 2^16=65536. Therefore, under default conditions, the maximum window size supported by the TCP header is 65536 bytes, or 64KB.

2. Len: Message Length refers to the data packet segment, as the entire TCP packet = Header + packSize, so this message length indicates the total length of the data packet to be transmitted, which in this analysis is the size of the HTTP message.

3. Mss: Maximum Segment Size: This specifies the maximum length of the packet that can be transmitted. To achieve optimal transmission efficiency, the TCP protocol typically negotiates the MSS value during connection establishment, which is often replaced by the MTU value (after subtracting the header sizes of IP packets 20Bytes and TCP segments 20Bytes). Therefore, the typical MSS value is 1460, which aligns with the value in our packet capture image.

4. Ws: Window Scaling Adjustment Factor: As mentioned earlier, the default maximum window size for TCP is 64KB, which is insufficient in today’s high-speed internet age. To support more buffered data, RFC 1323 specifies TCP extension options, one of which is the window scaling adjustment factor. How does this work? First, this parameter is negotiated during the [SYN] synchronization phase. Analyzing the above packet data, we see that the result of the first request negotiation is WS=256, and it takes effect in the ACK phase, adjusting the window size. The effective packet capture is as follows:

 60738 ->8081   [ACK] Seq=1 ACK=1 Win=66560 Len=0

We find that the window size has become 66560, larger than the default window. Upon examining the packet details:

We discover that the actual window declared in the request is 260, and the WS extension factor is 256. The final calculated window size is 260*256=66560.

5. SACK_PERM: SACK Option: We know that TCP transmission has a packet acknowledgment mechanism. By default, the receiving end sends an ACK confirmation after receiving a packet. However, it only supports sequential acknowledgment, meaning if packets A, B, and C are sent, and A and C are received but B is not, the acknowledgment for C cannot be sent until B is received. TCP has a timeout retransmission mechanism, so if a packet is not acknowledged for a long time, it is assumed to be lost and retransmitted, which can lead to unnecessary retransmissions and waste transmission space. To solve this problem, SACK proposes a selective acknowledgment mechanism. When SACK is enabled, the receiving end acknowledges all received packets, so the sending end only needs to retransmit genuinely lost packets.

After briefly introducing the above basic concepts, let’s summarize the HTTP request process based on the packet capture, where the local port for the HTTP request is 60378. The summarized process is as follows:

------------------------Request Connection--------------------------
1) 60738 -> 8081 [SYN] Seq=0 Win=64240 Len=0 Mss=1460 Ws=256 SACK_PERM=1
2) 8081 -> 60738 [SYN,ACK] Seq=0 ACK=1 Win=29200 Len=0 MSS=1420 SACK_PERM=1 WS=128
3) 60738 -> 8081  [ACK] Seq=1 ACK=1 Win=66560 Len=0
4) Get /test HTTP/1.1
5) 8081 -> 60738  [ACK] Seq=1 ACK=396 Win=30336 Len=0
6) HTTP/1.1 200 (text/html)
7) 60738 -> 8081  [ACK] Seq=396 ACK=120 Win=66560 Len=0

------------------Disconnect Connection-----------------------------
8) 60738 -> 8081 [FIN ACK] Seq=396 Ack=120 Win=66560 Len=0
9) 8081 -> 60738  [FIN ACK] Seq=120 Ack=397 Win=30336 Len=0
10) 60738 -> 8081 [ACK] Seq=397 Ack=121 Win=66560 Len=0

From the above process, we can see that Sequence 1 to Sequence 3 clearly indicates the three-way handshake. Then Sequence 4 represents an HTTP request, followed by Sequence 5 which is an acknowledgment of the HTTP request, Sequence 6 is the response to the HTTP request, and Sequence 7 is the acknowledgment of the response request.

B) Four-Way Handshake

Sequences 8, 9, and 10 were captured after I closed the browser. Since this is a closure of the browser, we know that the TCP connection has been broken. Some of you may have noticed the issue: the disconnection involves 4 handshakes, but you only captured three records. I assure you that I did not make a mistake; this is the actual packet capture. Why are there only three? Let’s analyze:

Under normal circumstances, a connection is broken with 4 handshakes. The 4 handshake process is illustrated in the image below:

Analyzing this image, the handshake process is as follows:

1. The client initiates a disconnect request, entering FIN-WAIT state
2. The server confirms the disconnect request
3. The server immediately sends a disconnect request, entering CLOSE-WAIT state
4. The client confirms the server's disconnect request, entering TIME-WAIT state

We find that both Process 2 and Process 3 are initiated by the server. Is it possible to combine these two requests and send them to the client in one go? The answer is: Yes. In section 4.2 of RFC 2581, it is mentioned that ACK can be delayed, as long as it is ensured that the acknowledgment packet reaches within 500ms. Under this standard, TCP acknowledgment may be merged with delayed acknowledgment. Based on this, we infer the following packet:

 9) 8081 -> 60738  [FIN ACK] Seq=120 Ack=397 Win=30336 Len=0

This packet combines the acknowledgment for the client and the server’s FIN disconnect signal. We click on the details of this packet, and the red box indicates that this Packet 9 is the ACK confirmation for Frame500. From the initial screenshot, we can see that this packet is Packet 8

 8) 60738 -> 8081 [FIN ACK] Seq=396 Ack=120 Win=66560 Len=0

Moreover, Packet 9 itself sends a FIN signal packet, so we can consider that Packet 9 merged the contents of ACK and FIN. Therefore, the typical 4 handshakes can be reduced to 3 handshakes after merging.

This concludes a complete HTTP request, and the entire process is illustrated as follows:

C) Keep-Alive

Some of you may ask, since this is a complete HTTP request, does every request involve three handshakes?

The answer is: Currently, the protocol does not require it.

In HTTP0.9 and HTTP1.0, each request-response required three handshakes. However, starting from HTTP1.0, persistent connections were attempted with the Keep-Alive parameter, but it was not officially supported. In the HTTP1.1 protocol, the Keep-Alive parameter is officially supported by default, allowing for persistent connections. The Keep-Alive serves two main purposes:

1. Check for dead nodes
2. Prevent connections from being closed due to inactivity

Check for Dead Nodes

This is primarily to quickly detect connection failures and reconnect. For instance, if nodes A and B have established a connection, and node B goes down for some reason, while node A is unaware, two scenarios can occur:

1. If node B has not yet recovered, node A will keep retrying until it realizes that node B is dead.

2. If node B recovers before node A sends data, when node A sends data, node B will not accept it and will send a RST signal (when data is received on a closed socket, a RST packet is sent, requesting the other end to close the abnormal connection without needing to reply with an ACK), allowing node A to know that node B needs to reconnect.

In both scenarios, only when data is sent does node A realize that the other end has encountered an issue. With Keep-Alive, heartbeat signals are sent periodically, allowing for quick detection of the server node’s status.

Prevent Connections from Being Closed Due to Inactivity

We know that establishing and maintaining network connections consume resources, and the number of connections a server can establish is limited. Therefore, firewalls or operating systems may release inactive connections to save resources. The Keep-Alive sends a heartbeat packet periodically to inform the firewall or operating system that this connection is active and should not be terminated.

Later, I captured packets with Keep-Alive included, as shown in the screenshot below:

In the image, the last two packets are the Keep-Alive packets, which the server confirms with an ACK. We see that the keep-alive packet actually sends a packet with one byte, which is the implementation of keep-alive.

Returning to the initial question, why does an HTTP request involve two port handshakes? This is unrelated to the protocol itself. The first packet capture screenshot was accessed using Google Chrome, while the last packet capture image was accessed using Firefox. A careful comparison reveals that Firefox only has one port for the three-way handshake. Therefore, this situation arises from the browser’s implementation. Why does Google Chrome implement it this way? My guess is: to ensure the availability of HTTP access as much as possible. If one port is unavailable, it can immediately switch to another port to complete the HTTP request and response. (This is a personal guess; if there is an authoritative answer, please feel free to discuss in the comments.)

Conclusion

HTTP requests rely on TCP connections, and during the first connection, a TCP three-way handshake occurs.
HTTP utilizes Keep-Alive for persistent connections, sending a heartbeat packet periodically to inform the server that it is still active.
HTTP connection disconnections will lead to TCP four-way handshakes, but if the server meets certain conditions, it can merge the ACK and FIN signals, resulting in three handshakes.

Long press to subscribe for more exciting content ▼

If you found this helpful, please give it a thumbs up. Thank you sincerely!

A) Three-Way Handshake

B) Four-Way Handshake

C) Keep-Alive

Related posts

Leave a Comment Cancel reply