What Happens During an HTTP Request?

Original link:https://blog.51cto.com/u_15155097/2719818

In this article, we will use a packet capture analysis tool to analyze how an HTTP request works.

Environment Setup

Initially, I wanted to find a website for packet capture analysis, but the HTTP requests in the production environment were too numerous and distracting, making analysis difficult. Therefore, I created a simple demo to return a string for the HTTP request.

Environment:

1. Demo service responding to HTTP requests
2. Client IP: 192.168.2.135
3. Server: 45.76.105.92
4. Packet capture tool: Wireshark

After deploying the demo on the server, it started successfully, and the access is as follows:

What Happens During an HTTP Request?

Open the packet capture tool Wireshark to capture packets, and the results are as follows:

What Happens During an HTTP Request?

From the above image, we can see that we successfully captured an HTTP request and response. However, we also see many TCP requests. Next, let’s analyze what these TCP requests are for.

Packet Capture Analysis

A) Three-Way Handshake

The initial step is that the local machine sends two requests to the server. I will explain why there are two requests later. For now, let’s focus on the HTTP corresponding port request, as follows:

192.168.2.135:60738---->45.76.105.92:8081

From the screenshot above, we know this is the first handshake of the TCP protocol. Those familiar with the TCP protocol know that establishing a connection requires a three-way handshake, while disconnecting requires a four-way handshake.

Let’s first look at the first request:

60738 -> 8081 [SYN] Seq=0 Win=64240 Len=0 Mss=1460 Ws=256 SACK_PERM=1

Let’s analyze this packet request information:

60738->8081 Port number: Source port —> Destination port
[SYN]: Synchronization handshake signal
Seq: Message number
Win: TCP window size
Len: Message length
Mss: Maximum segment size
Ws: Window scaling factor
SACK_PERM: SACK option, where 1 indicates SACK is enabled.

For the above concepts, I will briefly explain them. Before that, let’s look at the data structure diagram of the TCPHeader to have an intuitive understanding of the TCP header data structure.

What Happens During an HTTP Request?

1. Win: TCP Window Size refers to the maximum number of bytes that can be accepted for TCP transmission. This can be dynamically adjusted, which is the sliding window of TCP. By dynamically adjusting the window size, the sending rate of data can be controlled. The above image occupies 2 bytes, which is 16 bits, so the maximum number supported is 2^16=65536. Therefore, under default conditions, the maximum window size supported by the TCP header is 65536 bytes, which is 64KB.

2. Len: Message Length refers to the length of the data segment. Since the entire TCP packet = Header + packSize, this message length refers to the total length of the data packet to be transmitted, which in this analysis is the size of the HTTP message.

3. Mss: Maximum Segment Size: This specifies the maximum length of the packet that can be transmitted. To achieve optimal transmission efficiency, the TCP protocol usually negotiates the MSS value between both parties when establishing a connection. This value is often replaced by the MTU value (after subtracting the size of the IP packet header of 20Bytes and the TCP segment header of 20Bytes). Therefore, the general MSS value is 1460, which is consistent with the value in our packet capture image.

4. Ws: Window Scaling Factor: As mentioned earlier, the default maximum window size for TCP can only support 64KB of buffered data. In today’s high-speed internet era, this size is certainly insufficient. Therefore, to support more buffered data, RFC 1323 specifies the extension options for TCP, among which the window scaling factor is one. How does this work? First, this parameter is negotiated during the [SYN] synchronization phase. Referring to the packet data analysis above, we see that the result of the first request negotiation is WS=256, and then in the ACK phase, the scaling factor takes effect, adjusting the window size. The effective packet capture is as follows:

60738 ->8081   [ACK] Seq=1 ACK=1 Win=66560 Len=0

We find that the window has become 66560, larger than the default window. Let’s check the packet details:

What Happens During an HTTP Request?

We find that the actual declared window is 260, and the WS scaling factor is 256, so the final calculated window size is 66560. Thus, we know that the role of this scaling factor is to multiply the original window size by the scaling factor to obtain the final window size, which is 260*256=66560.

5. SACK_PERM: SACK Option: We know that TCP transmission has a packet acknowledgment mechanism. By default, when the receiving end receives a packet, it sends an ACK acknowledgment. However, it only supports sequential acknowledgment by default. This means that if packets A, B, and C are sent, and I receive packets A and C but not B, then for C, I will not acknowledge it until I receive packet B. Therefore, TCP has a timeout retransmission mechanism. If a packet is not acknowledged for a long time, it will be considered lost and retransmitted, which can lead to unnecessary retransmissions and waste of transmission space. To solve this problem, the SACK (Selective Acknowledgment) mechanism was proposed. When SACK is enabled, the receiving end acknowledges all received packets, so the sending end only needs to retransmit the truly lost packets.

After briefly introducing the above basic concepts, let’s summarize the process of the HTTP request based on the packet capture. The local port for the HTTP request is 60378, and the summarized process is as follows:

------------------------Request Connection--------------------------
1) 60738 -> 8081 [SYN] Seq=0 Win=64240 Len=0 Mss=1460 Ws=256 SACK_PERM=1
2) 8081 -> 60738 [SYN,ACK] Seq=0 ACK=1 Win=29200 Len=0 MSS=1420 SACK_PERM=1 WS=128
3) 60738 -> 8081  [ACK] Seq=1 ACK=1 Win=66560 Len=0
4) Get /test HTTP/1.1
5) 8081 -> 60738  [ACK] Seq=1 ACK=396 Win=30336 Len=0
6) HTTP/1.1 200 (text/html)
7) 60738 -> 8081  [ACK] Seq=396 ACK=120 Win=66560 Len=0

------------------Disconnect Connection-----------------------------
8) 60738 -> 8081 [FIN ACK] Seq=396 Ack=120 Win=66560 Len=0
9) 8081 -> 60738  [FIN ACK] Seq=120 Ack=397 Win=30336 Len=0
10) 60738 -> 8081 [ACK] Seq=397 Ack=121 Win=66560 Len=0

From the above process, we can see that 1 to 3 is the obvious three-way handshake. Then, 4 is an HTTP request, followed by 5, which is an acknowledgment of the HTTP request, and 6 is the response to the HTTP request, while 7 is the acknowledgment of the response.

B) Four-Way Handshake

The above sequence numbers 8, 9, and 10 are the packets captured after I closed the browser. Since we know that closing the browser means the TCP connection is closed, some may have noticed a problem: our disconnection involves 4 handshakes, but the captured packets only show three records. Did I make a mistake? I want to tell you that I did not make a mistake; this is the actual packet capture. As for why there are only three, let’s analyze it:

Under normal circumstances, disconnecting a connection involves 4 handshakes. The process of the 4 handshakes is as follows:

What Happens During an HTTP Request?

We analyze this diagram, and the handshake process is as follows:

1. The client initiates a disconnect request and enters the FIN-WAIT state.
2. The server acknowledges the disconnect request.
3. The server immediately sends a disconnect request and enters the CLOSE-WAIT state.
4. The client acknowledges the server's disconnect request and enters the TIME-WAIT state.

We find that the above process 2 and process 3 are both initiated by the server. So is it possible to merge these two requests and send them to the client in one go?The answer is yes. In RFC 2581, section 4.2 mentions that ACK can be delayed, as long as it is guaranteed that the acknowledgment packet arrives within 500ms. Under this standard, TCP acknowledgment can be merged and delayed. Therefore, based on this, we deduce the following packet:

9) 8081 -> 60738  [FIN ACK] Seq=120 Ack=397 Win=30336 Len=0

This packet merges the acknowledgment to the client and the server’s FIN disconnect signal. We click on the details of this packet, and as shown in the red box, this packet 9 is the acknowledgment for Frame500. From the initial screenshot, we know that this packet is packet 8.

8) 60738 -> 8081 [FIN ACK] Seq=396 Ack=120 Win=66560 Len=0

What Happens During an HTTP Request?

Moreover, packet 9 itself is a FIN signal packet, so we can consider that packet 9 merges the contents of ACK and FIN. Therefore, the usual 4 handshakes, after merging, become 3 handshakes.

This concludes a complete HTTP request, and the entire process is illustrated as follows:

What Happens During an HTTP Request?

C) Keep-Alive

Some may ask, since this is a complete HTTP request, does every request involve three-way handshakes?

The answer is: Currently, the protocol does not require it.

In the HTTP0.9 and HTTP1.0 versions, every request-response required three-way handshakes. However, starting from HTTP1.0, persistent connections were attempted, which is the Keep-Alive parameter. However, it was not officially supported. In the HTTP1.1 protocol, the Keep-Alive parameter is officially supported by default, allowing for persistent connections.

The main functions of Keep-Alive are as follows:

1. Check for dead nodes
2. Prevent connections from being closed due to inactivity

Check for Dead Nodes

This is mainly to quickly detect failed connections for reconnection. For example, if nodes A and B have established a connection, and node B goes down for some reason, while node A is unaware, there are two scenarios:

1. If node B has not yet recovered, then node B will not reply with an ACK, and node A will keep retrying until it realizes that node B is a dead node.

2. If node B successfully restarts before node A sends data, when node A sends data, node B will not accept it but will send a RST signal (when data is received on a closed socket, a RST packet is sent to request the other end to close the abnormal connection without requiring an ACK response), and only then will node A know that node B needs to reconnect.

In both scenarios, it will only be known that the other party has encountered an exception when data is sent. However, Keep-Alive sends a heartbeat at regular intervals, allowing for quick awareness of the server node’s status.

Prevent Connections from Being Closed Due to Inactivity

We know that establishing and maintaining network connections consumes resources, and the number of connections that can be established on a server is limited. Therefore, firewalls or operating systems may release inactive connections to save resources. The Keep-Alive sends a heartbeat packet at regular intervals to inform the firewall or operating system that this connection is active and should not be terminated.

I later captured packets with Keep-Alive, and the screenshot is as follows:

What Happens During an HTTP Request?

In the above image, the last two packets are the sent Keep-Alive packets, and the server acknowledges them. We see that the keep-alive packet actually sends a packet with one byte, which is the implementation of keep-alive.

After discussing Keep-Alive, let’s return to the initial question: why does an HTTP request involve two port handshakes? In fact, this has nothing to do with the protocol itself. The first packet capture screenshot was taken using Google Chrome, while the last packet capture was taken using Firefox. A careful comparison reveals that Firefox only has one port for the three-way handshake. Therefore, this situation arises from the browser’s own implementation. Why does Google Chrome implement it this way? It is speculated that it aims to ensure the availability of HTTP access. If one port is unavailable, it can immediately switch to another port to complete the HTTP request and response. (This is a personal guess; if there is an authoritative answer, please feel free to discuss in the comments.)

Summary

HTTP requests rely on TCP connections, and the first connection involves a TCP three-way handshake.
HTTP uses Keep-Alive for persistent connections, sending a heartbeat packet at regular intervals to inform the server that it is still active.
HTTP connection disconnection also involves a TCP four-way handshake, but if the server determines that conditions are met, it can merge the ACK and FIN signals, thus converting it into a three-way handshake.

Previous Recommendations

Why Can’t Your Nginx Handle High Concurrency?

Batch Management Tool for Servers: Ansible, Very Detailed!
Is Our CMDB Model Wrong in Operations?
Keepalived: The Magic Tool for Hot Standby, Achieving Zero Downtime!
Linux Shell Tricks: sed
Automation Operations Tool in the Cloud: Terraform
How to Handle Mining Viruses on Online Linux Servers?
Jenkins Setup, Permission Management, Parameterization, Pipeline, etc., Very Detailed!
Detailed Explanation of Jenkins Pipeline, Recommended for Collection!
Finally Understood Nginx Reverse Proxy!
How to Troubleshoot Network Packet Loss in Linux? Finally Understood!
How to Achieve Canary/Gray Release in K8s?
More than 20 Linux Commands for Log Analysis, Very Comprehensive!
Deploying a MySQL Cluster in K8s, Stable!
40 Common Nginx Interview Questions (with Answers)
Jenkins Production Environment Notes, Very Detailed!
19 Common Issues in K8s Clusters, Recommended for Collection
Nginx Working Principles and Optimization Summary (Super Detailed)
Six High-Frequency Linux Operations Troubleshooting Notes!
17 Common Metrics for Operations, 90% of People Don’t Know!
Why Does Performance Drop So Much After Containerizing Applications?
Detailed Explanation of Nginx Rate Limiting, Responding to Traffic Surges and Malicious Attacks
How to Troubleshoot When Linux Disk is Full?
What to Do When the CPU Spikes to 900%?
Nginx Performance Optimization is Covered in This Article!
K8s Pod “OOM Killer”, Found the Reason
Summary of Nginx High Concurrency Performance Optimization Notes from Major Companies
Building a CI/CD System Based on Jenkins
Seven Major Application Scenarios of Nginx (with Configuration)
The Most Comprehensive Detailed Explanation of Jenkins Pipeline
Learning Guide for Mainstream Monitoring System Prometheus
40 Common Nginx Interview Questions
Common Linux Operations Interview Questions, A Must-Read for Job Seekers!
25 Common MySQL Interview Questions and Answers

Light it up, and the server will not go down for three years What Happens During an HTTP Request?

A) Three-Way Handshake

B) Four-Way Handshake

C) Keep-Alive

Related posts

Leave a Comment Cancel reply