Daily Insight | The Evolution of the HTTP Protocol

Hello everyone, this is the Programming Cookbook. Today’s Daily Insight is about computer networking knowledge: the evolution of the HTTP protocol.

Table of Contents

• HTTP/1.1
• What is Pipelining?
• HTTP 2.0
• 1. Binary Framing
• 2. Multiplexing
• 3. Header Compression
• 4. Server Push
• 5. Prioritization and Flow Control
• HTTP 3.0
• 1. QUIC Protocol based on UDP
• 2. Built-in TLS 1.3
• 3. 0-RTT Handshake
• 4. Improved Flow Control
• 5. Connection Migration

The HTTP (HyperText Transfer Protocol) protocol has evolved through multiple versions to enhance performance, security, and reliability. Below are the evolutions and optimizations of HTTP 1.1, 2.0, and 3.0:

HTTP/1.1

What is Pipelining?

Pipelining in HTTP/1.1 is an optimization technique that allows multiple HTTP requests to be sent in parallel over the same TCP connection without waiting for the response to each request.

Function:

• In traditional HTTP/1.0, each request/response must occur sequentially: send a request, wait for a response, and then send the next request.
• Pipelining in HTTP/1.1 reduces waiting time and TCP connection overhead by allowing multiple requests to be sent simultaneously, theoretically increasing request throughput.

How it works:

• On a single TCP connection, the client can send multiple HTTP requests continuously without waiting for each response. In other words, multiple requests can be “queued” (pipelining), and the server will return responses in order.
• Pipelined requests and responses are completed over the same TCP connection, allowing the client to continue sending the next request without waiting for each response to return.

Limitations:

• Response Order Issue: Pipelined requests in HTTP/1.1 must<span> return responses in order</span>. Even if some requests are processed quickly, subsequent requests must wait for the responses of previous requests, causing Head-of-Line Blocking.
• Server Support: Not all servers and intermediaries support pipelining.
• Browser Support: Many modern browsers no longer fully support HTTP/1.1 pipelining due to its tendency to cause performance bottlenecks.

Summary: Pipelining in HTTP/1.1 attempts to improve efficiency by sending multiple requests in parallel over the same connection, but due to ordering and blocking issues, it has not widely resolved performance bottlenecks.

HTTP 2.0

HTTP/2 significantly enhances web performance through features such as binary framing, multiplexing, header compression, server push, and prioritization and flow control. These improvements make HTTP/2 a crucial foundation for modern web applications, especially in high-concurrency and high-latency scenarios. However, HTTP/2 still relies on TCP, which may limit performance in certain situations (e.g., network packet loss), and this is an area where HTTP/3 further improves.

1. Binary Framing

Core Concept:

• HTTP/2 breaks communication data into smaller binary frames, with each frame belonging to a specific stream.
• Frames are the smallest communication unit in HTTP/2, and all requests and responses are split into<span> multiple frames for transmission</span>.

Frame Structure:

• Each frame contains the following fields:
• Length: The length of the frame.
• Type: The type of the frame (e.g., HEADERS for transmitting HTTP header information, DATA for transmitting the body data of HTTP requests or responses, PRIORITY for setting stream priority, etc.).
• Flags: Used to control the behavior of the frame (e.g., whether to end the stream).
• Stream Identifier: Identifies the stream to which the frame belongs.
• Frame Payload: The actual data of the frame.

Advantages:

• Efficient Parsing: The binary format is more compact and faster to parse than the text format of HTTP/1.x.
• Flexibility: Frames can be transmitted out of order, allowing the receiver to reassemble them based on stream ID.
• Scalability: New frame types can be supported through protocol extensions for additional functionality.

Example:

HTTP/1.x (e.g., HTTP/1.1) transmits requests and responses in plain text format, with all content represented in human-readable characters.

Example (HTTP/1.1 Request):

GET /index.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0
Accept: text/html

HTTP/2 converts messages into binary encoded frames, with all data transmitted as a compact binary byte stream.

• An HTTP request may be split into:
• One <span>HEADERS</span> frame (containing the request headers).
• Multiple <span>DATA</span> frames (containing the request body).
• These frames can be transmitted in parallel over the same connection.

Example (HTTP/2 Frame Structure):

+-----------------------------------------------+
| Length (24 bits)                              |  # Frame Length
+---------------+---------------+---------------+
| Type (8 bits) | Flags (8 bits)                |  # Frame Type and Flags
+-+-------------+---------------+---------------+
|R| Stream Identifier (31 bits)                |  # Stream Identifier
+=+=============================================+
| Frame Payload (0 or more bytes)               |  # Frame Data
+-----------------------------------------------+

2. Multiplexing

Core Concept:

• HTTP/2 allows <span>multiple requests and responses to be transmitted simultaneously</span> over the same TCP connection.
• Each request and response is assigned a unique stream ID for identification and differentiation.

How it Works:

• The client and server can simultaneously send frames of multiple streams on a single connection.
• Frames are distinguished by stream ID, and the receiver reassembles the frames into complete requests or responses.
• Details are as follows:

1. Frame: HTTP/2 divides data into multiple small “frames”. Each frame has an identifier and can belong to different requests and responses. Request and response data is broken down into multiple small frames and transmitted over a single connection.
2. Stream: Each request/response has a unique stream identifier. HTTP/2 allows data to be sent simultaneously across multiple streams, enabling multiple requests/responses to be handled over a single connection.
3. No Order Dependency: Unlike pipelining in HTTP/1.1, HTTP/2’s multiplexing allows requests and responses to be processed completely independently in parallel, avoiding head-of-line blocking issues. The order of responses is no longer constrained by the order of request sending.
4. Prioritization and Flow Control: HTTP/2 supports prioritization control for requests and responses, allowing the client to declare the priority of certain requests to optimize resource scheduling.

Advantages:

• Solves Head-of-Line Blocking: In HTTP/1.1, a slow request blocks subsequent requests. In HTTP/2, multiple requests can be processed in parallel.
• Reduces Connection Count: HTTP/1.1 requires multiple TCP connections to process requests in parallel, while HTTP/2 only needs one connection.
• Reduces Latency: In high-latency networks, multiplexing significantly reduces the waiting time for requests.

Disadvantages:

• Head-of-Line Blocking at the TCP Level:
• HTTP 2.0 itself solves application-layer head-of-line blocking, but since all requests still share a single TCP connection, if<span> TCP packet loss</span><code><span>, all requests on the entire connection will be affected.</span>

Example:

• The client requests HTML, CSS, and JavaScript files simultaneously.
• The server sends frames of these files in parallel over a single connection, and the client reassembles them based on stream ID.

3. Header Compression

Core Concept:

• HTTP/2 uses HPACK algorithm to compress HTTP headers.
• Header fields are encoded as indexed values, reducing duplicate transmission.

How HPACK Works:

• Static Table: Contains 61 common HTTP header fields (e.g., <span>:method</span>,<span>:path</span> etc.).
• Dynamic Table: A table dynamically updated during the connection to store recurring header fields.
• Huffman Coding: Compresses header values.

Advantages:

• Reduces Redundancy: For example,<span> Cookie</span><span> </span><span>and</span><span> </span><code><span> User-Agent</span><span> </span><span>header fields only need to be sent once, and subsequent requests can reuse them.</span>
• Saves Bandwidth: Header compression significantly reduces data transmission volume, especially in high-concurrency scenarios.

Example:

• First Request:
• Header:<span>:method: GET, :path: /index.html, user-agent: Chrome</span>
• Dynamic table updated to:<span>[:method: GET, :path: /index.html, user-agent: Chrome]</span>
• Second Request:
• Header:<span>:method: GET, :path: /style.css, user-agent: Chrome</span>
• Only <span>:path: /style.css</span> needs to be sent, with other fields referenced from the dynamic table.

4. Server Push

Core Concept:

• The server can proactively push resources before the client requests them.
• The pushed resources share the same connection as the original request.

How it Works:

• When the server receives a request, it can predict other resources the client may need.
• The server sends a <span>PUSH_PROMISE</span> frame, informing the client of the resources that will be pushed.
• The client can choose to accept or reject the pushed resources.

Advantages:

• Reduces Latency: The client does not need to make additional requests for resources, as the server can push them in advance.
• Optimizes Loading: For example, pushing CSS and JavaScript files reduces page rendering time.

Example:

• The client requests <span>index.html</span>.
• The server pushes <span>style.css</span> and <span>script.js</span>, as these resources are typically used with the HTML file.

5. Prioritization and Flow Control

Prioritization:

• Core Concept: The client can set priorities for each stream, and the server processes requests based on these priorities.
• Priority Fields:
• Stream Dependency: Specifies the dependency relationship of streams.
• Weight: Specifies the relative priority of streams.
• Advantages:
• Ensures critical resources (like CSS and JavaScript) are loaded first.
• Optimizes page rendering performance.

Flow Control:

• Core Concept: HTTP/2 provides a flow control mechanism based on streams.
• How it Works:
• Each stream has a flow control window that limits the amount of data the sender can send.
• The receiver can dynamically adjust the window size.
• Advantages:
• Prevents one end from sending too much data, causing overload on the other end.
• Ensures each stream can fairly use bandwidth.

Example:

• The client requests HTML, CSS, and JavaScript files, setting a higher priority for the CSS file.
• The server prioritizes sending the CSS file to ensure the page renders quickly.

HTTP 3.0

HTTP 3.0 is based on the QUIC protocol, significantly improving transmission performance through the following enhancements:

1. Based on UDP: Solves TCP’s head-of-line blocking issue, providing lower latency and higher concurrency performance.
2. Built-in TLS 1.3: Enforces encrypted communication, enhancing security.
3. 0-RTT Handshake: Reduces latency for the first request, improving user experience.
4. Improved Flow Control: Independent flow control mechanisms ensure that packet loss in one stream does not affect others.
5. Connection Migration: Supports seamless network switching, enhancing mobile experience.

HTTP 3.0 is an important direction for future web communication, especially in high-latency and high-packet-loss environments, providing faster and more stable transmission performance.

1. QUIC Protocol Based on UDP

Core Concept:

• HTTP/3 no longer relies on TCP but is based on QUIC (Quick UDP Internet Connections) protocol.
• QUIC runs on UDP and combines TCP’s reliability with UDP’s flexibility.

How it Works:

1. UDP as the Transport Layer:

• QUIC uses UDP as the underlying protocol, avoiding the overhead of TCP connection establishment and maintenance.
• UDP is connectionless, and QUIC implements reliability, congestion control, and flow control at the application layer.

Multiplexing and Independent Streams:

• QUIC divides data into multiple independent streams, each with its own sequence number and flow control.
• Even if one stream experiences packet loss, the data transmission of other streams will not be affected.

Fast Retransmission and Recovery:

• QUIC uses a more efficient retransmission mechanism, quickly detecting and recovering from packet loss through packet numbering (Packet Number) and acknowledgment mechanisms (ACK).

Advantages:

• Solves TCP Head-of-Line Blocking Issue: Independent flow control ensures that packet loss in one stream does not affect others.
• Lower Latency: The connectionless nature of UDP reduces protocol overhead.
• Better Congestion Control: QUIC implements more flexible congestion control algorithms.

Example:

• In HTTP/2, TCP-level packet loss causes the entire connection to block.
• In HTTP/3, only the streams experiencing packet loss are affected, while others remain unaffected.

2. Built-in TLS 1.3

Core Concept:

• QUIC integrates TLS 1.3, enforcing<span> encrypted communication</span>.
• TLS 1.3 is the current<span> most secure</span> encryption protocol, providing stronger security and faster handshake speeds.

How it Works:

1. TLS 1.3 Handshake:

• QUIC combines the TLS 1.3 handshake with connection establishment, reducing additional handshake latency.
• TLS 1.3 supports 1-RTT and 0-RTT handshakes.

Encryption and Decryption:

• QUIC uses TLS 1.3 to encrypt all communication data, preventing man-in-the-middle attacks and data tampering.
• Encryption keys are dynamically generated during the handshake.

Session Recovery:

• Fast session recovery is achieved through session tickets, supporting 0-RTT data transmission.

Advantages:

• Enhanced Security: Mandatory encryption prevents man-in-the-middle attacks and data tampering.
• Reduced Latency: The handshake process of TLS 1.3 is more efficient.

Example:

• In HTTP/2, the TLS handshake and TCP handshake are separate, adding latency.
• In HTTP/3, TLS 1.3 is directly integrated into the QUIC protocol, resulting in faster handshakes.

3. 0-RTT Handshake

Core Concept:

• QUIC supports 0-RTT (zero round-trip time) connection establishment.
• The client can send data directly during the first request, without waiting for the handshake to complete.

How it Works:

1. First Connection (1-RTT):

• The client and server need to complete a full TLS 1.3 handshake.
• The server generates a session ticket that the client can cache.

Subsequent Connections (0-RTT):

• The client uses the cached session ticket to send data directly.
• The server verifies the ticket and immediately begins processing the request.

Security:

• 0-RTT data may be subject to replay attacks, so QUIC restricts the use of 0-RTT data (e.g., only for idempotent requests).

Advantages:

• Reduced Latency: 0-RTT significantly reduces the latency of the first request.
• Improved User Experience: Page loading speeds are faster.

Example:

• When a user first visits a website, a 1-RTT handshake is required.
• When the user visits again, data can be sent directly without waiting for the handshake.

4. Improved Flow Control

Core Concept:

• In QUIC, streams are independent, with each stream having its own flow control window.
• Packet loss in one stream does not affect the data transmission of other streams.

How it Works:

1. Independent Flow Control:

• Each stream has its own flow control window, allowing the client and server to dynamically adjust the window size.
• Even if one stream experiences packet loss, other streams can continue to transmit.

Multiplexing:

• QUIC supports parallel transmission of multiple streams over the same connection.
• Packets from each stream can arrive out of order, and the receiver reassembles them based on stream ID.

Fast Retransmission:

• QUIC uses packet numbering (Packet Number) and acknowledgment mechanisms (ACK) to quickly detect and recover from packet loss.

Advantages:

• Higher Concurrency Performance: Multiple streams can be transmitted in parallel without interference.
• More Flexible Flow Control: Dynamically adjusting window sizes adapts to different network environments.

Example:

• In HTTP/2, TCP-level packet loss causes the entire connection to block.
• In HTTP/3, only the streams experiencing packet loss are affected, while others remain unaffected.

5. Connection Migration

Core Concept:

• QUIC supports connection migration, allowing connections to remain uninterrupted even if the device’s IP address changes.
• This is achieved by using connection IDs instead of IP addresses to identify connections.

How it Works:

1. Connection ID:

• QUIC uses connection IDs to identify connections instead of relying on IP addresses and ports.
• Even if the device’s IP address changes, the connection ID remains unchanged.

Seamless Migration:

• When a device switches networks (e.g., from WiFi to 4G), QUIC can seamlessly migrate the connection.
• The client and server re-establish communication using the connection ID.

Security:

• During connection migration, the TLS 1.3 encryption keys remain unchanged, ensuring communication security.

Advantages:

• Seamless Network Switching: Enhances the mobile user experience.
• Increased Reliability: Connections do not drop in unstable network environments.

Example:

• The user is watching a video and switches from WiFi to mobile data.
• In TCP, the connection would drop, requiring the video to reload.
• In QUIC, the connection can migrate seamlessly, allowing the video to continue playing.

Complete HTTP Knowledge

Currently, the Computer Networking Cookbook has been mostly updated, and you are welcome to click the image below to access the collection directory.

Discussion Group