Analysis of Streaming Technology for Large Language Models: From SSE, HTTP/2, gRPC to Nginx Applications

What is SSE

Server-Sent Events (SSE) is a unidirectional communication protocol based on HTTP that allows the server to actively push data to the client. Unlike the traditional request-response model, SSE establishes a long connection, allowing the server to continuously send event streams to the client.

Core Features

Client initiates SSE request

Receives 200 OK response

Starts receiving SSE events

Continuously receives Token

Data reception complete

New data arrives

Network exception/timeout

Browser auto-reconnects

Reconnect successful

Reconnect failed

Session established
Session active
Receiving data
Session maintained
Session disconnected
Auto-reconnect
Session failed
Each Token is transmitted as an independent event
data: {"token": "Hello"}
data: {"token": " world"}
data: {"token": "!"}

Based on Standard HTTP

  • • Uses standard HTTP protocol, no additional handshake required
  • • Firewall and proxy server friendly
  • • Can reuse existing HTTP infrastructure

Automatic Reconnect Mechanism

  • • Browsers natively support automatic reconnection after disconnection
  • • Configurable reconnect interval and maximum reconnect attempts
  • • No need for clients to implement complex reconnection logic

Comparison with the familiar WebSocket from the Internet era:

Feature SSE WebSocket
Communication Direction Unidirectional (Server → Client) Bidirectional
Protocol Complexity Simple Complex
Browser Support Natively supported Requires additional handling
Automatic Reconnect Built-in Needs manual implementation
Data Format Text Binary/Text

LLM Chooses SSE

In the application scenarios of large language models (LLM), SSE has become the preferred solution for streaming output, which is backed by profound technical reasons.

Streaming Output

LLM generates text in a token-by-token process, and users expect to see real-time generation effects. The traditional request-response model requires waiting for the complete generation before returning results, while SSE can push each generated token in real-time, greatly enhancing user experience.

000 ms000 ms000 ms000 ms000 ms000 ms000 ms000 ms000 msUser InputStart GeneratingToken1 PushToken1 TransmissionDisplay Token1Token2 PushToken2 TransmissionDisplay Token2Token3 PushToken3 TransmissionDisplay Token3User InputLLM GenerationNetwork TransmissionUser SeesLLM Streaming Output Timeline (including network delay)

LLM Streaming Output Timeline (including network delay)

Advantages of Protocol Simplicity

Development Efficiency

  • • Clients only need to listen for events, no need to handle complex protocol handshakes
  • • Servers can reuse existing HTTP infrastructure
  • • Debugging and monitoring are more intuitive

Compatibility Assurance

  • • All modern browsers natively support SSE
  • • Good support on both mobile and desktop
  • • Network proxies and CDNs can handle it properly

The Value of Automatic Reconnect

In LLM dialogue scenarios, network interruptions are common issues. The automatic reconnect mechanism of SSE ensures continuity of user experience:

const eventSource = new EventSource('/v1/chat/completions');
eventSource.onmessage = function(event) {
    const data = JSON.parse(event.data);
    displayToken(data.content);
};

// Automatic reconnect, no manual handling required
eventSource.onerror = function(event) {
    console.log('Connection lost, will retry automatically');
};

Multiplexing: HTTP/2 + gRPC

SSE is designed based on HTTP/1.1, and when combined with the multiplexing capabilities of HTTP/2 and gRPC, it can significantly enhance performance and concurrency handling.

HTTP/2 Multiplexing

Supports server push, multiplexing reduces the number of TCP connections, lowers server resource consumption, avoids the head-of-line blocking issue of HTTP/1.1, supports header compression, reduces network overhead, and significantly improves performance.

HTTP/2 + SSE
Server 2
Connection Layer 2
Client 2



LLM Service
User A Dialogue 1
Stream 1
User A Dialogue 2
Stream 2
User B Dialogue 1
Stream 3
Single Connection
HTTP/1.1 + SSE
Server 1
Connection Layer 1
Client 1



LLM Service
User A Dialogue 1
Connection 1
User A Dialogue 2
Connection 2
User B Dialogue 1
Connection 3

gRPC Streaming Enhancement

gRPC provides more powerful streaming capabilities on top of HTTP/2, supporting not only server push (similar to SSE) but also client streaming input, enabling true bidirectional real-time communication.

LLM Service Layer
gRPC Middleware Layer
Client Layer

User Input ↔ Token Output

User Input ↔ Token Output

User Input ↔ Token Output


Request Dispatch ↔ Response Aggregation

Request Dispatch ↔ Response Aggregation

Request Dispatch ↔ Response Aggregation

User A Dialogue 1
User A Dialogue 2
User B Dialogue 1
gRPC Connection Pool
Stream 1: Bidirectional
Stream 2: Bidirectional
Stream 3: Bidirectional
LLM Service 1
LLM Service 2
LLM Service 3

Comparison and Selection of HTTP/2 + gRPC

In LLM scenarios, both HTTP/2 and gRPC have their advantages, and the choice depends on specific requirements.

Feature HTTP/2 + SSE gRPC
Protocol Basis HTTP/2 + Text Event Stream HTTP/2 + Protocol Buffers
Data Format Plain Text Binary
Bidirectional Communication Unidirectional (Server → Client) Bidirectional
Debugging Difficulty Simple Complex
Performance Medium High
Compatibility Excellent Requires additional support

Performance Comparison

Protocol First Token Latency Concurrent Stream Support Bandwidth Usage Compatibility
HTTP/2 + SSE 100-150ms ~1000 Text format, higher bandwidth usage Excellent, natively supported by all modern browsers
gRPC 80-120ms 1000+ Binary format, saves 20-30% bandwidth Requires gRPC-Web or proxy support

Detailed Feature Comparison Table

Feature HTTP/2 + SSE gRPC
Latency Performance First token latency between 100 and 150 milliseconds First token latency between 80 and 120 milliseconds
Throughput Single connection supports about 1000 concurrent streams Single connection supports over 1000 concurrent streams
Bandwidth Usage Text format transmission, higher bandwidth usage Binary format transmission, saves 20% to 30% bandwidth
Compatibility Natively supported by all modern browsers, excellent compatibility Requires gRPC-Web or proxy support, additional configuration needed in browser environments
Development Complexity Simple implementation, easy debugging, suitable for rapid development Requires defining protocol files, higher learning cost, but provides stronger type safety
Resource Consumption Requires more memory and CPU resources to handle text data Binary serialization is more efficient, relatively lower resource consumption

Nginx and SSE

Nginx, as a high-performance reverse proxy server, plays a key role in SSE scenarios, providing complete SSE support and optimization.

Core SSE Parameters (Must Configure)

Parameter Function Importance Default Value Recommended Value SSE Relevance
<span>proxy_buffering</span> Disable proxy buffering Very High on off Directly determines SSE real-time performance
<span>proxy_cache</span> Disable proxy caching Very High on off Ensures real-time data transmission
<span>proxy_http_version</span> HTTP version Very High 1.0 1.1 SSE requires persistent connections
<span>proxy_read_timeout</span> Read timeout Very High 60s 300s+ Prevents long connections from disconnecting
<span>proxy_send_timeout</span> Send timeout Very High 60s 300s+ Ensures complete streaming data
<span>proxy_set_header Connection</span> Connection header Very High – ” Supports HTTP/1.1 persistent connections

Streaming Parameters (Highly Relevant)

Parameter Function Importance Default Value Recommended Value SSE Relevance
<span>add_header Cache-Control</span> Cache control High – “no-cache” Prevents client from caching SSE streams
<span>add_header X-Accel-Buffering</span> Accelerated buffering High – “no” Disables Nginx internal buffering
<span>add_header Content-Type</span> Content type High – “text/event-stream” SSE standard content type
<span>add_header Connection</span> Connection type High – “keep-alive” Maintains persistent connections
<span>sendfile</span> Zero-copy transmission Medium off on Enhances streaming transmission efficiency
<span>tcp_nopush</span> TCP optimization Medium off on Reduces the number of network packets
<span>tcp_nodelay</span> TCP delay Medium off on Reduces streaming transmission delay

Connection Management Parameters (Moderately Relevant)

Parameter Function Importance Default Value Recommended Value SSE Relevance
<span>worker_connections</span> Number of worker connections High 1024 4096+ Supports more SSE connections
<span>worker_rlimit_nofile</span> File descriptors High – 65535+ Supports long connection file handles
<span>keepalive_timeout</span> Long connection timeout Medium 75s 300s+ Maintains SSE connection time
<span>keepalive_requests</span> Long connection requests Medium 100 1000+ Single connection handling capacity
<span>use</span> Event model Medium select epoll Efficiently handles long connections
<span>multi_accept</span> Multi-connection acceptance Low off on Enhances connection acceptance efficiency

Failover Parameters (SSE Stability Related)

Parameter Function Importance Default Value Recommended Value SSE Relevance
<span>proxy_next_upstream</span> Failover conditions High – error timeout Switch when SSE connection fails
<span>proxy_next_upstream_tries</span> Retry count Medium 0 3 SSE connection retry mechanism
<span>proxy_next_upstream_timeout</span> Retry timeout Medium 0 10s SSE retry time limit
<span>max_fails</span> Maximum failure count Medium 1 3 Backend server health check
<span>fail_timeout</span> Failure timeout Medium 10s 30s Server recovery time

SSE Specific Configuration Template

Basic SSE Configuration

location /sse {
    proxy_pass &lt;http://backend&gt;;

    # 🔴 Core SSE Parameters
    proxy_buffering off;
    proxy_cache off;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;

    # 🟠 Streaming Parameters
    add_header Cache-Control "no-cache";
    add_header X-Accel-Buffering "no";
    add_header Content-Type "text/event-stream";
    add_header Connection "keep-alive";
}

High-Performance SSE Configuration

# Global Optimization
worker_processes auto;
worker_rlimit_nofile 65535;

events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    # Connection Optimization
    keepalive_timeout 300s;
    keepalive_requests 1000;

    # Transmission Optimization
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    # SSE Specific Location
    location /sse {
        proxy_pass http://sse_backend;

        # Core SSE Parameters
        proxy_buffering off;
        proxy_cache off;
        proxy_http_version 1.1;
        proxy_set_header Connection '';
        proxy_read_timeout 86400s;  # 24 hours
        proxy_send_timeout 86400s;

        # Streaming Headers
        add_header Cache-Control "no-cache";
        add_header X-Accel-Buffering "no";
        add_header Content-Type "text/event-stream";
        add_header Connection "keep-alive";

        # Failover
        proxy_next_upstream error timeout;
        proxy_next_upstream_tries 3;
        proxy_next_upstream_timeout 10s;
    }
}

Nginx Plus: GPU Sticky Sessions + Gray Release

In production environments, LLM services face two core challenges: how to ensure session consistency and how to achieve zero-downtime updates. Nginx Plus provides a complete solution.

GPU Sticky Sessions: Ensuring Session Consistency

Background of the ProblemIn LLM dialogues, the user’s contextual information needs to remain consistent. If different requests from the same dialogue are routed to different GPU nodes, it can lead to context loss, affecting dialogue quality.

Sticky Cookie Solution

upstream vllm_inference_pool {
    # 8 A100 GPU nodes
    server 192.168.1.101:8000 weight=1;
    server 192.168.1.102:8000 weight=1;
    server 192.168.1.103:8000 weight=1;
    server 192.168.1.104:8000 weight=1;
    server 192.168.1.105:8000 weight=1;
    server 192.168.1.106:8000 weight=1;
    server 192.168.1.107:8000 weight=1;
    server 192.168.1.108:8000 weight=1;

    # Key: Session stickiness based on chat_id
    sticky cookie chat_session expires=2h domain=.yourdomain.com path=/;
}

server {
    location /v1/chat/completions {
        proxy_pass http://vllm_inference_pool;

        # Pass chat_id to backend
        proxy_set_header X-Chat-ID $cookie_chat_session;
        proxy_set_header X-User-ID $arg_user_id;
    }
}

How It Works

  1. 1. On the first request, Nginx assigns a unique <span>chat_session</span> cookie to the client
  2. 2. Based on the cookie value, a hash is calculated to determine the target GPU node
  3. 3. Subsequent requests carry the same cookie, always routed to the same node
  4. 4. Ensures dialogue context is maintained on the same GPU

Gray Release: Zero-Downtime Updates

ChallengeModels iterate weekly, and updates need to occur without affecting ongoing user dialogues. Traditional rolling updates can interrupt dialogues that are currently being generated.

Dynamic Weight Adjustment Solution

# Define weight storage area
keyval_zone zone=server_weights:1M state=/var/lib/nginx/weights.json;
keyval $server_name $weight zone=server_weights;

# Dynamic weight upstream
upstream vllm_inference_pool {
    server 192.168.1.101:8000 weight=$weight_101;
    server 192.168.1.102:8000 weight=$weight_102;
    server 192.168.1.103:8000 weight=$weight_103;
    server 192.168.1.104:8000 weight=$weight_104;
    server 192.168.1.105:8000 weight=$weight_105;
    server 192.168.1.106:8000 weight=$weight_106;
    server 192.168.1.107:8000 weight=$weight_107;
    server 192.168.1.108:8000 weight=$weight_108;

    sticky cookie chat_session expires=2h domain=.yourdomain.com path=/;
}

Gray Release Process

#!/bin/bash
# 1. Deploy new version to 2 nodes
kubectl rollout restart deployment/vllm-v1.1 -n llm-prod

# 2. Wait for the new version to be ready
kubectl wait --for=condition=available deployment/vllm-v1.1 -n llm-prod --timeout=300s

# 3. Gradually adjust weights (complete within 1 second)
curl -X POST "&lt;http://localhost/nginx-api/upstream_conf?upstream=vllm_inference_pool&amp;server=192.168.1.107:8000&amp;weight=1&gt;"# New version
curl -X POST "&lt;http://localhost/nginx-api/upstream_conf?upstream=vllm_inference_pool&amp;server=192.168.1.108:8000&amp;weight=1&gt;"# New version

# 4. Monitor new version performance
sleep 60

# 5. If performance is normal, continue to increase new version traffic
if check_performance_metrics; then
    echo "Performance OK, increasing new version traffic..."
    # Continue adjusting weights...
fi

Seamless Update Principle

  1. 1. Session Persistence: Ongoing dialogues continue using old version nodes
  2. 2. Traffic Switching: New requests can be routed to new version nodes
  3. 3. Rapid Adjustment: Weight adjustments completed within 1 second, no restart required
  4. 4. Monitoring Rollback: Immediate rollback if issues are detected

Performance Data

Metric Value Description
Peak QPS 25,000 Processing 25,000 query requests per second
P99 Latency 280ms 99% of requests have latency within 280 milliseconds
First Token Latency <100ms First token generation time is less than 100 milliseconds
Downtime 0 No service interruption during model updates

Technical Advantages

Advantage Category Specific Features Technical Implementation
Session Consistency Same dialogue always on the same GPU Sticky Cookie + Hash Binding
Zero-Downtime Updates User completely unaware Dynamic Weight Adjustment + Gray Release
High Performance Multiplexing + Connection Reuse HTTP/2 + gRPC Streaming
High Availability Failover + Health Checks Automatic Fault Detection + Load Balancing

Conclusion

SSE, as the foundational protocol for streaming output, provides LLM with real-time push capabilities, allowing users to see the generation process token by token.

HTTP/2 and gRPC solve performance bottlenecks in high-concurrency scenarios through multiplexing technology, achieving efficient transmission of multiple dialogue streams over a single connection.

Nginx, as a unified gateway, not only provides basic support for SSE but also ensures high availability of services through load balancing, health checks, and other functions.

This technology stack not only addresses the core needs of LLM services but also provides a replicable and scalable solution for large-scale applications in the AI era. SSE, as the foundational protocol for streaming output, will maintain its importance for a long time to come.

Related posts

  1. Abandoning fasthttp Client: A Case Study
  2. How to Recharge IoT Gas Cards? Simple Steps to Eliminate Payment Worries!
  3. New HTTP Smuggling Attack Technique Allows Attackers to Inject Malicious Requests
  4. Spring MVC: The Ultimate Guide to HTTP Mastery from Bronze to King
  5. The Unorthodox Techniques of FPGA Development
  6. Introduction to Python
  7. Factors Influencing Myocardial Injury After Surgery in Sepsis Patients
  8. HTTPX: The Next-Generation HTTP Client in Python, More Powerful than Requests!
  9. Out-of-the-Box: NTLM Relaying Low-Privilege HTTP Authentication to LDAP
  10. HooRii Technology Launches Matter + BLE Dual-Mode Protocol Solution to Facilitate Seamless Compatibility and Upgrades for Smart Devices
  11. Is There Hope for SaaS Companies in the Age of AI Agents?

Leave a Comment