Who ‘Killed’ Your HTTP Connection?

Hello everyone, I am Tony Bai.

Have you encountered intermittent errors in a production environment such as <span>EOF</span>, <span>connection reset by peer</span>, or <span>unexpected end of stream</span>? Have you checked the code logic, firewall rules, or even captured packets, only to find that everything is normal at the application layer, yet requests occasionally fail? The most puzzling part is that this often happens in low-frequency request scenarios or when the system just “wakes up” from an idle state.

Many developers—whether writing for Android or Go—often limit their focus to the code logic layer. However, in the cloud-native era, application code is just one part of a vast network link. This article will use a real cross-cloud communication failure as a starting point to delve into the mechanism of the HTTP connection pool’s <span>Idle Timeout</span>, providing best practice configurations using the Go language as an example.

The Scene of the Incident: A “Ghost” Error

Recently, while troubleshooting a cross-cloud call failure, we discovered a classic phenomenon:

• Client: An application running in a container, using OkHttp’s HTTP connection pool (Keep-Alive).
• Server: A SaaS service deployed on a public cloud, with a load balancer (LB) mounted at the front end.
• Phenomenon: Intermittent network request failures, reporting <span>unexpected end of stream</span>.
• Troubleshooting: The client SNAT is set with a TCP keep-alive time of up to 1 hour, and the network link is very stable. However, the server logs show “no request received”.

The truth is: the connection was “silently” closed.

Under the HTTP Keep-Alive mechanism, for performance reasons, the client reuses idle TCP connections. However, each connection must pass through a complex network link: client -> NAT gateway -> Internet -> load balancer (LB) -> server.

This is a typical “barrel effect”: The effective lifespan of a connection depends on the shortest timeout among all nodes in the link.

If the client’s connection pool believes the connection can live for 300 seconds (the default value for OkHttp), while the intermediate cloud vendor’s LB is configured with an Idle Timeout of 60 seconds:

1. The connection is idle until the 61st second, and the LB silently cuts off the connection.
2. The client is unaware (because no packets were sent, it may not have received FIN/RST, or it received them but did not process them).
3. At the 100th second, the client reuses this “zombie connection” to send a request, hitting a wall and reporting EOF.

Default “Traps” in Go Language

In Go, the <span>net/http</span> standard library provides very powerful connection pool management, mainly controlled by the <span>http.Transport</span> struct. However, Go’s default configuration is not always safe in modern cloud environments.

Let’s take a look at the source code snippet of Go (1.25.3) for <span>DefaultTransport</span>:

var DefaultTransport RoundTripper = &Transport{
    Proxy: ProxyFromEnvironment,
    DialContext: defaultTransportDialContext(&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second, // TCP layer KeepAlive probe interval
    }),
    ForceAttemptHTTP2:     true,
    MaxIdleConns:          100,
    IdleConnTimeout:       90 * time.Second, // <--- The key point is here!
    TLSHandshakeTimeout:   10 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}

Note that <span>IdleConnTimeout: 90 * time.Second</span>.

This means that the Go HTTP client will keep idle connections for 90 seconds.

Conflict Eruption Point

What is the default <span>Idle Timeout</span> of mainstream public cloud load balancers (AWS ALB, Alibaba Cloud SLB, Google LB, etc.)?

• AWS ALB: Default is 60 seconds.
• Alibaba Cloud SLB: Default is 60 seconds (TCP listening may differ, but HTTP/7 layer is usually shorter).
• Nginx (default): <span>keepalive_timeout</span> is often set to 65 seconds or 75 seconds.

The risk is obvious: The Go client believes the connection is available between 60 and 90 seconds, but the cloud LB has already killed it at the 60-second mark. This leads to a 30-second window where reusing the connection is bound to fail.

The Golden Rule: Connection Pool Configuration Guide

To completely solve this problem, developers (whether in Go, Java, or Node.js) must follow a core configuration principle:

Client Idle Timeout < Infrastructure Idle Timeout < Server KeepAlive Timeout

The client’s idle timeout must be shorter than the timeout of any intermediate devices in the link (LB, NAT, Firewall).

It is recommended to set the client’s idle timeout to the intermediate device timeout minus 5-10 seconds as a safety buffer. For most public cloud environments, 30 seconds to 45 seconds is an extremely safe value.

Go Practical: How to Properly Configure http.Client

Do not directly use <span>http.Get()</span><span> </span>or <code><span>&http.Client{}</span><span> (they use the default Transport). In production-level code, you should always explicitly define </span><code><span>Transport</span>.

Recommended Configuration Example

package main

import (
    "net"
    "net/http"
    "time"
)

func NewProductionHttpClient() *http.Client {
    // Custom Transport
    t := &http.Transport{
        // 1. Optimize dialing logic
        DialContext: (&net.Dialer{
            Timeout:   5 * time.Second,  // Connection establishment timeout, do not set too long
            KeepAlive: 30 * time.Second, // TCP layer probe to prevent dead connections
        }).DialContext,

        // 2. Core configuration of the connection pool
        // The key here is: IdleConnTimeout must be less than the timeout of the cloud vendor's LB (usually 60s)
        // Setting it to 30s is a relatively safe choice
        IdleConnTimeout:       30 * time.Second, 
        
        // Control the maximum number of connections to prevent local resource exhaustion
        MaxIdleConns:          100, 
        MaxIdleConnsPerHost:   10,   // Adjust based on your concurrency, default is 2, too small will lead to frequent connection creation and destruction

        TLSHandshakeTimeout:   5 * time.Second, // TLS handshake timeout
        ResponseHeaderTimeout: 10 * time.Second, // Wait for response header timeout
    }

    return &http.Client{
        Transport: t,
        // Global request timeout, including connection + read/write, as a fallback
        Timeout: 30 * time.Second, 
    }
}

Key Parameter Explanation

1. <span>IdleConnTimeout</span> (most important):

• Meaning: How long a connection is allowed to remain idle after being returned to the connection pool.
• Recommendation: 30s – 45s. This ensures that the client actively closes the connection rather than passively waiting for the server to send RST, thus avoiding reusing “stale connections”.

<span>MaxIdleConnsPerHost</span>:

• Meaning: For the same target host, how many idle connections can be retained in the connection pool at most. Go’s default value is 2.
• Pitfall: In high concurrency scenarios of microservices, the default value <span>2</span> is too small. This will lead to a large number of connections being created when requests come in concurrently, and only 2 can return to the pool after processing, while the rest are all closed. When the next concurrent request comes in, it has to re-establish the handshake.
• Recommendation: Estimate based on your QPS, usually recommended to set it to 10 ~ 50 or even higher.

<span>DisableKeepAlives</span>:

• For debugging: If you really can’t resolve network issues, you can set it to <span>true</span>, forcing short connections (close after use). However, this will significantly reduce performance and should only be used for troubleshooting.

The Last Line of Defense: Retry Mechanism

Even if you have configured the perfect timeout, network jitter is still inevitable. Connection pool configuration can only reduce the probability of <span>stale connections</span>, not eliminate it 100%.

For idempotent requests (such as GET, PUT, DELETE), the application layer must have a retry mechanism.

The Go standard library <span>net/http</span> does not automatically retry by default. You can use excellent open-source libraries like hashicorp/go-retryablehttp, or implement a simple retry logic yourself:

// Simple retry logic pseudo code
var err error
for i := 0; i < 3; i++ {
    resp, err = client.Do(req)
    if err == nil {
        return resp, nil
    }
    // Only specific errors should be retried, such as connection reset
    if isConnectionReset(err) {
        continue
    }
    break
}

Summary

Infrastructure as Code does not mean that your code can ignore the physical limitations of Infrastructure.

Regarding the HTTP connection pool, remember these three points:

1. Do not trust default values: OkHttp’s 5 minutes, Go’s 90 seconds, are all hazards in front of a 60-second timeout public cloud LB.
2. Proactively show weakness: The client’s idle timeout must be shorter than the server and intermediate gateway.It is always safer for the client to actively reclaim connections than to be forcibly cut off by the server.
3. Embrace failure: Configuring a reasonable retry strategy is a must for building robust distributed systems.

The next time you encounter <span>unexpected end of stream</span><span>, don't rush to doubt life; check your </span><code><span>IdleTimeout</span><span> settings!</span>

If this article has helped you, please like, recommend, and share!

Click the title below to read more valuable content!

– Just knowing net/http is not enough; are you brave enough to venture into the “deep waters” of Go network programming?

– Detailed control of http.Client’s connection behavior

– The official Go HTTP/3 implementation finally sees the light: x/net/http3 proposal launched, QUIC foundation is in place

– Understanding how the Go standard library http package handles keep-alive connections through examples

– [Complete Guide to Go Network Programming] 13 From HTTP/1.1 to gRPC: The evolution of Web APIs and microservices

– WebRTC Lesson 1: Network architecture and NAT working principles

– Exploring the allocation and filtering behavior of Docker’s default network NAT mapping

– WebRTC Lesson 1: The full process of connection establishment from signaling, ICE to NAT traversal

🔥 Still troubled by “copy-pasting to feed AI”? My new Geek Time column “Practical Workflow for AI Native Development” will take you:

• Say goodbye to inefficiency and reshape the development paradigm
• Master AI Agent (Claude Code) to achieve workflow automation
• Evolve from an “AI user” to a “workflow conductor” driven by specifications

Scan the QR code below to start your AI native development journey.