Abandoning fasthttp Client: A Case Study

Recently, I developed a web service with strict latency requirements that needed to call external HTTPS services. The maximum allowed latency was 50ms, with a maximum QPS of about 1000 (the deployed machine was configured with 4 CPU cores, 8GB of memory, and an outbound bandwidth of 12Mbps). Since github.com/valyala/fasthttp claims that its client is 4 times faster than net/http, the initial technical selection was to use the fasthttp client instead of the net/http client.

Related Implementation Code

Connection Pool Configuration

func createFastHTTPClient() *fasthttp.Client {
    // Create a custom dialer and configure TCP parameters
    dialer := &fasthttp.TCPDialer{
        Concurrency:      1000,             // Maximum number of concurrent connections
        DNSCacheDuration: 10 * time.Minute, // DNS cache duration
    }

    return &fasthttp.Client{
        // Core performance parameters for the connection pool
        MaxConnsPerHost:     500,        // Maximum number of connections per host (active + idle)
        MaxIdleConnDuration: 30 * time.Second, // Idle connection retention time

        // Read and write timeouts (will be overridden by DoTimeout as default)
        ReadTimeout:  5 * time.Second,
        WriteTimeout: 5 * time.Second, // Write timeout, also serves as handshake timeout

        // TLS configuration
        TLSConfig: &tls.Config{
            InsecureSkipVerify: true,
            // ClientSessionCache: tls.NewLRUClientSessionCache(1000),
        },

        // Connection reuse optimization
        DisableHeaderNamesNormalizing: true, // Improve performance
        DisablePathNormalizing:        true, // Improve performance

        // Custom dialer - configure TCP low-level parameters
        Dial: func(addr string) (net.Conn, error) {
            conn, err := dialer.Dial(addr)
            if err != nil {
                return nil, err
            }

            // Set TCP parameters to reduce latency
            if tcpConn, ok := conn.(*net.TCPConn); ok {
                tcpConn.SetNoDelay(true)   // Disable Nagle's algorithm
                tcpConn.SetKeepAlive(true) // Enable TCP keep-alive
                tcpConn.SetKeepAlivePeriod(30 * time.Second)

                // Set socket options (Linux specific)
                setSocketOptions(tcpConn)
            }

            return conn, nil
        },
    }
}

HTTP Layer Keep-Alive Mechanism

Since the fasthttp client establishes connections and performs HTTPS handshakes only during business calls, to minimize latency during these calls, in addition to the 30-second periodic keep-alive at the TCP layer, an additional HTTP layer keep-alive was implemented every 15 seconds:

func keepAliveBackend(url string) {
    req := fasthttp.AcquireRequest()
    resp := fasthttp.AcquireResponse()
    defer func() {
        fasthttp.ReleaseRequest(req)
        fasthttp.ReleaseResponse(resp)
    }()

    // Set keep-alive request (using HEAD method to reduce overhead)
    req.SetRequestURI(url)
    req.Header.SetMethod("HEAD")

    var httpClient = getFastHTTPClient()
    err := httpClient.DoTimeout(req, resp, 2*time.Second)
    if err != nil {
        glog.WarnS("fasthttp_client", "Keep-alive request failed - URL: %s, Error: %v\n", url, err)
        return
    }

    // Must read and discard the response body to ensure the connection is reusable
    io.Copy(io.Discard, bytes.NewReader(resp.Body()))
}

Timeout Control

When making HTTP requests, the specific timeout values override the connection pool’s ReadTimeout and WriteTimeout (5 seconds) configuration:

err := httpClient.DoTimeout(req, resp, currentTimeout)

Issue Discovery and Analysis

During traffic gray testing, a critical issue was discovered: when the traffic approached the external bandwidth limit, DoTimeout did not return as expected within the currentTimeout time, sometimes returning in 5 seconds, and other times in 3 seconds.

By reading the source code, I found:

HTTPS Handshake Timeout: The HTTPS handshake timeout in fasthttp uses the connection pool’s WriteTimeout (5 seconds in this example)

func dialAddr(
    addr string, dial DialFunc, dialWithTimeout DialFuncWithTimeout, dialDualStack, isTLS bool,
    tlsConfig *tls.Config, dialTimeout, writeTimeout time.Duration,
) (net.Conn, error) {
    deadline := time.Now().Add(writeTimeout)
    conn, err := callDialFunc(addr, dial, dialWithTimeout, dialDualStack, isTLS, dialTimeout)
    if err != nil {
        return nil, err
    }
    if conn == nil {
        return nil, errors.New("dialling unsuccessful. Please report this bug")
    }

    // Omitted part of the code...

    if isTLS && !isTLSAlready {
        if writeTimeout == 0 {
            return tls.Client(conn, tlsConfig), nil
        }
        return tlsClientHandshake(conn, tlsConfig, deadline)
    }
    return conn, nil
}

Connection Timeout: The connection timeout is hardcoded to 3 seconds

func(d *TCPDialer) Dial(addr string) (net.Conn, error) {
    return d.dial(addr, false, DefaultDialTimeout)
}

This is the fundamental reason for the instability of the timeout (3 seconds or 5 seconds).

Attempts to Solve and New Problems

To achieve strict timeout control, I considered using additional goroutines in conjunction with context timeouts, as shown in the following code example:

timeout := remainingTimeout + MaxExecuteTimeoutPlus
// Create a context with timeout
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeout)*time.Millisecond)
defer cancel() // Ensure resources are released

// Create a channel to receive results
type result struct {
    res []string
    err error
}
ch := make(chan result, 1)

// Execute in a goroutine
go func() {
    // HTTPS interface call
    res, err := productSvc.Execute(time.Duration(remainingTimeout) * time.Millisecond)
    ch &lt;- result{res, err}
}()

// Wait for result or timeout
select {
case &lt;-ctx.Done():
    // Timeout case
    return nil, ctx.Err()
case ret := &lt;-ch:
    // Normal return result
    return ret.res, ret.err
}

However, this approach led to more severe issues:

Additional goroutines reduced performance
Goroutines continued running after timeout, continuously occupying bandwidth resources
When traffic approached the bandwidth limit, all 500 connections in the connection pool were filled, preventing handshakes and connections from being established, ultimately leading to complete service failure

Conclusion

Ultimately, due to the inability to effectively resolve the above issues, the fasthttp client was abandoned in favor of the net/http client.

#fasthttp #http client #golang #web server