01 Introduction

Before Java 11, the HttpURLConnection feature provided by the standard library was quite basic.

It was difficult to handle complex HTTP scenario requirements, such as connection pool management, asynchronous requests, and flexible retries.

Apache HttpClient, as one of the most mature and comprehensive HTTP client libraries in the Java ecosystem, has been widely used in enterprise-level development for a long time.

This article will systematically explain the technical details of Apache HttpClient, from design and basic usage to practical cases.

02 Introduction to Apache HttpClient

Apache HttpClient is an open-source HTTP client library maintained by the Apache Software Foundation.

It fully supports the HTTP/1.1 protocol and provides users with a series of powerful core capabilities.

This makes it an ideal choice for developing efficient and reliable network applications.

Its core capabilities are as follows:

• • Connection Pool Management: Apache HttpClient has a built-in efficient connection pool management mechanism.

It can reuse TCP connections, significantly reducing the handshake overhead of establishing and closing connections.

Through the connection pool, the client can share connections between multiple requests.

This not only improves performance but also reduces resource consumption.

• • Flexible Request Configuration: This library provides rich configuration options.

It supports setting request timeouts, configuring proxy servers, managing redirection policies, handling cookies, and implementing authentication, among others.

These configuration options allow HttpClient to flexibly respond to various complex network environments and request requirements.

• • Extensibility: Apache HttpClient allows users to customize logging, request retries, response caching, and other logic through an interceptor mechanism.

The interceptor mechanism gives HttpClient a high degree of extensibility, making it easy to integrate into various complex systems.

• • Support for Synchronous/Asynchronous Requests: HttpClient is compatible with the traditional blocking I/O model.

It allows users to initiate synchronous requests and wait for responses, and also provides support for asynchronous non-blocking requests (in conjunction with an asynchronous HTTP client).

The support for asynchronous requests allows HttpClient to handle a large number of concurrent requests while maintaining application responsiveness and scalability.

When choosing a version of HttpClient, users should weigh their specific project needs, target Java version, and requirements for new features.

The version selection is as follows:

• • HttpClient 4.x: This is a widely used stable version.

HttpClient 4.x has been thoroughly tested and validated, suitable for most application scenarios.

• • HttpClient 5.x: This is the next-generation API.

HttpClient 5.x has made several improvements in performance and functionality, and supports the HTTP/2 protocol (requires Java 8 or higher).

03 Basic Usage of HttpClient

Adding Dependencies

To use Apache HttpClient in a Java project, you need to add the corresponding dependency in the project’s build file.

The Maven configuration is as follows (using version 4.5.13 as an example):

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.13</version>
</dependency>

Creating an HttpClient Instance

Creating a Default Instance:

To send HTTP requests, you need to create a CloseableHttpClient instance.

You can directly obtain a pre-configured HttpClient instance using the HttpClients.createDefault() method.

This instance uses the following default parameters:

// Create a CloseableHttpClient instance with default configuration
CloseableHttpClient httpClient = HttpClients.createDefault();

Non-default Configured Instance:

The following example shows how to customize the configuration of CloseableHttpClient using HttpClientBuilder.

It overrides key parameters such as connection timeout, proxy, retry strategy, and SSL certificate validation:

import org.apache.http.HttpHost;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.StandardHttpRequestRetryHandler;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.ssl.SSLContextBuilder;
import javax.net.ssl.SSLContext;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.util.UUID;
import java.util.concurrent.TimeUnit;

public class ApacheHttpClientExample {
    // Custom HttpClient configuration method
    public static CloseableHttpClient createCustomHttpClient() throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
        // 1. Configure connection pool
        PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
        connManager.setMaxTotal(200);          // Set maximum total connections in the pool
        connManager.setDefaultMaxPerRoute(50); // Set maximum concurrent connections per route (target host)
        // Close idle connections after 60 seconds of inactivity
        connManager.closeIdleConnections(60, TimeUnit.SECONDS);

        // 2. Configure timeout parameters
        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(5000)    // Connection establishment timeout (5 seconds)
                .setSocketTimeout(15000)    // Data transfer timeout (15 seconds)
                .setConnectionRequestTimeout(3000) // Timeout for obtaining a connection from the pool (3 seconds)
                .build();

        // 3. Configure SSL (trust self-signed certificates)
        SSLContext sslContext = SSLContextBuilder.create()
                .loadTrustMaterial(new TrustSelfSignedStrategy()) // Trust self-signed certificates
                .build();
        SSLConnectionSocketFactory sslSocketFactory = new SSLConnectionSocketFactory(
                sslContext,
                new String[]{"TLSv1.2", "TLSv1.3"}, // Supported TLS protocol versions
                null,
                SSLConnectionSocketFactory.getDefaultHostnameVerifier() // Default hostname verifier
        );

        // 4. Configure retry strategy
        StandardHttpRequestRetryHandler retryHandler = new StandardHttpRequestRetryHandler(
                3,      // Maximum retry count is 3
                true    // Retry for non-idempotent requests (like POST) as well (business must ensure safety)
        );

        // 5. Build and return the custom HttpClient
        return HttpClientBuilder.create()
                .setConnectionManager(connManager)          // Set connection pool
                .setDefaultRequestConfig(requestConfig)     // Set default timeout configuration
                .setSSLSocketFactory(sslSocketFactory)      // Set SSL configuration
                .setRetryHandler(retryHandler)              // Set retry strategy
                .setProxy(new HttpHost("proxy.example.com", 8080)) // Set proxy server
                .disableCookieManagement()                  // Disable cookie management (enable as needed)
                .setUserAgent("Custom-Client/1.0")          // Custom User-Agent header
                // Add custom request interceptor to add unique request ID
                .addInterceptorFirst((HttpRequestInterceptor) (request, context) ->
                        request.addHeader("X-Request-ID", UUID.randomUUID().toString()))
                .build();
    }

    public static void main(String[] args) {
        try {
            // Create custom configured HttpClient
            CloseableHttpClient httpClient = createCustomHttpClient();
            // Use httpClient for HTTP requests (subsequent steps)
            // ...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Sending GET Requests

Sending GET requests is one of the most common usages of HttpClient.

The following is an example of sending an HTTP GET request using the Apache HttpClient library:

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class ApacheHttpClientExample {
    public static void main(String[] args) {
        // Create HttpClient using try-with-resources to ensure resources are automatically closed
        try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
            // Create HttpGet request object, specifying the target URL
            HttpGet request = new HttpGet("https://jsonplaceholder.typicode.com/posts/1");
            // Execute request and get response (also using try-with-resources to ensure response is automatically closed)
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                // Print HTTP response status code
                System.out.println("Status Code: " + response.getStatusLine().getStatusCode());
                // Convert response entity to string and print
                String responseBody = EntityUtils.toString(response.getEntity());
                System.out.println("Response Body: " + responseBody);
            }
        } catch (Exception e) {
            // Catch and print exceptions
            e.printStackTrace();
        }
    }
}

This example creates a CloseableHttpClient instance in the main method using a try-with-resources statement.

This instance is responsible for sending HTTP requests and creates an HttpGet request object pointing to https://jsonplaceholder.typicode.com/posts/1 (an API providing fake data).

Then, it again uses a try-with-resources statement to execute the request and obtain a CloseableHttpResponse response object.

Within the scope of the response object, the program prints the HTTP response status code.

It also uses the EntityUtils.toString method to convert the response entity to a string and finally prints the response body content.

If an exception occurs during the request execution or response handling, it will be caught and printed to standard error output.

Sending POST Requests

Sending POST requests is similar to GET requests, except that data to be sent must be included in the request body.

Typically, POST requests are used to submit form data or JSON data.

The following example demonstrates how to send an HTTP POST request containing a JSON request body using the Apache HttpClient library in Java and handle the response.

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;

public class ApacheHttpClientExample {
    public static void main(String[] args) {
        // Create HttpClient using try-with-resources to automatically close resources
        try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
            // Create HttpPost request object, specifying the target URL
            HttpPost request = new HttpPost("https://jsonplaceholder.typicode.com/posts");
            // Define JSON formatted request body content
            String json = "{\"title\":\"foo\",\"body\":\"bar\",\"userId\":1}";
            // Set request body, specifying content type as JSON
            request.setEntity(new StringEntity(json, ContentType.APPLICATION_JSON));
            // Add custom request header
            request.addHeader("X-Custom-Header", "value");
            // Execute request and get response (automatically closing response resource)
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                // Print response status code
                System.out.println("Status Code: " + response.getStatusLine().getStatusCode());
                // Convert response entity to string and print
                String responseBody = EntityUtils.toString(response.getEntity());
                System.out.println("Response Body: " + responseBody);
            }
        } catch (Exception e) {
            // Catch and handle exceptions
            e.printStackTrace();
        }
    }
}

In this method, a CloseableHttpClient instance is created using a try-with-resources statement to ensure resources are properly closed.

Then, an HttpPost request object is created pointing to the target URL, with a JSON formatted request body and corresponding content type set.

A custom request header is also added, and the request is executed while capturing the response.

The program prints the HTTP response status code and response body content.

Running result:

Status Code: 201
Response Body: {"title":"foo","body":"bar","userId":1,"id":101}

04 Advanced Configuration and Practical Cases of HttpClient

Connection Pool Optimization

In high-performance HTTP client applications, managing the connection pool is crucial.

The Apache HttpClient library provides a powerful PoolingHttpClientConnectionManager tool.

It allows developers to efficiently manage and reuse HTTP connections, significantly improving application performance and resource utilization.

An example of managing the connection pool using PoolingHttpClientConnectionManager is as follows:

import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;

public class ApacheHttpClientExample {
    public static void main(String[] args) {
        // Create connection pool manager
        PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
        connManager.setMaxTotal(200);          // Set maximum total connections in the pool
        connManager.setDefaultMaxPerRoute(20); // Set maximum connections per route (target host)

        // Create HttpClient based on connection pool
        try (CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(connManager)
                .build()) {
            // Execute high concurrency requests (subsequent implementation)...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Timeout and Retry Configuration

When building high-performance, highly reliable HTTP client applications, timeout and retry configurations are two key aspects.

They not only affect the application’s response speed but also directly relate to the application’s stability and user experience.

An example of setting timeout and retry configurations is as follows:

import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;

public class ApacheHttpClientExample {
    public static void main(String[] args) {
        // Configure timeout parameters
        RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(5000)    // Connection timeout: 5 seconds
                .setSocketTimeout(10000)    // Data transfer timeout: 10 seconds
                .build();

        // Build HttpClient, configure timeout and retry strategy
        HttpClientBuilder builder = HttpClients.custom()
                .setDefaultRequestConfig(requestConfig)
                // Custom retry strategy: retry up to 3 times, support retry for non-idempotent requests
                .setRetryHandler(new DefaultHttpRequestRetryHandler(3, true));

        // Create HttpClient and use it
        try (CloseableHttpClient httpClient = builder.build()) {
            // Execute request (subsequent implementation)...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

File Upload (Multipart)

To use Apache HttpClient for file uploads in a Java project, you need to add the corresponding dependency in the project’s build file.

The Maven configuration is as follows (using version 4.5.13 as an example):

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpmime</artifactId>
    <version>4.5.13</version>
</dependency>

The following is an example of sending an HTTP POST request containing file upload using Apache HttpClient:

import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

import java.io.File;

public class ApacheHttpClientExample {
    public static void main(String[] args) {
        // Create HttpClient (automatically closing resources)
        try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
            // Create POST request object for file upload
            HttpPost request = new HttpPost("https://example.com/upload");

            // Build Multipart type request body
            MultipartEntityBuilder builder = MultipartEntityBuilder.create();
            // Add the file to upload (form field name "file")
            builder.addPart("file", new FileBody(new File("test.txt")));
            // Add text parameter (form field name "comment")
            builder.addTextBody("comment", "Sample File");

            // Set the built entity to the request
            request.setEntity(builder.build());

            // Execute request and get response (automatically closing response)
            try (CloseableHttpResponse response = httpClient.execute(request)) {
                // Handle response (subsequent implementation, such as getting status code, response body, etc.)...
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Conclusion

Apache HttpClient has been the preferred solution for handling complex HTTP communications in versions of Java prior to 11.

With flexible configurations, efficient connection pool management, and rich extensibility, it can meet the high-performance needs of enterprise applications.

However, with the popularity of the standard library HttpClient in Java 11+, new projects are recommended to prioritize using the native solution, while legacy systems can still rely on the stability and mature ecosystem of Apache HttpClient.

Recommended Articles:

1. Explanation of 100 Java Interview Questions in Markdown Format

2.CICD+Docker+Dockerfile Three Major Series 37 Articles Systematic Explanation