Super Simple: Implementing an NIO HTTP Client in Java with 100 Lines of Code and No Third-Party Dependencies

This article was published on Ruzhila (WeChat Official Account: ruzhila). Everyone can visit Ruzhila to learn more about practical programming.

🎉 Implement an NIO HTTP client in Java with just 100 lines of code. The code is simple and clear, helping you understand the working principle of NIO and the completely asynchronous HTTP protocol parsing process.

Project Address

The code is open source at https://github.com/ruzhila/java-nio-http-download. 👏 Feel free to Star it!

All projects are open source on GitHub: https://github.com/ruzhila/100-line-code. Welcome to Star! 👏

Implementing the project in 100 lines of different languages (Java, Python, Go, JavaScript, Rust) helps everyone learn programming through project explanations.

We will regularly share the latest practical project codes in the group, including implementations in different languages.

The instructor will also explain code optimization ideas in detail. Scan the code to join the practical group:

Working Principle of NIO

In most cases, our IO calls are blocking calls. For example, when you call Java’s URLConnection, it will wait indefinitely for the server to return data before continuing to execute the subsequent code:

URL url = new URL("http://example.org");URLConnection connection = url.openConnection();InputStream inputStream = connection.getInputStream();....

This code is a typical blocking IO call. When we call connection.getInputStream(), the program will wait for the server to return data before continuing to execute the subsequent code.

This approach is simple and easy to understand, but there is a problem: each IO request logic requires a separate thread. When there are a large number of IO requests, it can lead to thread resource exhaustion and decreased program performance.

Non-blocking IO was created to solve this problem. It is an asynchronous IO solution provided by Java. With NIO, we can handle multiple IO requests with a single thread, improving program performance:

This is a typical program structure diagram of an NIO program. We can see that the workflow of NIO is as follows:

(1) Selector listens to all Channels through the select() method. When events such as a Channel being readable, writable, or a new connection occur, the Selector will return these events.
(2) The system calls the select method, and the kernel’s Socket will be monitored. When an event occurs, the Socket with the event will be placed into the result set of select.
(3) Iterate through the result set of select to handle events, such as writing data.
(4) Call the write method to write data into the Socket.
(5) The kernel’s Socket will send the data over the network.

Selector and Channel

In NIO, we mainly use Selector and Channel to implement asynchronous IO.

Selector is a multiplexer that can listen to events from multiple Channels simultaneously. When an event occurs, Selector will return these events, which can be read and written through the Channel:

Selector selector = Selector.open();SocketChannel channel = SocketChannel.open();channel.configureBlocking(false);channel.register(selector, SelectionKey.OP_CONNECT);while(true) {    selector.select();    Set<SelectionKey> keys = selector.selectedKeys();    for(SelectionKey key: keys) {        if(key.isConnectable()) {            // do something        }    }}

By registering the OP_CONNECT event with the selector, when the channel successfully connects, we can handle the connection success event.

Let’s Get to the Code

Code Analysis

We implemented a class called NioHTTPClient, which provides several methods:

sendRequest, to create a connection and send an HTTP request.
HTTPResponseListener, since the NIO client can only obtain data through callbacks, we use this interface to retrieve data.

onResponse will be called when there is a response to the request.
onData will be called when the body data starts to return.

After introducing NioHTTPClient, let’s take a look at how the overall program works:

Create a Selector object.
Create a NioHTTPClient object.
Call the sendRequest method to send the HTTP request.
Continuously call the selector.select() method to listen for events. If an event occurs, call the corresponding methods of NioHTTPClient such as onCanWrite and onCanRead to process data.

`NioHTTPClient` Connection Creation and Request Sending Process:

Lines 27-31 sendRequest creates a connection and initiates the socket’s connect operation.
Line 40 waits for the connection success event.
Line 43 when the Selector detects that this SocketChannel has successfully connected, it calls the onConnect function.

Line 46 tells the Selector to listen for writable events.
Lines 50-54 when the next time the Selector finds that this SocketChannel can write, it calls the onCanWrite function.

The process from line 43 to 46 to 50 is the most important asynchronous concept: every IO operation should wait until it can operate before calling.

In other words, in the onConnect function, we tell the Selector that we need to listen for write events. When the Selector detects that this SocketChannel can write, we then call the onCanWrite function to send the data.

`NioHTTPClient` Data Reading Process:

Line 53 after sending the request, it will register a read event with the Selector, meaning that when data returns, it will call the onCanRead function.

The onCanRead function needs to handle a relatively complex state:

Each TCP data packet is within 1500 bytes. The kernel will place the network transmission data into a buffer. If the interval between calls to onCanRead is relatively long, it is possible to read a complete HTTP request data in one read.

However, the code cannot be so optimistic; it needs to handle data in the most conservative way. After reading data, it should be placed into a buffer and then check if complete HTTP request data has been read. If not, continue reading until complete HTTP request data is obtained before executing the parsing work.

HTTP Protocol Parsing

The format of HTTP response data is as follows:

HTTP/1.1 200 OKContent-Type: text/htmlContent-Length: 1256
PNG....

According to the HTTP request specification, each HTTP request header ends with \r\n\r\n, so we can use this to determine whether complete HTTP request data has been read.

Line 67 when isResponseParsed is false, it indicates that the complete HTTP request data has not been parsed yet. We need to check if \r\n\r\n appears. If it does, it indicates that complete HTTP request data has been read.

Lines 77-90 parse the HTTP response and call the listener.onResponse function to inform the caller that the request has been responded to.

Lines 93-97 when isResponseParsed is true, it indicates that the complete HTTP request data has been parsed, and we can directly read the body data.

This part of the process is the most difficult to understand in asynchronous programming because data does not return in complete accordance with your application protocol (such as HTTP) but rather in batches. Therefore, we need to handle data stitching and parsing ourselves.

Let’s take a look at the actual calling code:

This code can implement an asynchronous NioHTTPClient class that can serve as a simple HTTP client. Through the HTTPResponseListener interface, we can obtain the response data of the HTTP request.

Conclusion

NIO is an essential technology for backend programming, as it requires a deep understanding of system operations and protocols.

It also involves handling Zero-Copy and multithreading, which are advanced technologies that require extensive practice.

High-performance server programming is based on NIO, as seen in classic products like Redis and Nginx.

This version only supports the HTTP protocol and does not support HTTPS. The HTTPS protocol is more complex and involves more considerations, such as certificate verification, encryption, and decryption.

Follow our official account and join our project practice group to obtain the HTTPS version of the code and tutorial.

Communication

We have built a practical group for the 100-line code project. You can scan the code to join and learn programming together.

You can also visit Ruzhila to learn more about practical programming.

All code is open source on GitHub: https://github.com/ruzhila/100-line-code. Welcome to Star! 👏

Recommended Previous Articles:

👏Introduction to Network Programming: 100 Lines of Go Code to Implement WebSocket to TCP Proxy, Supporting Docker and Complete Unit Testing.

👏100 Lines to Implement Multithreaded Download Without Any Third-Party Libraries.

👏100 Lines of Python Code to Implement a Brick Breaker Game Without Third-Party Dependencies.

👏100 Lines of Practical Series: Python Implementation of Tetris Without Dependencies.

👏100 Lines of JS Code to Implement the 2048 Game, Playable in the Browser.

👏100 Lines of Go Code to Convert Excel/CSV to JSON and Markdown Tables.

Super Simple: Implementing an NIO HTTP Client in Java with 100 Lines of Code and No Third-Party Dependencies

Project Address

Working Principle of NIO

Selector and Channel

Let’s Get to the Code

Code Analysis

`<span>NioHTTPClient</span>` Connection Creation and Request Sending Process:

`<span>NioHTTPClient</span>` Data Reading Process:

HTTP Protocol Parsing

Let’s take a look at the actual calling code:

Conclusion

Communication

Leave a Comment Cancel reply

Project Address

Working Principle of NIO

Selector and Channel

Let’s Get to the Code

Code Analysis

<span>NioHTTPClient</span> Connection Creation and Request Sending Process:

<span>NioHTTPClient</span> Data Reading Process:

HTTP Protocol Parsing

Let’s take a look at the actual calling code:

Conclusion

Communication

Related posts

Leave a Comment Cancel reply

`<span>NioHTTPClient</span>` Connection Creation and Request Sending Process:

`<span>NioHTTPClient</span>` Data Reading Process: