Implementing a Simple HTTP Proxy in Golang

This article provides a detailed introduction to implementing an HTTP proxy using Golang. Those who have a need in practical applications can learn from it!

A proxy is an important function in networking, serving to retrieve network information on behalf of network users. Figuratively speaking, it acts as a transfer station for network information. For clients, the proxy plays the role of a server, receiving request messages and returning response messages; for web servers, the proxy acts as a client, sending request messages and receiving response messages.

There are various types of proxies. If classified based on network users, they can be divided into forward proxies and reverse proxies:

  • Forward Proxy: The client acts as the network user. When the client accesses the server, it first accesses the proxy server, which then accesses the server. This process requires the client to configure the proxy, making it transparent to the server.
  • Reverse Proxy: The server acts as the network user. The access process is the same as that of a forward proxy, but this process is transparent to the client, requiring the server to configure the proxy (which can also be left unconfigured).

Different proxy protocols exist for forward and reverse proxies, which are the protocols used for communication between the proxy server and the network user:

  • Forward Proxy:
    • HTTP
    • HTTPS
    • SOCKS4
    • SOCKS5
    • VPN: Functionally, a VPN can also be considered a proxy.
  • Reverse Proxy:
    • TCP
    • UDP
    • HTTP
    • HTTPS

Next, let’s discuss HTTP proxies.

Overview of HTTP Proxies

HTTP proxies are a relatively simple type of forward proxy that uses the HTTP protocol as the transport protocol between the client and the proxy server.

HTTP proxies can carry HTTP, HTTPS, FTP protocols, etc. The data format between the client and the proxy server varies slightly depending on the protocol.

HTTP Protocol

First, let’s take a look at the HTTP header sent from the client to the proxy server under the HTTP protocol:

// Direct connection
GET / HTTP/1.1
Host: staight.github.io
Connection: keep-alive

// HTTP proxy
GET http://staight.github.io/ HTTP/1.1
Host: staight.github.io
Proxy-Connection: keep-alive

As we can see, compared to a direct connection, the HTTP proxy:

  • The URL becomes a complete path, / -> http://staight.github.io/
  • The Connection field changes to Proxy-Connection field
  • The rest remains the same

Why use a complete path?

To identify the target server. Without a complete path and without a Host field, the proxy server will not know the address of the target server.

Why use the Proxy-Connection field instead of the Connection field?

This is to maintain compatibility with outdated proxy servers that use the HTTP/1.0 protocol. The long connection feature was introduced in HTTP/1.1. In a direct connection, if the HTTP header sent by the client contains the Connection: keep-alive field, it indicates the use of a long connection for HTTP communication with the server. However, if there is an outdated proxy server in between, that proxy server will not be able to maintain a long connection with both the client and the server, causing both to wait unnecessarily, wasting time.

Therefore, the Proxy-Connection field is used instead of the Connection field. If the proxy server uses the HTTP/1.1 protocol and can recognize the Proxy-Connection field, it will convert that field to Connection before sending it to the server; if it cannot recognize it, it will send it directly to the server, as the server also cannot recognize it, thus using a short connection for communication.

The interaction process of the HTTP proxy under the HTTP protocol is shown in the figure:

Implementing a Simple HTTP Proxy in Golang

HTTP Proxy HTTP Protocol

HTTPS Protocol

Next, let’s look at the HTTP header sent from the client to the proxy server under the HTTPS protocol:

CONNECT staight.github.io:443 HTTP/1.1
Host: staight.github.io:443
Proxy-Connection: keep-alive

As shown, compared to the HTTP protocol, the HTTPS protocol:

  • The request method changes from GET to CONNECT
  • The URL does not have a protocol field

In fact, since the communication between the client and server under HTTPS is encrypted except for the initial negotiation, the proxy server no longer modifies the HTTP messages for forwarding. Instead, it negotiates the server’s address with the client from the beginning, and the subsequent TCP encrypted data is forwarded directly.

The interaction process of the HTTP proxy under the HTTPS protocol is shown in the figure:

Implementing a Simple HTTP Proxy in Golang

HTTP Proxy HTTPS Protocol

Code Implementation

First, create a TCP service, and for each TCP request, call the handle function:

// TCP connection, listening on port 8080
l, err := net.Listen("tcp", ":8080")
if err != nil { log.Panic(err) }
// Infinite loop, call handle for each connection
for { client, err := l.Accept() if err != nil { log.Panic(err) }
 go handle(client) }

Then, place the received data into a buffer:

// Buffer to store client data
var b [1024]byte
// Read data from the client
n, err := client.Read(b[:])
if err != nil { log.Println(err) return }

Read the HTTP request method, URL, and other information from the buffer:

var method, URL, address string
// Read method and URL from client data
fmt.Sscanf(string(b[:bytes.IndexByte(b[:], '\n')]), "%s%s", &method, &URL)
hostPortURL, err := url.Parse(URL)
if err != nil { log.Println(err) return }

The methods for obtaining addresses under HTTP and HTTPS protocols are different, handled separately:

// If the method is CONNECT, it is HTTPS
if method == "CONNECT" { address = hostPortURL.Scheme + ":" + hostPortURL.Opaque} else { // Otherwise, it is HTTP
 address = hostPortURL.Host // If host does not include port, default to 80
 if strings.Index(hostPortURL.Host, ":") == -1 { // Host does not include port, default 80
 address = hostPortURL.Host + ":80" } }

Use the obtained address to initiate a request to the server. If it is an HTTP protocol, directly forward the client’s request to the server; if it is an HTTPS protocol, send an HTTP response:

// Obtained the request's host and port, initiate a TCP connection to the server
server, err := net.Dial("tcp", address)
if err != nil { log.Println(err) return }
// If using HTTPS, first indicate to the client that the connection has been established
if method == "CONNECT" { fmt.Fprint(client, "HTTP/1.1 200 Connection established\r\n\r\n") } else { // If using HTTP, forward the HTTP request obtained from the client to the server
 server.Write(b[:n]) }

Finally, forward all client requests to the server and all server responses back to the client:

// Forward client requests to the server and server responses to the client. io.Copy is a blocking function, it won't stop until the file descriptors are closed
go io.Copy(server, client)
io.Copy(client, server)

The complete source code is as follows:

package main
import ( "bytes" "fmt" "io" "log" "net" "net/url" "strings")
func main() { // TCP connection, listening on port 8080
 l, err := net.Listen("tcp", ":8080")
 if err != nil { log.Panic(err) }
 // Infinite loop, call handle for each connection
 for { client, err := l.Accept() if err != nil { log.Panic(err) }
  go handle(client) }}
func handle(client net.Conn) { if client == nil { return } defer client.Close()
 log.Printf("remote addr: %v\n", client.RemoteAddr())
 // Buffer to store client data
 var b [1024]byte // Read data from the client
 n, err := client.Read(b[:]) if err != nil { log.Println(err) return }
 var method, URL, address string // Read method and URL from client data
 fmt.Sscanf(string(b[:bytes.IndexByte(b[:], '\n')]), "%s%s", &method, &URL)
 hostPortURL, err := url.Parse(URL) if err != nil { log.Println(err) return }
 // If the method is CONNECT, it is HTTPS
 if method == "CONNECT" { address = hostPortURL.Scheme + ":" + hostPortURL.Opaque } else { // Otherwise, it is HTTP
 address = hostPortURL.Host // If host does not include port, default to 80
 if strings.Index(hostPortURL.Host, ":") == -1 { // Host does not include port, default 80
 address = hostPortURL.Host + ":80" } }
 // Obtained the request's host and port, initiate a TCP connection to the server
 server, err := net.Dial("tcp", address) if err != nil { log.Println(err) return }
 // If using HTTPS, first indicate to the client that the connection has been established
 if method == "CONNECT" { fmt.Fprint(client, "HTTP/1.1 200 Connection established\r\n\r\n") } else { // If using HTTP, forward the HTTP request obtained from the client to the server
 server.Write(b[:n]) }
 // Forward client requests to the server and server responses to the client. io.Copy is a blocking function, it won't stop until the file descriptors are closed
 go io.Copy(server, client)
io.Copy(client, server)}

Add the proxy and run:

Implementing a Simple HTTP Proxy in Golang

Implementing a Simple HTTP Proxy in Golang

Source: https://staight.github.io/Implementing a Simple HTTP Proxy in GolangEnd of Article BenefitsImplementing a Simple HTTP Proxy in Golang

Today, I would like to share aComplete Learning Material Set for Go Language Engineers,[Comprehensive Collection of Go Interview Questions + Go Learning Roadmap + Go Learning Documentation Notes + Go Project Practical Videos + Go Introductory Videos] A set covering most of the core knowledge that Go programmers need to master.

Essential Technologies Overview

Implementing a Simple HTTP Proxy in Golang

Implementing a Simple HTTP Proxy in GolangScan the QR code above

to get free access to[Go Material Collection]

Related posts

  1. Analysis of HTTP2 Features
  2. How to Receive Messages in Real-Time via HTTP Like TCP?
  3. Introduction to Networking Basics
  4. Smooth Upgrades for HTTP Services in Go: Signal Handling Mechanism | Zero Downtime Deployment
  5. High-Performance HTTPS Server Optimization and Improvements Based on Hyper with Public Network Access and Multi-Domain Support for HTTP/3 and HTTP/2
  6. A Senior Developer’s Year-Long Journey Learning Rust
  7. Introduction to Cryptography and Encryption Algorithms
  8. Essential Knowledge of HTTP
  9. Enterprise-Level IoT Platform with Open Source Code for TCP/UDP/MQTT/HTTP
  10. Comparison and Analysis of Container Network Plugins and the Impact of Bare Metal vs Virtualized Network Performance
  11. Httpx: The Amazing HTTP Client Library for Python!
  12. Can HTTP/2 + SSE Completely Replace WebSocket?
  13. Master Huawei Interview Questions: Understanding HTTP
  14. A Quick Overview: Can We Discuss HTTP and HTTPS?
  15. 7 Technical Trends of MQTT Protocol in 2023
  16. Developing a Data Compression Tool in C Language
  17. C Language Coding Standards: Naming Conventions and Commenting Styles
  18. Common Errors in C Language: Pointer Errors and Memory Leaks
  19. Detailed Explanation of Bit Fields in C Language

Leave a Comment