Hello everyone, I am Octopus Cat.
Those who have worked with web scraping know that the HTTP request step is often the “key to success or failure”.
If you are just requesting some public APIs, Go’s net/http standard library is completely sufficient. However, once you encounter websites with anti-scraping detection, the situation becomes complicated:
- Some websites detect browser fingerprints to distinguish between “real users” and “script access”.
- Others identify client characteristics through JA3/JA4 TLS fingerprints to determine if it is a scraper.
- Some have fully adopted HTTP/3 and QUIC, strictly verifying transport layer characteristics.
At this point, relying solely on the standard library to simulate requests often exposes your intentions.
I recently discovered an open-source project called Surf, which can be said to be an HTTP client “born for anti-scraping”, with capabilities far beyond the standard library.
The highlights of Surf mainly focus on two words: Disguise.
It is based on advanced TLS fingerprint technology, capable of “confusing the genuine” at the protocol level, simulating the request characteristics of a real browser. In contrast, the standard library is more limited to the level of “being able to send requests”, and struggles with complex anti-scraping measures.
Surf offers the following features:
1. Browser Disguise
Surf has a complete fingerprint library of mainstream browsers built-in, such as Chrome v131, Firefox v131, etc. Developers can switch to the request characteristics of a specified browser with one click, simulating a real user’s access from start to finish.
2. JA3/JA4 Fingerprint Customization
JA3/JA4 are the “fingerprint identifiers” of TLS handshake characteristics, which websites often use to identify clients. Surf supports fully customizable TLS handshake characteristics, allowing developers to debug or forge as needed to bypass fingerprint detection.
3. HTTP/3 and QUIC Support
As more and more websites upgrade to HTTP/3/QUIC, clients that cannot be compatible will appear “suspicious”. Surf provides complete support in this area, including QUIC transport parameter fingerprints.
4. Middleware System
Surf has a built-in extensible middleware system, making it easy for developers to insert custom logic into the request/response chain, such as logging, retrying, proxy switching, etc.
5. Connection Pool Optimization
Using a singleton pattern for the connection pool can significantly improve performance in high-concurrency scenarios. This is a very practical optimization for scrapers that require frequent requests.
6. Proxy Rotation
Supports multiple types of proxies including HTTP, HTTPS, SOCKS5, and can automatically switch to reduce the risk of IP bans.
It is developed based on the Go language and can be used after installation via go get. Developers with scraping needs can take a look.
Open source project address: https://github.com/enetx/surf
Open source project author: enetx
