▼ Click the card below to follow me
▲ Click the card above to follow me
Who is still struggling with requests? This year, httpx has quietly reached the threshold and become the new “top player” in Python network requests. Don’t be fooled by its name, which sounds like requests’ younger brother; it actually has explosive functionality and handles asynchronous requests with ease. At this point, not learning it would be a mistake. Today, I will focus on practical content: the usage of httpx, asynchronous requests, common pitfalls, and practical applications.
What exactly is httpx?
httpx is essentially an HTTP client library. Its standout feature is that it supports both synchronous and asynchronous requests seamlessly. If you want to use it synchronously, the interface is almost identical to requests; if you want to play with asynchronous requests, just add async/await, and the speed is impressive. It also supports HTTP/2, connection pools, cookies, proxies, and more—it’s incredibly versatile.
Let’s start with a basic request, easy to understand:
import httpx
resp = httpx.get('https://httpbin.org/get')
print(resp.status_code)
print(resp.json())Output 200, mission accomplished. It feels like a close sibling to requests, right? This is its cleverness; beginners can get started with zero pressure.
Asynchronous requests: a new way to speed up
Synchronous requests one after another can be painfully slow, especially when dealing with web scraping or batch fetching, making you want to smash your computer. The asynchronous mode of httpx takes off directly; just add async/await, and the efficiency doubles.
Here’s an example of three consecutive requests, fighting asynchronously:
import asyncio
import httpx
async def fetch(url):
async with httpx.AsyncClient() as client:
resp = await client.get(url)
return resp.text
async def main():
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/2',
'https://httpbin.org/delay/3'
]
tasks = [fetch(u) for u in urls]
results = await asyncio.gather(*tasks)
for r in results:
print(r[:50])
asyncio.run(main())Three web pages run concurrently; just wait for the slowest one. No more fear of “waiting in line for food”.
Friendly reminder: When using asynchronous requests, remember to use AsyncClient; don’t mix it up with Client, which is synchronous. If async/await is not set up correctly, the errors can be quite frustrating.
Session management: handling logins and cookies
Long-time users of requests love to use Session; in httpx, it’s called Client and AsyncClient. It handles cookies, headers, authentication, etc., saving you from having to input them every time. For example, simulating a login to a website and automatically carrying over cookies:
import httpx
with httpx.Client() as client:
login_data = {"user": "test", "pwd": "1234"}
client.post('https://httpbin.org/cookies/set', data=login_data)
resp = client.get('https://httpbin.org/cookies')
print(resp.json())Cookies are automatically saved and carried over to the next request.
Switching to asynchronous is quite similar; just replace with async with.
Friendly reminder: Can’t remember which is synchronous and which is asynchronous? A simple trick: AsyncClient has async in its name, while the others are synchronous. Don’t accidentally mix them up; bugs will chase you down.
HTTP/2, proxies, timeouts? It’s all there!
httpx is not only great for asynchronous requests but also supports many “advanced features”. HTTP/2, proxies, SSL verification, and more are all included.
Want to use a proxy? It’s just a line of code:
import httpx
proxies = {
"http://": "http://127.0.0.1:8080",
"https://": "http://127.0.0.1:8080",
}
resp = httpx.get('https://httpbin.org/get', proxies=proxies)
print(resp.text)Need HTTP/2? As long as the server supports it, httpx will handle it automatically. You can also customize the timeout to prevent requests from hanging:
import httpx
try:
httpx.get('https://httpbin.org/delay/5', timeout=2)
except httpx.TimeoutException:
print("Timeout!")Friendly reminder: If you misspell proxy or timeout parameters, httpx won’t be forgiving; it will throw an error. If you have spelling sensitivity issues, check the documentation.
Error handling: Don’t let exceptions trip you up
In the world of networking, anything can happen. 404, 500, timeouts, network disconnections—httpx can gracefully capture them. Use try/except for exception handling:
import httpx
try:
resp = httpx.get('https://httpbin.org/status/404')
resp.raise_for_status()
except httpx.HTTPStatusError as e:
print("Error occurred:", e.response.status_code)
except httpx.RequestError as e:
print("Connection failed:", e)raise_for_status() can automatically throw status code exceptions, so you don’t have to write a bunch of if statements.
Friendly reminder: httpx has two types of exceptions: RequestError for network issues and HTTPStatusError for incorrect response codes. Don’t confuse them; it can lead to messy code.
Tips and common pitfalls
Don’t mix asynchronous and synchronous code; be clear about which Client to use, and don’t switch between async and normal.
asyncio.run() cannot be used in an existing event loop, such as in Jupyter notebooks, which will cause errors. You can use await main() or nest_asyncio as a fallback.
Parameters like headers, cookies, and timeout can be used with dictionaries or specific types; don’t write them as strings, or you’ll likely run into issues.
Don’t mix json and data parameters; if you want to send JSON in a POST request, use json=, and for form data, use data=.
In conclusion
httpx has now become the new favorite for Python network requests, especially in asynchronous scenarios. Its synchronous usage is compatible with requests, while its asynchronous mode offers outstanding performance, making it suitable for batch web scraping, API testing, and microservice development. With session management, cookies, proxies, HTTP/2, exceptions, and timeouts, it excels in every aspect. Your coding speed will increase, and bugs will decrease. Stop struggling with the old ways; using httpx will double your efficiency and joy.
Like and share
Let money and love flow to you