▼ Click the card below to follow me

▲ Click the card above to follow me

High-Performance HTTP Client with asyncio and aiohttp: Implementing Asynchronous Requests

Recently, I’ve been enjoying asynchronous programming, and today I want to share something interesting – using asyncio and aiohttp to implement asynchronous HTTP requests.

In simple terms, it allows your program to do other tasks while waiting for network responses instead of idly standing by. This approach is significantly faster than the traditional requests library!

What is Asynchronous?

You can think of asynchronous as “idle” while playing a game. Regular synchronous code is like standing still waiting for experience points, unable to do anything. But with asynchronous, your character can idle to gain experience while also completing other quests or tasks.

Let’s look at a basic asynchronous code example:

import asyncio
async def hello():
    print("Preparing to say hello...")
    await asyncio.sleep(1)
    print("hello!")
asyncio.run(hello())

Do you see the two keywords async and await? They are essential for asynchronous programming. async tells Python, “This is an asynchronous function,” while await indicates, “Please wait here; you can go do something else in the meantime.”

Getting Started with aiohttp

Enough talk, let’s dive into a practical example:

import asyncio
import aiohttp
async def get_page(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()
async def main():
    urls = [
        'http://python.org',
        'http://example.com',
        'http://httpbin.org'
    ]
    tasks = [get_page(url) for url in urls]
    results = await asyncio.gather(*tasks)
    for url, html in zip(urls, results):
        print(f"Fetched {url}, page size: {len(html)} bytes")
asyncio.run(main())

Friendly reminder: Don’t forget to run <span>pip install aiohttp</span> first, or the code won’t run!

Pitfalls of Asynchronous Code

Writing asynchronous code isn’t always smooth sailing; there are a few pitfalls to watch out for:

1. Not all operations can be asynchronous; for example, file I/O is not asynchronous by default.
2. await can only be used inside async functions.
3. Asynchronous functions return coroutine objects, not actual results.

Here’s an example of a mistake:

async def bad_example():
    # This will block
    time.sleep(1)  # Wrong! Should use await asyncio.sleep(1)
    # This also won't work
    result = requests.get('http://example.com')  # Wrong! Should use aiohttp

Concurrency Control

Sometimes, too many concurrent requests can overwhelm the server, so we need to exercise restraint:

import asyncio
from aiohttp import ClientSession, TCPConnector
async def controlled_fetch():
    # Limit to 5 concurrent connections
    connector = TCPConnector(limit=5)
    async with ClientSession(connector=connector) as session:
        tasks = []
        for i in range(20):
            url = f'http://httpbin.org/delay/{i % 3}'
            tasks.append(session.get(url))
        responses = await asyncio.gather(*tasks)

That’s the essence of asynchronous programming. At first, it may seem a bit convoluted, but with practice, you’ll become proficient. Remember a key principle: whenever you see an operation that requires waiting (like network requests or sleep), use await to let the program do something else. This way, your code will run quickly and efficiently!

On a side note, if you find these concepts difficult to digest at first, don’t rush it; take your time. Just like learning to ride a bike, you need to practice with support before you can ride freely. Once you master asynchronous programming, it becomes a powerful tool for writing web scrapers and APIs!

Previous Articles

◆ Airflow: The Ultimate Tool for Building Data Workflows!

◆

◆

Like and Share

Let money and love flow to you