When it comes to asynchronous network programming, the library I have enjoyed using the most over the years is aiohttp.
Those who have written web scrapers know that while requests are easy to use, they can easily get blocked by a high volume of requests.
aiohttp perfectly addresses this pain point; it leverages Python’s asynchronous features, allowing us to handle multiple network requests simultaneously with great efficiency!
What is Asynchronous Programming?
Asynchronous programming sounds sophisticated, but it essentially allows your program to multitask. For example, when cooking rice, you wouldn’t just stand in front of the rice cooker waiting; instead, you would check your phone or wash vegetables. This is the essence of asynchronous thinking. The same applies to code; you don’t have to wait for one task to finish before starting another.
import asyncio
import aiohttp
async def get_page(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
async def main():
urls = [
'http://example1.com',
'http://example2.com',
'http://example3.com'
]
tasks = [get_page(url) for url in urls]
results = await asyncio.gather(*tasks)
print('Done!')
asyncio.run(main())
The Power of Session Management
A small tip for using aiohttp is to make good use of ClientSession. Despite its intimidating name, it is simply a session manager that helps you maintain connections and reuse cookies.
async def better_way():
async with aiohttp.ClientSession() as session:
# Use the same session to send requests, which is very efficient
async with session.get('http://example.com') as resp1:
data1 = await resp1.json()
async with session.get('http://example.com/api') as resp2:
data2 = await resp2.json()
Tip: Remember to use async with to manage the session; do not close it manually, as it can lead to leaks. This is similar to file operations where using with is always a good practice.
A Variety of Request Types
aiohttp supports various request methods, including GET, POST, and PUT. When sending POST requests, the data format is also very flexible:
async def post_stuff():
async with aiohttp.ClientSession() as session:
# Sending form data
form_data = {'name': 'Xiao Ming', 'age': 18}
async with session.post('http://api.com', data=form_data) as resp:
result = await resp.text()
# Sending JSON data
json_data = {'message': 'Hello!'}
async with session.post('http://api.com', json=json_data) as resp:
result = await resp.json()
Concurrency Control
When writing web scrapers, sending too many requests at once can overwhelm the target server. We need to exercise restraint and use Semaphore to limit the number of concurrent requests:
async def controlled_requests():
# Allow a maximum of 5 concurrent requests
sem = asyncio.Semaphore(5)
async def safe_get(url):
async with sem: # Control concurrency with semaphore
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
urls = ['http://example.com'] * 20
tasks = [safe_get(url) for url in urls]
await asyncio.gather(*tasks)
Tip: Don’t start with a high concurrency; test with a small number first, and gradually increase the concurrency once you confirm it works.
Error Handling
Network requests can always encounter issues, such as timeouts and connection errors, which need to be handled properly:
async def handle_errors():
try:
timeout = aiohttp.ClientTimeout(total=10) # 10 seconds timeout
async with aiohttp.ClientSession(timeout=timeout) as session:
async with session.get('http://slowapi.com') as resp:
return await resp.text()
except asyncio.TimeoutError:
print('Oops, timeout!')
except aiohttp.ClientError as e:
print(f'Error occurred: {e}')
The most common mistake when writing asynchronous code is forgetting to add await. Remember, whenever you see a function defined with async, you must prefix the call with await. Also, avoid using synchronous operations like time.sleep() inside asynchronous functions; use asyncio.sleep() instead.
By effectively using aiohttp, your web scraping speed will significantly increase, and API calls will become much faster.
However, remember to write your code elegantly; don’t skimp on comments where necessary, and don’t be lazy about handling exceptions.
After all, the code is written for others to read, and you will need to maintain it later!
Previous Reviews
◆
◆
◆
Like and Share
Letmoney and love flow to you