Analyzing Matrices with Python: Achieving 509 Times Data Processing Throughput with Requests

Last year, I did something amusing; I ran a single-threaded crawler for an entire day just to scrape data from a small e-commerce website. My boss saw the progress bar on my desktop and laughed, “At this rate, you’ll be delivering the report to the client in the year of the monkey!” Well, he was right. This prompted me to start researching how to fundamentally improve Python request processing efficiency, especially when handling large-scale API calls. After three months of technical iterations and practical testing, I improved the data acquisition speed by 509 times! Not 509%, but 509 times! Below, I will share this practical experience.

Pain Points in Data Scraping

To be honest, the Python Requests library is my true love; it’s so simple and intuitive that it makes me want to cry. But when you need to handle thousands of requests in parallel, the traditional serial approach is maddeningly slow. My initial crawler code looked something like this:

for url in url_list:
    response = requests.get(url)
    # Process data...

This method is a nightmare when dealing with large-scale requests. For example, in our e-commerce project, we needed to scrape about 140,000 product pages, and the single-threaded approach was estimated to take over 40 hours! Who can wait that long? Not to mention, if the network drops midway, it would be devastating.

After some research, I found that the core of the problem lies in IO blocking – when one request is waiting for a server response, the entire program is stuck there, completely wasting CPU resources. There are many ways to solve this problem, but after practical testing, I summarized three effective strategies.

Strategy One: Asynchronous Coroutines – Single-Core Speeding

The combination of asyncio and aiohttp in Python 3.7+ is simply a super weapon for data processing. Check out this implementation:

async with aiohttp.ClientSession() as session:
    tasks = [fetch(session, url) for url in urls]
    await asyncio.gather(*tasks)

This looks simple, but it can actually keep the CPU busy. The principle is that when one request is waiting for a response, the program can switch to handle other requests, instead of just waiting idly. In my actual project, using coroutines improved the processing speed by about 40 times. But wait, I encountered a new problem – when the number of requests exceeded a few thousand, some requests would time out or fail because, although coroutines can handle concurrency, they still execute in the same thread.

To be honest, if your machine is multi-core (who isn’t these days?), just using coroutines is a waste of resources. This leads us to the second strategy.

Strategy Two: Process Pool + Thread Pool Combination

Isn’t it funny to only use coroutines? It’s like driving a Ferrari but only using one cylinder! Modern CPUs are multi-core, and not utilizing them is a shame. My solution is: to use multiprocessing to fully leverage multi-core performance while using a thread pool or coroutine pool within each process to handle IO-intensive tasks.

with ProcessPoolExecutor(max_workers=cpu_count()) as executor:
    future_to_url = {executor.submit(process_batch, batch): batch for batch in batches}

This method caused my crawler speed to explode again, reaching about 120 times that of the original version. Each core was operating efficiently, and tasks were allocated reasonably. However, during actual debugging, I found a pitfall – the overhead of inter-process communication is also significant, and if task division is not reasonable, it can negatively impact performance.

Additionally, there is an annoying issue where the target server may limit request frequency or outright refuse service. I was shocked to find that after optimizing, my IP got banned. What a hassle!

Strategy Three: Intelligent Request Management

After implementing basic concurrency, I began to focus on how to intelligently manage requests. I developed a self-adaptive request scheduler that can dynamically adjust request frequency based on server responses, incorporating features like retry mechanisms, proxy IP rotation, and request header randomization.

class SmartScheduler:
    def __init__(self, max_rate=10):
        self.rate_limit = max_rate
        self.success_count = 0
        # Other initialization code...

This scheduler automatically adjusts the concurrency based on the success rate of requests. When it detects a high failure rate, it reduces the request frequency and activates backup proxies. When handling a particularly sensitive financial data API, this solution helped me increase the success rate from 68% to 99.7%.

Getting back to the main point, combining these three strategies, my final solution achieved a throughput increase of 509 times in actual projects. What used to take 40 hours can now be completed in under 5 minutes for all 140,000 pages, truly achieving a leap in data processing capability.

To be honest, these technical combinations are not complicated, but considering Python’s GIL limitations, achieving such performance improvements is quite remarkable. In the company’s data analysis team, my solution has now become the standard configuration, and even my boss, who previously mocked me, is now amazed.

As a real case, we developed a competitive analysis tool that needs to scrape the latest prices and review data from 37 different e-commerce platforms daily. Tasks that used to require overnight runs can now be completed during lunch, and then directly generate analysis reports to send to clients, improving business response speed by a whole order of magnitude.

Finally, I want to share a little tip: when debugging concurrent code, print statements are really not enough. I usually combine the logging and tqdm modules to see detailed logs and visually display progress, greatly enhancing the development experience.

This method is not only applicable to web scraping but also suitable for any scenario requiring large-scale API calls, such as batch data processing and distributed computing. I hope my practical experiences can help you, and feel free to discuss any questions!

Before you go, remember to click “Looking”~