Requests: A Python Library That Simplifies HTTP Requests!

In today’s digital age, whether it’s software development, data analysis, or everyday automation tasks, interaction with web services is indispensable. For instance, you might need to fetch weather data from an API to create a weather application, or batch download images from a website, or automate testing a web interface. The core of these operations is sending HTTP requests and processing responses. If you use Python’s built-in urllib library, you’ll find that you need to write a lot of code to handle tedious details like URL encoding, cookies, request headers, and response parsing. The emergence of the requests library has completely changed this situation. With its simple and elegant API, it encapsulates the complex HTTP protocol, allowing developers to accomplish powerful web request functionalities with minimal code, making it one of the most popular HTTP libraries in the Python ecosystem.

1. Introduction to the Library

requests is a Python HTTP library based on the Apache2 Licensed open-source protocol, designed with the philosophy of being “user-friendly”. It builds upon the Python standard library but provides a more intuitive and friendly interface. In practical life and work, requests has a wide range of application scenarios:

Web Development and Testing: Developers can use it to simulate client requests, perform interface testing, automate deployment, and monitoring.
Data Collection (Web Scraping): Scraping data from various websites and APIs is a fundamental step in data analysis and machine learning.
Automation Scripts: Automatically fetching information from the web, such as stock prices and news, and processing or notifying.
Third-party Service Integration: Almost all modern applications need to integrate with third-party services (such as payment, maps, social platforms) via APIs, and requests is the preferred tool for achieving this functionality.

In simple terms, as long as your task involves dealing with the HTTP/HTTPS protocol, requests can make your work easy and enjoyable.

2. Installing the Library

requests library installation is very simple, and it is highly recommended to use the pip tool for installation.

pip install requests

Once installed, you can use it in your Python scripts by importing it with the import statement.

import requests

3. Basic Usage (in 4 Steps)

requests library’s core functionality revolves around several basic HTTP request methods, with the most commonly used being GET and POST.

Step 1: Sending a Simple GET Request

This is the most common scenario, used to fetch data from the server.

# Target URL, here we use a test API provided by JSONPlaceholderurl = 'https://jsonplaceholder.typicode.com/posts/1'# Send GET requestresponse = requests.get(url)# response is a Response object containing all information returned by the serverprint(f"Response Status Code: {response.status_code}") # 200 indicates successprint(f"Response Content Type: {response.headers['Content-Type']}")

Step 2: Handling Response Content

requests provides various convenient ways to handle response content.

# 1. Get the raw binary contentraw_content = response.contentprint(f"Raw Content (first 50 characters): {raw_content[:50]}")# 2. Get the text content (automatically decodes based on response encoding)text_content = response.textprint(f"Text Content (first 50 characters): {text_content[:50]}")# 3. Get JSON formatted data (most commonly used)# If the response's Content-Type is application/json, the json() method will automatically parsejson_data = response.json()print(f"Parsed JSON Data: {json_data}")print(f"Article Title: {json_data['title']}")

Step 3: Sending a GET Request with Parameters

Often, a GET request needs to include query parameters in the URL, such as for search or pagination.

url = 'https://jsonplaceholder.typicode.com/comments'# Define query parameters as a dictionaryparams = {    'postId': 1,  # Get all comments for article ID 1    '_limit': 5   # Only get the first 5}# Pass parameters to the params argument of the get methodresponse = requests.get(url, params=params)if response.status_code == 200:    comments = response.json()    print(f"Retrieved {len(comments)} comments:")    for comment in comments:        print(f"- {comment['email']}: {comment['body'][:30]}...")

Step 4: Sending a POST Request to Submit Data

When you need to submit data to the server (such as user registration or form submission), a POST request is used.

url = 'https://jsonplaceholder.typicode.com/posts'# Data to be submitted, represented as a dictionarypayload = {    'title': 'I am a new title',    'body': 'This is the content of the article.',    'userId': 1}# Send POST request, passing data to the data parameter# requests will automatically encode the dictionary as application/x-www-form-urlencoded formatresponse = requests.post(url, data=payload)print(f"POST Request Status Code: {response.status_code}") # 201 Created indicates successprint("Server returned created resource:")print(response.json())

4. Advanced Usage

requests is powerful in its elegant support for complex scenarios.

1. Custom Request Headers

Some APIs require specific User-Agent or Authorization tokens.

url = 'https://api.github.com/user'# Custom request headersheaders = {    'User-Agent': 'MyPythonScript/1.0',    'Authorization': 'token YOUR_GITHUB_PERSONAL_ACCESS_TOKEN' # Replace with your Token}response = requests.get(url, headers=headers)if response.status_code == 200:    user_info = response.json()    print(f"GitHub Username: {user_info['login']}")else:    print(f"Request failed: {response.status_code} - {response.text}")

2. Handling Cookies and Sessions

In scenarios where you need to maintain login status or sessions (such as scraping websites that require login), the Session object is the best choice.

# Create a Session objects = requests.Session()# 1. First log in, the Session will automatically save the cookies after loginlogin_url = 'https://example.com/login'login_data = {'username': 'myuser', 'password': 'mypassword'}s.post(login_url, data=login_data)# 2. Access a page that requires login, the Session will automatically carry the cookiesprofile_url = 'https://example.com/profile'response = s.get(profile_url)# Now the response content is the user's personal homepage after login# print(response.text)

3. Timeout Settings and Exception Handling

Network requests may fail for various reasons, and robust code must handle these exceptions.

url = 'https://jsonplaceholder.typicode.com/posts/1'try:    # Set timeout (seconds) to prevent requests from waiting indefinitely    response = requests.get(url, timeout=5)    # If the request fails (status code 4xx or 5xx), raise_for_status() will throw an HTTPError exception    response.raise_for_status()    data = response.json()    print("Successfully retrieved data.")except requests.exceptions.Timeout:    print("Request timed out, please check your network connection.")except requests.exceptions.ConnectionError:    print("Connection error, please check if the URL is correct.")except requests.exceptions.HTTPError as err:    print(f"HTTP error occurred: {err}")except Exception as e:    print(f"An unknown error occurred: {e}")

5. Practical Application Scenarios

Scenario 1: Automatically Fetching and Saving Daily Wallpapers

Many websites provide high-quality daily wallpapers, and we can use requests to download them automatically.

import requestsimport osfrom datetime import datetimedef download_wallpaper():    # Assume this is an API that returns the daily wallpaper URL    api_url = "https://api.example.com/daily-wallpaper"    try:        response = requests.get(api_url)        response.raise_for_status()        data = response.json()        wallpaper_url = data['url']        wallpaper_title = data['title']        print(f"Today's Wallpaper: {wallpaper_title}")        print(f"Download URL: {wallpaper_url}")        # Download the image        img_response = requests.get(wallpaper_url, stream=True) # stream=True for large file downloads        img_response.raise_for_status()        # Set save path        filename = f"wallpaper_{datetime.now().strftime('%Y%m%d')}.jpg"        save_path = os.path.join(os.getcwd(), 'wallpapers', filename)        # Ensure the directory exists        os.makedirs(os.path.dirname(save_path), exist_ok=True)        # Write to file        with open(save_path, 'wb') as f:            for chunk in img_response.iter_content(chunk_size=8192):                f.write(chunk)        print(f"Wallpaper successfully saved to: {save_path}")    except Exception as e:        print(f"Failed to download wallpaper: {e}")# Execute functiondownload_wallpaper()

Scenario 2: Batch Querying IP Address Locations

Using third-party APIs, we can quickly query the geographical location information of a large number of IP addresses.

import requestsdef get_ip_location(ip_address):    """Query the location of a single IP address"""    url = f"http://ip-api.com/json/{ip_address}"    try:        response = requests.get(url, timeout=10)        response.raise_for_status()        data = response.json()        if data['status'] == 'success':            return {                'ip': data['query'],                'country': data['country'],                'region': data['regionName'],                'city': data['city'],                'isp': data['isp']            }        else:            return {'ip': ip_address, 'error': data['message']}    except Exception as e:        return {'ip': ip_address, 'error': str(e)}# List of IP addresses to queryip_list = [    '8.8.8.8',      # Google DNS    '1.1.1.1',      # Cloudflare DNS    '202.108.22.5', # Baidu    '93.184.216.34' # Example IP]print("IP Address Location Query Results:")print("-" * 60)for ip in ip_list:    result = get_ip_location(ip)    if 'error' in result:        print(f"IP: {result['ip']} -> Query failed: {result['error']}")    else:        print(f"IP: {result['ip']} -> {result['country']} - {result['region']} - {result['city']} (ISP: {result['isp']})")print("-" * 60)

requests library, with its unparalleled simplicity and powerful functionality, has become the de facto standard for HTTP communication in Python. It liberates developers from the underlying protocol details, allowing them to focus on the business logic itself. Whether for rapid prototyping, writing automation scripts, or building complex distributed systems, requests is an indispensable tool.

Mastering requests is not just about learning to use a library; it is about understanding the basic patterns of modern web service interactions. I hope this article helps you quickly get started and leverage its immense capabilities in real projects.

What complex HTTP request scenarios have you encountered in your work? How has the requests library helped you solve them? Or do you have any advanced tips about requests that you would like to share? Feel free to leave comments for discussion, let’s learn and progress together!