Requests: A Python Library That Simplifies HTTP Requests!

In today’s digital age, web requests have become an indispensable part of software development. Whether it’s fetching API data, submitting forms, downloading files, or interacting with web services, we need to send HTTP requests.

The Python requests library was created to solve these problems, providing a simple and elegant API that makes HTTP requests effortless.

1. Introduction to the Requests Library

requests is the most popular HTTP client library in Python, developed and open-sourced by Kenneth Reitz in 2011. It is built on top of Python’s urllib3 library, providing a higher-level, more user-friendly API that makes sending HTTP requests exceptionally simple. Compared to Python’s built-in urllib module, requests has the following significant advantages:

Concise API: requests uses intuitive method names (like get() and post()) and parameters, enhancing code readability.
Automatic handling of complex situations: Automatically manages URL encoding, response decoding, cookie management, session persistence, etc.
Rich feature support: Supports various features of HTTP requests, such as SSL verification, proxy settings, timeout control, streaming responses, etc.
Extensive community support: As an important part of the Python ecosystem, requests has a large user community and abundant documentation resources.

In real life, the requests library has a wide range of application scenarios. For example:

Web scraping: Fetching web page content and extracting information
API calls: Accessing various RESTful APIs to retrieve data
Automated testing: Testing the interfaces of web services
Data collection: Collecting data from various web services
Application integration: Interacting with third-party services (like GitHub, Twitter, WeChat, etc.)

2. Installing the Requests Library

requests is not a built-in library in Python, so it needs to be installed using pip:

pip install requests

If you are using Python version 3.4 or below, you may need to upgrade pip first:

pip install --upgrade pip

Once installed, you can import and use it in your Python code:

import requests

3. Basic Usage

1. Sending a Simple GET Request

Sending a GET request is one of the most common HTTP operations, and can be easily achieved using the requests.get() method:

import requests# Send GET request
response = requests.get('https://api.github.com/users/octocat')# Check response status code
if response.status_code == 200:
    # Get response content (JSON format)
    data = response.json()
    print(f"Username: {data['login']}")
    print(f"Followers: {data['followers']}")
else:
    print(f"Request failed, status code: {response.status_code}")

2. Passing Query Parameters

In a GET request, it is often necessary to pass query parameters (URL parameters). This can be done using the params parameter:

import requests# Define query parameters
params = {
    'q': 'python programming',
    'page': 1,
    'per_page': 10
}# Send GET request with parameters
response = requests.get('https://api.github.com/search/repositories', params=params)# Handle response
if response.status_code == 200:
    data = response.json()
    print(f"Found {data['total_count']} repositories")
    for item in data['items'][:3]:
        print(f"- {item['name']}: {item['description']}")

3. Sending a POST Request

Sending a POST request is typically used to submit data to the server, such as form data or JSON data. This can be done using the requests.post() method:

import requests# Define data to submit (as a dictionary)
data = {
    'username': 'john_doe',
    'password': 'secret123'
}# Send POST request
response = requests.post('https://example.com/login', data=data)# Handle response
if response.status_code == 200:
    print("Login successful")
else:
    print("Login failed")

4. Handling Responses

requests returns a response object that contains rich information, making it easy to access response content, status codes, headers, etc.:

import requests
response = requests.get('https://www.example.com')# Get response status code
print(f"Status code: {response.status_code}")  # e.g., 200
# Get response headers
print(f"Response headers: {response.headers}")  # Includes server type, content type, etc.
# Get response content (text format)
print(f"Response content: {response.text[:100]}...")  # First 100 characters
# Get response content (binary format, for downloading files)
print(f"Response content (binary): {response.content[:10]}...")# Get JSON format response (if the response is JSON)
# data = response.json()

4. Advanced Usage

1. Customizing Request Headers

Sometimes it is necessary to add custom request headers, such as setting user agents or authentication information:

import requests
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Authorization': 'Bearer your_access_token'
}
response = requests.get('https://api.example.com/protected', headers=headers)

2. Session Management

Using requests.Session() allows you to create a session object that maintains session state (like cookies), suitable for scenarios requiring multiple requests:

import requests# Create session object
session = requests.Session()# Login (set cookies)
login_data = {'username': 'user', 'password': 'pass'}
session.post('https://example.com/login', data=login_data)# Access a page that requires login
response = session.get('https://example.com/dashboard')# Close session
session.close()

3. File Upload

Using requests makes file uploads easy:

import requests# Prepare files to upload
files = {
    'file': open('example.txt', 'rb'),
    'image': ('photo.jpg', open('photo.jpg', 'rb'), 'image/jpeg')
}# Upload files
response = requests.post('https://example.com/upload', files=files)

4. Timeout Settings and Exception Handling

To prevent requests from hanging for a long time, you can set timeout parameters and handle potential exceptions:

import requests
from requests.exceptions import Timeout, ConnectionError
try:
    # Set timeout to 5 seconds
    response = requests.get('https://example.com', timeout=5)
    response.raise_for_status()  # Check response status code, raise exception if not 200
except Timeout:
    print("Request timed out")
except ConnectionError:
    print("Connection error")
except requests.exceptions.HTTPError as http_err:
    print(f"HTTP error: {http_err}")
except Exception as err:
    print(f"Other error: {err}")
else:
    print("Request successful")

5. Practical Application Scenarios

1. Data Collection and API Calls

Fetching data from open APIs is one of the most common application scenarios for requests. For example, fetching weather data:

import requests# OpenWeatherMap API
API_KEY = 'your_api_key'
CITY = 'Beijing'
URL = f'http://api.openweathermap.org/data/2.5/weather?q={CITY}&appid={API_KEY}&units=metric'# Send request
response = requests.get(URL)# Handle response
if response.status_code == 200:
    data = response.json()
    print(f"City: {data['name']}")
    print(f"Weather: {data['weather'][0]['description']}")
    print(f"Temperature: {data['main']['temp']}°C")
    print(f"Humidity: {data['main']['humidity']}%")
else:
    print(f"Unable to fetch weather data: {response.status_code}")

2. Web Scraping

Using requests in conjunction with parsing libraries (like BeautifulSoup) can implement simple web scraping:

import requests
from bs4 import BeautifulSoup  # Requires beautifulsoup4 library
# Target URL
url = 'https://news.ycombinator.com'
# Send request
response = requests.get(url)# Parse HTML content
if response.status_code == 200:
    soup = BeautifulSoup(response.text, 'html.parser')
    # Extract article titles and links
    articles = []
    for item in soup.select('.titleline'):
        a_tag = item.find('a')
        title = a_tag.text
        link = a_tag['href']
        articles.append({'title': title, 'link': link})
    # Print the first 5 articles
    for i, article in enumerate(articles[:5], 1):
        print(f"{i}. {article['title']}")
        print(f"   {article['link']}")
else:
    print(f"Request failed: {response.status_code}")

3. Automated Testing

Using requests allows you to write automated test scripts to validate the APIs of web services:

import requests# Base URL for testing API
BASE_URL = 'https://api.example.com/v1'
def test_user_creation():
    # Test user creation API
    data = {
        'username': 'test_user',
        'email': '[email protected]',
        'password': 'test123'
    }
    response = requests.post(f'{BASE_URL}/users', json=data)
    if response.status_code == 201:
        print("User created successfully")
        user_id = response.json()['id']
        return user_id
    else:
        print(f"User creation failed: {response.json()}")
        return None
def test_user_deletion(user_id):
    # Test user deletion API
    response = requests.delete(f'{BASE_URL}/users/{user_id}')
    if response.status_code == 204:
        print("User deleted successfully")
    else:
        print(f"User deletion failed: {response.status_code}")# Execute tests
user_id = test_user_creation()
if user_id:
    test_user_deletion(user_id)

6. Conclusion and Interaction

Through this introduction, you should now recognize the powerful role of the requests library in daily Python programming. It not only simplifies the process of HTTP requests but also provides rich features to handle various complex situations.

Have you also used the requests library in your projects? What interesting application scenarios or technical challenges have you encountered?

Feel free to share your experiences and insights in the comments! If you have any questions about the requests library or want to learn more about advanced usage, feel free to ask, and we can discuss and learn together.