Requests: A Powerful HTTP Request Library for Python!

In today’s digital age, web requests have become an indispensable part of software development. Whether it’s fetching web content, calling API interfaces, uploading and downloading files, or implementing automated testing, web requests are essential.

For Python programmers, the Requests library is the preferred tool for making HTTP requests. Requests is an elegant and simple HTTP library that provides an intuitive API for Python, making it exceptionally easy to send HTTP requests.

Whether it’s common request methods like GET, POST, PUT, DELETE, or handling request headers, parameters, response content, and cookies, Requests can handle it all with ease.

Requests: A Powerful HTTP Request Library for Python!

Installing the Requests Library

To install the Requests library, simply use the pip command:

pip install requests

Basic Usage

  1. Sending a GET RequestSending a GET request using Requests is very simple; just call the requests.get() method:
import requests
# Send GET request
response = requests.get('https://api.github.com/users/octocat')
# Get response status code
print(response.status_code)  # Output: 200
# Get response content
print(response.text)  # Output JSON formatted response content
# Parse response content as JSON
data = response.json()
print(data['login'])  # Output: octocat

2. Sending a GET Request with ParametersYou can pass query parameters through the params argument:

# Define query parameters
params = {'q': 'python', 'page': 1}
# Send GET request with parameters
response = requests.get('https://api.github.com/search/repositories', params=params)
# Print request URL
print(response.url)  # Output: https://api.github.com/search/repositories?q=python&page=1

3. Sending a POST RequestUse the requests.post() method to send a POST request and pass form data through the data argument:

# Define form data
data = {'username': 'doubao', 'password': '123456'}
# Send POST request
response = requests.post('https://example.com/login', data=data)
# Get response content
print(response.text)

4. Handling Response HeadersYou can access response header information through response.headers:

# Get response headers
headers = response.headers
# Print Content-Type
print(headers['Content-Type'])  # Output: application/json; charset=utf-8

Advanced Usage

  1. Setting Request HeadersYou can set request headers through the headers argument:
# Define request headers
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}
# Send GET request with headers
response = requests.get('https://www.example.com', headers=headers)

Handling CookiesRequests can automatically handle cookies:

# Send request and get cookies
response = requests.get('https://www.example.com')
# Get cookies
cookies = response.cookies
# Send request with cookies
response = requests.get('https://www.example.com/profile', cookies=cookies)

3. Session PersistenceUsing a Session object can maintain session state:

# Create Session object
session = requests.Session()
# Login request
session.post('https://www.example.com/login', data={'username': 'doubao', 'password': '123456'})
# Access a page that requires login
response = session.get('https://www.example.com/dashboard')
# Close session
session.close()

4. Timeout SettingsYou can set the request timeout using the timeout argument:

# Set timeout to 5 seconds
response = requests.get('https://www.example.com', timeout=5)

Practical Application Scenarios

  1. Web ScrapingIn web scraping development, the Requests library is a fundamental tool for obtaining web content. By sending HTTP requests to fetch the source code of web pages, you can then use parsing libraries to extract the needed information.
  2. API CallsMany websites and services provide API interfaces, and the Requests library makes it easy to call these APIs to retrieve data or perform operations. For example, calling a weather API to get weather information or a social media API to post content.
  3. Automated TestingIn automated testing, the Requests library can be used to test the API interfaces of web services. By sending various requests, you can verify the correctness and stability of the interfaces.
  4. Data CollectionBusinesses and individuals often need to collect data from the web, and the Requests library can help automate this process. For example, collecting product information from e-commerce platforms or news content from news websites.

Final Example Code

Below is an example code that implements a simple web scraper using the Requests library:

import requests
from bs4 import BeautifulSoup
import csv
import time
def get_book_info(url):    """Get information about a single book"""    headers = {        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
    try:        # Send request        response = requests.get(url, headers=headers, timeout=10)
        # Check response status code        if response.status_code == 200:            # Parse HTML content            soup = BeautifulSoup(response.text, 'html.parser')
            # Extract book information            title = soup.find('h1').text.strip()            price = soup.find('p', class_='price_color').text.strip()            availability = soup.find('p', class_='instock availability').text.strip()
            return {                'title': title,                'price': price,                'availability': availability            }        else:            print(f"Request failed, status code: {response.status_code}")            return None    except Exception as e:        print(f"An error occurred: {e}")        return None
def get_all_books(base_url, pages=50):    """Get information about books across multiple pages"""    all_books = []
    for page in range(1, pages + 1):        page_url = f"{base_url}/catalogue/page-{page}.html"        print(f"Scraping page {page}: {page_url}")
        # Get book information for the current page        page_books = get_book_info(page_url)        if page_books:            all_books.extend(page_books)
        # Delay to avoid frequent requests        time.sleep(1)
    return all_books
def save_to_csv(books, filename='books.csv'):    """Save book information to a CSV file"""    if not books:        print("No data to save")        return
    # Define CSV file header    fieldnames = ['title', 'price', 'availability']
    # Write to CSV file    with open(filename, 'w', newline='', encoding='utf-8') as csvfile:        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        # Write header        writer.writeheader()
        # Write data        writer.writerows(books)
    print(f"Data has been saved to {filename}")
if __name__ == "__main__":    base_url = 'https://books.toscrape.com'
    # Get book information    books = get_all_books(base_url, pages=5)
    # Save to CSV file    save_to_csv(books)

This code implements a simple web scraper that collects book information from a book website. The scraper sends HTTP requests to fetch web content, then uses BeautifulSoup to parse the HTML, extract the required book titles, prices, and availability information, and finally saves the data to a CSV file.

The powerful features of the Requests library make it the go-to tool for Python programmers when making HTTP requests. Whether for simple API calls or complex web scraping development, Requests provides efficient and convenient solutions.

Through this article, I believe you now have a deeper understanding of the Requests library. Have you used the Requests library in your actual projects? What other features of the Requests library would you like to learn about? Feel free to leave a comment to share your experiences and thoughts!

Leave a Comment