Requests: A Simple HTTP Request Library

In Python, Requests is a very popular and easy-to-use library that allows us to conveniently communicate with websites, send HTTP requests, and retrieve website data. It helps us access data on the internet, perform data scraping, submit forms, and more. It is an essential tool for web programming and web scraping development.

1. What is Requests?

Requests is a library in Python used to send HTTP requests. HTTP (Hypertext Transfer Protocol) is the foundational protocol for communication between clients and servers on the internet. With Requests, we can easily send HTTP requests (such as GET and POST requests) and receive the response data. Whether it’s fetching webpage content, submitting forms, or accessing APIs, Requests enables us to accomplish these tasks efficiently.

2. Installing Requests

First, we need to install the Requests library using Python’s package manager pip:

pip install requests

Once installed, you can import and use it in your code:

import requests

3. Sending a GET Request

GET requests are one of the most commonly used types of HTTP requests. They are used to retrieve data from the server. For example, we can use a GET request to fetch the content of a webpage.

import requests

# Send a GET request
response = requests.get('https://www.example.com')

# Print the returned status code
print(response.status_code)  # Outputs 200, indicating success

# Print the first 500 characters of the webpage content
print(response.text[:500])  # Prints the first 500 characters of the webpage

Here, response is the response object returned by Requests. We can obtain the request’s status code through response.status_code and the content of the webpage through response.text.

4. Sending a POST Request

POST requests are typically used to send data to the server, such as submitting forms or uploading files. Unlike GET requests, POST requests usually send data in the request body.

import requests

# Prepare the data to send
data = {'username': 'user123', 'password': 'mypassword'}

# Send a POST request
response = requests.post('https://httpbin.org/post', data=data)

# Print the returned response content
print(response.text)

Using the requests.post() method, we can send a POST request to a specified URL and pass the data to be submitted. httpbin.org is a tool website for testing HTTP requests, and the returned data will display the content you submitted.

5. Handling Request Parameters

In practical use, we sometimes need to send query parameters or custom request headers. Requests also provides convenient ways to handle these parameters.

5.1 Sending Query Parameters

Sometimes, we need to include query parameters (like search criteria) in a GET request. This can be easily achieved using the params parameter:

import requests

# Send a GET request with query parameters
params = {'q': 'Python', 'page': 2}
response = requests.get('https://www.google.com/search', params=params)

# Print the returned status code
print(response.status_code)

params is a dictionary that contains the query parameters to be passed in the request.

5.2 Setting Request Headers

Request headers are part of the HTTP request and contain relevant information about the request, such as the User-Agent, which tells the server what kind of device or browser the request is coming from. You can set the request headers through the headers parameter:

import requests

# Set custom request headers
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get('https://www.example.com', headers=headers)

# Print the returned webpage content
print(response.text[:500])

6. Handling Response Data

After sending a request, Requests returns a response object through which we can access various information about the response, such as the status code, response content, and response headers.

6.1 Getting JSON Data

Many APIs return data in JSON format. In Requests, we can use the response.json() method to convert JSON data into a Python dictionary.

import requests

# Request an API that returns JSON data
response = requests.get('https://jsonplaceholder.typicode.com/todos/1')

# Get JSON data
data = response.json()
print(data)

In this example, response.json() converts the returned JSON data into a Python dictionary object for further processing.

6.2 Getting Response Headers

Response headers contain information related to the response, such as server type and content type. You can access them through response.headers:

import requests

# Send a GET request
response = requests.get('https://www.example.com')

# Get response headers
print(response.headers)

The response headers are a dictionary that contains various header information returned by the server.

7. Error Handling

When sending HTTP requests, you may encounter errors such as network issues or request timeouts. Requests provides a simple way to handle these exceptions.

import requests

try:
    # Send the request
    response = requests.get('https://www.example.com', timeout=5)
    response.raise_for_status()  # Raises an exception if the status code is not 2xx
    print(response.text)
except requests.exceptions.RequestException as e:
    # Handle all exceptions
    print(f"Request error: {e}")

In this example, timeout=5 sets the timeout for the request; if there is no response after 5 seconds, an exception will be raised. raise_for_status() checks the returned status code, and if it is not 2xx, indicating a failure, it will raise an exception.

8. Summary

Requests is a powerful and easy-to-use HTTP request library that allows you to interact with various services on the internet in just a few lines of code. With GET requests, you can easily scrape webpage content; with POST requests, you can submit data to the server. Additionally, Requests supports handling query parameters, request headers, JSON data, and more. Its simplicity and ease of use make it one of the most popular web programming tools in Python.

Whether you are performing web scraping, API calls, or network data analysis, Requests is an indispensable tool. Mastering it will make it easier for you to interact with the world of the internet.