Requests: Making HTTP Requests So Easy!

Today, I want to introduce you to a Python library that I use every day – the Requests library. I remember when I first started learning web scraping, I was overwhelmed by the complex syntax of urllib. Until I discovered Requests, it was like meeting an old friend! It makes sending HTTP requests as natural as breathing. Today, let’s dive into this super useful library!

Why Use Requests?

In our daily programming, we often need to interact with the web: calling APIs, scraping web pages, downloading files… If we use Python’s built-in urllib, the code can be quite cumbersome. Just look at this comparison:

# Using urllib
from urllib import request, parse
url = 'https://api.example.com/get?name=tom'
headers = {'User-Agent': 'Mozilla/5.0'}
req = request.Request(url, headers=headers)
response = request.urlopen(req)
data = response.read().decode('utf-8')

# Using requests
import requests
response = requests.get('https://api.example.com/get', 
                       params={'name': 'tom'},
                       headers={'User-Agent': 'Mozilla/5.0'})
data = response.text

Doesn’t it feel like requests is so much simpler? That’s why I recommend it to everyone!

Installation and Basic Usage

First, we need to install requests:

pip install requests

Now let’s see how to make a basic GET request:

import requests

# Sending a GET request
response = requests.get('https://httpbin.org/get')
print(response.text)  # View response content
print(response.status_code)  # View status code

# GET request with parameters
params = {'name': 'tom', 'age': 18}
response = requests.get('https://httpbin.org/get', params=params)
print(response.json())  # Directly parse JSON response

Tip: response.text returns a string, while response.json() directly returns the parsed JSON data, which is super convenient!

POST Requests Are Not Difficult Either

Sometimes we need to submit data to the server, and that’s where POST requests come in:

# Sending form data
data = {'username': 'tom', 'password': '123456'}
response = requests.post('https://httpbin.org/post', data=data)

# Sending JSON data
json_data = {'name': 'tom', 'skills': ['Python', 'Java']}
response = requests.post('https://httpbin.org/post', json=json_data)

Note: Use the data parameter to send form data and the json parameter to send JSON data. Requests will automatically set the correct Content-Type header.

Handling File Uploads and Downloads

# Uploading a file
files = {'file': open('photo.jpg', 'rb')}
response = requests.post('https://httpbin.org/post', files=files)

# Downloading a file
response = requests.get('https://example.com/photo.jpg')
with open('downloaded_photo.jpg', 'wb') as f:
    f.write(response.content)

Tip: When downloading files, use response.content to get binary content, not response.text!

Session and Cookie Handling

If you need to maintain a login state, using the Session object is super convenient:

# Creating a session
session = requests.Session()

# Logging in
session.post('https://example.com/login', 
             data={'username': 'tom', 'password': '123'})

# Accessing a page that requires login, cookies are automatically included
response = session.get('https://example.com/profile')

Useful Tips

1. Set a timeout to avoid the program hanging:

# Set 5 seconds timeout
response = requests.get('https://example.com', timeout=5)

1. Error Handling:

try:
    response = requests.get('https://example.com')
    response.raise_for_status()  # Check response status
except requests.RequestException as e:
    print(f'Request failed: {e}')

1. Using Proxies:

proxies = {
    'http': 'http://10.10.10.1:1080',
    'https': 'http://10.10.10.1:1080'
}
response = requests.get('https://example.com', proxies=proxies)

Practice Exercises

1. Try accessing https://httpbin.org/get and print out the headers returned by the server.
2. Use requests to download an image and save it locally.
3. Try sending a POST request to https://httpbin.org/post with your name and age included.

Friends, that’s it for today’s Python learning journey! Remember to get your hands on the code, and feel free to ask questions in the comments. Happy learning, and may your Python skills soar!