Requests: An Ultimate HTTP Request Library

Sometimes, programs need to interact with the outside world, such as fetching web content, submitting forms, downloading files… These operations rely on HTTP requests. The Python standard library also has a urllib that can be used to send HTTP requests, but it always feels a bit awkward to use. Thus, requests emerged! It is simply the ultimate answer for HTTP requests, incredibly simple and powerful.

Let’s have a good chat about this incredible HTTP request library today.

Making a Simple GET Request

The most common HTTP request is the GET request. For example, if you want to scrape a webpage, you can do it in one line of code with requests:

import requests

response = requests.get('https://jsonplaceholder.typicode.com/posts/1')
print(response.text)

The result is a segment of text in JSON format, which looks something like this:

{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

You can directly access the response content using the .text attribute. Isn’t that straightforward? No more worrying about encoding formats or decoding.

Adding Parameters to Requests

Often, requests need to include parameters, such as when you want to search for a specific keyword or filter certain conditions. In requests, you can simply place the parameters in params, making it incredibly easy.

response = requests.get('https://jsonplaceholder.typicode.com/posts', params={'userId': 1})
print(response.url)

The output will be:

https://jsonplaceholder.typicode.com/posts?userId=1

As you can see, params automatically converts the dictionary into a URL query string, without you having to worry about any &.

Tip: params is for GET requests. If you use POST requests, don’t mistakenly pass params; it won’t recognize it. POST parameters should use data or json, which we will discuss later.

POST Requests: Submitting Data

POST requests are generally used to submit data, such as login forms or file uploads. You simply need to stuff the data into data, and requests will handle it for you.

response = requests.post('https://httpbin.org/post', data={'key': 'value'})
print(response.json())

After running, you will see that the response contains a large JSON with the data you just submitted.

There is also a more advanced way to submit JSON formatted data (which is common since JSON is the universal language of the internet). You can directly use the json parameter, and requests will handle the encoding issues for you.

response = requests.post('https://httpbin.org/post', json={'key': 'value'})
print(response.json())

Be careful: Do not use both data and json in a POST request, as requests will get confused.

Request Headers and Cookies

Sometimes, sending a request requires including headers, such as setting a user agent (User-Agent) to masquerade as a browser or passing cookies. Requests makes this clear as well.

headers = {'User-Agent': 'Mozilla/5.0'}
cookies = {'session_id': '123456'}

response = requests.get('https://httpbin.org/headers', headers=headers, cookies=cookies)
print(response.text)

After running, you will find that headers and cookies are both attached to the request.

File Uploads and Downloads

Downloading files is one of the routine tasks for programmers. Using requests is incredibly convenient. Let’s look at a simple example:

response = requests.get('https://via.placeholder.com/150')
with open('image.png', 'wb') as f:
    f.write(response.content)

You can directly use .content to get the binary data and write it to a file. Images, PDFs, videos, anything can be handled.

Uploading files is also easy; you just use the files parameter:

files = {'file': open('image.png', 'rb')}
response = requests.post('https://httpbin.org/post', files=files)
print(response.json())

Once the file is uploaded, the server will echo back the information about the file you sent. Isn’t it easy and enjoyable?

Timeouts and Retries

Network requests are not always smooth; sometimes they timeout or fail. In such cases, requests provides a timeout parameter to prevent you from waiting indefinitely.

try:
    response = requests.get('https://httpbin.org/delay/3', timeout=2)
    print(response.text)
except requests.exceptions.Timeout:
    print('Request timed out!')

In this example, the server intentionally delays for 3 seconds, but we set our timeout to only 2 seconds, which directly raises a Timeout exception.

If you want the request to automatically retry a few times upon failure, you can configure it using requests.adapters. This is a bit advanced, and we won’t expand on it in this article.

SSL Verification and Ignoring

When accessing HTTPS sites, requests will verify the SSL certificate by default. This is good, but sometimes you may encounter invalid certificates or self-signed certificates. What to do? Simple, turn off verification.

response = requests.get('https://expired.badssl.com/', verify=False)
print(response.text)

Warning: Disabling SSL verification poses security risks; avoid doing so unless you are very sure it is safe.

Common Errors and Exception Handling

A benefit of using requests is that its error exceptions are quite user-friendly. For instance, if the network is down or the server returns a strange status code, requests will raise an exception.

try:
    response = requests.get('https://httpbin.org/status/404')
    response.raise_for_status()  # Raise an exception if the status code is not 200
except requests.exceptions.HTTPError as e:
    print(f'HTTP Error: {e}')
except requests.exceptions.ConnectionError:
    print('Network connection error!')
except requests.exceptions.RequestException as e:
    print(f'Request failed: {e}')

raise_for_status() is a great tool that helps you quickly identify HTTP errors, saving you from checking each status code.

Some Learning Tips

1. Try https://httpbin.org, which is a great tool for testing HTTP requests; it has almost everything you might want to test.
2. Don’t forget to check the official documentation; the requests documentation is particularly clear, https://docs.python-requests.org.
3. When encountering problems, don’t rush to blame requests itself; it’s likely an issue with your network environment or the server.

Requests is the kind of library you can’t live without once you use it. It is simple enough to make you feel that HTTP requests should be this way, and powerful enough to meet almost all your needs.