Fundamentals of HTTP Protocol

Overview

HTTP (HyperText Transfer Protocol) is the most widely used network protocol on the Internet. As a web crawler developer, a deep understanding of the HTTP protocol is an essential foundational skill. This article will start from basic concepts and provide a detailed introduction to various aspects of the HTTP protocol.

1. Introduction to HTTP Protocol

1.1 What is HTTP Protocol

HTTP is a stateless application layer protocol based on a request-response model. It defines how clients (such as browsers and web crawlers) communicate with servers.

Characteristics of HTTP Protocol:

Statelessness: Each request is independent, and the server does not retain client state information.
Based on TCP/IP: The HTTP protocol is built on reliable TCP connections.
Request-Response Model: The client sends a request, and the server returns a response.
Text Protocol: HTTP messages are transmitted in plain text format (except for HTTP/2).

1.2 HTTP Workflow

Client  Server|||1. Establish TCP connection||----------------------->||||2. Send HTTP request||----------------------->||||3. Server processes request||||4. Return HTTP response||<-----------------------||||5. Close TCP connection||<-----------------------|

2. Detailed Explanation of HTTP Request Methods

HTTP defines several request methods, each with specific purposes and semantics.

2.1 GET Method

Purpose: Request to retrieve a specified resourceCharacteristics:

Parameters are passed via URL
Requests can be cached
Requests are retained in browser history
Data length is limited
Used only for requesting data

Example:

GET /api/users?page=1&limit=10 HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0

2.2 POST Method

Purpose: Submit data to the serverCharacteristics:

Parameters are passed in the request body
Requests are not cached
Requests are not retained in browser history
No limit on data length
Can transmit various types of data

Example:

POST /api/users HTTP/1.1
Host: example.com
Content-Type: application/json
Content-Length: 45
{"name":"Zhang San","email":"[email protected]"}

2.3 PUT Method

Purpose: Update or create a specified resourceCharacteristics:

Idempotence: Multiple executions of the same PUT request yield the same result
Typically used to update the entire resource

Example:

PUT /api/users/123 HTTP/1.1
Host: example.com
Content-Type: application/json
{"id":123,"name":"Li Si","email":"[email protected]"}

2.4 DELETE Method

Purpose: Delete a specified resourceCharacteristics:

Idempotence: Multiple deletions of the same resource yield the same result
Used to delete resources on the server

Example:

DELETE /api/users/123 HTTP/1.1
Host: example.com

2.5 Other Methods

HEAD: Similar to GET, but only returns response headers, not the response body
OPTIONS: Query the HTTP methods supported by the server
PATCH: Partially modify a resource
TRACE: Echo the request received by the server (rarely used)
CONNECT: Establish a tunnel to the server (used for HTTPS proxies)

3. Detailed Explanation of Request and Response Headers

3.1 Common Request Headers

User-Agent

Identifies the type and version information of the client

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36

Accept

Specifies the content types that the client can accept

Accept: text/html, application/xhtml+xml, application/xml;q=0.9, */*;q=0.8

Accept-Language

Specifies the preferred language of the client

Accept-Language: zh-CN, zh;q=0.9, en;q=0.8

Accept-Encoding

Specifies the encoding formats supported by the client

Accept-Encoding: gzip, deflate, br

Referer

Indicates the URL of the referring page

Referer: https://www.google.com/

Cookie

Sends stored cookie information from the client

Cookie: sessionid=abc123; csrftoken=xyz789

Authorization

Contains authentication information

Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

3.2 Common Response Headers

Content-Type

Specifies the media type of the response body

Content-Type: application/json; charset=utf-8

Content-Length

Specifies the length of the response body (in bytes)

Content-Length: 1024

Set-Cookie

Sets cookie information

Set-Cookie: sessionid=abc123; Path=/; HttpOnly; Secure

Cache-Control

Controls caching behavior

Cache-Control: no-cache, no-store, must-revalidate

Location

Used for redirection, specifies the new URL

Location: https://www.example.com/new-page

4. Detailed Explanation of HTTP Status Codes

HTTP status codes are standardized representations of the results of server request processing.

4.1 1xx Informational Status Codes

100 Continue: The client should continue sending the request
101 Switching Protocols: The server is switching protocols

4.2 2xx Success Status Codes

200 OK: The request was successful
201 Created: The request was successful and a new resource was created
202 Accepted: The request has been accepted but has not yet been completed
204 No Content: The request was successful but no content was returned

4.3 3xx Redirection Status Codes

301 Moved Permanently: Permanent redirection
302 Found: Temporary redirection
304 Not Modified: Resource not modified, can use cache
307 Temporary Redirect: Temporary redirection (maintaining request method)

4.4 4xx Client Error Status Codes

400 Bad Request: Request syntax error
401 Unauthorized: Authentication required
403 Forbidden: Server refuses the request
404 Not Found: The requested resource does not exist
405 Method Not Allowed: Request method not allowed
429 Too Many Requests: Requests are too frequent

4.5 5xx Server Error Status Codes

500 Internal Server Error: Internal server error
502 Bad Gateway: Gateway error
503 Service Unavailable: Service unavailable
504 Gateway Timeout: Gateway timeout

5. Cookie and Session Mechanisms

5.1 Cookie Mechanism

A cookie is a small piece of data sent from the server to the user’s browser and stored locally.

Cookie Workflow:

The server sends cookies via the Set-Cookie response header
The browser saves the cookie
Subsequent requests automatically carry the cookie

Cookie Attributes:

Name/Value: The name and value of the cookie
Domain: The scope of the cookie
Path: The path for which the cookie is valid
Expires/Max-Age: The expiration time of the cookie
HttpOnly: Prevents JavaScript access
Secure: Transmitted only over HTTPS connections
SameSite: Controls cookie sending during cross-site requests

Example:

Set-Cookie: username=zhangsan; Domain=.example.com; Path=/; Expires=Wed, 09 Jun 2024 10:18:14 GMT; HttpOnly; Secure

5.2 Session Mechanism

A session is a server-side session management mechanism, typically used in conjunction with cookies.

Session Workflow:

When a user first visits, the server creates a session
The server generates a unique session ID
The session ID is sent to the client via a cookie
The client carries the session ID in subsequent requests
The server identifies the user based on the session ID

6. HTTPS and Certificate Verification

6.1 Introduction to HTTPS

HTTPS (HTTP Secure) is the secure version of HTTP, providing encrypted communication through the SSL/TLS protocol.

Advantages of HTTPS:

Data Encryption: Prevents data from being eavesdropped
Data Integrity: Prevents data from being tampered with
Authentication: Verifies the identity of the server

6.2 SSL/TLS Handshake Process

Client  Server|||1. ClientHello||----------------------->||||2. ServerHello||3. Certificate||4. ServerHelloDone||<-----------------------||||5. ClientKeyExchange||6. ChangeCipherSpec||7. Finished||----------------------->||||8. ChangeCipherSpec||9. Finished||<-----------------------||||Encrypted communication begins|

6.3 Certificate Verification

Digital certificates are used to verify the identity of the server and contain the following information:

Server Public Key
Server Domain Name
Certificate Authority (CA)
Certificate Validity Period
Digital Signature

Certificate Verification Process:

Check if the certificate is expired
Verify the integrity of the certificate chain
Check if the certificate has been revoked
Verify if the domain name matches

7. URL Encoding and Decoding

7.1 Why URL Encoding is Necessary

URLs can only contain specific characters from the ASCII character set. When a URL contains special characters or non-ASCII characters, encoding is required.

7.2 URL Encoding Rules

Reserved Characters: Characters with special meanings need to be encoded
: → %3A
/ → %2F
? → %3F
# → %23
& → %26
= → %3D
Non-ASCII Characters: Use UTF-8 encoding followed by percent encoding
中 → %E4%B8%AD
文 → %E6%96%87

7.3 URL Encoding in Python

import urllib.parse
# URL encoding
original = "https://example.com/search?q=Python爬虫&type=tutorial"
encoded = urllib.parse.quote(original, safe=':/?#[]@!$&\'()*+,;=')
print(encoded)
# URL decoding
decoded = urllib.parse.unquote(encoded)
print(decoded)
# Query parameter encoding
params = {'q': 'Python爬虫', 'type': 'tutorial'}
query_string = urllib.parse.urlencode(params)
print(query_string)
# q=Python%E7%88%AC%E8%99%AB&type=tutorial

8. Comparison of HTTP Versions

8.1 HTTP/1.0

Each request requires establishing a new TCP connection
No support for persistent connections
No support for pipelining

8.2 HTTP/1.1

Supports persistent connections (Connection: keep-alive)
Supports pipelined requests
Increased caching control options
Supports chunked transfer encoding

8.3 HTTP/2

Binary protocol
Multiplexing
Server push
Header compression

8.4 HTTP/3

Based on QUIC protocol
Faster connection establishment
Better network adaptability

9. Practical Examples

9.1 Sending HTTP Requests with Python

import requests
# GET request
response = requests.get('https://httpbin.org/get',
                       params={'key': 'value'},
                       headers={'User-Agent': 'My Spider 1.0'})
print(f"Status Code: {response.status_code}")
print(f"Response Headers: {response.headers}")
print(f"Response Content: {response.text}")
# POST request
data = {'username': 'admin', 'password': '123456'}
response = requests.post('https://httpbin.org/post',
                        data=data,
                        headers={'Content-Type': 'application/x-www-form-urlencoded'})
print(f"Response JSON: {response.json()}")

9.2 Handling Cookies and Sessions

import requests
# Using Session to maintain a session
session = requests.Session()
# Login
login_data = {'username': 'admin', 'password': '123456'}
response = session.post('https://example.com/login', data=login_data)
# Accessing a page that requires login
response = session.get('https://example.com/dashboard')
print(response.text)
# View cookies
print(session.cookies)

10. Summary

The HTTP protocol is the foundation of web crawling, and understanding its various aspects is crucial for developing efficient and stable web crawler programs. This article covered:

Basic Concepts of HTTP: Request-response model, stateless characteristics
Request Methods: Usage scenarios for GET, POST, PUT, DELETE, etc.
Request and Response Headers: Meanings and functions of various header fields
Status Codes: Meanings and handling of different status codes
Cookies and Sessions: Session management mechanisms
HTTPS: Secure communication protocol
URL Encoding: Handling of special characters

By mastering these foundational concepts, you will better understand how network requests work, laying a solid foundation for future web crawling development. In practical web crawling development, it is often necessary to analyze website HTTP requests, set appropriate request headers, and handle various status codes and redirections, all of which require a deep understanding of the HTTP protocol.

Next Steps in Learning

After mastering the basics of the HTTP protocol, it is recommended to continue learning:

Basic principles and workflows of web crawling
Usage of Python networking libraries
Web content parsing techniques
Anti-crawling mechanisms and countermeasures

Through gradual learning, you will be able to develop powerful web crawler programs.

Fundamentals of HTTP Protocol

Overview

1. Introduction to HTTP Protocol

1.1 What is HTTP Protocol

1.2 HTTP Workflow

2. Detailed Explanation of HTTP Request Methods

2.1 GET Method

2.2 POST Method

2.3 PUT Method

2.4 DELETE Method

2.5 Other Methods

3. Detailed Explanation of Request and Response Headers

3.1 Common Request Headers

User-Agent

Accept

Accept-Language

Accept-Encoding

Referer

Cookie

Authorization

3.2 Common Response Headers

Content-Type

Content-Length

Set-Cookie

Cache-Control

Location

4. Detailed Explanation of HTTP Status Codes

4.1 1xx Informational Status Codes

4.2 2xx Success Status Codes

4.3 3xx Redirection Status Codes

4.4 4xx Client Error Status Codes

4.5 5xx Server Error Status Codes

5. Cookie and Session Mechanisms

5.1 Cookie Mechanism

5.2 Session Mechanism

6. HTTPS and Certificate Verification

6.1 Introduction to HTTPS

6.2 SSL/TLS Handshake Process

6.3 Certificate Verification

7. URL Encoding and Decoding

7.1 Why URL Encoding is Necessary

7.2 URL Encoding Rules

7.3 URL Encoding in Python

8. Comparison of HTTP Versions

8.1 HTTP/1.0

8.2 HTTP/1.1

8.3 HTTP/2

8.4 HTTP/3

9. Practical Examples

9.1 Sending HTTP Requests with Python

9.2 Handling Cookies and Sessions

10. Summary

Next Steps in Learning

Related posts

Leave a Comment Cancel reply