Chapter 2: HTTP Protocol and Network Basics
Introduction to HTTP Protocol
HTTP (HyperText Transfer Protocol) is one of the most widely used protocols on the Internet. Simply put, it is the “dialogue rules” between the browser and the server.
HTTP Workflow:
1. Browser sends request → 2. Server processes request → 3. Server returns response → 4. Browser displays content
URL Structure Analysis
URL (Uniform Resource Locator) is like the “house number” in the online world:
|
Plain Text https://www.example.com:8080/search?q=python&page=1#results │ │ │ │ │ │ │ │ │ │ │ └── Anchor │ │ │ │ └── Query Parameters │ │ │ └── Port Number │ │ └── Domain Name │ └── Protocol └── Complete URL |
Code Example: Parsing URL
|
Python from urllib.parse import urlparse url = “https://www.baidu.com/s?wd=python爬虫&ie=utf-8” parsed = urlparse(url) print(f”Protocol: {parsed.scheme}”) print(f”Domain: {parsed.netloc}”) print(f”Path: {parsed.path}”) print(f”Query Parameters: {parsed.query}”) |
HTTP Request Methods
GET Request – Retrieve Data
The most commonly used request method for retrieving information:
|
Python import requests # GET request example response = requests.get(‘https://httpbin.org/get’, params={‘name’: ‘python’, ‘type’: ‘crawler’}) print(f”Request URL: {response.url}”) print(f”Status Code: {response.status_code}”) print(f”Response Content: {response.json()}”) |
POST Request – Submit Data
Used to submit data to the server (e.g., forms, logins):
|
Python # POST request example data = { ‘username’: ‘testuser’, ‘password’: ‘testpass’ } response = requests.post(‘https://httpbin.org/post’, data=data) print(f”Submitted Data: {response.json()[‘form’]}”) |
Understanding HTTP Status Codes
Status codes tell us the result of the request:
Click the image to view the complete spreadsheet
Status Code Check Example:
|
Python def check_url_status(url): try: response = requests.get(url, timeout=5) if response.status_code == 200: print(f”✅ {url} is accessible”) elif response.status_code == 404: print(f”❌ {url} page does not exist”) else: print(f”⚠️ {url} status code: {response.status_code}”) except Exception as e: print(f”🚫 {url} access failed: {e}”) # Test different websites urls = [‘https://www.baidu.com’, ‘https://httpbin.org/status/404’] for url in urls: check_url_status(url) |
Detailed Explanation of HTTP Request Headers
Request headers contain important metadata:
|
Python # View request headers import requests response = requests.get(‘https://httpbin.org/headers’) headers_info = response.json() print(“Default Request Headers:”) for key, value in headers_info[‘headers’].items(): print(f”{key}: {value}”) |
Custom Request Headers:
|
Python # Simulate browser access headers = { ‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36’, ‘Accept’: ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8’, ‘Accept-Language’: ‘zh-CN,zh;q=0.9,en;q=0.8’, ‘Accept-Encoding’: ‘gzip, deflate, br’, ‘Connection’: ‘keep-alive’, } response = requests.get(‘https://httpbin.org/headers’, headers=headers) print(“Custom request headers applied!”) |
Cookie and Session Mechanism
Cookie – Client Storage
|
Python # Cookie usage example session = requests.Session() # The first request will set the cookie response1 = session.get(‘https://httpbin.org/cookies/set/name/python_spider’) # The second request will automatically carry the cookie response2 = session.get(‘https://httpbin.org/cookies’) print(f”Current Cookies: {response2.json()}”) |
Session – Maintaining State
Session can maintain state across multiple requests:
|
Python # Simulate login to maintain session def simulate_login(): session = requests.Session() # Simulate login login_data = {‘username’: ‘test’, ‘password’: ‘test’} login_response = session.post(‘https://httpbin.org/post’, data=login_data) # Subsequent requests will maintain login state profile_response = session.get(‘https://httpbin.org/cookies’) return session session = simulate_login() print(“Session maintained successfully!”) |
Browser Developer Tools Practical Application
Mastering browser debugging tools is an essential skill for web scraping development:
Steps to use Chrome Developer Tools:
Press F12 to open Developer Tools
Switch to the Network tab
Refresh the page or perform an action
Observe HTTP requests and responses
Key information to focus on:
Request URL
Request Method
Request Headers
Response Headers
Response Content
Practical Exercise
|
Python def analyze_website(url): “””Analyze basic information of the website””” try: response = requests.get(url, timeout=10) print(f”🌐 Website Analysis: {url}”) print(f”📊 Status Code: {response.status_code}”) print(f”🕒 Response Time: {response.elapsed.total_seconds():.2f} seconds”) print(f”📏 Content Size: {len(response.content)} bytes”) print(f”🔤 Encoding: {response.encoding}”) print(f”🍪 Number of Cookies: {len(response.cookies)}”) except Exception as e: print(f”❌ Analysis failed: {e}”) # Test analysis analyze_website(“https://www.python.org”) |
Chapter Summary
Through this chapter, you have learned:
✅ The basic working principles of the HTTP protocol
✅ The structure of URLs and the meaning of each part
✅ The differences and uses of GET and POST requests
✅ The meanings of HTTP status codes
✅ The roles of request and response headers
✅ The mechanisms of Cookies and Sessions
✅ The use of browser developer tools
Next Chapter Preview: We will delve into Python network programming, focusing on advanced usage of the requests library.
Homework:
Analyze a website you frequently use using developer tools
Write code to check the access status of 10 websites
Try sending different types of data to https://httpbin.org/post