Web Data Acquisition and Processing with Python

1. Using requests to Acquire Web Page Content and Handle Encoding

Scenario: Acquire HTML content from a web page and save it correctly (solving Chinese garbled text issues)

import requests
url = 'https://www.tjwenming.cn/'
res = requests.get(url)

# The Chinese characters in the response data are garbled, so encoding operations need to be performed on the response object
# Response object.encoding = 'charset value'
res.encoding = 'gb2312'  # Set according to the actual encoding of the web page

# The reason for garbled text in the browser: the browser displays normally when the two encodings are consistent (the encoding of the saved file and the parsing encoding of the browser)
# Solution: Make the two encodings consistent, modify the charset value before saving the data
s = res.text  # Get the string type response content
s = s.replace('gb2312', 'utf-8')  # Change the encoding declaration in the web page meta information to utf-8

# Save the file with utf-8 encoding to ensure Chinese characters are displayed correctly
with open('天津文明网.html', 'w', encoding='utf-8') as f:
    f.write(s)

Knowledge Point Explanation:

1. <span><span>requests.get(url)</span></span>: Sends a GET request to acquire web page content

2. Encoding Handling: Set the response encoding through<span><span>res.encoding</span></span> to solve Chinese garbled text

3. File Saving: Modify the encoding declaration in the web page meta information to ensure the saved file encoding is consistent with the declaration, avoiding garbled text in browser parsing

4. <span><span>res.text</span></span>: Get the string type response content (suitable for text data such as HTML)

2. Using requests to Download Binary Files

Scenario: Download binary resources such as images and videos

import requests
url = 'http://pic.enorth.com.cn/005/026/920/00502692031_21660ab6.jpg'

res = requests.get(url)

# Common file extensions:
# Text document: txt; HTML file: html; Python file: py
# Images: png, jpg, gif; Videos: mp4; Audio: mp3
# Excel: xls, xlsx (requires special modules to handle, cannot be opened directly)

# When handling binary data (images, videos, etc.), use res.content, and the write mode should be wb
# When dealing with byte operations, do not add the encoding parameter
with open('1.png', 'wb') as f:
    f.write(res.content)

Knowledge Point Explanation:

1. <span><span>res.content</span></span>: Get the byte type response content (suitable for binary resources such as images, videos, audio)

2. Binary Writing: Use<span><span>open(..., 'wb')</span></span> mode to write byte data

3. Storage Characteristics of Different Types of Files: Text files use<span><span>res.text</span></span> and<span><span>'w'</span></span> mode, binary files use<span><span>res.content</span></span> and<span><span>'wb'</span></span> mode

3. Handling JSON Format Responses

Scenario: Parse JSON data returned from the requested URL (dictionary/list structure)

import requests
''' 
Ways to get response content:
- Text data (such as HTML): response object.text → string type
- Binary data (such as images): response object.content → byte type
- JSON format data (dictionary/list): response object.json() → dictionary or list type
'''

# Tencent recruitment target data request address
url = 'https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1756296971366&countryId=&cityId=&bgIds=&productId=&categoryId=40001001,40001002,40001003,40001004,40001005,40001006&parentCategoryId=&attrId=1&keyword=&pageIndex=3&pageSize=10&language=zh-cn&area=cn'
res = requests.get(url)

# Parse JSON data (convert to dictionary type)
res_data = res.json()

# Extract recruitment information: title, city, job responsibilities
for i in res_data['Data']['Posts']:
    RecruitPostName = i['RecruitPostName']  # Job title
    LocationName = i['LocationName']        # City
    Responsibility = i['Responsibility']    # Job responsibilities
    print(RecruitPostName, LocationName, Responsibility)
    print('========================')

# Example of accessing values in a nested dictionary structure
res_data_demo = {
    'Code':200,
    'Data':{'Count':818,'Posts':[
        {'LocationName':'Beijing'},
        {'LocationName':'Changsha1'},
        {'LocationName':'Changsha2'},
        {'LocationName':'Changsha3'}
    ]}
}

# Access values layer by layer (from outer dictionary to inner dictionary)
print(res_data_demo['Data']['Posts'][0]['LocationName'])  # Get the first city: Beijing

# Loop to get all data in the list
for item in res_data_demo['Data']['Posts']:
    print(item['LocationName'])  # Print all cities one by one

Knowledge Point Explanation:

1. <span><span>res.json()</span></span>: Converts JSON format response content to a Python dictionary or list for easy data extraction

2. Accessing Nested Structures: Access target data layer by layer through key names (dictionaries) and indices (lists)

3. Loop Traversal: For list type data, use for loop to batch extract information

4. Paginated Data Acquisition

Scenario: Acquire multi-page data by looping and changing URL parameters

import requests

# Pagination principle: Change the pageIndex parameter in the URL (starting from 1)
# Example: Get data from pages 1-10
for page in range(1, 11):  # page takes values from 1 to 10
    # Format URL, replace pageIndex parameter
    url = f'https://careers.tencent.com/tencentcareer/api/post/Query?timestamp=1756296971366&countryId=&cityId=&bgIds=&productId=&categoryId=40001001,40001002,40001003,40001004,40001005,40001006&parentCategoryId=&attrId=1&keyword=&pageIndex={page}&pageSize=10&language=zh-cn&area=cn'
    
    res = requests.get(url)
    res_data = res.json()  # Parse current page JSON data
    
    # Extract and print recruitment information for the current page
    for i in res_data['Data']['Posts']:
        RecruitPostName = i['RecruitPostName']
        LocationName = i['LocationName']
        Responsibility = i['Responsibility']
        print(RecruitPostName, LocationName, Responsibility)
        print('========================')
    
    print(f'Page {page} data has been fully acquired')

Knowledge Point Explanation:

1. Pagination Parameters: Most pagination is controlled by<span><span>pageIndex</span></span> (page number) or<span><span>offset</span></span> (offset) parameters

2. Loop Implementation: Use for loop to generate URLs for different page numbers, batch request multi-page data

3. Batch Processing: Parse each page of data separately to achieve full data acquisition