This resource package gathers 30 carefully selected practical Python web scraping projects, ranging from easy to difficult, covering the core knowledge points of scraping technology and various application scenarios. Whether you are a beginner just starting out or a developer looking to deepen your skills, these projects will help you quickly enhance your abilities in data capture, parsing, storage, and anti-scraping techniques.
Project Features and Covered Technologies:
-
Step-by-Step Progression: The project difficulty progresses from simple static page scraping to more complex dynamic web pages, API analysis, and advanced techniques such as CAPTCHA recognition.
-
Comprehensive Technology: The projects cover mainstream scraping libraries and frameworks such as
<span>requests</span>,<span>BeautifulSoup</span>,<span>lxml</span>,<span>Scrapy</span>,<span>Selenium</span>, and<span>Playwright</span>. -
Diverse Scenarios: The projects cover data scraping in multiple popular fields, including e-commerce, social media, video, news, literature, recruitment, and real estate, making them highly valuable references.
-
Clear Source Code: Each project provides complete, runnable source code with comments on key steps, making it easy to understand and modify.
Case 1: Scraping Douban Movie Top 250
Objective: To obtain the movie names, ratings, and number of reviews for the Douban Movie Top 250. Method: Use the requests library to send HTTP requests, the BeautifulSoup library to parse web content, and the csv library to save data to a CSV file.
import requests
from bs4 import BeautifulSoup
import time
import csv
import re
def crawl_douban_top250(): # Set request headers to simulate browser access headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' }
# Store all movie information movies = []
# Douban Top 250 has a total of 10 pages, with 25 movies per page for page in range(0, 250, 25): url = f'https://movie.douban.com/top250?start={page}'
try: # Send request response = requests.get(url, headers=headers) response.raise_for_status() # Raise an exception if the request fails
# Parse HTML soup = BeautifulSoup(response.text, 'html.parser')
# Find all movie entries items = soup.find_all('div', class_='item')
for item in items: # Extract movie information rank = item.find('em').text # Ranking title = item.find('span', class_='title').text # Chinese title
# Possible English title other_title = item.find('span', class_='other') if other_title: title += ' ' + other_title.text
# Extract information line: director, leading actors, year, region, genre info = item.find('div', class_='bd').find('p').text.strip() info_parts = info.split('\n')
# Parse director and leading actors director_actors = info_parts[0].strip() if len(info_parts) > 0 else ''
# Parse year, region, and genre year_region_genre = info_parts[1].strip().split('/') if len(info_parts) > 1 else ['', '', ''] year = year_region_genre[0].strip() if len(year_region_genre) > 0 else '' region = year_region_genre[1].strip() if len(year_region_genre) > 1 else '' genre = year_region_genre[2].strip() if len(year_region_genre) > 2 else ''
# Extract rating rating = item.find('span', class_='rating_num').text
# Extract number of reviews rating_count = item.find('div', class_='star').find_all('span')[-1].text rating_count = re.search(r'\d+', rating_count).group()
# Extract summary quote_elem = item.find('span', class_='inq') quote = quote_elem.text if quote_elem else ''
# Store movie information movie = { 'Ranking': rank, 'Title': title, 'Information': director_actors, 'Year': year, 'Region': region, 'Genre': genre, 'Rating': rating, 'Number of Reviews': rating_count, 'Summary': quote }
movies.append(movie) print(f"Scraped movie {rank}: {title}")
# Delay to avoid sending requests too frequently time.sleep(1)
except requests.RequestException as e: print(f"Request failed: {e}") break
# Save to CSV file with open('douban_top250.csv', 'w', newline='', encoding='utf-8-sig') as f: writer = csv.DictWriter(f, fieldnames=['Ranking', 'Title', 'Information', 'Year', 'Region', 'Genre', 'Rating', 'Number of Reviews', 'Summary']) writer.writeheader() writer.writerows(movies)
print("Scraping completed, data saved to douban_top250.csv") return movies
if __name__ == '__main__': crawl_douban_top250()Case 2: Scraping Maoyan Movie Top 100
Objective: To obtain the movie names, leading actors, and release dates for the Maoyan Movie Top 100.
Method: Use the requests library to send HTTP requests, regular expressions to parse web content, and save data to a txt file.
def get_movie_detail(url, headers): try: response = requests.get(url, headers=headers) response.encoding = 'utf-8' soup = BeautifulSoup(response.text, 'html.parser')
# Extract director information director_elem = soup.find('li', class_='celebrity') director = director_elem.find('a', class_='name').text.strip() if director_elem else 'Unknown'
# Extract genre type_elems = soup.find_all('li', class_='ellipsis')[0] movie_type = type_elems.text.strip() if type_elems else 'Unknown'
# Extract duration duration_elems = soup.find_all('li', class_='ellipsis')[1] if len(soup.find_all('li', class_='ellipsis')) > 1 else None duration = duration_elems.text.strip() if duration_elems else 'Unknown'
return { 'Director': director, 'Genre': movie_type, 'Duration': duration } except Exception as e: print(f"Failed to get details: {e}") return { 'Director': 'Unknown', 'Genre': 'Unknown', 'Duration': 'Unknown' }Remaining Python Scraping Projects
How to Obtain All Materials:1. Like + Revisit2. Follow the editor’s public account, then message for materials.