Parsel is a lightweight and efficient web data extraction library in the Python ecosystem, developed based on lxml (a high-performance XML/HTML parser). It supports CSS selectors, XPath expressions, and regular expressions, making it flexible for various web data extraction scenarios (either used independently or in conjunction with the Scrapy framework).
Advantages
- 1. Multi-selector support: Compatible with CSS, XPath, and regular expressions, allowing you to choose the most convenient extraction method as needed;
- 2. High performance: Based on lxml, the parsing speed far exceeds that of BeautifulSoup;
- 3. Lightweight and flexible: Minimal dependencies, can be used independently (no need for the Scrapy framework);
Installation
pip install parselExample
The core of Parsel is the <span>Selector</span> class — it encapsulates the parsed HTML/XML document and provides methods for data extraction using CSS, XPath, etc.
1. Initializing Selector
from parsel import Selector
# Example HTML text
html = """
<div class="movie-list">
<div class="movie" id="movie1">
<h3 class="title">The Wandering Earth 2</h3>
<p class="score">9.2</p>
<a href="/movie/296767" class="link">View Details</a>
</div>
<div class="movie" id="movie2">
<h3 class="title">Full River Red</h3>
<p class="score">8.5</p>
<a href="/movie/301284" class="link">View Details</a>
</div>
</div>
"""
# Create Selector object
# Method 1: Initialize from text
selector = Selector(text=html)
# Method 2: Read from requests response
import requests
response = requests.get("https://example.com/movies")
selector = Selector(text=response.content.decode("utf-8"))
selector = Selector(text=response.text)Data Extraction Methods
<span>Selector</span> provides three core extraction methods, all returning results as <span>SelectorList</span>
1. Extraction Functions
Whether using CSS or XPath, you ultimately need to use the following functions to obtain specific data:
| Method | Function | Return Type | Example |
<span>get()</span> |
Get the first matching result, returns <span>None</span> if no match |
String / None | <span>selector.css(".title::text").get()</span> |
<span>getall()</span> |
Get all matching results | List (String) | <span>selector.css(".title::text").getall()</span> |
<span>extract()</span> |
Same as <span>getall()</span> (deprecated method, use <span>getall()</span> instead) |
List (String) | <span>selector.xpath("//h3/text()").extract()</span> |
<span>extract_first()</span> |
Same as <span>get()</span> (deprecated method, use <span>get()</span> instead) |
String / None | <span>selector.xpath("//h3/text()").extract_first()</span> |
2. CSS Selectors
The CSS selector syntax is consistent with front-end development, suitable for extracting elements with known class names or IDs.
Common syntax examples
| Requirement | CSS Syntax | Example Code | Result (based on previous HTML) |
Select text with class <span>title</span> |
<span>.title::text</span> |
<span>selector.css(".movie .title::text").getall()</span> |
<span>["The Wandering Earth 2", "Full River Red"]</span> |
Select element with ID <span>movie1</span> |
<span>#movie1</span> |
<span>selector.css("#movie1 .score::text").get()</span> |
<span>"9.2"</span> |
Select <span>a</span> tag’s <span>href</span> attribute |
<span>a.link::attr(href)</span> |
<span>selector.css(".movie a.link::attr(href)").getall()</span> |
<span>["/movie/296767", "/movie/301284"]</span> |
Select the 2nd <span>movie</span> element |
<span>.movie:nth-child(2) .title::text</span> |
<span>selector.css(".movie:nth-child(2) .title::text").get()</span> |
<span>"Full River Red"</span> |
Chained calls: first select the parent element, then select child elements within the parent
# 1. First select all .movie parent elements
movie_selectors = selector.css(".movie")
# 2. Iterate over parent elements to extract each movie's information (chained calls)
for movie in movie_selectors:
title = movie.css(".title::text").get() # Only search within the current .movie
score = movie.css(".score::text").get()
link = movie.css("a.link::attr(href)").get()
print(f"Movie: {title}, Score: {score}, Link: {link}")3. XPath Selectors
XPath supports more complex node location (such as by text content, hierarchical relationships), suitable for scenarios where CSS is difficult to implement.
Common syntax examples
| Requirement | XPath Syntax | Example Code | Result (based on previous HTML) |
Select all <span>h3</span> text |
<span>//h3/text()</span> |
<span>selector.xpath("//h3/text()").getall()</span> |
<span>["The Wandering Earth 2", "Full River Red"]</span> |
Select text with class <span>score</span> |
<span>//p[@class="score"]/text()</span> |
<span>selector.xpath("//p[@class='score']/text()").getall()</span> |
<span>["9.2", "8.5"]</span> |
Select <span>a</span> tag’s <span>href</span> attribute |
<span>//a/@href</span> |
<span>selector.xpath("//a/@href").getall()</span> |
<span>["/movie/296767", "/movie/301284"]</span> |
| Select movie titles with score > 9 | <span>//div[./p[@class="score"]/text() > 9]/h3/text()</span> |
<span>selector.xpath("//div[./p[@class='score']/text() > 9]/h3/text()").get()</span> |
<span>"The Wandering Earth 2"</span> |
Relative XPath
When traversing parent elements, XPath needs to start with <span>.</span> to indicate “current node”; otherwise, it will match globally:
movie_selectors = selector.xpath("//div[@class='movie']")
for movie in movie_selectors:
# Use .// to indicate "search within the current movie node"
title = movie.xpath(".//h3/text()").get()
score = movie.xpath(".//p[@class='score']/text()").get()
print(f"{title}: {score}")4. Method 3: Regular Expressions
When the text format is irregular (such as containing extra characters), use <span>re()</span> or <span>re_first()</span> combined with regular expressions for extraction.
# If the HTML contains score as <p class="score">Score: 9.2</p>
score_text = selector.css(".score::text").get() # "Score: 9.2"
# Use regex to extract the numeric part
score = selector.css(".score::text").re_first(r"Score:(\d+\.\d+)") # "9.2"
# Link format: /movie/296767 → extract 296767
links = selector.css("a.link::attr(href)").getall() # ["/movie/296767", "/movie/301284"]
# Use re() to extract numbers from each link
movie_ids = selector.css("a.link::attr(href)").re(r"/movie/(\d+)") # ["296767", "301284"]Example: Playwright + Parsel Handling Dynamic Web Pages
from parsel import Selector
from playwright.sync_api import sync_playwright
# Use Playwright to render dynamic web pages
with sync_playwright() as p:
browser = p.chromium.launch(headless=True) # Headless mode
page = browser.new_page()
page.goto("https://example.com/dynamic-movies") # Dynamic web page
page.wait_for_selector(".movie") # Wait for target element to load
html = page.content() # Get rendered HTML
browser.close()
# Use Parsel to extract data
selector = Selector(text=html)
titles = selector.css(".movie .title::text").getall()
print(titles)Parsel is a great helper for parsing HTML