Hello everyone! The fifteenth article in the web scraping series is here! When dealing with complex JSON data, we often need to extract specific information from deeply nested structures. Today, we will introduce JSONPath, which is like ‘XPath’ tailored for JSON data, making data extraction easy and efficient.
1. JSONPath: The Query Language for JSON Data
What is JSONPath?
JSONPath is an information extraction tool specifically designed to extract parts of data from JSON documents. It functions similarly to XPath for XML, providing a concise syntax to locate and extract specific elements within JSON.
Why do we need JSONPath?
-
Deep Nesting: JSON data often has multiple layers of nested structures.
-
Precise Location: It is necessary to extract data that meets specific conditions.
-
Simplified Code: More concise and intuitive than manual traversal.
-
Universality: Implementations exist in various programming languages.
2. Basic Syntax of JSONPath
1. Installation and Import
# Install JSONPath library
# pip install jsonpath
from jsonpath import jsonpath # Import jsonpath function
2. Basic Selectors
# Sample data (Four Great Classical Novels of China)
data ={
"store":{
"book":[
{
"category":"reference",
"author":"Wu Cheng'en",
"title":"Journey to the West",
"price":8.95
},
{
"category":"fiction",
"author":"Cao Xueqin",
"title":"Dream of the Red Chamber",
"price":12.99
},
{
"category":"fiction",
"author":"Luo Guanzhong",
"title":"Romance of the Three Kingdoms",
"isbn":"0-553-21311-3",
"price":8.99
},
{
"category":"fiction",
"author":"Shi Nai'an",
"title":"Water Margin",
"isbn":"0-395-19395-8",
"price":22.99
}
],
"bicycle":{
"color":"red",
"price":19.95
}
}
}
# $ Root node (all JSONPath expressions start with $)
matches = jsonpath(data,'$.store.bicycle')
print("Bicycle information:", matches)
# . Select child nodes
matches = jsonpath(data,'$.store.bicycle.color')
print("Bicycle color:", matches)
# .. Recursive descent, search all matching nodes
matches = jsonpath(data,'$..price')
print("All prices:", matches)
# * Wildcard, match all elements
matches = jsonpath(data,'$.store.book[*]')
print("All books:",len(matches),"items")
3. Advanced JSONPath Query Techniques1. Array Indexing and Slicing
# Array indexing (starts from 0)
matches = jsonpath(data,'$..book[1]')# Second book
print("Second book:", matches[0]['title'])
# Array slicing (similar to Python slicing)
matches = jsonpath(data,'$..book[0:2]')# First two books
print("First two books:",[book['title']for book in matches])
# Negative indexing (from the end)
matches = jsonpath(data,'$..book[-1]')# Last book
print("Last book:", matches[0]['title'])
# Step selection
matches = jsonpath(data,'$..book[0:4:2]')# First and third books
print("First and third books:",[book['title']for book in matches])
2. Conditional Filtering# [?(expression)] Conditional filtering
# @ represents the current node
# Find books with ISBN numbers
matches = jsonpath(data,'$..book[?(@.isbn)]')
print("Books with ISBN:",[book['title']for book in matches])
# Find books priced below 10 yuan
matches = jsonpath(data,'$..book[?(@.price < 10)]')
print("Books priced below 10 yuan:",[book['title']for book in matches])
# Find books authored by someone with the surname 'Wu'
matches=jsonpath(data,'$..book[?(@.author.startswith("Wu"))]')
print("Books authored by 'Wu':",[book['title']for book in matches])
3. Practical Application: Baidu Image Scraperimport requests
from jsonpath import jsonpath
# Set request headers to simulate browser access
headers ={
'Accept':'application/json, text/plain, */*',
'Accept-Language':'zh-CN,zh;q=0.9',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36',
'Referer':'https://image.baidu.com/'
}
# Request parameters
params ={
'tn':'resultjson_com',
'word':'kitten',# Search keyword
'ie':'utf-8',
'pn':'60',# Start from the 60th image (pagination parameter)
'rn':'30',# 30 images per page
}
# Send request to get JSON data
response = requests.get(
'https://image.baidu.com/search/acjson',
params=params,
headers=headers
)
# Parse JSON response
data = response.json()
# Use JSONPath to extract image URLs and formats
urls = jsonpath(data,'$.data..thumbURL')# Extract all image URLs
formats = jsonpath(data,'$.data..type')# Extract all image formats
# Filter out None values (JSONPath may return None)
urls =[url for url in urls if url]if urls else[]
formats =[fmt for fmt in formats if fmt]if formats else[]
print(f"Found {len(urls)} images")
print("First 5 image URLs:", urls[:5])
print("First 5 image formats:", formats[:5])
4. Combining JSONPath with Regular Expressionsimport re
# Combine with regular expressions for more complex filtering
# Find books with titles containing the character '游'
matches = jsonpath(data,'$..book[?(@.title)]')# First get all books with titles
# Use regular expressions for further filtering
travel_books =[
book for book in matches
if re.search(r'游', book['title'])
]
print("Books with titles containing '游':",[book['title']for book in travel_books])
5. Error Handling and Best Practices1. Handling cases where JSONPath may return False
# JSONPath returns False if no matches are found instead of an empty list
result = jsonpath(data,'$..nonexistent')
if result is False:
print("No matches found")
elif result:
print(f"Found {len(result)} matches")
2. Encapsulating a Safe JSONPath Query Functiondef safe_jsonpath(data, expression, default=None):
"""Safe JSONPath query to avoid returning False"""
result = jsonpath(data, expression)
if result is False:
return default if default is not None else[]
return result
# Use safe query
urls = safe_jsonpath(data,'$.data..thumbURL',[])
print("Image URLs:", urls)
6. Comparing JSONPath with Other Techniques1. JSONPath vs Manual Traversal
# Manually traversing to find books priced below 10 yuan
manual_result =[]
for book in data['store']['book']:
if 'price' in book and book['price'] < 10:
manual_result.append(book)
# JSONPath method
jsonpath_result = jsonpath(data,'$..book[?(@.price < 10)]')
print("Manual traversal result:",[book['title']for book in manual_result])
print("JSONPath result:",[book['title']for book in jsonpath_result])
2. JSONPath vs Dictionary get Method# Traditional way to get bicycle color
color = data.get('store',{}).get('bicycle',{}).get('color')
# JSONPath method
color = jsonpath(data,'$.store.bicycle.color')[0]
print("Bicycle color:", color)
7. Practical Suggestions-
First validate the JSON structure: Use browser developer tools to view the JSON response.
-
Build expressions step by step: Start simple and gradually increase complexity.
-
Use online testing tools: Tools like JSONPath Online Evaluator can help with debugging.
-
Pay attention to performance: Complex JSONPaths may be slow for very large JSONs.
-
Combine with other technologies: JSONPath combined with regular expressions and conditional statements is more powerful.
JSONPath is a powerful tool for handling JSON data, especially suitable for extracting specific information from complex API responses. Mastering JSONPath will greatly enhance your web scraping data processing capabilities.
Next time, we will continue exploring Python, making a little progress every day. Let’s keep learning together! If you have any questions, feel free to leave a comment for discussion!