Introduction

Recently, I wrote a web scraper for work to extract information from the National Standards website, which naturally involved Python. When it comes to parsing HTML, XPath undoubtedly stands out as the top weapon.

You may have heard of or used other parsing methods, such as Beautiful Soup, which seems to have a lot of users. However, compared to XPath:

• The syntax is simpler

• The parsing speed is faster

Like regular expressions, it requires some learning at first, but over time, the rules become second nature, and once you are proficient, they are hard to forget.

Installing lxml

XPath is just a parsing rule; it requires a corresponding library to implement its functionality.

In Python, <span>lxml</span> is the library that implements XPath parsing.

Installing <span>lxml</span> is very simple:

pip install lxml

Next, let’s see how to leverage its power in a real project.

Loading HTML Content

First, import the <span>lxml</span> library:

from lxml import etree

Loading HTML content is done through the <span>etree</span> method, which has three specific methods:

• <span>parse()</span> – loads from a file

• <span>fromstring()</span> – loads from a string

• <span>HTML()</span> – also loads from a string, but in an HTML-compatible way

Which method should you choose?

Don’t hesitate, choose <span>etree.HTML()</span>, even if your HTML content comes from a file. Why is that?

First, it’s important to note that HTML is a type of XML, and the XML standard requires it to have a root tag; otherwise, the XML is invalid. The HTML content we load:

may not be complete, just a fragment, and lacks a root tag

or the loaded HTML may appear complete from start to finish, but some nodes in between may be missing closing tags.

These situations are actually invalid XML. Therefore, when using <span>parse()</span> or <span>fromstring()</span> to load such incomplete HTML, errors will occur; however, using <span>etree.HTML()</span> will not.

This is because <span>etree.HTML()</span> has excellent HTML compatibility; it will complete the incomplete HTML, turning it into a complete, valid HTML.

Example: Loading HTML from a file

from lxml import etree

with open('test.html', 'r') as f:
    html = etree.HTML(f.read())
    print(html, type(html))

Output:

<Element html at 0x7f7efa762040> <class 'lxml.etree._Element'>

The loaded HTML string has already been transformed into an <span>Element</span> object. We will later use XPath to find HTML nodes, all operations will be performed on this <span>Element</span> object.

Finding the HTML Nodes You Need

Assuming we have the following HTML structure:

<table>
    <tbody>...</tbody>  <!-- Table Header -->
    <tbody>...</tbody>  <!-- Table Content -->
</table>

How to locate the second tbody?

We typically call the <span>Element</span> object’s <span>xpath()</span> method to search using the provided XPath path. There are two ways to write the path:

1. <span>/</span> at the beginning – searches down from the root HTML tag along child nodes

2. <span>//</span> at the beginning – finds the node regardless of its position (most commonly used)

Example:

html.xpath('//table/tbody')  # Find all direct child tbody nodes under table
html.xpath('//table/tbody[2]')  # Find the second tbody (index starts from 1)

Powerful Attribute Selectors

If the HTML content contains more than one table, we can use attribute selectors to locate elements more precisely.

Example:

<div class="table-responsive">
    <table>...</table>
</div>

XPath syntax:

html.xpath('//div[@class="table-responsive"]/table/tbody[2]')

Note: In XPath, all HTML attribute matches start with <span>@</span>.

Fuzzy Matching: If the class is not unique, you can use the <span>contains()</span> function:

html.xpath('//table[contains(@class, "result_list")]/tbody[2]')

Tip:

XPath returns a list of elements, so you need to access elements by index.

Relative Positioning

Assuming we want to iterate through all content rows in the table to get the standard number and standard name:

from lxml import etree

with open('test.html', 'r') as f:
    html = etree.HTML(f.read())
    rows = html.xpath('//table[contains(@class, "result_list")]/tbody[2]/tr')

    for row in rows:
        td_list = row.xpath('...')  # How should this be written?

Question:

If you use <span>row.xpath('//td')</span>, you will find that it returns all td nodes, not just the current row’s td. This is because <span>//</span> is absolute positioning, starting from the root node.

Solution:

Use relative positioning (<span>.</span> represents the current node):

td_list = row.xpath('./td')  # Correct syntax

Getting Attributes and Text

Getting Attributes:

a1 = td.xpath('./a')[0]
onclick = a1.get('onclick')  # Get onclick attribute

Getting Text:

text = a1.text  # Directly get node text

Sibling Node Selector

If a row has 1000 td elements but you only need 2, you can optimize using sibling node selectors:

Finding the preceding sibling node:

title_td = row.xpath('./td[@class="mytxt"]')[0]
id_td = title_td.xpath('./preceding-sibling::td[2]')[0]  # Find the 2nd preceding sibling node

Finding the following sibling node:

following_td = title_td.xpath('./following-sibling::td[1]')[0]  # Find the 1st following sibling node

Complete Example Code

from lxml import etree

with open('test.html', 'r') as f:
    html = etree.HTML(f.read())
    rows = html.xpath('//table[contains(@class, "result_list")]/tbody[2]/tr')

    for row in rows:
        title_td = row.xpath('./td[@class="mytxt"]')[0]
        title_link = title_td.xpath('./a')[0]
        title_onclick = title_link.get('onclick')

        print(title_onclick, title_link.text)

        id_td = title_td.xpath('./preceding-sibling::td[2]')[0]
        id_link_text = id_td.xpath('./a/text()')[0]

        print(id_link_text)

XPath Quick Reference

The rules of XPath are not complicated; the commonly used ones are just those. Once you get familiar with them, you will naturally remember them. Here is a highly recommended quick reference: https://devhints.io/xpath

Although the content may not seem rich, it is clear at a glance, and using CSS syntax as a comparison helps better understand the practical use of each XPath rule.

Author: bananaplan

Source: cnblogs.com/bananaplan/p/18576728/python-xpath

Disclaimer: Online content for learning purposes only, respecting copyright, infringement will be deleted promptly, apologies and thanks!

ENDPrevious highlights: Ready-to-use data synchronization tool: supports 20+ data sources, automatic table creation, real-time monitoring, millisecond responseZero threshold to get started! Open-source smart city unified management AI platform, free deployment for immediate useHigh-performance low-code BI platform: rapid reports, integrated large screens and data middle platformA flexible and easy-to-use custom soft keyboard, easily handling ID cards, license plates, phone numbers, etc.Free and open-source! MES system supporting warehouse, production, quality, and equipment managementLightweight Web SCADA solution for industrial IoTThis is the enterprise-level low-code platform: plugin + process engine + intelligent reportsReady-to-use React + ECharts + DataV large screen visualization solutionMicroservice practical case of energy management system based on Vue3 + Spring CloudWeb-based multifunctional network debugging assistant, easily handling TCP/UDP and serial portsOpen-source cloud-native MQTT Broker: easily connect massive IoT devicesFirst in China! iotView open-source lightweight industrial IoT low-code platformLightweight and efficient file management system implemented with Vue 3 + TypeScriptReady-to-use enterprise-level microservice framework, integrating SSO, payment center, and monitoring trackingWeb configuration software Vue+Go to create B/S architecture real-time monitoring systemA B/S structure, open-source, free production execution management system (MES)Open-source zero-code data visualization, drag-and-drop analysis, BI tool for everyoneThree.js meets Vue3 to develop a modern 3D visualization editing systemVue3 + Vite7 + TS high-quality backend management template | Fresh and elegant, ready to useBrand new open-source SAAS architecture: simple, beautiful, and truly implementable multi-tenant cloud platformIoT card operation comprehensive platformA documentation management tool based on Vue and ElectronOpen-source and free! This data visualization platform supports multi-source free combination, freely creating dashboardsA network video platform implemented based on the GB28181-2016 standardH.264/H.265 video management + cloud-edge collaboration: practical case of monitoring system based on RuoyiOpen-source fire visualization platform supporting Windows and Linux is here!Open-source free human resource management system, local deployment, secure and controllable, flexible and customizableA free, efficient, and stable open-source enterprise-level questionnaire form systemOpen-source edge computing platform: real-time behavior risk warning system integrating video and IoTDomestic open-source general AI-CRM: next-generation intelligent customer management platformA free, open-source, and privately deployable online knowledge base management platformReactive IoT platform based on Spring Boot + Reactor supports secondary developmentA truly ready-to-use high-quality Vue3 backend management template, completely open-source and freeOne-stop integrated intelligent production management system covering MES, WMS, ERP, QMS, and IoTSReady-to-use open-source workflow system: flexible configuration supporting leave requests, work orders, and other scenariosA high-performance open-source private cloud document system supporting online preview of hundreds of formatsOpen-source free for commercial use! Business-level backend management system based on popular technology stackDevelopment framework integrating IoT, video surveillance, and low-code (supports GB28181/RTSP)Ready-to-use! Lightweight and efficient front-end and back-end separated examination system, supporting Web and mini-programsRust implements an efficient and secure GB28181 video surveillance platform (compatible with 2016/2022)A powerful enterprise-level content management system based on SpringBoot3 + Vue2AI + low-code dual-engine: SpringCloud + Vue3 enterprise platform, one-click zero-code generation!EasyAIoT: AI zero-threshold, all-scenario customizable IoT platformFully open-source AI + low-code dual-engine intelligent office systemEnterprise-level low-code platform, easily handling 99.99% of programming challengesOpen-source electronic contract platform, supporting evidence chain signing and private deploymentA high-performance network framework suitable for IoT, IM, and customer service systemsFully open-source BI platform to create cool data dashboards in three steps (multi-data source + drag-and-drop design)Lightweight comprehensive IoT management platform (card module management, three-network integration, full-process business)Open-source smart property management system, microservice architecture supports mini-programs, H5 multi-end, and secondary developmentOne-stop IoT access middleware covering all protocols, seamlessly connecting mainstream databasesNote【Open-source】

Facilitating communication, resource sharing, and mutual growth

Pure technical exchange group, please scan the code if you need

Gained something? Feel free to share to benefit more people

Follow “Programmer Open Source Stack”, and improve technical strength together

Click to shareClick to collectClick to viewClick to like