Hello everyone! Today I bring you the thirty-first learning note, to discuss a very powerful text processing tool in Python—Regular Expressions. Regular expressions are like the “Swiss Army knife” of text processing, helping us quickly match, search, and extract complex text patterns.

1. Basics of Regular Expressions

1. What are Regular Expressions?

Regular expressions are “rule strings” composed of specific character combinations, used for text:

• Matching Check: Validating whether a string conforms to a certain format

• Search: Finding specific patterns in text

• Extract and Replace: Extracting or replacing text that meets the rules

import re  # Python's regular expression module

# Simple example: Check if the string starts with 'h'

result = re.match('h','hello')

print(result)  # Match successful, returns Match object

2. Comparison of Five Matching Methods

1. re.match() – Match from the start

# Only matches the beginning of the string

result = re.match('h','hello')  # Success

print(result.group())  # Output: h

result = re.match('h','world')  # Failure

print(result)  # Output: None

2. re.search() – Search for a match

# Search for the first match in the entire string

result = re.search('h','hello world')  # Found 'h' at the start

result = re.search('h','world hello')  # Found 'h' in the middle

3. re.findall() – Find all matches

# Find all matches, return a list

results = re.findall('h','hello world hi')

print(results)  # Output: ['h', 'h']

4. re.finditer() – Return an iterator

# Find all matches, return MatchObject iterator

matches = re.finditer(r'\d+','There are 3 apples and 5 oranges')

for match in matches:

    print(f"Found number: {match.group()}, position: {match.span()}")

# Output:

# Found number: 3, position: (2, 3)

# Found number: 5, position: (8, 9)

5. re.sub() – Replace matches

# Replace all matched text

text ="Today is 2023-12-15, tomorrow is 2023-12-16"

new_text = re.sub(r'\d{4}-\d{2}-\d{2}','XXXX-XX-XX', text)

print(new_text)  # Output: Today is XXXX-XX-XX, tomorrow is XXXX-XX-XX

3. Character Matching Rules

1. Single Character Matching

# . matches any character (except newline)

re.match('.','a')  # Matches

re.match('.','1')  # Matches

re.match('.','\n')  # Does not match

# [] matches a set of characters

re.match('[abc]','a')  # Matches a/b/c

re.match('[a-z]','m')  # Matches any lowercase letter

re.match('[0-9]','5')  # Matches a digit

2. Predefined Character Classes

# \d matches a digit ≡ [0-9]

re.match(r'\d','123')  # Matches 1

# \D matches a non-digit ≡ [^0-9]

re.match(r'\D','abc')  # Matches a

# \s matches whitespace characters (spaces, tabs, etc.)

re.match(r'\s',' hello')  # Matches space

# \w matches word characters (letters, digits, underscores)

re.match(r'\w','hello')  # Matches h

4. Quantity Matching Rules

1. Repeated Matching

# * matches 0 or more times

re.match(r'a*','aaa')  # Matches aaa

re.match(r'a*','')  # Matches empty string

# + matches 1 or more times

re.match(r'a+','aaa')  # Matches aaa

re.match(r'a+','')  # Does not match

# ? matches 0 or 1 time

re.match(r'a?','a')  # Matches a

re.match(r'a?','')  # Matches empty

2. Exact Count Matching

# {n} matches exactly n times

re.match(r'a{3}','aaa')  # Matches aaa

# {n,} matches at least n times

re.match(r'a{2,}','aaa')  # Matches aaa

# {n,m} matches between n and m times

re.match(r'a{2,4}','aaa')  # Matches aaa

5. Position Matching Rules

1. Boundary Matching

# ^ matches the start of the string

re.match('^h','hello')  # Matches

re.match('^h','world')  # Does not match

# $ matches the end of the string

re.search('d$','world')  # Matches

re.search('d$','hello')  # Does not match

# \b matches word boundaries

re.search(r'\bword\b','hello word test')  # Matches

re.search(r'\bword\b','hello world test')  # Does not match

2. Group Matching

# | matches either

re.match('a|b','a')  # Matches a

re.match('a|b','b')  # Matches b

# () captures groups

result = re.match(r'(\w+)-(\d+)','order-12345')

print(result.group(1))  # Output: order

print(result.group(2))  # Output: 12345

6. Greedy and Non-Greedy Matching

1. Greedy Mode (default)

# Matches as many characters as possible

text ='Learning Python courses, Learning Java courses'

result = re.search('学习.*课程', text)

print(result.group())  # Output: 学习python课程，学习Java课程

2. Non-Greedy Mode (add ?)

# Matches as few characters as possible

result = re.search('学习.*?课程', text)

print(result.group())  # Output: 学习python课程

7. Practical Tips and Examples

1. Extracting URLs

def extract_urls(text):

    pattern =r'https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+[/\w\.-]*'

    return re.findall(pattern, text)

text ='Visit the official website https://www.example.com or http://test-site.org/path'

urls = extract_urls(text)

print(urls)  # Output: ['https://www.example.com', 'http://test-site.org/path']

2. Extracting Chinese Text

text ='Hello世界！Python编程很有趣。123数字'

chinese_chars = re.findall(r'[\u4e00-\u9fff]+', text)

print(chinese_chars)  # Output: ['世界', '编程很有趣']

3. Data Cleaning

def clean_data(text):

    # Remove extra whitespace

    text = re.sub(r'\s+',' ', text)

    # Remove non-alphanumeric characters (keep Chinese and basic punctuation)

    text=re.sub(r'[^-\u4e00-\u9fff\s\.\,\!\\?]','', text)

    return text.strip()

dirty_text ='  Hello   世界！@#￥%   Python编程...  '

clean_text = clean_data(dirty_text)

print(clean_text)  # Output: 'Hello 世界！ Python编程...'

8. Advanced Techniques

1. Named Groups

# Use named groups for better readability

text ='Name: Zhang San, Age: 25, City: Beijing'

pattern =r'Name: (?P<name>\w+), Age: (?P<age>\d+), City: (?P<city>\w+)'

match= re.search(pattern, text)

if match:

    print(f"Name: {match.group('name')}")

    print(f"Age: {match.group('age')}")

    print(f"City: {match.group('city')}")

2. Compiling Regular Expressions

# Pre-compiling improves performance (suitable for repeated use)

pattern = re.compile(r'\b\w{4,}\b')

# Matches words with more than 4 letters

text ='Regular expressions are a very powerful text processing tool'

long_words = pattern.findall(text)

print(long_words)

# Output: ['Regular', 'expressions', 'powerful', 'processing', 'tool']

9. Notes

1. Use raw strings: Prefix regular expressions with<span><span>r</span></span> to avoid escape issues

2. Compile for reuse: Regular expressions used frequently can be compiled first to improve performance

3. Error handling: Matching may return<span><span>None</span></span>, check to avoid exceptions

4. Performance considerations: Complex regular expressions may affect performance, try to simplify and optimize

5. Readability: Overly complex regular expressions are hard to maintain, add comments appropriately

# Use verbose mode to add comments

pattern = re.compile(r"""

    ^                   # Start of string

    (\w+)               # Username

    @                   # @ symbol

    ([a-zA-Z0-9.-]+)    # Domain name

    \.                  # Dot

    ([a-zA-Z]{2,4})     # Top-level domain

    $                   # End of string

""", re.VERBOSE)

The basic learning notes officially end here. Let’s make progress together every day on our learning journey! If you have any questions, feel free to leave a comment for discussion~PS: Regular expressions require a lot of practice. They may seem complex at first, but mastering them can greatly improve text processing efficiency. It is recommended to start with simple patterns and gradually build complex expressions.