Introduction to the Linux Text Processing Trio: Understanding sed

Related Articles:

• Introduction to the Linux Text Processing Trio: Starting with vi
• Introduction to the Linux Text Processing Trio: Understanding grep

In daily work with Linux, text processing is a fundamental yet long-used skill. From modifying configuration files, analyzing logs, to batch replacing code, almost all automation tasks rely on efficient text processing tools. vi/vim is an excellent editor, but it is more suitable for interactive operations; when we need to implement unattended text editing in scripts, we must rely on another type of tool — sed.

sed (Stream Editor) is one of the most powerful non-interactive text processing tools in Linux, capable of making precise modifications to text without opening an editor. Compared to vi/vim, sed is more suitable for automation scenarios, with the following advantages:

• Batch replace text content, suitable for automated script processing
• Delete, insert, or modify specific lines
• Filter text content based on conditions
• Directly modify files without temporary files
• Stream processing, with very low memory usage when handling large files

In this issue, we will systematically introduce the usage of sed, adding a true “magic weapon” to your command line toolbox.

Basic Concepts of sed

What is sed?

sed is a stream editor that reads input line by line, processes the text according to specified commands, and then outputs the result.

Core Features:

• Non-interactive editing (unlike vi/vim which requires opening files)
• Line-by-line processing (reads one line at a time)
• Default output to standard output (does not modify the original file)
• Supports regular expressions (powerful pattern matching)

Working Principle

The workflow of sed:

1. Read a line into the pattern space (Pattern Space)
2. Execute editing commands on the content of the pattern space
3. Output the content of the pattern space (if not deleted)
4. Clear the pattern space
5. Repeat steps 1-4 until the end of the file

Illustration:

Input File → [Read a line] → [Pattern Space] → [Execute Command] → [Output] → Standard Output
              ↑                                      ↓
              └────────────── Loop Processing ──────────────┘

Basic Syntax

sed [options] 'command' filename
sed [options] -e 'command1' -e 'command2' filename
sed [options] -f scriptfile filename

Common Options:

• -n: Silent mode, does not automatically print the pattern space
• -e: Execute multiple editing commands
• -f: Read sed commands from a file
• -i: In-place editing (directly modify the file)
• -i.bak: Backup the original file before modification
• -r or -E: Use extended regular expressions

Basic sed Commands

Replacement Command (s)

This is the most commonly used command in sed, syntax:

sed 's/pattern/replacement/flags' file

Basic Examples:

# Replace the first match in each line
echo "hello world hello" | sed 's/hello/hi/'
# Output: hi world hello

# Replace all matches (g flag)
echo "hello world hello" | sed 's/hello/hi/g'
# Output: hi world hi

# Replace only the 2nd match
echo "hello world hello again hello" | sed 's/hello/hi/2'
# Output: hello world hi again hello

Common Flags:

• g: Global replacement (replace all matches)
• number: Replace only the Nth match
• p: Print the replaced line
• w file: Write the replaced line to a file
• i: Ignore case

Practical Examples:

# Case-insensitive replacement
echo "Hello HELLO hello" | sed 's/hello/hi/gi'
# Output: hi hi hi

# Print the replaced lines (with -n)
sed -n 's/error/ERROR/p' log.txt
# Only outputs lines containing error and replaced

# Write the replacement result to a new file
sed 's/old/new/gw output.txt' input.txt

Delete Command (d)

Delete matching lines:

# Delete the 3rd line
sed '3d' file.txt

# Delete lines 2 to 5
sed '2,5d' file.txt

# Delete the last line
sed '$d' file.txt

# Delete empty lines
sed '/^$/d' file.txt

# Delete lines containing pattern
sed '/pattern/d' file.txt

# Delete lines not containing pattern
sed '/pattern/!d' file.txt

# Delete comment lines (starting with #)
sed '/^#/d' file.txt

# Delete empty lines and comment lines
sed '/^$/d;/^#/d' file.txt

Practical Example:

# Clean configuration file (delete comments and empty lines)
sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config

# Delete HTML tags
echo "&lt;p&gt;Hello&lt;/p&gt;" | sed 's/&lt;[^&gt;]*&gt;//g'
# Output: Hello

Print Command (p)

Print matching lines (usually used with -n):

# Print the 3rd line
sed -n '3p' file.txt

# Print lines 2 to 5
sed -n '2,5p' file.txt

# Print matching lines
sed -n '/pattern/p' file.txt

# Print non-matching lines
sed -n '/pattern/!p' file.txt

# Simulate grep
sed -n '/error/p' log.txt
# Equivalent to: grep error log.txt

Append, Insert, and Change (a, i, c)

Append (a): Add content after matching lines

# Append after the 2nd line
sed '2a\This is new line' file.txt

# Append after matching lines
sed '/pattern/a\New content' file.txt

# Append multiple lines
sed '/pattern/a\
Line 1\
Line 2\
Line 3' file.txt

Insert (i): Insert content before matching lines

# Insert before the 2nd line
sed '2i\This is new line' file.txt

# Insert before matching lines
sed '/pattern/i\New content' file.txt

# Insert at the beginning of the file
sed '1i\Header line' file.txt

Change (c): Replace the entire line

# Change the 3rd line
sed '3c\New line content' file.txt

# Change matching lines
sed '/pattern/c\Replacement line' file.txt

# Change a range of lines
sed '2,4c\Single replacement line' file.txt

Practical Example:

# Add description at the beginning of the configuration file
sed '1i\# Configuration File' config.txt

# Add a separator line before each section
sed '/^[/i\# ========================================' config.ini

# Replace the entire erroneous configuration line
sed '/old_config/c\new_config=value' settings.conf

Line Numbers and Address Ranges

sed supports various address representations:

# Single line
sed '5d' file.txt              # 5th line
sed '$d' file.txt              # last line

# Line range
sed '2,5d' file.txt            # lines 2 to 5
sed '2,$d' file.txt            # from line 2 to the end
sed '2,+3d' file.txt           # line 2 and the next 3 lines

# Pattern matching
sed '/start/,/end/d' file.txt  # from start to end
sed '/pattern/d' file.txt      # lines containing pattern

# Interval lines
sed '1~2d' file.txt            # delete odd lines (1,3,5...)
sed '2~2d' file.txt            # delete even lines (2,4,6...)

# Combined conditions
sed '2,5{s/old/new/g; s/foo/bar/g}' file.txt

Advanced Usage

Using Regular Expressions

sed uses basic regular expressions (BRE) by default; use -r or -E to enable extended regular expressions (ERE).

Basic Regular Expression Examples:

# Match the beginning of a line
sed -n '/^Error/p' log.txt

# Match the end of a line
sed -n '/ERROR$/p' log.txt

# Match any character
sed 's/a.c/abc/g' file.txt     # . matches any single character

# Match repeated characters
sed 's/a*/x/g' file.txt        # * means the preceding character repeats 0 or more times
sed 's/a\+/x/g' file.txt       # \+ means repeats 1 or more times (needs escaping)
sed 's/a\{3\}/x/g' file.txt    # \{3\} means exactly 3 times

# Character classes
sed 's/[0-9]/X/g' file.txt     # replace all digits
sed 's/[a-zA-Z]/X/g' file.txt  # replace all letters

# Grouping and back-referencing
sed 's/\([0-9]\{3\}\)-\([0-9]\{4\}\)/(
\1) \2/g' phone.txt
# Convert 123-4567 to (123) 4567

Extended Regular Expression Examples:

# Use -E to enable extended regex
sed -E 's/a+/x/g' file.txt            # + does not need escaping
sed -E 's/a{3}/x/g' file.txt          # {3} does not need escaping
sed -E 's/(abc)+/x/g' file.txt        # () does not need escaping

# Or operator
sed -E 's/(cat|dog)/animal/g' file.txt

# Match email
sed -En 's/.*([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}).*/\1/p' file.txt

# Match IP address
sed -En 's/.*([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*/\1/p' log.txt

Back-referencing and Capture Groups

Use \1, \2, etc. to reference captured content:

# Swap two words
echo "hello world" | sed 's/\(.*\) \(.*\)/\2 \1/'
# Output: world hello

# Extract and reformat date format
echo "2024-01-15" | sed 's/\([0-9]\{4\}\)-\([0-9]\{2\}\)-\([0-9]\{2\}\)/\3\/\2\/\1/'
# Output: 15/01/2024

# Add parentheses to numbers
echo "Tel: 12345678" | sed 's/\([0-9]\+/\(\1\)/g'
# Output: Tel: (12345678)

# Use extended regex (more concise)
echo "hello world" | sed -E 's/(.*) (.*)/\2 \1/'
echo "2024-01-15" | sed -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})/\3\/\2\/\1/'

Multiple Commands

Method 1: Use semicolons to separate

sed 's/old/new/g; s/foo/bar/g' file.txt

Method 2: Use -e option

sed -e 's/old/new/g' -e 's/foo/bar/g' file.txt

Method 3: Use braces (for specific lines)

# Execute multiple commands on lines 2-5
sed '2,5{s/old/new/g; s/foo/bar/g}' file.txt

Method 4: Use script files

# Create a sed script file script.sed
cat &gt; script.sed &lt;&lt; 'EOF'
s/old/new/g
s/foo/bar/g
/^$/d
EOF

# Use the script
sed -f script.sed file.txt

Hold Space

sed has two buffers:

• Pattern Space: The current line being processed
• Hold Space: Temporary storage area

Related Commands:

• h: Copy pattern space to hold space
• H: Append pattern space to hold space
• g: Copy hold space to pattern space
• G: Append hold space to pattern space
• x: Swap pattern space and hold space

Practical Example:

# Output file in reverse order (similar to tac)
sed '1!G;h;$!d' file.txt

# Add an empty line after each line
sed 'G' file.txt

# Merge two adjacent lines
sed 'N;s/\n/ /' file.txt

# Delete duplicate consecutive lines (similar to uniq)
sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt

In-place Editing (-i option)

-i option allows direct modification of files:

# Directly modify the file (dangerous!)
sed -i 's/old/new/g' file.txt

# Backup before modification (recommended!), not necessarily bak, can be any string
sed -i.bak 's/old/new/g' file.txt
# This will create a backup file.txt.bak

# Custom backup suffix
sed -i.backup-$(date +%Y%m%d) 's/old/new/g' file.txt

# sed syntax differs on macOS
sed -i '' 's/old/new/g' file.txt           # macOS
sed -i.bak 's/old/new/g' file.txt          # macOS with backup

Batch Modify Multiple Files:

# Modify all .txt files in the current directory
sed -i.bak 's/old/new/g' *.txt

# Use find with sed
find . -name "*.conf" -exec sed -i.bak 's/old/new/g' {} \;

# Safer way (test first)
find . -name "*.conf" -exec sed 's/old/new/g' {} \; | less
# Confirm correctness before adding -i

Practical Cases

Configuration File Processing

Scenario 1: Modify Configuration Items

# Modify Nginx port
sed -i 's/listen 80;/listen 8080;/' /etc/nginx/nginx.conf

# Modify multiple configurations (ensure idempotency)
sed -i '/^Port/c\Port 2222' /etc/ssh/sshd_config

# Modify key-value pair configuration
sed -i 's/^DEBUG=.*/DEBUG=true/' config.sh

Scenario 2: Add Configuration Items

# Add configuration after [section]
sed -i '/^\[section\]/a\new_option=value' config.ini

# Add at the end of the file
sed -i '$a\new_line' file.txt

# Insert before a specific line (e.g., include statement)
sed -i '/^include/i\# Custom includes' nginx.conf

Scenario 3: Comment and Uncomment

# Comment out a line
sed -i 's/^Port 22/#&amp;/' /etc/ssh/sshd_config
# &amp; represents the entire matched content

# Uncomment
sed -i 's/^#\(Port 22\)/\1/' /etc/ssh/sshd_config

# Batch comment multiple lines
sed -i '10,20s/^/#/' file.txt

# Comment lines containing keywords
sed -i '/DEBUG/s/^/#/' config.txt

Log Processing

Scenario 1: Extract Specific Information

# Extract all IP addresses
sed -n 's/.*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/p' access.log

# Extract timestamps
sed -n 's/.*\[\(.*\)\].*/\1/p' nginx_access.log

# Extract ERROR level logs
sed -n '/ERROR/p' application.log

# Extract and format
sed -n 's/\([0-9-]\{10\}\) \([0-9:]\{8\}\).*ERROR.*\(.*\)/[\1 \2] \3/p' app.log

Scenario 2: Filter and Clean Logs

# Delete debug logs
sed '/DEBUG/d' app.log &gt; cleaned.log

# Keep the last 1000 lines
sed -n '$-999,$p' large.log &gt; recent.log

# Delete empty lines and timestamps
sed -e '/^$/d' -e 's/^[0-9: -]\{19\} //' app.log

Scenario 3: Log Statistics

# Count errors per hour
sed -n 's/.*\([0-9]\{2\}\):[0-9:]\{6\}.*ERROR.*/\1/p' app.log | sort | uniq -c

# Extract and count status codes
sed -n 's/.* \([0-9]\{3\}\) .*/\1/p' access.log | sort | uniq -c | sort -rn

# Count the most accessed IPs
sed -n 's/^\([0-9.]+\).*/\1/p' access.log | sort | uniq -c | sort -rn | head -10

Code Processing

Scenario 1: Batch Modify Code

# Replace function names
sed -i 's/\boldFunction\b/newFunction/g' *.js

# Modify import paths
sed -i 's|from "\./old/|from "./new/|g' *.ts

# Modify version numbers
sed -i 's/"version": "[^"]*"/"version": "2.0.0"/' package.json

# Add file header comments
sed -i '1i/**\n * Copyright 2024\n * Author: Your Name\n */' *.js

Scenario 2: Code Formatting

# Delete trailing whitespace
sed -i 's/[[:space:]]\+$$//' *.py

# Standardize indentation (convert tabs to 4 spaces)
sed -i 's/\t/    /g' *.py

# Delete extra empty lines (consecutive empty lines become single)
sed -i '/^$/N;/^\n$/D' code.py

# Add empty lines before functions
sed -i '/^def /i\' code.py

Scenario 3: Code Auditing

# Find hard-coded passwords
sed -n 's/.*password.*=.*["\x27]\(.*\)["\x27].*/Found: \1/p' *.config

# Find TODO comments
sed -n 's/.*//.*TODO:\(.*\)/\1/p' *.js

# Extract API endpoints
sed -n 's/.*@app\.route(["\x27]\(.*\)["\x27].*/\1/p' *.py

Data Processing

Scenario 1: CSV Processing

# Extract specific columns (2nd column)
sed 's/[^,]*,\([^,]*\),.*/\1/' data.csv

# Delete CSV header
sed '1d' data.csv

# Add quotes to all fields
sed 's/\([^,]*\)/"\1"/g' data.csv

# Replace delimiter (comma to tab)
sed 's/,/\t/g' data.csv

Scenario 2: HTML/XML Processing

# Delete all HTML tags
sed 's/&lt;[^&gt;]*&gt;//g' page.html

# Extract links
sed -n 's/.*href=["\x27]\([^"\x27]*\)["\x27].*/\1/p' page.html

# Extract titles
sed -n 's/.*&lt;title&gt;\(.*\)&lt;\/title&gt;.*/\1/p' page.html

Scenario 3: Text Format Conversion

# Convert Windows line endings to Unix
sed -i 's/\r$$//' windows.txt

# Unix to Windows
sed -i 's/$$/\r/' unix.txt

# Wrap every 80 characters
sed 's/.\{80\}/&amp;\n/g' long_line.txt

# Merge all lines
sed ':a;N;$!ba;s/\n/ /g' multiline.txt

System Administration

Scenario 1: Users and Permissions

# Extract all regular users
sed -n '/\/home/s/^\([^:]*\).*/\1/p' /etc/passwd

# Find users with UID >= 1000
sed -n 's/^\([^:]*\):[^:]*:\([0-9]\+\):.*/\2 \1/p' /etc/passwd | awk '$1>=1000'

# Modify user shell
sed -i 's/^\(username:.*:\)\/bin\/bash$/\1\/bin\/zsh/' /etc/passwd

Scenario 2: Batch Operations

# Batch rename files (used in scripts)
for file in *.txt; do
    newname=$(echo "$file" | sed 's/old/new/')
    mv "$file" "$newname"
done

# Generate batch commands
sed 's/^/mysql -e "DROP TABLE /' tables.txt | sed 's/$/"/' &gt; drop_tables.sh

# Generate configuration from template
sed "s/{{SERVER_NAME}}/$HOSTNAME/g; s/{{PORT}}/$PORT/g" template.conf &gt; server.conf

Performance Optimization Tips

Early Termination of Processing

# Only process the first 100 lines
sed '100q' large_file.txt

# Exit after finding the first match
sed '/pattern/q' file.txt

# Exit after processing a specified range
sed '1,1000{s/old/new/g}; 1000q' huge_file.txt

Avoid Unnecessary Operations

# ❌ Inefficient: Try to replace on every line
sed 's/rare_pattern/new/g' large_file.txt

# ✅ Efficient: Filter first then replace
sed '/rare_pattern/s/rare_pattern/new/g' large_file.txt

# ❌ Inefficient: Read the file multiple times
sed 's/old/new/g' file.txt &gt; temp &amp;&amp; sed 's/foo/bar/g' temp &gt; output

# ✅ Efficient: Process once
sed -e 's/old/new/g' -e 's/foo/bar/g' file.txt &gt; output

Use More Specific Patterns

# ❌ Vague matching, may cause unintended operations
sed 's/test/TEST/g' file.txt

# ✅ Precise matching (word boundaries)
sed 's/\&lt;test\&gt;/TEST/g' file.txt
sed 's/\btest\b/TEST/g' file.txt  # Extended regex

# ❌ Greedy matching may be excessive
sed 's/.*error.*/ERROR/' file.txt

# ✅ Non-greedy or more precise patterns
sed 's/\(.*\)error\(.*\)/\1ERROR\2/' file.txt

Common Errors and Debugging

Common Errors

Error 1: Forgetting to Escape Special Characters

# ❌ Error: . matches any character
sed 's/192.168.1.1/10.0.0.1/' file.txt

# ✅ Correct: Escape the dot
sed 's/192\.168\.1\.1/10.0.0.1/' file.txt

# ✅ Or use a different delimiter
sed 's|192.168.1.1|10.0.0.1|' file.txt

Error 2: Replacement Contains &

# ❌ & has a special meaning (represents matched content)
echo "hello" | sed 's/hello/hi & bye/'
# Output: hi hello bye

# ✅ If you want to output literal &, you need to escape

echo "hello" | sed 's/hello/hi \& bye/'
# Output: hi & bye

Error 3: Unescaped Grouping (BRE Mode)

# ❌ In basic regex, brackets need to be escaped
sed 's/(test)/[\1]/' file.txt

# ✅ Correct escaping
sed 's/\(test\)/[\1]/' file.txt

# ✅ Or use extended regex
sed -E 's/(test)/[\1]/' file.txt

Error 4: Incorrect Usage of -i Option

# ❌ This will cause an error on macOS
sed -i 's/old/new/g' file.txt

# ✅ macOS requires a suffix (can be empty)
sed -i '' 's/old/new/g' file.txt

# ✅ Linux can be used without a suffix
sed -i 's/old/new/g' file.txt

# ✅ Cross-platform compatible way with backup
sed -i.bak 's/old/new/g' file.txt

Error 5: Improper Use of Quotes

# ❌ Variables will expand inside double quotes
name="test"
sed "s/old/$name/g" file.txt  # If $name contains / it will cause an error

# ✅ Use other delimiters
sed "s|old|$name|g" file.txt

# ✅ Or escape first
name_escaped=$(echo "$name" | sed 's/[\/&]/\\&/g')
sed "s/old/$name_escaped/g" file.txt

Debugging Tips

Tip 1: Use -n and p to View Processing Results

# Test first without modifying the original file
sed -n 's/old/new/gp' file.txt

# See which lines will be deleted
sed -n '/pattern/!p' file.txt

Tip 2: Build Complex Commands Step by Step

# Step 1: Test matching
sed -n '/pattern/p' file.txt

# Step 2: Test replacement
sed -n '/pattern/s/old/new/p' file.txt

# Step 3: Apply to file
sed -i.bak '/pattern/s/old/new/g' file.txt

Tip 3: Use = to Show Line Numbers

# Show line numbers of matching lines
sed -n '/pattern/=' file.txt

# Show both line numbers and content
sed -n '/pattern/{=;p}' file.txt

# Format output
sed -n '/pattern/{=;p}' file.txt | sed 'N;s/\n/: /'

Tip 4: Use l Command to View Special Characters

# Show invisible characters
sed -n 'l' file.txt

# View specific lines
sed -n '5l' file.txt

Tip 5: Test on Small Datasets

# Test on a small file first
head -100 large_file.txt &gt; test.txt
sed 's/old/new/g' test.txt

# Confirm correctness before processing large files
sed 's/old/new/g' large_file.txt &gt; output.txt

Practical Script Examples

Configuration File Management Script

#!/bin/bash
# config_manager.sh - Configuration file management tool

CONFIG_FILE="app.conf"

# Get configuration value
get_config() {
    local key="$1"
    sed -n "s/^${key}=\(.*\)/\1/p" "$CONFIG_FILE"
}

# Set configuration value
set_config() {
    local key="$1"
    local value="$2"
    
    # If the key exists, update it; otherwise, append it
    if grep -q "^${key}=" "$CONFIG_FILE"; then
        sed -i "s|^${key}=.*|${key}=${value}|' "$CONFIG_FILE"
    else
        echo "${key}=${value}" &gt;&gt; "$CONFIG_FILE"
    fi
}

# Delete configuration item
delete_config() {
    local key="$1"
    sed -i "/^${key}=/d" "$CONFIG_FILE"
}

# Usage example
set_config "DEBUG" "true"
echo "DEBUG = $(get_config DEBUG)"
delete_config "OLD_KEY"

Log Analysis Script

#!/bin/bash
# log_analyzer.sh - Log analysis tool

LOG_FILE="$1"
OUTPUT_DIR="./analysis"

mkdir -p "$OUTPUT_DIR"

echo "Analyzing log file: $LOG_FILE"

# Extract error logs
sed -n '/ERROR/p' "$LOG_FILE" &gt; "$OUTPUT_DIR/errors.log"
echo "Number of errors: $(wc -l &lt; "$OUTPUT_DIR/errors.log")"

# Extract warning logs
sed -n '/WARN/p' "$LOG_FILE" &gt; "$OUTPUT_DIR/warnings.log"
echo "Number of warnings: $(wc -l &lt; "$OUTPUT_DIR/warnings.log")"

# Count errors by hour
echo "Hourly error statistics:" &gt; "$OUTPUT_DIR/hourly_errors.txt"
sed -n 's/.*\([0-9]\{2\}\):[0-9:]\{6\}.*ERROR.*/\1/p' "$LOG_FILE" | \
    sort | uniq -c | sort -rn &gt;&gt; "$OUTPUT_DIR/hourly_errors.txt"

# Extract top 10 most common errors
echo "Top 10 error messages:" &gt; "$OUTPUT_DIR/top_errors.txt"
sed -n 's/.*ERROR - \(.*\)/\1/p' "$LOG_FILE" | \
    sort | uniq -c | sort -rn | head -10 &gt;&gt; "$OUTPUT_DIR/top_errors.txt"

echo "Analysis complete, results saved in $OUTPUT_DIR"

Batch File Processing Script

#!/bin/bash
# batch_process.sh - Batch file processing

SOURCE_DIR="$1"
BACKUP_DIR="./backup_$(date +%Y%m%d_%H%M%S)"

# Create backup
mkdir -p "$BACKUP_DIR"
cp -r "$SOURCE_DIR"/* "$BACKUP_DIR/"

echo "Backup created at: $BACKUP_DIR"

# Batch operations
find "$SOURCE_DIR" -type f -name "*.txt" | while read file; do
    echo "Processing: $file"
    
    # 1. Delete trailing whitespace
    sed -i 's/[[:space:]]\+$$//' "$file"
    
    # 2. Delete empty lines
    sed -i '/^$/d' "$file"
    
    # 3. Standardize line endings
    sed -i 's/\r$$//' "$file"
    
    # 4. Replace specific content
    sed -i 's/old_term/new_term/g' "$file"
    
    echo "  ✓ Done"
done

echo "All files processed"

Text Template Engine

#!/bin/bash
# template_engine.sh - Simple template engine

TEMPLATE_FILE="$1"
OUTPUT_FILE="$2"

# Define variables
export APP_NAME="MyApp"
export APP_VERSION="1.0.0"
export SERVER_PORT="8080"
export DATABASE_HOST="localhost"

# Process template
sed -e "s/{{APP_NAME}}/$APP_NAME/g" \
    -e "s/{{APP_VERSION}}/$APP_VERSION/g" \
    -e "s/{{SERVER_PORT}}/$SERVER_PORT/g" \
    -e "s/{{DATABASE_HOST}}/$DATABASE_HOST/g" \
    "$TEMPLATE_FILE" &gt; "$OUTPUT_FILE"

echo "Configuration file generated: $OUTPUT_FILE"

# Example template config.template:
# [application]
# name={{APP_NAME}}
# version={{APP_VERSION}}
# 
# [server]
# port={{SERVER_PORT}}
# 
# [database]
# host={{DATABASE_HOST}}

Code Refactoring Helper Script

#!/bin/bash
# refactor_helper.sh - Code refactoring helper tool

OLD_FUNC="oldFunction"
NEW_FUNC="newFunction"
FILE_PATTERN="*.js"

# Find all files using the old function
echo "Finding files using $OLD_FUNC..."
grep -rl "$OLD_FUNC" --include="$FILE_PATTERN" .

# Preview the modifications to be made
echo -e "\nPreview modifications:"
grep -rn "$OLD_FUNC" --include="$FILE_PATTERN" . | sed "s/$OLD_FUNC/$NEW_FUNC/g"

# Ask for confirmation
read -p "Confirm execution of modifications? (y/n): " confirm

if [ "$confirm" = "y" ]; then
    # Backup
    BACKUP_DIR="./refactor_backup_$(date +%Y%m%d_%H%M%S)"
    mkdir -p "$BACKUP_DIR"
    
    # Execute replacement
    find . -name "$FILE_PATTERN" -type f | while read file; do
        if grep -q "$OLD_FUNC" "$file"; then
            cp "$file" "$BACKUP_DIR/"
            sed -i "s/\b$OLD_FUNC\b/$NEW_FUNC/g" "$file"
            echo "Modified: $file"
        fi
    done
    
echo "Refactoring complete, backup saved at: $BACKUP_DIR"
else
    echo "Operation canceled"
fi

Best Practice Summary

Safety Practices

# ✅ 1. Always use a backup suffix with the -i option
sed -i.bak 's/old/new/g' file.txt

# ✅ 2. Test before executing
sed 's/old/new/g' file.txt | less
sed -i.bak 's/old/new/g' file.txt

# ✅ 3. Use version control
git add file.txt
git commit -m "Before sed modification"
sed -i 's/old/new/g' file.txt

# ✅ 4. Use date backups for critical files
sed -i.$(date +%Y%m%d) 's/old/new/g' important.conf

# ✅ 5. Limit the scope of modifications
sed '100,200s/old/new/g' file.txt  # Only modify specified lines

Performance Practices

# ✅ 1. Use more specific address ranges
sed '/start/,/end/s/old/new/g' file.txt

# ✅ 2. Exit early
sed '/pattern/q' file.txt

# ✅ 3. Merge multiple commands
sed -e 's/a/A/g' -e 's/b/B/g' file.txt

# ✅ 4. Avoid unnecessary pipes
# ❌ Inefficient
cat file.txt | sed 's/old/new/g'
# ✅ Efficient
sed 's/old/new/g' file.txt

# ✅ 5. Use stream processing when handling large files
sed 's/old/new/g' huge_file.txt &gt; output.txt

Maintainability Practices

# ✅ 1. Use extended regex (more readable)
sed -E 's/([0-9]+)/[\1]/g' file.txt

# ✅ 2. Save complex commands as scripts
cat &gt; cleanup.sed &lt;&lt; 'EOF'
# Delete comments
/^#/d
# Delete empty lines
/^$/d
# Standardize indentation
s/\t/    /g
EOF
sed -f cleanup.sed file.txt

# ✅ 3. Add comments for clarity
sed '
    # Extract IP addresses
    s/.*\([0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\).*/\1/
' log.txt

# ✅ 4. Use variables for readability
old_pattern="deprecated_func"
new_pattern="new_func"
sed "s/$old_pattern/$new_pattern/g" code.py

# ✅ 5. Process complex tasks step by step
sed 's/old1/new1/g' file.txt | \
sed 's/old2/new2/g' | \
sed '/pattern/d' &gt; output.txt

Cross-Platform Compatibility

# ✅ 1. Avoid using GNU sed specific features
# Use basic regex instead of extended regex
sed 's/\([0-9]\+/[\1]/g' file.txt

# ✅ 2. -i option compatibility
# Always provide a backup suffix
sed -i.bak 's/old/new/g' file.txt

# ✅ 3. Detect operating system
if [[ "$OSTYPE" == "darwin*" ]]; then
    # macOS
    sed -i '' 's/old/new/g' file.txt
else
    # Linux
    sed -i 's/old/new/g' file.txt
fi

# ✅ 4. Use POSIX compatible features
# Avoid \w \d and other GNU extensions
sed 's/[a-zA-Z0-9_]\+/WORD/g' file.txt

# ✅ 5. Test scripts on different systems
# Use virtual machines or containers for testing

Quick Reference

Common Command Quick Reference

Command	Description	Example
`<span>s/pattern/replacement/</span>`	Replace	`<span>sed 's/old/new/'</span>`
`<span>d</span>`	Delete	`<span>sed '3d'</span>`
`<span>p</span>`	Print	`<span>sed -n '/pattern/p'</span>`
`<span>a\text</span>`	Append	`<span>sed '2a\new line'</span>`
`<span>i\text</span>`	Insert	`<span>sed '2i\new line'</span>`
`<span>c\text</span>`	Change	`<span>sed '3c\replacement'</span>`
`<span>=</span>`	Print line number	`<span>sed -n '/pattern/='</span>`
`<span>q</span>`	Exit	`<span>sed '10q'</span>`
`<span>n</span>`	Read next line	`<span>sed 'n;d'</span>`
`<span>N</span>`	Append next line	`<span>sed 'N;s/\n/ /'</span>`

Address Range Quick Reference

Address	Description	Example
`<span>n</span>`	Line n	`<span>sed '5d'</span>`
`# Complete Guide to Linux sed Command: The Art of Stream Editing

Flag Quick Reference

Flag	Description	Example
`<span>g</span>`	Global replacement	`<span>sed 's/a/A/g'</span>`
`<span>number</span>`	Replace the Nth	`<span>sed 's/a/A/2'</span>`
`<span>p</span>`	Print modified line	`<span>sed -n 's/a/A/p'</span>`
`<span>w file</span>`	Write to file	`<span>sed 's/a/A/w out.txt'</span>`
`<span>i</span>`	Ignore case	`<span>sed 's/a/A/gi'</span>`
`<span>e</span>`	Execute replacement result	`<span>sed 's/^/echo /e'</span>`

Common Matching and Address Patterns

Pattern	Type	Description	Example
`<span>^</span>`	Regex	Start of line	`<span>sed -n '/^ERROR/p'</span>`
`<span>$</span>`	Regex	End of line	`<span>sed -n '/ERROR$/p'</span>`
`<span>.</span>`	Regex	Any character	`<span>sed 's/a.c/abc/'</span>`
`<span>*</span>`	Regex	0 or more times	`<span>sed 's/a*/x/'</span>`
`<span>\+</span>`	Regex (GNU extension)	1 or more times	`<span>sed 's/a\+/x/'</span>`
`<span>\?</span>`	Regex (GNU extension)	0 or 1 time	`<span>sed 's/a\?/x/'</span>`
`<span>\{n\}</span>`	Regex	Exactly n times	`<span>sed 's/a\{3\}/x/'</span>`
`<span>\{n,\}</span>`	Regex	At least n times	`<span>sed 's/a\{3,\}/x/'</span>`
`<span>\{n,m\}</span>`	Regex	From n to m times	`<span>sed 's/a\{2,4\}/x/'</span>`
`<span>[abc]</span>`	Regex	Character class	`<span>sed 's/[0-9]/X/g'</span>`
`<span>[^abc]</span>`	Regex	Non-character class	`<span>sed 's/[^0-9]/X/g'</span>`
`<span>$pattern$</span>`	Regex	Capture group	`<span>sed 's/$a$\1/\1/'</span>`
`<span>\1, \2...</span>`	Regex	Back-reference	`<span>sed 's/$.$$.$/\2\1/'</span>`
`<span>\<</span>`, `<span>\></span>`	Regex (GNU extension)	Word boundary	`<span>sed 's/\<word\>/WORD/'</span>`
`<span>n,$</span>`	Address	From line n to the end	`<span>sed '10,$d'</span>`
`<span>/pattern/</span>`	Address	Matching regex line	`<span>sed '/error/d'</span>`
`<span>/p1/,/p2/</span>`	Address range	From p1 to p2	`<span>sed '/start/,/end/d'</span>`
`<span>n~m</span>`	Address (GNU extension)	Every m lines, starting from n	`<span>sed '1~2d'</span>`
`<span>!</span>`	Address control	Address negation	`<span>sed '/pattern/!d'</span>`

Learning Resources

Practice Suggestions

# Create test files for practice
cat &gt; test.txt &lt;&lt; 'EOF'
Line 1: Hello World
Line 2: hello world
Line 3: HELLO WORLD
Line 4: 123-456-7890
Line 5: [email protected]
EOF

# Practice various sed operations
# 1. Text replacement
# 2. Line operations
# 3. Regular expression matching
# 4. Complex transformations

Online Tools

• sed Online Testing:

• https://sed.js.org
• https://regex101.com (supports sed syntax)

• Regular Expression Testing:

• https://regexr.com
• https://www.regexpal.com

In Conclusion

sed, along with grep and awk, is known as the “three musketeers” of Linux text processing, and is a core tool that every Linux user must master. Mastering sed means you have a more efficient capability in automation operations, log management, and text processing tasks. Its usage scenarios include but are not limited to:

✅ Improving Efficiency – Batch processing text, saving a lot of time✅ Automated Operations – Writing scripts for configuration management✅ Data Processing – Handling logs, CSV, configuration files✅ Code Refactoring – Batch modifying code files✅ System Management – Processing system configurations and user data

Learning Suggestions

1. Start Simple – First master basic replacements and deletions
2. Progress Gradually – Learn regular expressions and advanced features step by step
3. Practice More – Apply what you have learned in actual work
4. Refer to Documentation – Consult the man pages when encountering problems
5. Share and Communicate – Discuss learning experiences with others

Advanced Directions

• Deepen your understanding of regular expressions
• Combine awk and grep to build powerful text processing pipelines
• Learn sed scripting programming
• Study performance optimization techniques
• Explore sed applications in specific fields

I hope this article helps you comprehensively master the sed command! If you have any questions, feel free to leave a comment for discussion 💬

#Linux #sed #StreamEditor #RegularExpressions #ShellScript

Basic Concepts of sed

What is sed?

Working Principle

Basic Syntax

Basic sed Commands

Replacement Command (s)

Delete Command (d)

Print Command (p)

Append, Insert, and Change (a, i, c)

Line Numbers and Address Ranges

Advanced Usage

Using Regular Expressions

Back-referencing and Capture Groups

Multiple Commands

Hold Space

In-place Editing (-i option)

Practical Cases

Configuration File Processing

Log Processing

Code Processing

Data Processing

System Administration

Performance Optimization Tips

Early Termination of Processing

Avoid Unnecessary Operations

Use More Specific Patterns

Common Errors and Debugging

Common Errors

Debugging Tips

Practical Script Examples

Configuration File Management Script

Log Analysis Script

Batch File Processing Script

Text Template Engine

Code Refactoring Helper Script

Best Practice Summary

Safety Practices

Performance Practices

Maintainability Practices

Cross-Platform Compatibility

Quick Reference

Common Command Quick Reference

Address Range Quick Reference

Flag Quick Reference

Common Matching and Address Patterns

Learning Resources

Recommended Reading

Practice Suggestions

Online Tools

In Conclusion

Learning Suggestions

Advanced Directions

Related posts

Leave a Comment Cancel reply