Linux Stream Editing Commands: sed and awk

1. Overview of Stream Editors

Stream editors are powerful tools for text processing in Linux, efficiently completing text transformation tasks in a non-interactive, streaming manner. Unlike interactive editors (such as Vim), stream editors automatically process text based on predefined rules, making them particularly suitable for batch processing and automation tasks.

Working Principle: A stream editor reads one line of data from the input at a time, matches and modifies the data based on the provided commands, outputs to standard output, and then continues processing the next line until all data has been processed. This mode is highly efficient for handling large files.

Selection Principles:

Simple replacements, deletions → Use sed
Complex data extraction, calculations → Use awk
Combined use → Maximize power

2. Detailed Explanation of sed Command

2.1 Purpose

sed (Stream Editor) is a stream text editor used for line-by-line processing and transformation of text. Its main functions include:

Text Replacement: Batch replacement of strings
Line Operations: Deleting, inserting, appending lines
Pattern Matching: Text processing based on regular expressions
Batch Processing: Configuration file modifications, log cleaning

2.2 Syntax

sed [options] 'command' input_file

Common Options:

-n: Suppress default output, only print processed text
-i: Modify file content directly (use with caution, backup recommended)
-e: Allow specifying multiple editing commands
-r or -E: Use extended regular expressions
-f: Read sed scripts from a specified file

Common Commands:

s: Substitute (replace)
d: Delete
p: Print
i: Insert (before the line)
a: Append (after the line)

2.3 Examples

Replacement Operations:

# Basic replacement (only replace the first match in each line)
sed 's/old/new/' file.txt

# Global replacement (replace all matches)
sed 's/old/new/g' file.txt

# Replace specific lines
sed '3s/old/new/' file.txt      # Only replace line 3
sed '2,4s/old/new/g' file.txt   # Replace lines 2-4

# Case-insensitive replacement
sed 's/error/ERROR/gi' file.txt

# Directly modify the file (backup recommended)
sed -i.bak 's/old/new/g' file.txt

Deletion Operations:

# Delete specified lines
sed '3d' file.txt          # Delete line 3
sed '2,5d' file.txt        # Delete lines 2-5
sed '$d' file.txt          # Delete the last line

# Delete matching lines
sed '/pattern/d' file.txt  # Delete lines containing pattern
sed '/^$/d' file.txt       # Delete empty lines
sed '/^#/d' file.txt       # Delete comment lines (starting with #)

# Delete line range
sed '/start/,/end/d' file.txt  # Delete lines between start and end

Insertion and Appending:

# Insert before line 3
sed '3i\Inserted content' file.txt

# Append after line 3
sed '3a\Appended content' file.txt

# Insert title at the beginning of the file
sed '1i\# Configuration file' config.txt

# Append after matching line
sed '/pattern/a\New content' file.txt

Print Operations:

# Print specified lines
sed -n '3p' file.txt           # Print line 3
sed -n '2,5p' file.txt         # Print lines 2-5

# Print matching lines
sed -n '/error/p' file.txt     # Print lines containing error

# Print line numbers
sed -n '/pattern/=' file.txt   # Show line numbers of matching lines

Multi-command Editing:

# Using -e option
sed -e 's/foo/bar/g' -e 's/hello/world/g' file.txt

# Using semicolon to separate
sed 's/foo/bar/g; s/hello/world/g' file.txt

# Using script file
sed -f script.sed file.txt

Regular Expression Applications:

# Using extended regex (-r or -E)
sed -r 's/[0-9]+/NUM/g' file.txt        # Replace all numbers
sed -r 's/\s+/ /g' file.txt             # Replace multiple spaces with a single space

# Group references
sed 's/\(hello\) \(world\)/\2 \1/' file.txt  # Swap order
sed -r 's/(hello) (world)/\2 \1/' file.txt   # Extended regex syntax

# Match at the beginning and end of lines
sed 's/^/# /' file.txt          # Add # at the beginning of each line
sed 's/$/ END/' file.txt        # Add END at the end of each line

Practical Scenario Examples:

# Modify port number in configuration file
sed -i 's/port=8080/port=9090/' config.conf

# Delete comments and empty lines from the file
sed '/^#/d; /^$/d' config.conf

# Extract error messages from logs
sed -n '/ERROR/p' app.log

# Batch modify multiple files
sed -i 's/old_domain/new_domain/g' *.html

# Add configuration after specified line
sed -i '/\[server\]/a\    listen 80;' nginx.conf

2.4 Source Code

sed is part of the GNU toolset, and the source code can be obtained as follows:

Official homepage: GNU sed
Git repository: Savannah sed
GitHub mirror: mirror/sed

2.5 References

System manual: man sed
Official documentation: GNU sed manual
Online manual: man7.org sed

(Links available at the bottom of the article)

3. Detailed Explanation of awk Command

3.1 Purpose

awk is a powerful text processing tool developed by Alfred Aho, Peter Weinberger, and Brian Kernighan in 1977 (the name is derived from the initials of their last names). Its main functions include:

Column-based text processing: Supports field splitting and extraction
Data statistics and calculations: Summation, averages, counts, etc.
Conditional filtering: Filtering data based on pattern matching
Report generation: Formatting output data
Log analysis: Extracting and counting log information

3.2 Syntax

Basic Format:

awk [options] 'program' input_file

Where program has the structure:

'pattern {action}'

Complete Format:

awk -F'field_separator' -v variable=value 'BEGIN {initialization} pattern {action} END {finalization}' file

Common Options:

-F: Specify field separator (default is space or tab)
-v: Define variables, e.g.,-v name=value
-f: Read awk scripts from a file

Pattern Explanation: The pattern specifies when to execute the action and can be:

Regular expression: /error/ matches lines containing error
Relational expression: $3 > 100 third column greater than 100
Pattern matching: $1 ~ /^root/ first column starts with root
BEGIN: Execute before processing any lines
END: Execute after processing all lines
Omitting pattern: Execute action for all lines

Action Explanation: The action is a statement enclosed in curly braces {} that specifies the operation to perform on matching lines:

{print $1}: Print the first column
{sum += $2}: Accumulate the second column
{print $1, $3}: Print the first and third columns
Omitting action defaults to{print $0} (print the entire line)

Pattern and Action Combination Examples:

awk '/error/ {print $0}' file.txt      # Print lines containing error
awk '$3>100 {print $1}' file.txt       # Print first column when third column>100
awk '{print $1}' file.txt              # Omit pattern, process all lines
awk '/error/' file.txt                 # Omit action, print matching lines

Program Structure:

awk 'BEGIN {initialization} pattern {processing} END {finalization}' file

BEGIN Block: Executes before reading any input lines, used for initializing variables, printing headers, etc.
Main Block: Executes pattern matching and action operations for each line
END Block: Executes after processing all input lines, used for outputting statistics, etc.

Built-in Variables:

$0: Entire line content
$1, $2, ...: Content of the 1st, 2nd, … columns
NR: Current line number (Number of Record)
NF: Number of fields in the current line (Number of Field)
FS: Input field separator (Field Separator)
OFS: Output field separator (Output Field Separator)
RS: Record separator (default newline)
ORS: Output record separator
FILENAME: Name of the currently processed file

3.3 Examples

Field Extraction:

# Print the first column
awk '{print $1}' file.txt

# Print multiple columns
awk '{print $1, $3}' file.txt

# Print the last column
awk '{print $NF}' file.txt

# Print the second to last column
awk '{print $(NF-1)}' file.txt

# Specify field separator
awk -F':' '{print $1, $3}' /etc/passwd

# Custom output format
awk '{printf "% -10s %-8s\n", $1, $2}' file.txt

Conditional Filtering:

# Numerical comparison
awk '$3 > 100' file.txt              # Lines where third column is greater than 100
awk '$3 > 100 {print $1, $3}' file.txt

# String matching
awk '/error/' file.log               # Lines containing error
awk '$1 == "root"' /etc/passwd       # First column equals root

# Combined conditions
awk '$1 == "root" && $3 > 1000' file.txt
awk '$3 > 100 || $4 < 50' file.txt

# Regular matching
awk '$1 ~ /^[A-Z]/' file.txt         # First column starts with uppercase letter
awk '$1 !~ /^#/' file.txt            # First column does not start with #

Calculations and Statistics:

# Summation
awk '{sum += $1} END {print "Total:", sum}' file.txt

# Average
awk '{sum += $1; count++} END {print "Average:", sum/count}' file.txt

# Count lines
awk 'END {print "Total lines:", NR}' file.txt

# Count matching lines
awk '/error/ {count++} END {print "Error count:", count}' file.log

# Maximum and minimum values
awk 'NR==1 {max=$1; min=$1} {if($1>max) max=$1; if($1<min) min=$1} END {print "Max:", max, "Min:", min}' file.txt

Advanced Data Processing:

# Count occurrences of fields
awk '{count[$1]++} END {for(item in count) print item, count[item]}' file.txt

# Group statistics
awk '{sales[$1] += $2} END {for(product in sales) print product, sales[product]}' sales.txt

# Remove duplicates
awk '!seen[$0]++' file.txt

# Sort by column and output
awk '{print $2, $1}' file.txt | sort -n

Formatted Output:

# Use printf for formatting
awk '{printf "% -15s %10d\n", $1, $2}' file.txt

# Add header
awk 'BEGIN {print "Name\t\tCount"} {print $1, "\t\t", $2}' file.txt

# Generate report
awk 'BEGIN {
    print "========== Sales Report =========="
}
NR > 1 {
    sales[$1] += $2
    total += $2
}
END {
    for(product in sales) {
        printf "% -15s %10d\n", product, sales[product]
    }
    print "=============================="
    printf "Total: %20d\n", total
}' sales_data.txt

Practical Scenario Examples:

# Analyze /etc/passwd file
awk -F':' '{print "User:", $1, "UID:", $3}' /etc/passwd

# Count HTTP status codes
awk '{status[$9]++} END {for(s in status) print s, status[s]}' access.log

# Extract logs for a specific time period
awk '/2025-11-26 10:[0-9][0-9]:/' app.log

# CSV to TSV
awk 'BEGIN {FS=","; OFS="\t"} {$1=$1; print}' data.csv

# Calculate total file size (with ls -l)
ls -l | awk 'NR>1 {sum += $5} END {print "Total size:", sum/1024/1024, "MB"}'

# Count access times for each IP
awk '{ip[$1]++} END {for(i in ip) print ip[i], i}' access.log | sort -rn | head -10

# Extract process information
ps aux | awk 'NR>1 {print $1, $2, $11}'

3.4 Source Code

awk has multiple implementation versions, the most commonly used is GNU awk (gawk):

Official homepage: GNU gawk
Git repository: Savannah gawk
One True Awk: onetrueawk/awk (original awk implementation)

3.5 References

System manual: man awk or man gawk
Official documentation: GNU awk manual
Online manual: man7.org gawk