The Three Musketeers of Text Processing in Linux

Core Concepts

• grep: Global Regular Expression Print. Used for searching and filtering text. Its core function is to match patterns (regular expressions) and output the matching lines.
• sed: Stream EDitor. A stream editor used for basic text transformations such as replacing, deleting, and inserting text. It processes text line by line and is very efficient.
• awk: Named after its three founders Alfred Aho, Peter Weinberger, and Brian Kernighan. It is not just a tool, but a powerful text processing programming language. It excels at slicing, filtering, counting, and formatting output for structured text (such as logs, CSV). Its capabilities are the most powerful.

Their design philosophy is “a tool should do one thing well”. By combining them through pipes (|), extremely complex text processing problems can be solved.

1. grep – The King of Search and Filter

The core of grep is pattern matching using regular expressions.

Common Options

• -i: Ignore case
• -v: Invert selection, i.e., output lines that do not match
• -n: Show line numbers of matching lines
• -c: Only output the total number of matching lines (count)
• -r or -R: Recursively search files in directories
• -l: Only output the names of files containing matches
• -E: Use extended regular expressions (equivalent to egrep), supporting more meta-characters like +, ?, |, ()
• -P: Enable PCRE (supports \\d, lookahead, etc.)
• -A num: Show matching lines and the following num lines
• -B num: Show matching lines and the preceding num lines
• -C num: Show matching lines and the surrounding num lines

Examples

Assuming we have a file test.txt with the following content:

Hello world
Linux is great
Hello again
12345
goodbye

1. Basic Search: Find lines containing “Hello”

grep "Hello" test.txt
# Output:
# Hello world
# Hello again

2. Ignore Case and Show Line Numbers: Find “linux”
```
grep -in "linux" test.txt
# Output:2:Linux is great
```

3. Inverse Search: Find lines that do not contain “Hello”

grep -v "Hello" test.txt
# Output:
# Linux is great
# 12345
# goodbye

4. Count: Count occurrences of “Hello” (number of lines)
```
grep -c "Hello" test.txt
# Output:2
```
5. Recursive Search in Directory: Search for “error” in all .log files in the current directory
```
grep -r "error" *.log
```
6. Using Pipes: View all processes related to “nginx” in the currently running processes
```
ps aux | grep nginx
```

2. sed – Stream Editor

The core of sed is the s (substitute) command, with the syntax s/old_pattern/new_pattern/flags.

Basic Structure

sed 'range+action' file

• Range: Numbers, regex addresses, comma ranges;
• Action: p print, d delete, s substitute, a/i append/insert.

Common Options and Commands

• -e: Specify sed commands to execute (can specify multiple)
• -i: Directly modify file content (very dangerous and useful, always test results without -i)
• -n: Silent mode, only output lines processed by sed commands
• s: Substitute command, the core command
• d: Delete command
• p: Print command
• a: Append text after specified line
• i: Insert text before specified line

Examples

Assuming we have a file sed_test.txt:

apple
banana
apple
orange

1. Simple Replacement: Replace the first “apple” in each line with “APPLE” (output to screen, does not modify file)
```
sed 's/apple/APPLE/' sed_test.txt
```
2. Global Replacement: Use g flag to replace all “apple”
```
sed 's/apple/APPLE/g' sed_test.txt
```

3. Directly Modify File: The -i option will actually change the original file

sed -i 's/apple/APPLE/g' sed_test.txt
# After execution, the file content is permanently changed

4. Delete Lines: Delete all lines containing “banana”
```
sed '/banana/d' sed_test.txt
```
5. Specify Line Number Operation: Delete the second line
```
sed '2d' sed_test.txt
```

6. Combine Commands: Replace and delete

sed -e 's/apple/APPLE/' -e '/orange/d' sed_test.txt

7. Multi-Pattern: Print from the first pattern to the second pattern
```
sed -n '/^apple/,/^orange/p' sed_test.txt
```

3. awk – The Programming Language Processor

awk treats each line as a record composed of fields, with the default field separator being space.$1, $2, … $n represent the 1st, 2nd, n-th fields respectively, and $0 represents the entire line.

Basic Structure

awk 'pattern { action }' filename

• Pattern: Can be a regular expression (e.g., /error/), a condition (e.g., $1 > 100), or special patterns (BEGIN, END).
• Action: A series of commands, commonly print.

Built-in Variables

• FS: Input field separator, default is space
• OFS: Output field separator, default is space
• RS: Input record (line) separator, default is newline
• ORS: Output record separator
• NF: Number of fields in the current line
• NR: Current line number (total)
• FNR: Line number in the current file

Examples

Assuming we have a structured file awk_test.txt with the following content:

Alice 90 85
Bob 78 92
Charlie 85 88

1. Print Specific Fields: Print the first and third columns (name and second score)
```
awk '{print $1, $3}' awk_test.txt
# Output:
# Alice 85
# Bob 92
# Charlie 88
```

2. Set Output Separator: Connect fields with a colon :

awk '{print $1 " : " $3}' awk_test.txt
# Or use a variable
awk 'BEGIN{OFS=" : "} {print $1, $3}' awk_test.txt

3. Conditional Filtering: Print lines where the first score is greater than 80

awk '$2 &gt; 80 {print $0}' awk_test.txt
# Output:
# Alice 90 85
# Charlie 85 88

4. Using Built-in Variables: Print the filename, line number, and number of fields for each line
```
awk '{print FILENAME, NR, NF}' awk_test.txt
```

5. BEGIN and END Patterns: Execute actions before and after processing, commonly used for initialization and summarization

# Calculate the average of the second score
awk 'BEGIN{sum=0} {sum+=$2} END{print "Average: ", sum/NR}' awk_test.txt
# Output:Average:  84.3333

6. Change Input Separator: Process the /etc/passwd file (fields separated by :), print the username (1st column) and login shell (7th column)
```
awk -F: '{print $1, $7}' /etc/passwd
# -F: specifies the input separator as colon
```

7. Formatted Printing: printf similar to C

awk '{printf "IP:%-15s Count:%4d\n",$1,$2}' file

8. Action File: Write awk scripts into script.awk file
```
awk -f script.awk file
```

Combining Usage: Infinite Power

This is where the true power of the “Three Musketeers” lies. By piping simple commands together, complex problems can be solved.

Example Task:

Scenario: From 10,000 lines of Apache logs, ① find 404 records; ② extract IP and UA; ③ deduplicate and count; ④ output Top 10.

A single line flow:

awk '$9==404 {print $1,$14}' access.log |
  sed 's/"//g' |
  sort -k2 |
  uniq -c |
  sort -nr |
  head -10 |
  awk '{printf "% -15s %-40s %4d\n",$2,$3,$1}'

Explanation:

1. awk filters 404 and takes IP (and14);
2. sed removes double quotes;
3. sort+uniq deduplicates and counts;
4. then sorts and takes the top 10;
5. awk formats and aligns the columns.

Quick Reference Table (Recommended for Desktop Use)

Search:   grep -Pn 'REGEX' file
Delete:   sed '/REGEX/d' file
Replace:   sed -i 's/A/B/g' file
Column:   awk -F: '{print $1,$3}' file
Count:   awk '{a[$1]++} END{for(k in a) print a[k],k}' file | sort -nr

Core Concepts

1. grep – The King of Search and Filter

Common Options

Examples

2. sed – Stream Editor

Basic Structure

Common Options and Commands

Examples

3. awk – The Programming Language Processor

Basic Structure

Built-in Variables

Examples

Combining Usage: Infinite Power

Quick Reference Table (Recommended for Desktop Use)

Related posts

Leave a Comment Cancel reply