Today, I will introduce three commands that are frequently used for text processing in our Linux operating system: grep, awk, and sed, collectively known as the “Three Musketeers of Text” in the community.
Each of these commands has its strengths, and when used together, they can efficiently meet various needs for text searching, filtering, and editing.
1. grep
Search for lines that match a regular expression from the input stream (file or standard input).
1. Syntax:
grep [options] pattern [file...]Pattern: can be a plain string or a regular expression. File: can specify one or more files; if omitted, it reads from standard input.
2. Common options:
| Option | Description |
| ————- | ———————————————— |
| `-i` | Ignore case |
| `-v` | Invert match (output lines that do not contain the pattern) |
| `-n` | Show line numbers |
| `-r` / `-R` | Recursively search directories |
| `-l` | Only show filenames containing matches |
| `-c` | Count matching lines |
| `-E` | Use extended regular expressions (equivalent to `egrep`) |
| `-F` | Treat pattern as a fixed string (do not parse regex, equivalent to `fgrep`) |
| `-A n` | Show matching line and the following n lines |
| `-B n` | Show matching line and the preceding n lines |
| `-C n` | Show matching line and n lines before and after |
3. Supports regular expressions
`grep` supports basic regular expressions (BRE), and using `-E` enables extended regular expressions (ERE):
# Match lines starting with error or failed
grep -E "^(error|failed)" test.txt# Match numbers
grep "[0-9]\+" test.txt4. Some usage scenarios
1. Find error messages in logs: grep -i "error" /var/log/messages
2. Count occurrences of a keyword: grep -c "yidiandian" test.txt
3. Filter out comments and empty lines: grep -v "^\s*#\|^\s*$" test.txt2. sed
Perform **non-interactive editing** on the input stream (file or pipe), supporting operations such as replacement, deletion, insertion, and appending.
1. Syntax:
sed [options] 'command' [file...][address]Address: can be line numbers, regular expressions, or ranges (e.g., 1,5, /pattern/). Command: such as s (replace), d (delete), a (append), i (insert), p (print), etc.
2. Core command descriptions
| Command | Description |
| —————- | —————————- |
| `s/old/new/g` | Replace (`g` means global replacement) |
| `d` | Delete matching lines |
| `p` | Print line (often used with `-n`) |
| `a\text` | Append text after matching line |
| `i\text` | Insert text before matching line |
| `c\text` | Replace entire line with specified text |
| `=` | Print line number |
3. Common options
| Option | Description |
| —— | ——————————————— |
| `-n` | Suppress default output, only output results of explicit `p` commands |
| `-e` | Specify multiple editing commands |
| `-f` | Read sed script from file |
| `-i` | Edit the original file directly (⚠️ Use with caution) |
| `-r` | Use extended regular expressions (GNU sed) |
4. Some usage scenarios
# Replace all "yi" with "YI"
sed 's/yi/YI/g' test.txt
# Delete empty lines
sed '/^$/d' test.txt
# Delete comment lines starting with #
sed '/^#/d' test.txt
# Insert content after matching line
sed '/pattern/a\New line added' test.txt
# Use extended regex (match multiple digits)
sed -r 's/[0-9]+/NUM/g' test.txt3. awk
A **powerful text analysis and report generation language**, particularly adept at handling **columnar structured data**.
1. Syntax:
awk [options] 'BEGIN {initialization} pattern {action} END {finalization}' [file...]BEGIN: executed before processing input, often used to initialize variables and set delimiters. Pattern: matching condition (can be omitted, indicating action on all lines). {action}: operations performed on matching lines, such as print, assignment, calculation, etc. END: executed after all input has been processed, often used for summary output.
2. Core variables
| Variable | Meaning |
| ————— | ———————————————— |
| `$0` | Current line |
| `$1, $2, …` | 1st, 2nd, … fields |
| `NF` | Number of fields in the current line |
| `NR` | Current record number (line number) |
| `FNR` | Line number of the current file (distinguished from NR in multi-file scenarios) |
| `FS` | Input field separator (Field Separator, default is space or tab) |
| `OFS` | Output field separator |
| `RS` | Input record separator (default is newline) |
| `ORS` | Output record separator |
3. Common options
| Option | Description |
| —————- | ——————————- |
| `-F fs` | Specify field separator (e.g., `-F:`) |
| `-v var=value` | Define awk variable |
| `-f script.awk` | Load awk program from file |
4. Some usage scenarios
# Print the first and third columns of each line
awk '{print $1, $3}' test.txt
# Set delimiter to colon (e.g., /etc/passwd), print the first and sixth columns
awk -F: '{print $1, $6}' /etc/passwd
# Conditional filtering: print lines where the second column is greater than 100
awk '$2 > 100' test.txt
# Pattern matching: lines containing "error"
awk '/error/ {print $0}' test.txt
# Count lines
awk 'END {print NR}' test.txt4. Combined Usage
Generally, we first use `grep` to find, `sed` to modify, and `awk` to calculate, connecting them with pipes to manipulate text freely.
1. Clean and format `/etc/passwd`, generating a user report
grep -v "^#" /etc/passwd | sed 's/:/ /g' | awk '$3 >= 1000 {print "User: " $1 ", UID: " $3 ", Shell: " $7}'1. grep -v "^#": Remove comment lines.2. sed 's/:/ /g': Replace colons : with spaces for easier separation by awk.3. awk '$3 >= 1000': Only keep ordinary users with UID ≥ 1000 (Linux typically starts from 1000). Format output to show username, UID, Shell.2. Remove comments and empty lines from configuration files, keeping only valid configurations
grep -v "^\s*#\|^\s*$" nginx.conf | sed 's/\s*#.*$//' | awk -F'=' '{gsub(/"/, "", $2); print $1 " = " $2}'1. grep -v "^\s*#\|^\s*$": Remove comment lines starting with # (\s* means any whitespace). Remove pure empty lines.2. sed 's/\s*#.*$//': Remove comments at the end of lines (e.g., port=8080 # web port → port=8080).3. awk -F'=' '{gsub(/"/, "", $2); print $1 " = " $2}': Use = as the delimiter.4. gsub(/"/, "", $2): Remove quotes from the second column (e.g., "8080" → 8080). Format output.