Practical Linux Text Processing Tools: Essential Skills for Log Analysis

tail, grep, and awk are three commonly used command-line tools in Linux/Unix environments. Each has its own focus, but they are often used together to process text data.

1. <span><span>tail</span></span>: View the end of a file

Function: Displays the end portion of a file (default last 10 lines), commonly used for viewing logs or real-time monitoring of file updates.

Common Usage:

# View the last 20 lines of a file
tail -n 20 filename.log
# Real-time tracking of file updates (e.g., monitoring logs)
tail -f /var/log/nginx/access.log
# Display from line 100 to the end of the file
tail -n +100 filename.log

Practical Example:

# Monitor real-time login logs for ssh
tail -f /var/log/secure | grep ssh

2. <span><span>grep</span></span>: Text search tool

Function: Searches text based on regular expressions, supports filtering, reverse matching, context display, etc.

Common Usage:

# Search for lines containing "error" (case-sensitive)
grep "error" server.log
# Case-insensitive search for "error"
grep -i "error" server.log
# Reverse matching (exclude lines containing "debug")
grep -v "debug" server.log
# Display matching lines and their next 3 lines (After context)
grep -A 3 "panic" app.log
# Recursively search all files in a directory
grep -r "TODO" /path/to/code/

Practical Example:

# Find unclosed parentheses in all PHP files
grep -rn "if (" *.php | grep -v ")"

3. <span><span>awk</span></span>: Text processing and data analysis

Function: Processes structured text (like CSV, logs) by columns, supports calculation, filtering, and formatted output.

Common Usage:

# Print the 1st and 3rd columns of a file
awk '{print $1, $3}' data.txt
# Filter rows by condition (rows where the 2nd column is greater than 100)
awk '$2 > 100 {print $0}' sales.csv
# Calculate the sum of the 3rd column
awk '{sum += $3} END {print sum}' data.txt
# Use a delimiter (e.g., colon to split /etc/passwd)
awk -F: '{print $1, $6}' /etc/passwd
# Combine with regular expressions (match lines containing "404")
awk '/404/ {print $7, $9}' access.log

Practical Example:

# Count the number of accesses for each IP in Nginx logs
awk '{ip_count[$1]++} END {for (ip in ip_count) print ip, ip_count[ip]}' access.log
# Extract rows from a CSV file where the second column is greater than 50
awk -F, '$2 > 50 {print $1, $3}' data.csv

Example of Combined Usage

Scenario: Real-time log monitoring, extracting specific error information for analysis

tail -f app.log | grep --line-buffered "ERROR" | awk '{print $2, $5}' | tee errors.txt

• <span>tail -f</span>: Real-time tracking of log updates.

• <span>grep --line-buffered</span>: Immediately output matching lines (avoiding buffer delays).

• <span>awk</span>: Extract the 2nd column (time) and the 5th column (error code).

• <span>tee</span>: Output the results to both the screen and the file <span>errors.txt</span>.

Appendix: Quick Reference for Common Parameters

Command	Parameter	Description
tail	-n <number of lines>	Specify the number of lines to display
-f	Real-time tracking of file updates
grep	-i	Ignore case
Reverse matching (exclude)
-A/-B/-C <>	Display matching lines after/before/both
awk	-F <delimiter>	Specify field delimiter (default is space)
NR	Current line number
NF	Number of fields in the current line

<span><span>Follow me:</span></span><span><span>Previous articles recommended:</span></span><span><span>Key points for data center inspection -- Ensuring stable operation of the data center</span></span><span><span>How to become an excellent engineer: Strategies for quickly locating and solving faults</span></span><span><span>Say goodbye to searching files in each directory! Master the find command to locate files instantly and avoid wasting time at work!</span></span><span><span>How to reset the administrator password in Linux -- Don’t ask others when it’s critical</span></span><span><span>Essential tools for Linux network detectives: tcpdump installation and efficient packet capture techniques revealed</span></span><span><span>Learn to say goodbye to running around the data center! Detailed explanation of server BMC functions, default management address/username/password all mastered</span></span><span><span>Is the internal network device's time always slow? No professional NTP server? Uncover a zero-cost time synchronization solution</span></span><span><span>Essential for Linux system monitoring: Easily master CPU, memory, and disk usage</span></span>