Linux Special: Content Filtering in System Usage

1. man

View command usage documentation

man unzip

2. tail

View the end of a file

# By default, view the last 10 lines of the file
 tail access.log

# View the last 100 lines of the file
tail -100 access.log

# View from line 50 to the end of the file
tail -n +50 access.log

# Monitor the file in real-time, outputting immediately when the file is updated
tail -f access.log

3. head

View the beginning of a file

# By default, view the first 10 lines of the file
 head access.log

# View the first 100 lines of the file
head -100 access.log

4. cat

Output the entire content of the file, printing from the first line to the last line to the console.

cat access.log

5. grep

Parameter	Description
-c	Count the number of matching lines
-E	Use extended regular expressions
-i	Ignore case differences
-v	Invert the match
-o	Only output the matching part of the file

# Filter all lines containing the string 'payment'
grep 'payment' access.log

# Filter all lines containing the string 'payment', ignoring case
grep -i 'payment' access.log

# Count the number of lines containing 10.8.0.6
# grep -c '10.8.0.6' access.log
219

# Filter lines starting with root
# grep '^root' /etc/passwd
root:x:0:0:root:/root:/bin/bash

# Use extended regex to filter lines starting with root or nginx
# grep -E  '^(root|nginx)' /etc/passwd
root:x:0:0:root:/root:/bin/bash
nginx:x:1000:1000::/home/nginx:/sbin/nologin

6. awk

Statement blocks can be divided into 3 types:

BEGIN statement block

Executed before reading the file, and executed once
In the BEGIN block, special variables like $0 cannot be used

main statement block

Executed in a loop while reading the file, executing once for each line read (default behavior)
main blocks can have multiple instances

END statement block

Executed after reading the file is complete, and executed once
There must be data to read (can be standard input) if there is an END block
END block can use special variables like $0, but these variables hold the data from the last round of the awk loop

awk 'BEGIN{
        statement1;
        statement2;
    }{
        statement1;
        statement2;
    }END{
        statement1;
        statement2;
    }' filename

In which the BEGIN and END statement blocks are formatted as BEGIN{...} and END{...}; while the main statement block is a general term, its pattern part has no fixed format and can be omitted, the main block is executed for each line read from the file.

# Use ':' as a delimiter, print the first column of the file
awk -F':' '{print $1}' /etc/passwd

# Use ':' as a delimiter, when the string root is found, print the corresponding line
# awk -F':' '/root/{print $0}' /etc/passwd
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

# Use ':' as a delimiter, when a string starting with root is found, print the first and last columns
# awk 'BEGIN{FS=":"} /^root/ {print $1, $NF}' /etc/passwd
root /bin/bash

# Equivalent
# awk -F':' '/^root/{print $1, $NF}' /etc/passwd
root /bin/bash

7. sed

sed [options] 'command' file(s)

Parameter	Description
`<span><span>a\</span></span>`	Insert text below the current line
`<span><span>i\</span></span>`	Insert text above the current line
`<span><span>d</span></span>`	Delete, remove selected lines
`<span><span>g</span></span>`	Indicates full line replacement
`<span><span>p</span></span>`	Indicates print line
`<span><span>w</span></span>`	Indicates writing the line to a file

# Copy a file to the current directory for testing
cp /etc/passwd .
# Search for lines starting with root, only print matching lines
# sed -n '/^root/p' /etc/passwd
root:x:0:0:root:/root:/bin/bash

# Delete lines starting with the string root
sed '/^root/d' passwd

# Add a line of text `maobaoinfo.com` below lines starting with the string root
# sed '/^root/a\maobaoinfo.com' passwd
root:x:0:0:root:/root:/bin/bash
maobaoinfo.com
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

# Replace lowercase strings starting with root with uppercase ROOT
# sed 's/^root/ROOT/g' passwd
ROOT:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

All of the above operations will not modify the original file; if you need to modify the original file, add the -i parameter, use with caution.

8. shell

Filter out the system IP addresses

# ip addr show | grep inet | awk '{print $2}'
127.0.0.1/8
192.168.25.73/24
172.17.0.1/16

| Pipe symbol
awk defaults to space or Tab

Combined application, involving grep / sed / awk regex matching and multidimensional array comprehensive application.

#!/bin/bash

[[ -z ${1} ]] && echo "\$1 for log directory"

logs=($(ls ${1} | egrep ".*\.log"))

cd ${1}

for log in ${logs[@]}; do
    printf "&lt;--------------%s--------------&gt;\n" ${log}
    egrep "lastMsgDsrp=.* diffTs=" ${log} | jq .msg | awk '{if ($2 ~ /lastMsgDsrp=.*/){print $2" "$3} else if ($3 ~ /lastMsgDsrp=.*/){pirnt $3" "$4}}' | sed 's/=//g' &gt; temp
    awk '{interface[$2][$NF] += 1}END{
            PROCINFO["sorted_in"] = "@ind_num_asc"
            for(i in interface){
                printf "% -30s: ", i;
                for(j in interface[i]){
                    printf "%1ss: %-8s", j, interface[i][j];
                }
                printf "\n"
            }
        }' temp
    printf "&lt;--------------%s--------------&gt;\n" ${log}
    printf "\n\n"
    rm -f temp
done