The Three Musketeers of Linux
grep
-n：Print line numbers
-A：After, print the N lines after the matched content
-B：Before, print the N lines before the matched content
-C：Center, print N lines before and after the matched content
-E: Support extended regex, e.g., grep -E 'root|nginx' /etc/passwd
-v：Invert the match
-o：Only print the matched content
-w：Exact match
-P：Support Perl regex
-i：Ignore case
-r：Recursively search through files, e.g., grep -r 'www.baidu.com' ./
-R：Recursively search through files, including symlink files, e.g., grep -R 'www.baidu.com' ./
-l：Only show filenames
-h：Only show file contents
-f：Compare file contents, showing the file with fewer contents first, and the one with more contents second; using invert will show differing file contents
-c：Count lines, similar to wc -l
-G：Support basic regex
-m：Show the first N lines, similar to head -n
[root@m01 web]# grep -r 'www.zls.com' /root/web/
/root/web/css/style.css:www.zls.com
/root/web/js/main.js:www.zls.com
/root/web/index.html:www.zls.com
[root@m01 web]# grep -lr 'www.zls.com' /root/web/|xargs sed -i.zls 's#www.zls.com#www.baidu.com#g'awk
sed
In sed, our core content is mainly divided into four parts:
-
Add
-
Delete
-
Modify
-
Search
Of course, we also have some advanced content: pattern space and hold space.
sed Command Execution Process
For example:
# The following is the content of zls.txt
1,zls,666
2,wls,777
3,cls,888
4,lls,999
After executing <span>sed -n '3p' zls.txt</span>, what does sed do?
1. sed first reads the file content line by line.2. For each line read, it checks if it is the desired line.3. If not, it checks if the -n option is added.4. If -n is added, it reads the next line.5. If -n is not added, it outputs all content to the command line (default output).6. If it is the desired line (the third line), it checks the subsequent actions (p d s a i c).7. After processing the actions, it outputs the specified content.8. Even after reading is complete and content is output, sed continues to read until the last line of the file.
sed – Search
| sed Command Options | Option Meaning | sed Command Action | Action Meaning |
|---|---|---|---|
| -n | Cancel default output | p | |
| -r | Support extended regex | d | Delete |
| a | Append | ||
| i | Insert |
p：Print, display
## sed displays a single line
[root@m01 ~]# sed '3p' zls.txt
1,zls,666
2,wls,777
3,cls,888
3,cls,888
4,lls,999
## sed cancels default output
[root@m01 ~]# sed -n '3p' zls.txt
3,cls,888
## sed displays multiple lines and cancels default output
[root@m01 ~]# sed -n '1,3p' zls.txt
1,zls,666
2,wls,777
3,cls,888
## sed fuzzy search
[root@m01 ~]# sed -n '/zls/p' zls.txt
1,zls,666
[root@m01 ~]# sed -nr '/zls|cls/p' zls.txt
1,zls,666
3,cls,888
[root@m01 ~]# sed -n '/zls/,/cls/p' zls.txt
1,zls,666
2,wls,777
3,cls,888
## sed implements grep -A
[root@m01 ~]# sed -n '/zls/,+2p' zls.txt
1,zls,666
2,wls,777
3,cls,888
[root@m01 ~]# grep 'zls' -A 2 zls.txt
1,zls,666
2,wls,777
3,cls,888
## sed reads a line every specified number of lines
[root@m01 ~]# sed -n '1~2p' zls.txt
1,zls,666
3,cls,888
[root@m01 ~]# sed -n '1~3p' zls.txt
1,zls,666
4,lls,999
sed – Delete
d：Delete
## Delete specified line without modifying the original file
[root@m01 ~]# sed '2d' zls.txt
## Delete the last line
[root@m01 ~]# sed '$d' zls.txt
1,zls,666
3,cls,888
## Delete all lines from zls to cls
[root@m01 ~]# sed -n '/zls/,/cls/d' zls.txt
sed – Add
Content
c：Replace entire line content
[root@m01 ~]# cat zls.txt
1,zls,666
3,cls,888
4,lls,999
[root@m01 ~]# sed '2c2,huanglong,438' zls.txt
1,zls,666
2,huanglong,438
4,lls,999
a：Append
[root@m01 ~]# sed '$a5,huanglong,438' zls.txt
1,zls,666
3,cls,888
4,lls,999
5,huanglong,438
[root@m01 ~]# sed '2a5,huanglong,438' zls.txt
1,zls,666
3,cls,888
5,huanglong,438
4,lls,999
[root@m01 ~]# sed '1a2,huanglong,438' zls.txt
1,zls,666
2,huanglong,438
3,cls,888
4,lls,999
i：Insert
[root@m01 ~]# sed '$i2,huanglong,438' zls.txt
1,zls,666
3,cls,888
2,huanglong,438
4,lls,999
[root@m01 ~]# sed '1i2,huanglong,438' zls.txt
2,huanglong,438
1,zls,666
3,cls,888
4,lls,999
sed – Modify
s：Substitute
g：Global
s###g
s@@@g
sAnything is fineg
## Basic usage
[root@zabbix01 ~]# sed 's#zls#ZLS#g' zls.txt
1,ZLS,666
2,wls,777
3,cls,888
4,lls,999
## Using regex
[root@m01 ~]# sed 's#[0-9]#666#g' zls.txt
666,zls,666666666
666,wls,666666666
666,cls,666666666
666,lls,666666666
## Back reference
[root@m01 ~]# ifconfig eth0|sed -nr 's#^.*inet (.*) net.*#\1#gp'
10.0.0.61
[root@m01 ~]# ip a s eth1|sed -nr 's#^.*inet (.*)/24.*#\1#gp'
172.16.1.61
sed’s Pattern Space
Replace all newline characters in the file with spaces.
N：When reading the file, let sed read the next line content together.awk
awk’s Built-in Variables, Actions, and Options
| awk Built-in Variables | Variable Meaning | awk Options | Option Meaning | awk Actions | Action Meaning |
|---|---|---|---|---|---|
| NR | Number of Record (line number) | -F | Specify delimiter | gsub | Replace |
| RS | Record Separator (line separator, \n) | -v | Specify variable (built-in or custom) | ||
| FS | Field Separator (space) | ||||
| NF | Number of Fields (how many columns in each line) |
Note: awk outputs variables using single quotes, while bash outputs variables using double quotes.
<span>awk</span> is not a command, it is a language.
<span>awk</span> is also known as GNU awk, gawk.
[root@m01 ~]# ls -l $(which awk)
lrwxrwxrwx. 1 root root 4 Jul 5 2021 /usr/bin/awk -> gawk
awk Execution Process
Three Stages
-
Before Reading the File
-
BEGIN{}
-
1. Before reading the file, check the command options, e.g., -F, -v.
-
2. If BEGIN{} is written, execute the instructions in BEGIN{} first.
-
While Reading the File
-
{}
-
1. awk reads the file line by line.
-
2. After reading a line, it checks if it meets the condition; if so, it executes.
-
3. If the condition is not met, awk continues to read the next line until the condition is met or the end of the file is reached.
-
After Reading the File
-
END{}
-
1. After all files are read, execute the instructions in END{}.
[root@m01 ~]# awk 'BEGIN{xxx}{print $1}END{print 1/3}' zls.txt
[root@m01 ~]# awk -F: 'BEGIN{print "name","uid"}{print $1,$3}END{print "File processing complete"}' /etc/passwd|column -t
[root@m01 ~]# awk -F: 'BEGIN{print "name","uid","gid"}{print $1,$3,$4}END{print "sb"}' /etc/passwd|column -t
name uid gid
root 0 0
bin 1 1
daemon 2 2
adm 3 4
lp 4 7
sync 5 0
shutdown 6 0
halt 7 0
mail 8 12
operator 11 0
games 12 100
ftp 14 50
awk Rows and Columns
Row: Record
Column: Field
awk Get Row
NR：Number of Record
[root@m01 ~]# awk 'NR==1' /etc/passwd
root:x:0:0:root:/root:/bin/bash
## Range Get Row
[root@m01 ~]# awk 'NR>=1 && NR<=3' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[root@m01 ~]# awk 'NR==1,NR==3' /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
## Lines containing zls and cls
[root@m01 ~]# awk '/zls|cls/' zls.txt
1,zls,666
3,cls,888
zls,111
[root@m01 ~]# awk 'NR>=3' zls.txt
/tmp/check_2.txt
/tmp/check_3.txt
/tmp/check_4.txt
/tmp/check_5.txt
3,cls,888
zls,111
4,lls,999
[root@m01 ~]# awk '/zls/,/cls/' zls.txt
1,zls,666
/tmp/check_1.txt
/tmp/check_2.txt
/tmp/check_3.txt
cls
zls
4,lls,999
### awk Control End Mark
Record Separator
[root@m01 ~]# awk -vRS=, 'NR==1' zls.txt
awk Get Column
FS：Built-in variable, column separator -F: = -vFS=:
[root@m01 ~]# awk -vFS=: '{print $1}' /etc/passwd
[root@m01 ~]# awk -vFS=: '{print $1,$NF}' /etc/passwd
[root@m01 ~]# awk -F: '{print $1"#"$2"#"$3"#"$4"#"$5"|"$6","$NF}' /etc/passwd
## Modify output content separator
[root@m01 ~]# awk -F: -vOFS=# '{print $1,$2,$3,$4,$5,$6,$NF}' /etc/passwd
[root@m01 ~]# awk -F: '{print $0}' /etc/passwd
awk Get Row and Column
## Get the running time from top
[root@m01 ~]# top -n1 |awk 'NR==1{print $5}'
1
## Get the IP address from the network card configuration file
[root@m01 ~]# awk -F= '/IPADDR/{print $2}' /etc/sysconfig/network-scripts/ifcfg-eth0
10.0.0.61
# Please find the people with the surname Zhang, their second donation amount and name
##### Problematic Writing ############
[root@m01 ~]# cat user.txt
1 Zeng Laoshi 133411023 :110:100:75
2 Deng Ziqi 44002231 :250:10:88
3 Zhang Xinyu 877623568 :120:300:200
4 Gu Linazha 11029987 :120:30:79
5 Di Lireba 253097001 :220:100:200
6 Jiang Shuying 535432779 :309:10:2
7 Ju Jingyi 68005178 :130:280:385
8 Zhang Yuqi 376788757 :500:290:33
9 Wen Zhang 259872003 :100:200:300
[root@m01 ~]# awk -F '[ :]+' 'BEGIN{print "Surname","Name","Donation Amount"}/^Zhang/{print $2,$3,$6}' user.txt
Surname Name Donation Amount
[root@m01 ~]# awk -F '[ :]+' 'BEGIN{print "Surname","Name","Donation Amount"}/Zhang/{print $2,$3,$6}' user.txt
Surname Name Donation Amount
Zhang Xinyu 300
Zhang Yuqi 290
Wen Zhang 200
############# Correct Writing ################
[root@m01 ~]# awk -F '[ :]+' 'BEGIN{print "Surname","Name","Donation Amount"}$2~/Zhang/{print $2,$3,$6}' user.txt
Surname Name Donation Amount
Zhang Xinyu 300
Zhang Yuqi 290
[root@m01 ~]# awk -F '[ :]+' 'BEGIN{print "Name","Donation Amount"}$2~/Zhang/ && $3~/X/{print $2,$3,$6}' user.txt |column -t
Name Donation Amount
ZhangXinyu 300
[root@m01 ~]# awk -F '[ :]+' 'BEGIN{print "Name","Donation Amount"}$2~/Zhang/ && $3~/X/{print $2 $3,$6}' user.txt |column -t
Name Donation Amount
ZhangXinyu 300
### Display all QQ numbers starting with 25 and their names
[root@m01 ~]# awk '$4~/^25/{print $2 $3,$4}' user.txt
DiLireba 253097001
WenZhang 259872003
### Display all QQ numbers whose last digit is 1 or 3, full name and QQ
[root@m01 ~]# awk '$4~/1$|3$/{print $2$3,$4}' user.txt
ZengLaoshi 133411023
DengZiqi 44002231
DiLireba 253097001
WenZhang 259872003
[root@m01 ~]# awk '$4~/(1|3)$/ {print $2$3,$4}' user.txt
ZengLaoshi 133411023
DengZiqi 44002231
DiLireba 253097001
WenZhang 259872003
[root@m01 ~]# awk '$4~/[13]$/{print $2$3,$4}' user.txt
ZengLaoshi 133411023
DengZiqi 44002231
DiLireba 253097001
WenZhang 259872003
### Display that each donation value starts with $ $110:$00$75
[root@m01 ~]# awk '{gsub(/:/,"$");print $0}' user.txt
1 Zeng Laoshi 133411023 $110$100$75
2 Deng Ziqi 44002231 $250$10$88
3 Zhang Xinyu 877623568 $120$300$200
4 Gu Linazha 11029987 $120$30$79
5 Di Lireba 253097001 $220$100$200
6 Jiang Shuying 535432779 $309$10$2
7 Ju Jingyi 68005178 $130$280$385
8 Zhang Yuqi 376788757 $500$290$33
9 Wen Zhang 259872003 $100$200$300
[root@m01 ~]# awk '{gsub(/:/,"$",$5);print $0}' user.txt
1 Zeng Laoshi 133411023 $110 :100:75
2 Deng Ziqi 44002231 $250 :10:88
3 Zhang Xinyu 877623568 $120 :300:200
4 Gu Linazha 11029987 $120 :30:79
5 Di Lireba 253097001 $220 :100:200
6 Jiang Shuying 535432779 $309 :10:2
7 Ju Jingyi 68005178 $130 :280:385
8 Zhang Yuqi 376788757 $500 :290:33
9 Wen Zhang 259872003 $100 :200:300
function gsub(){
$1
$2
$3
xxxx
}
gsub(xx,aaa,d)
gsub("Content to be replaced","New content to replace",Column N)
## Comprehensive Application: Find the range of numbers from 1-255 in ifconfig
[root@m01 ~]# ifconfig |awk -vRS='[^0-9]+' '$0>=1 && $0<=255'
10
61
255
255
255
10
255
6
80
20
29
3
64
20
29
3
2
5
2
6
1
172
16
1
61
255
255
255
172
16
1
255
6
80
20
29
3
9
64
20
29
3
9
46
9
117
29
7
73
127
1
255
6
1
128
10
awk Patterns and Actions
awk -F: 'NR==1{print $1,$3}' /etc/passwd
From the above command, we can see that <span>'NR==1{print $1,$3}'</span> can be understood as <span>'Pattern{Action}'</span> == <span>'Condition{Instruction}'</span>
Patterns in awk
-
Regular Expressions
# Regular expression syntax
'/regular expression/flag'
'$1~/regular expression/flag'
'$1!~/regular expression/flag'
However, we rarely use flags in awk.-
Comparison Expressions
NR==1
NR>=10
NR<=100
NR>=1 && NR<=10
$1>=100
-
Range Patterns
## Exact match line number: from line 10 to line 20
NR==10,NR==20
## Exact match string: from the line containing this string to the line containing another string
'/root/,/zls/'
'/from which string/,/to which string/'# Include all lines in between
## Fuzzy match string: from the line containing this string to the line containing another string
'$1~/oo/,$1~/zl/'
-
Special Patterns
BEGIN
END
Actions in awk
In awk, the most commonly used action is <span>print</span>
Of course, we also have other actions available:
-
print (print)
-
gsub (replace)
-
Variable assignment
-
Statistical calculations
useradd name;pass=`echo$RANDOM|md5sum|cut -c 1-10`;echo$pass|passwd --stdin name;echo$pass:$user >> /tmp/user.txt
seq 100|awk '{print "useradd test"$1";pass=`echo $RANDOM|md5sum|cut -c 1-10`;echo $pass|passwd --stdin test"$1";echo $pass:test"$1" >> /tmp/user.txt"}'|bash
If you want to use the BEGIN pattern, it must appear in pairs:<span>BEGIN{}</span>
We need to know that the content inside the braces of BEGIN{} will be executed before reading the file content.
Main application scenarios:
-
1. Calculations
[root@zabbix01 ~]# awk 'BEGIN{print 1/3}'
0.333333
-
2. awk function testing
-
3. Output table headers
END Pattern
Generally speaking, END{} is more important than BEGIN{} because BEGIN{} is optional; calculations can actually be done while reading the file, or executed afterwards.
In END{}, the content inside the braces will be processed after awk has read the last line of the file.
Function: We generally use END{} to display the results of log content analysis.Of course, there are other functions, such as displaying some tail information after file reading is complete.
# 1. Count how many lines are in the /etc/service file
[root@m01 ~]# awk '{hang++;print hang}' /etc/services
1
...
11176
No process, just results
[root@m01 ~]# awk '{hang++}END{print hang}' /etc/services
11176
### Can only count the total number of lines in the file
[root@m01 ~]# awk 'END{print NR}' /etc/services
11176
### Rogue writing
[root@m01 ~]# sed -n '$=' /etc/services
11176
[root@m01 ~]# grep -c '.*' /etc/services
11176
[root@m01 ~]# wc -l /etc/services
11176 /etc/services
# 2. Count the number of empty lines in /etc/service
[root@m01 ~]# awk '/^$/{print}' /etc/services
[root@m01 ~]# awk '/^$/{i++}END{print i}' /etc/services
17
# 3. Count the ages of all people in the following file
### Script method
#!/usr/bin/bash
n=0
for line in `cat user.txt`;do
if [[ $line =~ [0-9]+ ]];then
((n+=$line))
fi
done
echo$n
### awk method
[root@m01 ~]# cat user.txt
Name Age
Zeng Laoshi 23
Cangjing Kong 18
Xi Ye Xiang 99
[root@m01 ~]# awk 'NR>1{print $2}' user.txt
23
18
99
[root@m01 ~]# awk 'NR>1{n+=$2}END{print n}' user.txt
140
# 4. Count the number of times the status code is 200 in the nginx log and the traffic used when the status code is 200
[root@m01 ~]# zcat blog.driverzeng.com_access.log-20220623.gz |awk 'BEGIN{print "Status Code 200 Count","Total Traffic"}$10~/200/{code++;byte+=$11}END{print code,byte}'|column -t
Status Code 200 Count Total Traffic
3100 190477111
Script writing:
awk '
BEGIN{
print "Status Code","Total Traffic"
}
$10~/200/{
code++;byte+=$11
}
END{
print code,byte
}'|column -t
# 5. Count the number of times the status codes are 4xx and 5xx and the total traffic
[root@m01 ~]# zcat blog.driverzeng.com_access.log-20220623.gz|awk '$10~/^[45]/{i++;n+=$11}END{print i,n}'
580 519243
# 6. Comprehensive application: Count the number of times each status code and the total traffic for each status code
zcat blog.driverzeng.com_access.log-20220623.gz |awk '
BEGIN{
print "Status Code","Total Traffic"
}
$10~/200/{
i1++;n1+=$11
}
$10~/^3/{
i2++;n2+=$11
}
$10~/^4/{
i3++;n3+=$11
}
$10~/^5/{
i4++;n4+=$11
}
END{
print "200 Count:"i1,"200 Traffic:"n1
print "3xx Count:"i2,"3xx Traffic:"n2
print "4xx Count:"i3,"4xx Traffic:"n3
print "5xx Count:"i4,"5xx Traffic:"n4
}'|column -t
awk Arrays
The array data type in awk is a very useful type, unlike in shell, although shell arrays have their own advantages.
awk arrays are specifically used to count different categories.
** For example:**1. Count the number of times each IP appears in the nginx log.2. Count the number of times each status code appears in the nginx log.3. Count the number of accesses for each URI in the nginx log.
[root@m01 ~]# zcat blog.driverzeng.com_access.log-20220623.gz|awk '{print $1}'|sort|uniq -c|sort -nr
Assigning Values to awk Arrays
[root@m01 ~]# awk 'BEGIN{array[0]="zls";array[1]="wyk"}'
Accessing Values in awk Arrays
[root@m01 ~]# awk 'BEGIN{array[0]="zls";array[1]="wyk";print array[0],array[1]}'
zls wyk
Looping through Arrays in Shell
array[0]='zls'
array[1]='cls'
for name in${array[*]};do
echo$name
done
Looping through Arrays in awk
for(condition){
action
}
for condition;do
action
done
[root@m01 ~]# awk 'BEGIN{array[0]="zls";array[1]="wyk";for(num in array){print num}}'
0
1
[root@m01 ~]# awk 'BEGIN{array[0]="zls";array[1]="wyk";for(num in array){print array[num]}}'
zls
wyk
## Count the number of accesses for each IP address in the nginx log
zcat blog.driverzeng.com_access.log-20220623.gz |awk '{array[$1]++}END{for(ip in array){print ip,array[ip]}}'
#1. Extract the following domain names and count and sort them based on the domain name
https://blog.driverzeng.com/index.html
https://blog.driverzeng.com/1.html
http://post.driverzeng.com/index.html
http://mp3.driverzeng.com/index.html
https://blog.driverzeng.com/3.html
http://post.driverzeng.com/2.html
[root@m01 ~]# awk -F/ '{domain[$3]++}END{for(name in domain){print name,domain[name]}}' 1.txt
blog.driverzeng.com 3
post.driverzeng.com 2
mp3.driverzeng.com 1
#2. Count the total traffic used by each IP in the nginx log
[root@m01 ~]# zcat blog.driverzeng.com_access.log-20220623.gz |awk '{ip[$1]++;liuliang[$1]+=$11}END{for(i in ip){print i,ip[i],liuliang[i]}}'
awk Judgments
Comparison of awk judgments and shell judgments
## shell
if [ condition ];then
action
fi
if [ condition ];then
else
action
fi
if [ condition ];then
elif [ condition ];then
else
action
fi
## awk
if(condition){
action
}
if(condition){
action
}else{
action
}
if(condition){
action
}elseif(condition){
action
}else{
action
}
awk '{}END{for(condition){if(condition){action}else if(condition){action}else{action}}}'
awk '{
read file action
}END{
for(condition){
if(condition){
action
}else if(condition){
action
}else{
action
}
}
}'
#1. Check if disk usage is greater than 70%, if so, display "Disk space insufficient", otherwise display "Normal"
## Extract disk usage
[root@m01 ~]# df -h|awk -F '[ %]+' 'NR==2{if($5>70){print "Disk space insufficient"}else{print "Disk space is okay"}}'
Disk space is okay
[root@m01 ~]# df -h|awk -F '[ %]+' 'NR==2{if($5>70){print "Disk space insufficient"}else{print "Disk space is okay, current disk usage:"$5"%"}}'
Disk space is okay, current disk usage:9%Link: https://www.cnblogs.com/wangchengww/p/16540731.html
Copyright belongs to the original author, please delete if infringed.
WeChat group
WeChat group
To facilitate better communication on operation and maintenance and related technical issues, a WeChat group has been created. Friends who need to join the group can scan the QR code below to add me as a friend (note: join the group).
Blog
Guest
Blog
CSDN Blog: https://blog.csdn.net/qq_25599925
Juejin Blog: https://juejin.cn/user/4262187909781751
Knowledge Planet: https://wx.zsxq.com/group/15555885545422
Long press to recognize the QR code to visit the blog website and see more high-quality original content.