Unveiling the Secrets of Major Companies: Mastering the Linux Triad

The Linux Triad consists of the three most important commands in the Linux system, renowned for their powerful functionalities and wide range of applications. The combination of these three tools can almost perfectly address data analysis scenarios in the Shell, hence they are collectively referred to as the Linux Triad.

1. grep

grep is a powerful text search tool used to find specified strings within file contents and output the matching lines to standard output. The grep command supports various options and pattern matching, making it very flexible and efficient for searching specific content in files.Text search command;

The command format is:

grep "search_string" filename

1) Basics

Options:-c: Count occurrences-i: Case insensitive-n: Output line numbers

2) Example

Count occurrencesgrep -c "xx" text.log

2. sed

sed is a stream editor primarily used for basic text transformations on input streams (or files). It can be used to find, add, delete, and modify data in text replacements. The power of the sed command lies in its ability to perform non-interactive editing of text, thereby improving the efficiency of processing large amounts of text.

1) Basic Concepts

sed is a stream editor, a text processing tool that supports regular expressions, executing corresponding commands line by line to process and edit text files.(1) Syntaxsed [options] ‘command’ filename

The options part includes common options such as -n, -e, -i, -f, -r.The command part includes: [address1, address2] [function] [parameters (flags)]

(2) The command format is:

sed [-nefri] 'command' text
cat text | sed [-nefr] 'command'
echo '.....' | sed [-nefr] 'command'

2) Options (nefri)

(1) Option -n

By default, sed outputs the content processed in the pattern space to standard output, which means it outputs to the screen. Adding the -n option sets it to quiet mode, meaning it will not output default print information unless specifically instructed to print in the subcommands, thus only the modified matching lines will be printed.(2) Option -e

If multiple operations are needed on the text content using sed, multiple subcommands must be executed.

(3) Option -i

By default, sed reads input lines into the pattern space, which can be simply understood as a memory buffer. The content processed by sed subcommands is from the pattern space, not directly from the file content. Therefore, after modifying the content in the pattern space, it does not directly write back to the input file but prints to standard output. If you need to modify the input file, you can specify the -i option.

(4) Option -f

Remember that the -e option can execute multiple subcommand operations, and separating multiple command operations with semicolons is also possible. If there are many command operations, it can become cumbersome, so you can write multiple subcommand operations into a script file and use the -f option to specify that script.

(5) Option -rThe matching patterns of the sed command support regular expressions, and by default, only basic regular expressions are supported. To support extended regular expressions, the -r option must be added.

3) Commands (acdipynNs)

a: append, c: change, d: delete, i: insert, p: print, y: translate characters, etc.

[root@localhost sed]# cat message
1a
2b
3c
4d
5e
6f
abcdABCD

(1) a

The subcommand a indicates inserting specified content below the specified line.

# Insert a line of content A below each line in the message file
sed 'a A' message
# Insert a line of content A below lines 1-2 in the message file
sed '1,2a A' message
# Insert three lines of content A, B, C below lines 1-2 in the message file, using \n to insert multiple lines.
sed '1,2a A\nB\nC' message

(2) i

The subcommand i works similarly to a, but it inserts specified content above the specified line.

# Insert a line of content A above each line in the message file
sed 'i A' message
# Insert a line of content A above lines 1-2 in the message file
sed '1,2i A' message
# Insert three lines of content A, B, C above lines 1-2 in the message file, using \n to insert multiple lines.
sed '1,2i A\nB\nC' message

(3) c

The subcommand c indicates replacing the content of specified lines with the desired content.

# Replace all lines in the message file with line A
sed 'c A' message
# Replace the content of lines 1-2 in the message file with A, noting that this means replacing all content of lines 1-2 with a single A, thus making lines 1-2 become one line.
sed '1,2c A' message
# Replace the content of lines 1-2 in the message file with A, adding \n to replace content with multiple lines.
sed '1,2c A\nA' message

(4) d

The subcommand d indicates deleting specified lines, which is straightforward and easy to understand.

# Delete all lines in the message file, as no addressing expression is added, so if you need to delete specified line content, you need to add an addressing expression before the subcommand.
sed 'd' message
# Delete the content of lines 1-3 in the message file
sed '1,2d' message

(5) y

The subcommand y indicates character translation, which can replace multiple characters, but can only replace characters and not strings, and does not support regular expressions.

# Replace all a characters in the message with A symbols, and all b characters with B symbols
sed 'y/aB/Ab/' message

(6) =

The subcommand = can print the line number.

# Display the line number above the specified line
sed '1,2=' message

(7) r

The subcommand r, similar to a, also appends content to the specified line, but r reads the content of the specified file and appends it below the specified line.

# Read the content of r.txt and insert it below line 1 of the message file
sed '1r r.txt' message

(8) s

The subcommand s is the replacement subcommand, which is the most commonly used subcommand in sed. It supports regular expressions, making it extremely powerful. Below is a detailed explanation of how to use the s subcommand.Basic syntax:[address]s/pattern/replacement/flags

flags

flags	Description
n	Can be 1-512, indicating the nth occurrence to be replaced
g	Global change
p	Print the content of the pattern space
w file	Write to a file named file

# Replace all b with x
sed 's/b/x/g' message

4) Example

(1) Insert “a new line” before the first line of test.txt

sed -i '1i\a new line' text.txt

(2) Add a new line at the third line

sed -i '3a \a new line' text

3. awk

awk is a text processing tool used for pattern scanning and processing of text or data. It can process segments of data based on the located data lines and perform data slicing or execute other operations. The awk command is very suitable for tasks such as data extraction and report generation, making it a powerful tool for text and data processing in Linux systems.

1) Basic Concepts

awk is a structured data processing program;awk is a language in the field of data processing;awk has a C-like syntax structure, for example, print is a statement, not a function;(1) Syntaxawk [options] 'matching rules and processing rules' [path to processing text]

The matching rules mainly include: regular expressions, strings
The processing rules mainly include: setting variables, setting arrays, defining functions (used less frequently), array loops, arithmetic operations, string concatenation

(2) The command format is:

awk [options] 'BEGIN{definition1;definition2;}/pattern/{operation1;operation2}END{operation1;operation2}' file1 file2

Options: such as-F:specify the delimiter as:BEGIN{} : BEGIN runs before awk processes the text/pattern/ : The matching rule used, the pattern can be a regular expression or string{} : Loop (processes one line of data at a time), for example,{print $1}prints the first columnEND{} : Executes related operations in END after all processing is completed

2) Operators: ~, ==, !=, >, etc.

For example:

# If the current line matches starting with a, then execute the following print
awk '/^a/ {print $1}'

The above regular expression matches the entire line content. If you only want to match the content of a specific field, you can use:

awk '$n~/^1/ {print $1}'

The ~ symbol specifies the range of the regular matching operation [here is field n], and ~ is called the matching operator.

3) BEGIN and END:

BEGIN can print some strings or define variables (variable_name=”value”, if the value is not a number, it must be enclosed in double quotes; when referencing the variable, do not add $ before the variable).END can print some strings or perform final calculations. If no variables are defined in BEGIN, the variable can still be referenced in the end, but its value will be empty; in awk, variables do not need to be pre-defined;

4) Options

Parameter	Explanation
-F	Specify the field separator
-v	Define or modify an internal awk variable
-f	Read awk commands from a script file

5) Common Variables:

Built-in Variables	Explanation
FS	Default input field separator (space and tab)
OFS	Default output field separator (space)
NF	Number of segments (columns) in each line (print $NF represents printing the last column)
RS	Default input record separator (\n)
ORS	Default output record separator (\n)
FILENAME	Name of the currently operated file
NR	Current line number when reading each line (for merged files, different file line numbers are displayed continuously)
FNR	Current line number when reading each line (for merged files, different file line numbers are displayed non-continuously)

$0: represents the entire current line, $1 represents the first field, $2 represents the second field, $n represents the nth field;

6) Conditions:

7) Examples

(1) Default separator is space and tab, print the second column

ps | awk '{print $2}'

(2) Separator is colon, print the first and second columns, separated by space

awk -F ":" '{print $1,$2}' /etc/passwd

(3) Separator is colon, print the first and second columns, separated by specified symbol

awk -F ":" '{print $1"--"$2}' /etc/passwd

(4) Separator is colon and comma, print the first and second columns, separated by specified symbol

awk -F "[:,]" '{print $1"--"$2}' /etc/passwd

(5) Default separator, print the first column of lines starting with d

docker ps | awk '/^d/ {print $1}'

(6) Default separator, print the first column of lines starting with d in the nth column

docker ps | awk '$n~/^d/ {print $1}'

(7) Specify the separator variable FS as colon and comma before printing

awk 'BEGIN{FS="[:,]"}{print $1}' /etc/passwd

(8) Specify the separator variable FS as colon and comma, define variable line

awk 'BEGIN{FS="[:,]";line=1}{print $line}' /etc/passwd

(9) Define variable sum to calculate the sum, output the calculation result each time, and finally output

awk 'BEGIN{FS="[,:]";sum=0}{sum+=$3;print sum}END{print "sum=",sum}' /etc/passwd

(10) Print the current file name

awk '{print FILENAME}' /etc/passwd

Link: https://www.cnblogs.com/hoaprox/p/18269422

(Copyright belongs to the original author, please delete if infringed)

WeChat group

WeChat group