Introduction

Although most of my work is related to Java development, I interact with the Linux system daily, especially after using a Mac, where I work in a command-line environment with a black background. My memory isn’t great, and I often forget many useful Linux commands, so I am gradually summarizing them for future reference.

Basic Operations

Linux Shutdown and Restart

# Shutdown
shutdown -h now

# Restart
shutdown -r now

View System and CPU Information

# View system kernel information
uname -a

# View system kernel version
cat /proc/version

# View current user environment variables
env

cat /proc/cpuinfo

# View the number of logical CPUs, including CPU model
cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c

# View the number of CPUs and the number of cores for each
cat /proc/cpuinfo | grep physical | uniq -c

# Check if the current CPU is running in 32-bit or 64-bit mode; running in 32-bit does not mean the CPU does not support 64-bit
getconf LONG_BIT

# A result greater than 0 indicates support for 64-bit computation. 'lm' indicates long mode, supporting 'lm' means 64-bit
cat /proc/cpuinfo | grep flags | grep ' lm ' | wc -l

Create a Symbolic Link

ln -s /usr/local/jdk1.8/ jdk

RPM Related

# Check if the software was installed via RPM
rpm -qa | grep software_name

SSH Key

# Create SSH key
ssh-keygen -t rsa -C [email protected]

# Copy the content of id_rsa.pub to the home/username/.ssh/authorized_keys of the server to be controlled; create it if it doesn't exist (.ssh permissions should be 700, authorized_keys permissions should be 600)

Command Renaming

# Add renaming configuration in each user's .bash_profile
alias ll='ls -alF'

Synchronize Server Time

sudo ntpdate -u ntp.api.bz

Run Commands in the Background

# Run in the background with nohup.out output
nohup xxx &

# Run in the background without any log output
nohup xxx > /dev/null &

# Run in the background and redirect error messages to a log
nohup xxx >out.log 2>&1 &

Force Active User Logout

# Command to force an active user logout. TTY represents the terminal name
pkill -kill -t [TTY]

View Command Path

which <command>

View Maximum Open File Descriptor Count for Processes

ulimit -n

Configure DNS

vim /etc/resolv.conf

nslookup, View Domain Routing Table

nslookup google.com

last, Recent Login Information List

# Last 5 logged-in accounts
last -n 5

Set Static IP

ifconfig em1  192.168.5.177 netmask 255.255.255.0

View Loaded Environment Variables in a Process

# You can also go to the /proc directory to see what is loaded in the process memory
ps eww -p XXXXX(process_id)

View Process Tree to Find Server Processes

ps auwxf

View Process Startup Path

cd /proc/xxx(process_id)
ls -all
# cwd corresponds to the startup path

Add User and Configure Sudo Permissions

# Add a new user
useradd username
passwd username

# Add sudo permissions
vim /etc/sudoers
# Modify the file to include
# root    ALL=(ALL)       ALL
# username ALL=(ALL)       ALL

Force Close All Processes Containing ‘xxx’

ps aux|grep xxx | grep -v grep | awk '{print $2}' | xargs kill -9

Disk, File, and Directory Operations

vim Operations

# In normal mode, g means global, x means the content to find, y means the content to replace
:%s/x/y/g
 
# In normal mode
0  # Move cursor to the beginning of the line (number 0)
$  # Move cursor to the end of the line
shift + g # Jump to the end of the file
gg # Jump to the beginning of the file
 
# Show line numbers
:set nu
 
# Remove line numbers
:set nonu
 
# Search
/xxx(search content)  # Search from the beginning, press n to find the next
?xxx(search content)  # Search from the end

Open Read-Only Files and Save After Modification (without switching users)

# In normal mode
:w !sudo tee %

View Basic Information of Disk and File Directories

# View disk mount status
mount

# View disk partition information
df

# View directory and subdirectory sizes
du -H -h

# View the size of each file and folder in the current directory, without recursion
du -sh *

wc Command

# View how many lines are in a file
wc -l filename
 
# View how many words are in a file
wc -w filename

# View the length of the longest line in a file
wc -L filename

# Count bytes
wc -c

Common Compression and Decompression Commands

Compression Commands

tar czvf xxx.tar compress_directory
 
zip -r xxx.zip compress_directory

Decompression Commands

tar zxvf xxx.tar
 
# Decompress to a specified folder
tar zxvf xxx.tar -C /xxx/yyy/
 
unzip xxx.zip

Change File Ownership and User Group

chown eagleye.eagleye xxx.log

cp, scp, mkdir

# Copy
cp xxx.log

# Copy and force overwrite the same file
cp -f xxx.log

# Copy a folder
cp -r xxx(source_folder) yyy(target_folder)

# Remote copy
scp -P ssh_port [email protected]:/home/username/xxx /home/xxx

# Create directories in a cascading manner
mkdir -p /xxx/yyy/zzz

# Batch create folders, will create java and resources folders in both test and main
mkdir -p src/{test,main}/{java,resources}

Compare Two Files

diff -u 1.txt 2.txt

Log Output Byte Count, Can Be Used for Performance Testing

# For performance testing, you can output '.' to the log each time it runs, so the byte count in the log reflects the actual number of performance test runs, and you can also see the real-time rate.
tail -f xxx.log | pv -bt

View and Remove Special Characters

# View special characters
cat -v xxx.sh

# Remove special characters
sed -i 's/^M//g' env.sh  # Remove special characters from the file, e.g., ^M: needs to be input as: ctrl+v+enter

Handle Special Character Issues in Files Caused by System Reasons

# Convert to the file format of this system
cat file.sh > file.sh_bak

# First copy the content of file.sh and run it, then paste the content, and finally ctrl + d to save and exit
cat > file1.sh

# In vim, set file encoding and format as follows
:set fileencodings=utf-8, then w (save) to convert to utf8 format,
:set fileformat=unix

# Use dos2unix for file formatting on Mac
find . -name "*.sh" | xargs dos2unix

tee, Redirect Output to Screen While Outputting

awk '{print $0}' xxx.log | tee test.log

Search Related

grep

# Reverse match, find content that does not contain xxx
grep -v xxx

# Exclude all empty lines
grep -v '^$'

# If the result is 2, it indicates that the second line is empty
grep -n "^$" 111.txt    

# Query lines starting with abc
grep -n "^abc" 111.txt 

# List the line number where the word appears in the article
grep 'xxx' -n xxx.log

# Count how many times the substring appears
grep 'xxx' -c xxx.log

# Compare without considering case differences
grep 'xxx' -i xxx.log

awk

# Use ':' as the delimiter, if the fifth field contains user, output that line
awk -F ':' '{if ($5 ~ /user/) print $0}' /etc/passwd 

# Count how many times a character (string) appears in a single file (Chinese is invalid)
awk -v RS='character' 'END {print --NR}' xxx.txt

find Search Command

# Find files with the .mysql suffix in the directory
find /home/eagleye -name '*.mysql' -print

# Search from the /usr directory down for files accessed in the last 3 days.
find /usr -atime 3 –print
  
# Search from the /usr directory down for files modified in the last 5 days.
find /usr -ctime 5 –print

# Search from the /doc directory down for files owned by jacky, with filenames starting with j.
find /doc -user jacky -name 'j*' –print

# Search from the /doc directory down for files with names starting with ja or ma.
find /doc \( -name 'ja*' -o- -name 'ma*' \) –print

# Search from the /doc directory down for files ending with bak and delete them. The -exec option means execute, rm is the delete command, { } represents the filename, and "\;" is the command ending.
find /doc -name '*bak' -exec rm {} \;

Network Related

View Which Process is Using the Port

lsof -i:port

Get Local IP Address

/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"

iptables

# View iptables status
service iptables status

# To block an IP
iptables -I INPUT -s ***.***.***.*** -j DROP

# To unblock an IP, use the following command:
iptables -D INPUT -s ***.***.***.*** -j DROP

Note: The parameter -I means Insert (add), -D means Delete (remove). The following is the rule, INPUT means inbound, ***.***.***.*** is the IP to be blocked, DROP means to discard the connection.

# Open access to port 9090
/sbin/iptables -I INPUT -p tcp --dport 9090 -j ACCEPT 

# Firewall start, stop, restart
/etc/init.d/iptables status
/etc/init.d/iptables start
/etc/init.d/iptables stop
/etc/init.d/iptables restart

nc Command, TCP Debugging Tool

# Send TCP requests to a specific endpoint, sending the content of data to the other end
nc 192.168.0.11 8000 < data.txt

# nc can act as a server, listening on a certain port, storing the content of a request in received_data
nc -l 8000 > received_data

# The above only listens once; to listen multiple times, add the -k parameter
nc -lk 8000

tcpdump

# Dump TCP packets from local port 12301
tcpdump -i em1 tcp port 12301 -s 1500 -w abc.pcap

Trace Network Routing Path

# traceroute defaults to using UDP; if -I is used, it changes to ICMP
traceroute -I www.163.com

# Trace from the 3rd hop of ttl
traceroute -M 3 www.163.com  

# Add port tracing
traceroute -p 8080 192.168.10.11

ss

# Show all local open ports
ss -l 
 
# Show each process's specific open socket
ss -pl 
 
# Show all TCP sockets
ss -t -a 
 
# Show all UDP sockets
ss -u -a 
 
# Show all established SMTP connections
ss -o state established '( dport = :smtp or sport = :smtp )'  
 
# Show all established HTTP connections 
ss -o state established '( dport = :http or sport = :http )'  
 
# Find all processes connected to the X server
ss -x src /tmp/.X11-unix/*  
 
# List current socket statistics
ss -s 
 
# Explanation: netstat traverses each PID directory under /proc, while ss directly reads statistics from /proc/net. Therefore, ss consumes fewer resources and time than netstat.

netstat

# Output the connection count for each IP, as well as the total number of connections in each state
netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'

# Count all connection states, 
# CLOSED: No connections are active or in progress
# LISTEN: The server is waiting for incoming calls
# SYN_RECV: A connection request has been received, waiting for confirmation
# SYN_SENT: The application has started to open a connection
# ESTABLISHED: Normal data transmission state
# FIN_WAIT1: The application says it has completed
# FIN_WAIT2: The other side has agreed to release
# ITMED_WAIT: Waiting for all packets to die
# CLOSING: Both sides are trying to close
# TIME_WAIT: The actively closing connection side has not yet received feedback from the other side
# LAST_ACK: Waiting for all packets to die
netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

# Find connections with many time_wait states
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

Monitor Linux Performance Commands

top

Press the uppercase F or O key, then press a-z to sort the processes by the corresponding column, then press enter. The uppercase R key can reverse the current sorting.

Column Name	Meaning
PID	Process ID
PPID	Parent Process ID
RUSER	Real User Name
UID	User ID of the process owner
USER	Username of the process owner
GROUP	Group name of the process owner
TTY	Terminal name that started the process. Processes not started from a terminal will show as ?
PR	Priority
NI	Nice value. Negative values indicate high priority, positive values indicate low priority
P	Last used CPU, only meaningful in multi-CPU environments
%CPU	Percentage of CPU time used since the last update
TIME	Total CPU time used by the process, in seconds
TIME+	Total CPU time used by the process, in 1/100 seconds
%MEM	Percentage of physical memory used by the process
VIRT	Total amount of virtual memory used by the process, in KB. VIRT=SWAP+RES
SWAP	Size of virtual memory swapped out from the process, in KB.
RES	Size of physical memory used by the process that has not been swapped out, in KB. RES=CODE+DATA
CODE	Size of physical memory occupied by executable code, in KB
DATA	Size of physical memory occupied by parts other than executable code (data segment + stack), in KB
SHR	Size of shared memory, in KB
nFLT	Number of page faults
nDRT	Number of pages modified since the last write.
S	Process state. D=uninterruptible sleep state, R=running, S=sleeping, T=traced/stopped, Z=zombie process
COMMAND	Command name/command line
WCHAN	If the process is sleeping, it shows the name of the system function in which it is sleeping
Flags	Task flags, refer to sched.h

dmesg, View System Logs

dmesg

iostat, Monitor Disk IO Status

iostat -xz 1
 
# r/s, w/s, rkB/s, wkB/s: respectively indicate the number of read/write operations per second and the amount of data read/written per second (in kilobytes). Excessive read/write volume may cause performance issues.
# await: Average wait time for IO operations, in milliseconds. This is the time the application consumes when interacting with the disk, including IO wait and actual operation time. If this value is too high, it may indicate that the hardware device is bottlenecked or malfunctioning.
# avgqu-sz: Average number of requests sent to the device. If this value is greater than 1, it may indicate that the hardware device is saturated (some front-end hardware devices support parallel writing).
# %util: Device utilization. This value indicates the degree of busyness of the device; a common threshold is if it exceeds 60, it may affect IO performance (can refer to average wait time for IO operations). If it reaches 100%, it indicates that the hardware device is saturated.
# If the displayed data is for logical devices, then device utilization does not represent that the actual backend hardware device is saturated. It is worth noting that even if IO performance is not ideal, it does not necessarily mean that application performance will be poor; strategies such as pre-reading and write caching can enhance application performance.

free, Memory Usage Status

free -m
 
eg:
 
     total       used       free     shared    buffers     cached
Mem:          1002        769        232          0         62        421
-/+ buffers/cache:          286        715
Swap:          1153          0       1153
 
The first part Mem line:
total total memory: 1002M
used memory used: 769M
free free memory: 232M
shared currently deprecated, always 0
buffers buffer cache memory: 62M
cached page cache memory: 421M
 
Relationship: total(1002M) = used(769M) + free(232M)
 
The second part (-/+ buffers/cache):
(-buffers/cache) used memory: 286M (referring to used in the first part of Mem line - buffers - cached)
(+buffers/cache) free memory: 715M (referring to free in the first part of Mem line + buffers + cached)
 
It can be seen that -buffers/cache reflects the memory actually consumed by the program, while +buffers/cache reflects the total memory that can be allocated.
 
The third part refers to the swap partition

sar, View Network Throughput Status

# The sar command can be used to view the throughput of network devices. When troubleshooting performance issues, you can determine whether the network device is saturated by its throughput.
sar -n DEV 1
 
#
# The sar command can also be used to view TCP connection status, including:
# active/s: Number of TCP connections initiated locally per second, i.e., TCP connections created through the connect call;
# passive/s: Number of TCP connections initiated remotely per second, i.e., TCP connections created through the accept call;
# retrans/s: Number of TCP retransmissions per second;
# The number of TCP connections can be used to determine whether performance issues are due to too many connections being established, and further determine whether they are actively initiated connections or passively accepted connections. TCP retransmissions may be due to poor network conditions or excessive server load causing packet loss.
sar -n TCP,ETCP 1

vmstat, Monitor CPU Usage, Memory Usage, Virtual Memory Interaction, IO Read/Write at Given Time

# 2 indicates collecting status information every 2 seconds, 1 indicates collecting only once (ignoring continuous collection)
vmstat 2 1
 
eg:
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 3499840 315836 3819660 0 0 0 1 2 0 0 0 100 0
0 0 0 3499584 315836 3819660 0 0 0 0 88 158 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 2 86 162 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 10 81 151 0 0 100 0
1 0 0 3499732 315836 3819660 0 0 0 2 83 154 0 0 100 0

• r indicates the run queue (i.e., how many processes are actually allocated to the CPU); my tested server currently has a relatively idle CPU, with no programs running. When this value exceeds the number of CPUs, a CPU bottleneck will occur. This is also related to the load in top; generally, a load exceeding 3 is considered high, exceeding 5 is high, and exceeding 10 is abnormal, indicating that the server’s status is very dangerous. The load in top is similar to the running queue per second. If the running queue is too large, it indicates that your CPU is very busy, which generally leads to high CPU usage.
• b indicates blocked processes; this is self-explanatory, processes are blocked, everyone understands.
• swpd indicates the size of virtual memory used; if greater than 0, it indicates that your machine’s physical memory is insufficient. If this is not due to a memory leak in the program, then you should upgrade the memory or migrate memory-intensive tasks to other machines.
• free indicates the size of free physical memory; my machine has a total of 8G, with 3415M remaining.
• buff indicates the cache memory used by the Linux/Unix system to store the contents of directories, permissions, etc.; my machine occupies about 300M.
• cache is directly used to remember the files we open, providing a buffer for files; my machine occupies about 300M (this is the cleverness of Linux/Unix, taking a portion of free physical memory to cache files and directories to improve program execution performance; when a program uses memory, buffer/cached will be quickly utilized).
• si indicates the size of virtual memory read from the disk per second; if this value is greater than 0, it indicates that physical memory is insufficient or there is a memory leak; you need to find and resolve the memory-consuming process. My machine has sufficient memory, everything is normal.
• so indicates the size of virtual memory written to the disk per second; if this value is greater than 0, the same applies.
• bi indicates the number of blocks received per second from block devices; block devices refer to all disks and other block devices on the system, with a default block size of 1024 bytes. My machine has no IO operations, so it is always 0, but I have seen it reach 140000/s on machines handling large data copies (2-3T), with disk write speeds of about 140M per second.
• bo indicates the number of blocks sent per second from block devices; for example, when we read files, bo should be greater than 0. Generally, bi and bo should be close to 0; otherwise, IO is too frequent and needs adjustment.
• in indicates the number of interrupts per second for the CPU, including time interrupts.
• cs indicates the number of context switches per second; for example, when we call system functions, context switches occur, and thread switches also require process context switches. This value should be as small as possible; if it is too large, consider reducing the number of threads or processes. For example, in web servers like Apache and Nginx, we generally conduct performance tests with thousands or even tens of thousands of concurrent tests, and the choice of web server processes can be adjusted downwards until cs reaches a relatively small value; this process and thread count is the most suitable value. System calls also cause context switches, which are resource-intensive and should be avoided as much as possible. Excessive context switch counts indicate that your CPU is wasting most of its time on context switching, leading to less time for the CPU to do real work, which is undesirable.
• us indicates user CPU time; I once saw a server doing frequent encryption and decryption with us close to 100, and the running queue reached 80 (the machine was under stress testing, and performance was poor).
• sy indicates system CPU time; if too high, it indicates long system call times, such as frequent IO operations.
• id indicates idle CPU time; generally, id + us + sy = 100. I generally consider id as idle CPU usage, us as user CPU usage, and sy as system CPU usage.
• wt indicates CPU time waiting for IO.

Author: A Boy Who Picks Snails Source: https://t.1yb.co/K4OX