Basic Operations

Shutdown and Restart Linux

# Shutdown
shutdown -h now

# Restart
shutdown -r now

View System and CPU Information

# View system kernel information
uname -a

# View system kernel version
cat /proc/version

# View current user environment variables
env

cat /proc/cpuinfo

# View the number of logical CPUs and CPU model
cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c

# View the number of physical CPUs and the number of cores per CPU
cat /proc/cpuinfo | grep physical | uniq -c

# Check if the current CPU is running in 32-bit or 64-bit mode; if it runs in 32-bit, it does not mean the CPU does not support 64-bit
getconf LONG_BIT

# If the result is greater than 0, it indicates support for 64-bit computing. 'lm' means long mode, supporting 'lm' indicates 64-bit
cat /proc/cpuinfo | grep flags | grep ' lm ' | wc -l

Create Symbolic Links

ln -s /usr/local/jdk1.8/ jdk

RPM Related

# Check if the software is installed via RPM
rpm -qa | grep software_name

SSH Key

# Create SSH key
ssh-keygen -t rsa -C [email protected]

# Copy the content of id_rsa.pub to the home/username/.ssh/authorized_keys of the server to be controlled, create it if it does not exist (.ssh permissions should be 700, authorized_keys permissions should be 600)

Command Renaming

# Add renaming configuration in each user's .bash_profile
alias ll='ls -alF'

Synchronize Server Time

sudo ntpdate -u ntp.api.bz

Run Commands in Background

# Run in the background, and output to nohup.out
nohup xxx &

# Run in the background, no output logs
nohup xxx > /dev/null &

# Run in the background, and redirect error messages to log
nohup xxx >out.log 2>&1 &

Force Active User Logout

# Command to force an active user to log out. TTY refers to terminal name
pkill -kill -t [TTY]

View Command Path

which <command>

View Maximum Open File Descriptors for Processes

ulimit -n

Configure DNS

vim /etc/resolv.conf

nslookup, View Domain Routing Table

nslookup google.com

last, Recent Login Information List

# Last 5 logged in accounts
last -n 5

Set Static IP

ifconfig em1  192.168.5.177 netmask 255.255.255.0

View Environment Variables Loaded in Process Memory

# You can also go to the /proc directory to see what is loaded in the process memory
ps eww -p XXXXX(process_id)

View Process Tree to Find Server Processes

ps auwxf

View Process Startup Path

cd /proc/xxx(process_id)
ls -all
# cwd corresponds to the startup path

Add User and Configure Sudo Permissions

# Add new user
useradd username
passwd username

# Add sudo permissions
vim /etc/sudoers
# Modify the file to include
# root    ALL=(ALL)       ALL
# username ALL=(ALL)       ALL

Force Close All Processes Containing ‘xxx’

ps aux|grep xxx | grep -v grep | awk '{print $2}' | xargs kill -9

Disk, File, Directory Related Operations

vim Operations

# In normal mode, g means global, x means search content, y means content to be replaced
:%s/x/y/g

# In normal mode
0  # Move cursor to the beginning of the line (number 0)
$  # Move cursor to the end of the line
shift + g # Jump to the end of the file
gg # Jump to the beginning of the file

# Show line numbers
:set nu

# Remove line numbers
:set nonu

# Search
/xxx(search content)  # Search from the beginning, press n to find the next
?xxx(search content)  # Search from the end

Open Read-Only Files and Save After Modification Without Switching Users

# In normal mode
:w !sudo tee %

View Basic Information of Disk, File Directory

# View disk mounting status
mount

# View disk partition information
df

# View the size of directories and subdirectories
du -H -h

# View how much space each file and folder occupies in the current directory, will not recurse
du -sh *

wc Command

# View how many lines are in the file
wc -l filename

# View how many words are in the file
wc -w filename

# View the length of the longest line
wc -L filename

# Count bytes
wc -c

Common Compression and Decompression Commands

Compression Commands

tar czvf xxx.tar directory_to_compress

zip -r xxx.zip directory_to_compress

Decompression Commands

tar zxvf xxx.tar

# Decompress to a specified folder
tar zxvf xxx.tar -C /xxx/yyy/
unzip xxx.zip

Change File Ownership and User Group

chown eagleye.eagleye xxx.log

cp, scp, mkdir

# Copy
cp xxx.log

# Copy and force overwrite the same file
cp -f xxx.log

# Copy directory
cp -r xxx(source_directory) yyy(target_directory)

# Remote copy
scp -P ssh_port [email protected]:/home/username/xxx /home/xxx

# Create directories in a cascading manner
mkdir -p /xxx/yyy/zzz

# Batch create folders, will create java and resources folders in both test and main
mkdir -p src/{test,main}/{java,resources}

Compare Two Files

diff -u 1.txt 2.txt

Log Output Byte Count, Can Be Used for Performance Testing

# For performance testing, you can output '.' to the log each time, so the byte count in the log is the actual number of performance test runs, and you can also see the real-time rate.
tail -f xxx.log | pv -bt

View and Remove Special Characters

# View special characters
cat -v xxx.sh

# Remove special characters
sed -i 's/^M//g' env.sh  # Remove special characters from the file, e.g., ^M: this needs to be entered as: ctrl+v+enter

Handle Special Character Issues Caused by System Reasons in Files

# Can convert to the file format under this system
cat file.sh > file.sh_bak

# First, copy the contents of file.sh and run it, then paste the content, finally ctrl + d to save and exit
cat > file1.sh

# In vim, set file encoding and format as follows
:set fileencodings=utf-8, then w (save) to convert to utf8 format,
:set fileformat=unix

# Use dos2unix for file formatting on mac
find . -name "*.sh" | xargs dos2unix

tee, Redirect Output While Displaying on Screen

awk '{print $0}' xxx.log | tee test.log

Search Related

grep, WeChat Search Public Account: Architect Guide, Reply: Architect to Get Materials.

# Reverse match, find content that does not contain xxx
grep -v xxx

# Exclude all empty lines
grep -v '^/pre>

# If the result is 2, it indicates the second line is empty
grep -n "^$" 111.txt    

# Query lines starting with abc
grep -n "^abc" 111.txt 

# List which line the word appears in the article
grep 'xxx' -n xxx.log

# Count how many times the substring appears
grep 'xxx' -c xxx.log

# Compare without considering case differences
grep 'xxx' -i xxx.log

awk

# Use ':' as the delimiter, if the fifth field contains user, output that line
awk -F ':' '{if ($5 ~ /user/) print $0}' /etc/passwd 

# Count the occurrences of a character (string) in a single file (Chinese is invalid)
awk -v RS='character' 'END {print --NR}' xxx.txt

find Search Command

# Find files with .mysql suffix in the directory
find /home/eagleye -name '*.mysql' -print

# Will start searching from the /usr directory down, find files accessed within the last 3 days.
find /usr -atime 3 –print

# Will start searching from the /usr directory down, find files modified within the last 5 days.
find /usr -ctime 5 –print

# Will start searching from the /doc directory down, find files owned by jacky, whose filename starts with j.  
find /doc -user jacky -name 'j*' –print

# Will start searching from the /doc directory down, find files whose name starts with ja or ma.
find /doc \( -name 'ja*' -o- -name 'ma*' \) –print

# Will start searching from the /doc directory down, find and delete all files ending with bak. -exec option means execute, rm is the delete command, { } represents filename, "\;" is the command ending.
find /doc -name '*bak' -exec rm {} \;

Network Related

View Which Process is Using the Port

lsof -i:port

Get Local IP Address

/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"

iptables

# View iptables status
service iptables status

# Block an IP
iptables -I INPUT -s ***.***.***.*** -j DROP

# Unblock an IP, use the following command:
iptables -D INPUT -s ***.***.***.*** -j DROP

# Note: The parameter -I means Insert (add), -D means Delete (remove). The following is the rule, INPUT means inbound, ***.***.***.*** means the IP to be blocked, DROP means to discard the connection.

# Open access to port 9090
/sbin/iptables -I INPUT -p tcp --dport 9090 -j ACCEPT 

# Firewall start, stop, restart
/etc/init.d/iptables status
/etc/init.d/iptables start
/etc/init.d/iptables stop
/etc/init.d/iptables restart

nc Command, TCP Debugging Tool

# Send TCP requests to a certain endpoint, sending the content of data to the other end
nc 192.168.0.11 8000 < data.txt

# nc can act as a server, listen on a certain port, storing the content of a request to received_data
nc -l 8000 > received_data

# The above only listens once, if multiple times you can add the -k parameter
nc -lk 8000

tcpdump

# Dump TCP packets on local port 12301
tcpdump -i em1 tcp port 12301 -s 1500 -w abc.pcap

Trace Network Routing Path

# traceroute defaults to using UDP, if -I is used it changes to ICMP
traceroute -I www.163.com

# Trace from the 3rd hop of ttl
traceroute -M 3 www.163.com  

# Add port tracing
traceroute -p 8080 192.168.10.11

ss

# Display all local open ports
ss -l 

# Display each process's specific open socket
ss -pl 

# Display all TCP sockets
ss -t -a 

# Display all UDP sockets
ss -u -a 

# Display all established SMTP connections
ss -o state established '( dport = :smtp or sport = :smtp )'  

# Display all established HTTP connections
ss -o state established '( dport = :http or sport = :http )'  

# Find all processes connected to the X server
ss -x src /tmp/.X11-unix/*  

# List current socket statistics
ss -s 

# Explanation: netstat traverses each PID directory under /proc, while ss reads directly from /proc/net, so ss consumes less resources and time than netstat

netstat

# Output the number of connections for each IP, as well as the total number of connections in each state
netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'

# Count all connection states, 
# CLOSED: No connections are active or ongoing
# LISTEN: The server is waiting for incoming calls
# SYN_RECV: A connection request has been received, waiting for confirmation
# SYN_SENT: The application has started to open a connection
# ESTABLISHED: Normal data transfer state
# FIN_WAIT1: The application says it has completed
# FIN_WAIT2: The other side has agreed to release
# ITMED_WAIT: Waiting for all packets to die
# CLOSING: Both sides are trying to close
# TIME_WAIT: The actively closed connection end has not yet waited for feedback from the other end
# LAST_ACK: Waiting for all packets to die
netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

# Find many time_wait connections
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

Monitor Linux Performance Commands

top

Press the uppercase F or O key, then press a-z to sort the processes according to the corresponding column, then press enter. The uppercase R key can reverse the current sorting

dmesg, View System Logs

dmesg

iostat, Monitor Disk IO Situation

iostat -xz 1

# r/s, w/s, rkB/s, wkB/s: respectively represent the number of reads and writes per second and the amount of read and write data (in kilobytes). Excessively high read/write volume may cause performance issues.
# await: Average wait time for IO operations, in milliseconds. This is the time the application consumes when interacting with the disk, including IO wait and actual operation time. If this value is too high, it may indicate that the hardware device is encountering a bottleneck or has failed.
# avgqu-sz: Average number of requests sent to the device. If this value is greater than 1, it may indicate that the hardware device is saturated (some frontend hardware devices support parallel writing).
# %util: Device utilization. This value indicates how busy the device is, and the experience value is that if it exceeds 60, it may affect IO performance (refer to average wait time for IO operations). If it reaches 100%, it indicates that the hardware device is saturated.
# If the displayed data is for logical devices, then device utilization does not represent that the actual backend hardware device is saturated. It is worth noting that even if IO performance is not ideal, it does not necessarily mean that application performance will be poor, as strategies such as pre-reading and write caching can improve application performance.

free, Memory Usage

free -m

Example:

     total       used       free     shared    buffers     cached
Mem:          1002        769        232          0         62        421
-/+ buffers/cache:          286        715
Swap:          1153          0       1153

The first part Mem line:
total Total memory: 1002M
used Memory already used: 769M
free Free memory: 232M
shared Currently unused, always 0
buffers Buffer cache memory: 62M
cached Page cache memory:421M

Relationship: total(1002M) = used(769M) + free(232M)

The second part (-/+ buffers/cache):
(-buffers/cache) used memory:286M (referring to used in the first part of Mem line minus buffers and cached)
(+buffers/cache) free memory:715M (referring to free in the first part of Mem line plus buffers and cached)

It can be seen that -buffers/cache reflects the memory actually consumed by programs, while +buffers/cache reflects the total memory that can be allocated.

The third part refers to the swap partition

sar, View Network Throughput Status

# The sar command can be used here to view the throughput of network devices. When troubleshooting performance issues, you can determine whether the network device is saturated by the throughput of the network device.
sar -n DEV 1

#
# The sar command can also be used here to view TCP connection status, including:
# active/s: Number of TCP connections initiated locally per second, i.e., TCP connections created through the connect call;
# passive/s: Number of TCP connections initiated remotely per second, i.e., TCP connections created through the accept call;
# retrans/s: Number of TCP retransmissions per second;
# The number of TCP connections can be used to determine whether performance issues are due to too many connections being established, and further determine whether it is active or passive connections. TCP retransmissions may be due to poor network conditions or excessive server load resulting in packet loss.
sar -n TCP,ETCP 1

vmstat, Monitor CPU Usage, Memory Usage, Virtual Memory Interaction, IO Read/Write Over a Given Time

# 2 means collect status information every 2 seconds, 1 means collect once (neglecting continuous collection)
vmstat 2 1

Example:
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 3499840 315836 3819660 0 0 0 1 2 0 0 0 100 0
0 0 0 3499584 315836 3819660 0 0 0 0 88 158 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 2 86 162 0 0 100 0
0 0 0 3499708 315836 3819660 0 0 0 10 81 151 0 0 100 0
1 0 0 3499732 315836 3819660 0 0 0 2 83 154 0 0 100 0

• r indicates the running queue (how many processes are actually allocated to the CPU), I tested the server which is currently relatively idle, with not much running, when this value exceeds the number of CPUs, a CPU bottleneck will appear. This is also related to the load of top, generally, if the load exceeds 3, it is considered high, if it exceeds 5, it is high, if it exceeds 10, it is abnormal, and the server’s state is very dangerous. The load of top is similar to the running queue per second. If the running queue is too large, it indicates that your CPU is very busy, which generally leads to high CPU usage.
• b indicates blocked processes, this is self-explanatory, processes are blocked, everyone understands.
• swpd indicates the size of virtual memory used, if greater than 0, it indicates that your machine’s physical memory is insufficient, if not due to memory leaks, then you should upgrade the memory or migrate memory-intensive tasks to other machines.
• free indicates the size of free physical memory, my machine has a total of 8G, remaining 3415M.
• buff Linux/Unix systems use it to store what is in the directory, permissions, etc., my machine occupies about 300M
• cache cache is directly used to remember the files we opened, providing buffering for files, my machine occupies about 300M (this is the cleverness of Linux/Unix, taking part of the free physical memory to cache files and directories to improve program execution performance, when programs use memory, buffer/cached will be quickly used.)
• si indicates the size of virtual memory read from disk per second, if this value is greater than 0, it indicates insufficient physical memory or memory leaks, you need to find and solve memory-consuming processes. My machine has sufficient memory, everything is normal.
• so indicates the size of virtual memory written to disk per second, if this value is greater than 0, the same as above.
• bi indicates the number of blocks received per second by block devices, where block devices refer to all disks and other block devices on the system, the default block size is 1024 bytes, my machine has no IO operations, so it is always 0, but I have seen it reach 140,000/s when processing large amounts of data (2-3T), the disk write speed is about 140M per second.
• bo indicates the number of blocks sent per second by block devices, for example, when we read files, bo should be greater than 0. Bi and bo should generally be close to 0, otherwise, IO is too frequent and needs to be adjusted.
• in indicates the number of interrupts per second for the CPU, including time interrupts
• cs indicates the number of context switches per second, for example, when we call system functions, context switching occurs, thread switching also requires process context switching, this value should be as small as possible, if too large, consider lowering the number of threads or processes. For example, in web servers like apache and nginx, we generally conduct performance tests with thousands or even tens of thousands of concurrent connections, the choice of web server processes can be adjusted downwards until cs reaches a relatively small value, this process and thread count is a more appropriate value. System calls also cause context switching every time a system function is called, which is resource-intensive, and frequent calls should be avoided. Too many context switches indicate that your CPU is mostly wasted on context switching, resulting in less time for the CPU to do its actual work, which is undesirable.
• us indicates user CPU time, I once saw on a server that frequently performed encryption and decryption, us approached 100, and the running queue reached 80 (the machine was under stress testing, performance was poor).
• sy indicates system CPU time, if too high, it indicates long system call times, such as frequent IO operations.
• id indicates idle CPU time, generally, id + us + sy = 100, generally, I consider id to be idle CPU usage, us to be user CPU usage, and sy to be system CPU usage.
• wt indicates CPU time waiting for IO.

Source: siye1982.github.io/2016/02/25/linux-list