From WeChat Official Account: Efficient Operations
Introduction
Having worked in operations for several years, I still remember when I first started, I could only use some simple commands, and when writing scripts, I kept them as simple as possible, resulting in long and messy scripts.If I had known about some advanced commands, such as the xargs command, pipe commands, and automated response commands, I could have written concise and efficient scripts from the beginning.For whatever reason, I want to explain the usage of some advanced Linux commands, benefiting both myself and others, so that I can refer back to them in the future if I forget.
1. Practical xargs Command
In regular use, I find the xargs command to be quite important and convenient. We can use this command to pass the output of one command as parameters to another command.For example, if we want to find files ending with .conf in a certain path and categorize them, the usual approach would be to first find the files ending with .conf, output them to a file, then cat that file and use the file command to categorize the output files.This ordinary method can indeed be a bit cumbersome, and that’s where the xargs command comes in handy.Example 1: Find files ending with .conf in the / directory and categorize them
Command: # find / -name *.conf -type f -print | xargs file
The output is as follows:
After xargs, you can not only add file categorization commands but also many other commands. For instance, you can use the find command in conjunction with the tar command to find specific files in a designated path and directly package them with the tar command, as shown below:
# find / -name *.conf -type f -print | xargs tar cjf test.tar.gz
2. Running Commands or Scripts in the Background
Sometimes when we perform certain operations, we do not want our operations to terminate when the terminal session disconnects, especially for large database import/export operations. We cannot guarantee that our network will not have issues during our operations, so running scripts or commands in the background is a significant safeguard for us.For example, if we want to run a database export operation in the background and log the command output to a file, we can do it like this:
nohup mysqldump -uroot -pxxxxx --all-databases > ./alldatabases.sql & (xxxxx is the password)
If you do not want the password in plain text, you can do it like this:
nohup mysqldump -uroot -pxxxxx --all-databases > ./alldatabases.sql (do not add & symbol at the end)
After executing the above command, it will prompt you to enter the password. After entering the password, the command will still run in the foreground, but our goal is to run the command in the background. At this point, you can press Ctrl+Z and then type bg to achieve the effect of running the command in the background while also keeping the password hidden.The result of the command running in the background will leave a nohup.out file in the current directory where the command was executed, and you can check this file to see if there were any errors during execution.
3. Finding Processes with High Memory Usage in the Current System
In many operations, we find that memory usage is quite high. So how can we find and sort the processes consuming the most memory?
Command: # ps -aux | sort -rnk 4 | head -20
The 4th column of the output is the percentage of memory usage, and the last column corresponds to the process.
4. Finding Processes with High CPU Usage in the Current System
In many operations, we find that CPU usage is quite high. So how can we find and sort the processes consuming the most CPU?
Command: # ps -aux | sort -rnk 3 | head -20
The 3rd column of the output is the percentage of CPU usage, and the last column corresponds to the process.You may have noticed that the 3 and 4 after the sort command represent sorting by the 3rd and 4th columns, respectively.
5. Viewing Multiple Logs or Data Files Simultaneously
In daily work, we might use the tail command to view log files one by one in separate terminals, with one terminal for each log file. I, too, have done this, but sometimes it feels a bit cumbersome. There is actually a tool called multitail that allows you to view multiple log files in the same terminal.First, install multitail:
# wget ftp://ftp.is.co.za/mirror/ftp.rpmforge.net/redhat/el6/en/x86_64/dag/RPMS/multitail-5.2.9-1.el6.rf.x86_64.rpm
# yum -y localinstall multitail-5.2.9-1.el6.rf.x86_64.rpm
The multitail tool supports text highlighting, content filtering, and many more features you might need.
Here’s a useful example:
At this point, we want to view the secure log with specific filtering keywords while also monitoring real-time network ping:
The command is as follows:
# multitail -e "Accepted" /var/log/secure -l "ping baidu.com"
Isn’t that convenient? If we want to check the correlation between two logs, we can observe whether the log outputs trigger each other. If we separate them into two terminals, switching back and forth can be a bit of a waste of time, so using the multitail tool is a good method.
6. Continuous Ping and Logging Results
Often in operations, we hear someone say, “Is there a problem with the network?” leading to strange symptoms in the business. It’s often blamed on the server’s network issues. This is commonly referred to as “passing the buck”; when a business problem arises, and the relevant personnel cannot find the cause, they often attribute the issue to the server’s network.
At this point, if you ping a few packets and show the results, people might counter that there was just a problem for a short time, and now that the business is back to normal, the network must be fine. This can be quite frustrating.
If you bring out monitoring data from tools like Zabbix, it may not be appropriate, as you cannot set Zabbix to collect data every second. I have encountered this issue, and I used the following command for ping monitoring:
Then, when someone tries to pass the buck again, I can extract the ping database for the time period when the issue occurred, and we can discuss it openly. That time, I managed to turn the tables, and they no longer dared to easily pass the buck. That felt great.
Command:
ping api.jpush.cn | awk ‘{ print $0″ ” strftime(“%Y-%m-%d %H:%M:%S”,systime()) }’ >> /tmp/jiguang.log &
The output will be logged to /tmp/jiguang.log, with a new ping record added every second.
7. Checking TCP Connection Status
Specifically checking the TCP connection status on port 80 is helpful for analyzing whether connections are being released or for analyzing states during an attack.
Command:
netstat -nat | awk '{print $6}' | sort | uniq -c | sort -rn
8. Finding the Top 20 IPs with the Most Requests to Port 80
Sometimes when the request volume suddenly increases, we can check the source IPs of the requests. If they are concentrated on a few IPs, there may be an attack. We can use a firewall to block them. The command is as follows:
# netstat -anlp | grep 80 | grep tcp | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -nr | head -n20
9. SSH for Port Forwarding
Many people have heard of SSH as a secure remote login protocol for Linux, commonly used for remote server management. However, few may know that SSH can also be used for port forwarding. In fact, SSH’s port forwarding capability is quite powerful, and here’s an example.
Example Background: Our company has a bastion host, and all operations must be performed on the bastion host. Some developers need to access the ElasticSearch head panel to check the cluster status, but we do not want to expose the ElasticSearch port 9200. We still want to access it through the bastion host.
Therefore, we will forward requests to the bastion host (192.168.1.15) to the ElasticSearch server (192.168.1.19) on port 9200.
Example:
Forward access to port 9200 on the local machine (192.168.1.15) to port 9200 on 192.168.1.19
ssh -p 22 -C -f -N -g -L 9200:192.168.1.19:9200 [email protected]
Note: Make sure to perform key transfer first.
After executing the command, accessing port 192.168.1.15:9200 will actually access port 192.168.1.19:9200.