Keywords
Linux, rootkit, analysis, malware
1. Background Introduction
A rootkit is a dangerous type of malware that is difficult to detect once it is implanted in a host. Compared to other types of malware, it is harder to write, so developers often reuse open-source rootkit projects for development. Since rootkit analysis is quite interesting, the Avast research team has been attempting to capture wild samples.
Adore-Ng is an early, open-source, and widely used Linux kernel rootkit that was originally used to attack the 2.x version of the kernel and later expanded its attack surface to the 3.x version. Because it has the ability to hide processes, files, and even kernel modules, it is harder to detect. Authorized user-space processes can also interact with it to control the operation of the rootkit, allowing attackers to hide a large number of custom malicious components with just a simple rootkit.
In early 2022, the Avast research team captured a wild rootkit sample. This sample is primarily based on the Adore-Ng project and is under continuous development. Information in the sample’s .modinfo section indicates it was compiled based on a specific kernel version.
Figure 1: .modinfo Section Information
It is well known that even using the –force parameter with the linux command insmod to forcibly load a module, the loading operation may still fail due to missing required symbols in a certain version of the system kernel. If the loading fails, it can further lead to system crashes.
Research shows that this rootkit can run on Centos 6.10 distribution and its close kernel versions because its required kernel modules can be loaded smoothly without the –force parameter. Meanwhile, there is a special hard-coded filename PgSD93ql in the rootkit. This filename may serve as a disguise, as it looks like a legitimate PostgreSQL file.
Figure 2: Hard-Coded Strings in the Read-Only Data Section
By separating the hard-coded filename, the executable file hidden by the rootkit can be extracted. This file is a trojan backdoor written in C language, which Avast’s antivirus engine classifies as ELF:Rekoob, belonging to the Rekoobe malware family. Rekoobe is a code snippet implanted in legitimate servers. In this case, it lurks within a disguised SMTP server. When Rekoobe receives a specially crafted command, it will open a shell on the host. Since the command is written to the file /proc/syslogk, the Avast research team named this rootkit the Syslogk rootkit.
2. Syslogk Rootkit Analysis
The development of the Syslogk rootkit is mainly based on the Adore-Ng project, and on this basis, the developers have added some new features that make user-space applications and kernel rootkits harder to detect.
2.1 Kernel Module Loading
Loading the Syslogk rootkit into kernel space requires using a kernel version close to the one used at compile time. The Avast research team successfully loaded the rootkit using the insmod command on a Centos 6.10 virtual machine. Notably, after loading, the malicious driver could not be found in the list of loaded kernel modules using the lsmod command.
2.2 Discovering Syslogk Rootkit
The Syslogk rootkit contains a hide_module function that removes the module from the kernel module linked list by calling the list_del function in the Linux kernel API. Next, it specifically updates the internal module_hidden flag.
The good news is that the Syslogk rootkit has a feature that implements the proc_write function, exposing a write file interface in the /proc filesystem. By writing 1 to the /proc/syslogk file, the Syslogk rootkit can be discovered.
Figure 3: Example of Discovering the Syslogk Rootkit
Discovering the Syslogk rootkit also means that it can be removed from memory using the rmmod command. The “related research tools” section of this article will provide more details helpful for analyzing the Syslogk rootkit.
2.3 Overview of Syslogk Rootkit Features
In addition to hiding itself to make detection difficult, the Syslogk rootkit also thoroughly hides the malicious payload through the following measures:
-
Using the hk_proc_readdir function to hide directories containing malicious files on the operating system;
-
Hiding malicious processes through the hk_getpr function, which is a hybrid of the process hiding function in the Adore-Ng project;
-
Using the hk_t4_seq_show function to hide network traffic. When ports are opened, they do not appear in the system port list monitored by netstat;
-
The malicious code does not run persistently. The attacker can invoke the Syslogk rootkit by sending specially crafted TCP packets to the infected machine, a technique accomplished by installing netfilter hooks to listen to traffic;
-
The attacker can also remotely stop the payload’s operation. This behavior requires using a hard-coded key in the rootkit and knowledge of magic packets used to remotely wake the payload.
The Avast research team observed that when the Syslogk rootkit (and the Rekoobe payload) is used in conjunction with a spoofed SMTP server, its functionality can be perfectly demonstrated. That is, a backdoor program appears harmless under normal circumstances, but when certain specially crafted magic packets arrive, it transforms into a legitimate service hidden in memory, on disk, and on the network, capable of remote execution, even appearing as a legitimate SMTP server when scanned by network ports.
To successfully attack the operating system and implement the hidden functions mentioned above, Syslogk uses known rootkit functions set_addr_rw and set_addr_ro, which can be used to add or remove write permissions in the page table entry structure. After adding write permissions to the page table entries, the rootkit can hook functions declared in the internal rootkit structure of hks.
|
Page Table Entry Hook |
||
|
Function Type |
Offset |
Function Name |
|
Initial Function |
hks+(0x38) * 0 |
proc_root_readdir |
|
Hook Function 1 |
hks+(0x38) * 0 + 0x10 |
hk_proc_readdir |
|
Initial Function 2 |
hks+(0x38) * 1 |
tcp4_seq_show |
|
Hook Function 2 |
hks+(0x38) * 1 + 0x10 |
hk_t4_seq_show |
|
Initial Function 3 |
hks+(0x38) * 2 |
sys_getpriority |
|
Hook Function 3 |
hks+(0x38) * 2 + 0x10 |
hk_t4_seq_show |
The hooking technique mainly involves identifying hookable kernel symbols through /proc/kallsyms, which is implemented in the rootkit’s get_symbol_address function. After obtaining the symbol address, the Syslogk rootkit uses the udis86 project to hook the functions.
2.4 Directory Hiding Mechanism Analysis
The virtual file system is an abstraction layer that allows users to perform file system-like operations on non-traditional file systems. Since it is the entry point for all file system queries, it is a good alternative hook point for rootkits.
The Syslogk rootkit achieves the purpose of hiding the Rekoobe payload, which is stored in /etc/rc-Zobk0jpi/PgSD93ql, by hooking into the virtual file system functions.
This hook point is implemented through the hk_root_readdir function, which calls the nw_root_filldir function that has directory filtering capabilities.
Figure 4: Partial Decompiled Result of Directory Hiding Code
The function hk_get_vfs uses the kernel function filp_open to open the root directory of the file system. This operation returns a pointer to the file structure, which contains a structure called f_op that represents file operations, where the readdir function hooked by hk_root_readdir is stored.
However, this technique is not new; the Adore-Ng source code already has such practices.
2.5 Process Hiding Mechanism Analysis
In the control flow on the right side of Figure 5, the Syslogk rootkit is used to hide a process called PgSD93ql, making the control flow more intuitive compared to the original rootkit (the left side of Figure 5). In addition, Syslogk can further selectively hide authorized processes.
Figure 5: Control Flow Diagram of Process Hiding Mechanism
The Syslogk rootkit function hk_getpr is a hybrid of the adore_find_task and should_be_hidden functions from the Adore-Ng project, but it still uses the same mechanism to hide processes.
2.6 Network Traffic Hiding Mechanism Analysis
The Adore-Ng rootkit allows hiding a specified set of listening services to evade monitoring by Linux programs like Netstat. The specific implementation involves altering the running logic of the kernel tcp4_seq_show handle through the proc_net structure exported in the Adore-Ng project, which is called by the kernel when Netstat requests to listen to connections. By combining the adore_tcp4_seq_show function, the strnstr function searches for sub-strings in seq->buf that contain the hexadecimal representation of the hidden ports. If the required sub-string is found, that string is deleted to achieve the hiding purpose.
Figure 6: Control Flow Diagram of Network Traffic Hiding Mechanism
Through this method, the infected machine can avoid displaying the backdoor program when listing connections. The next chapters will describe other interesting features of the Syslogk rootkit.
2.7 Magic Packets Analysis
The Syslogk rootkit can be remotely started or terminated by receiving specially crafted network traffic packets, rather than running persistently.
Since the traffic packets have a special format and purpose, this technique is called magic packets. In the implementation of the Syslogk rootkit, the attacker can trigger the related behaviors of the Syslogk rootkit without needing to open listening ports on the infected machine, allowing commands to magically run within the system.
2.7.1 Starting the Rekoobe Payload
The magic packet verification logic for starting the Rekoobe spoofed SMTP server in the Syslogk rootkit is relatively simple: first, it checks if the traffic packet is of the TCP protocol; if so, it checks if the source port is 59318.
If the magic packet meets these two conditions, the Rekoobe spoofed SMTP server will start.
Figure 7: Condition Verification for Starting Rekoobe Payload
Before starting the spoofed service, the Syslogk rootkit calls the rootkit function pkill_clone_0 to terminate all instances of the rootkit. This termination includes hard-coded processes PgSD93ql and the Rekoobe process, and the termination is executed by calling send_sig to send a KILL signal.
To execute the command that starts the Rekoobe spoofed service in user space, the Syslogk rootkit combines kernel APIs to execute the following commands: call_usermodehelper_setup, call_usermodehelper_setfns, and call_usermodehelper_exec.
This article’s “related research tools” section provides how to start the Rekoobe payload using a specially crafted TCP magic packet in Python. The next section will present a more complex form of magic packets.
2.7.2 Terminating the Rekoobe Payload
Since the attacker does not want others on the network to stop Rekoobe, the magic packet used to stop Rekoobe needs to match certain fields of the previously started magic packet. Additionally, the traffic packet must meet extra requirements: it must include a hard-coded key with a dynamic offset in the rootkit. Before the termination operation, the following conditions will be checked:
1. Check if a certain flag is enabled. This flag will be enabled when the rootkit starts Rekoobe via magic packet.
2. Check if the Reserved field in the TCP header is 0x08.
3. The source port must be between 63400 and 63411.
4. The target port and source address need to match the corresponding fields of the magic packet used to start Rekoobe.
5. Check for the presence of a hard-coded key. In this case, the hard-coded key is D9sd87JMaij.
The dynamic offset of the hard-coded key is also set in the data offset field of the TCP packet header. This offset is not fixed but is calculated. After right-shifting the value by 4 bits and multiplying by 4, it points to the offset of the expected position of the key (as shown in Figure 8, the Syslogk rootkit compares the key in reverse order).
Figure 8: Condition Verification for Terminating Rekoobe Payload
In the Avast research team’s experiments, they used 0x50 as the value for the data offset field. The reason is that this value, after being right-shifted by 4 bits and multiplied by 4, equals 20. 20 is exactly the size of the TCP header, allowing the hard-coded key to be set at the beginning of the data segment of the traffic packet.
If readers are curious about how to implement the magic packet to terminate the Rekoobe payload, they can refer to the “related research tools” section of this article.
3. Rekoobe Analysis
When the infected machine receives a specially crafted magic packet, the Syslogk rootkit will start the hidden malware Rekoobe in user mode space.
It appears as a harmless SMTP server, but when it receives the starttls command, a backdoor command will be executed. In the context of legitimate services, the starttls command is sent by the client to the server to request a TLS handshake.
Figure 9: Establishment Process of TLS Handshake Request
To trigger the Rekoobe backdoor command, the attacker needs to send the byte 0x03 via the TLS protocol, followed by Tag Length Value (TLV) encoded data. The Tag is a symbol %, Length consists of four numeric characters, and the Value (note that length and value cannot be 0).
Figure 10: Attack Simulation to Trigger Rekoobe Backdoor Command
Additionally, to establish a TLS connection, a certificate embedded in Rekoobe needs to be used.
If interested, readers can check the certificate and the Python script used to connect with Rekoobe in the “related research tools” section.
4. Rekoobe and Syslogk Provenance
Rekoobe is evidently developed based on the TinyShell open-source project, as some of the character and variable declaration orders are highly consistent with those in the TinyShell project.
Figure 11: Comparison of Rekoobe and TinyShell Source Code
On the other hand, the Syslogk rootkit also includes references to TinyShell, which can be traced back to December 13, 2018.
Figure 12: Reference to TinyShell in Syslogk Rootkit
The above evidence indicates that the threat actors simultaneously developed Rekoobe and Syslogk and made the two tools work together. The Avast research team hopes this study will help protect its users and others.
5. Conclusion
One of the architectural advantages of security software is that components running at different privilege levels can handle malware running at lower privilege levels without easily interfering with processes running at higher privilege levels, allowing security software to deal with malware more directly.
On the other hand, because these malware run at high privilege levels, kernel rootkits can be difficult to detect and remove. System administrators and security companies must be aware of the existence of such malware and quickly formulate corresponding protective measures for their users.
6. Research Tools and Attack Indicators
6.1 Syslogk Research Tools
unhide_rootkit.chttps://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/unhide_rootkit.cmagic_packet_start_rekoobe.py:https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/magic_packet_start_rekoobe.pymagic_packet_kill_rekoobe.py：https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/magic_packet_kill_rekoobe.pyremove_syslogk_from_memory.sh：https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/remove_syslogk_from_memory.sh
6.2 Rekoobe Research Tools
rekoobe_backdor_client.pyhttps://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/rekoobe_backdoor_client.pycert.pem：https://github.com/avast/ioc/blob/master/SyslogkRootkit/Research Tools/cert.pem
6.3 Attack Indicators (IoCs)
Attack indicators can be found in the Avast research team’s GitHub project.
The project address is:
https://github.com/avast/ioc/tree/master/SyslogkRootkit
ENDReference link: https://decoded.avast.io/davidalvarez/linux-threat-hunting-syslogk-a-kernel-rootkit-found-under-development-in-the-wild/
Editor|Ni Kai
Proofread|He Shuangze, Jin Shi
Supervisor|Jiang Zhengwei
This article is compiled and organized by CNTIC and does not represent the views of this public account. Please retain the source and link when reprinting. Contact information can be found by clicking “About Us” after entering the public account.
