Ma Liang
Currently employed at NSFOCUS Technology Group, his research areas include IoT and industrial control security, with expertise in embedded systems. Before entering the security industry, he had ten years of extensive experience in embedded software development. This includes 4 years of IoT product development experience; 3 years of LTE development experience; and 3 years in industrial control system development.
The following content is the transcript of Ma Liang’s speech at the KSY 2018 Security Developer Summit:
Thank you very much for inviting me to speak here! The topic is “Several Breakthroughs in Vulnerability Mining for Smart Devices.”
I come from the Industrial IoT Security Laboratory of NSFOCUS Technology Group, and my specialty is firmware security for hardware. A quick advertisement: If you are interested in researching industrial IoT security, please feel free to contact us.
The outline of my talk today consists of these four points.
I have experience both as a developer and as a security professional, and at this conference, I take the opportunity to introduce hardware and firmware knowledge to security professionals and share security knowledge with developers.
1. Basic Knowledge of Smart Devices
This is the composition of smart devices, which consists of several parts. Smart devices must have several components, such as a CPU, memory, flash (equivalent to a hard drive), network, network interface, serial port, and sometimes an SD card slot for storage, as well as a display interface.
There are many architectures for CPUs, including X86, Linux, ARM, etc.; memory comes in several types. Storage also varies, such as flash, TF cards, MMC cards, and hard disks; generally, circuit boards will leave serial ports, especially for debugging. A problem that manufacturers face during maintenance is that if a smart device has issues during use, customers demand immediate resolution. If the developer does not have a maintenance interface, they may not be able to resolve the issue on-site. Therefore, these are essential; sometimes it is a serial port, sometimes a network port, and sometimes a USB port. The software running on smart devices is also called firmware, which includes a part for booting the system, commonly using Uboot, which occupies a large market share; the operating system used is often Linux due to its excellent network support.
Let me introduce what a serial port is. Serial ports are generally divided into two types: RS232 or TTL, where TTL is 5V or 3.3V, corresponding to the CPU voltage. The definition of the serial port is shown in the lower left corner, and a serial port actually uses three lines: RXD, TXD, and DCD. During serial transmission, when there are three lines, data is transmitted according to a certain timing, which has a specific cycle generated asynchronously. Asynchronous transmission has a baud rate concept: for example, a baud rate of 9600 translates to approximately 1000 bytes transmitted per second. The higher the baud rate, the faster the transmission.
Another important component of smart devices is flash, which comes in two types: Norflash, which is relatively expensive, has a small capacity, and has separate address and data lines, allowing the CPU to address directly. This is commonly used for code storage, with larger storage capacity requiring more address lines.
The other type is NAND Flash, which is cheaper and has a larger capacity, mainly used for data storage, but it is generally not addressable and requires a driver program.
Next, let’s talk about Uboot and Busybox in the software. Uboot in embedded systems serves as a bootloader and is also used for updates. It supports a wide variety of CPUs, such as ARM, Linux, MIPS, and PowerPC, as well as simple network commands. Busybox integrates over 300 commonly used Linux commands and tools. It is very compact, compiling to about 1-2 MB, but supports many commands and can be trimmed according to needs.
This is the collaborative relationship between software and hardware in smart devices. Taking the Linux operating system as an example, the memory is on top, the storage flash is below, and in the middle is the firmware. The Uboot in the firmware is used for booting, followed by the kernel, and then the file system, which includes RAMFS and FLASH FS.
This is the upgrade interface. We need to understand the maintenance interface for several reasons: First, it is essential for upgrades; second, it is used for debugging; and third, it is needed when upgrading firmware. The basic functions of the debugging interface are these.
2. Ten Approaches to Extracting Firmware
The ten approaches I am familiar with are methods I often use; not discussing other methods does not mean they do not exist. The ten methods are divided into hardware and software.
I just provided a brief introduction to the basic knowledge. For example, if I want to extract its firmware, it is stored in flash, so I need to understand it. I also need to study serial ports and interfaces. Additionally, I need to be familiar with the programs supported in the firmware, file systems, and basic commands, as these knowledge and skills can be very effective when applied well.
The ten approaches to extracting firmware from smart devices are:
Let me first discuss nine methods for extracting firmware; the last method will surprise you!
1. Obtain the upgrade package from the official website or customer service. I won’t elaborate on this.
2. Extract firmware through online upgrade methods. If there is a mobile application or computer application, you can click on “Upgrade”. When upgrading firmware, you can capture the firmware through packet capture. If there is a naming pattern for the download addresses of the old and new version firmware, you can guess the old version after capturing the new version. The PPT is just an example and is not specific to any manufacturer.
3. Reverse engineer upgrade software that has built-in unpacking and communication algorithms. If it is designed to decrypt on the lower machine, it is relatively secure; if it decrypts on the upper machine, it can be reverse-engineered.
4. Obtain firmware from debugging interfaces: JTAG/SWD, etc.
Last year, Xiao Ma from Tencent’s Xuanwu Lab gave two public explanations, but due to time constraints, I won’t describe them here.
5. Extract firmware from flash, SD/TF cards, hard drives, etc., using a programmer/card reader.
The common practice is to desolder the flash chip, read the firmware with a programmer, and then solder the chip back.
It is important to note that the chip has a direction, and the first pin has a dot. When soldering or reading with a programmer, be sure to clarify the chip’s orientation.
6. Obtain firmware from the serial port (UART) debugging interface.
This method is one I summarized while helping customers test products in security assessments. When manufacturers create products, they may pursue security, but may not have the resources, so they seek our help for security testing. Also, OEM products that are about to be launched often seek our testing to ensure they meet security requirements and do not damage the brand’s reputation.
During the security testing process, I discovered vulnerabilities in Uboot that allowed for firmware extraction, leading to a series of methods for extracting firmware.
I primarily focus on research regarding vulnerabilities in industrial control devices, as it is often difficult to obtain firmware, and even if obtained, it may be encrypted, which has troubled me for a long time. After some time of exploration, I have had about ten successful cases of firmware extraction. This year, I have also worked on cameras and similar devices like ordinary IoT routers, and after practical research, I found that extracting firmware through the serial port is the most stable and reliable method. Why do I say this? When I conduct tests for customers, they often say, “Do whatever you want; it doesn’t matter if it breaks, but we only have one sample!” I want to desolder the flash to extract the firmware, but if the device is the only one, what if I damage the chip during soldering? Then I won’t be able to validate the vulnerabilities later. The key is that industrial control devices can cost tens of thousands, and if I damage the device, it can be quite embarrassing. Therefore, I have gradually thought of some methods to share with everyone here.
Serial port identification: There are two standards for serial ports. When identifying a serial port, I generally open the device and find four pins. The first step is to find the ground (GND), which is easy to locate as it connects to power. After confirming the ground, I connect it to the USB interface, first connecting the receiving pin. Both pins may be high, so I connect the receiving pin to the ground and then connect a random pin. When the device starts, if there is output, it means the connection is correct; then I connect the last pin, and all three pins are connected. This is my thought process when identifying serial ports.
This PPT page modifies the Uboot startup script to enter a password-protected login system; I won’t elaborate on it. There are five methods, each providing a way to enter the system without a password.
As mentioned earlier, Uboot has a 3-second window during every startup, which can be set. Usually, there is a 3-second wait for input (or other shortcuts, generally with prompts). After entering Uboot mode, there are help commands, one of which caught my attention: md. What does md do? It displays memory and can show the contents of memory. This is interesting because it turns out that as long as it is Nor flash, md can display the contents of Nor flash (because Nor flash can be directly addressed by the CPU), which means that the md command can extract firmware. However, the md command requires knowledge of the starting address and length. If you know the starting address and length, you can extract the firmware. How to find out the starting address and command? I checked more command information and found that bdinfo and flinfo can show the starting address and capacity of the flash. If I still can’t find it, I can restart and not interrupt the process to see if it prints more information about the flash address distribution. Through this information, I can generally find the starting address to extract the firmware.
Finally, I summarized the process of extracting firmware from Uboot: First, obtain the storage information of the flash, including its size, model, and the CPU’s address range to access it. Second, use the md command to extract firmware information. Third, document it. Fourth, analyze the output information to obtain the firmware. The downside of this method is that it can take a bit longer, sometimes half an hour, sometimes two hours. However, that’s fine as long as it can reliably extract the firmware without damaging the hardware; just solder three lines, and it can be done.
Please let the staff play the video. (Video plays) I used simple commands to attempt extraction, and indeed, I could read the kernel. The next step was to read everything and extract the firmware. I deleted the above information, as the memory address information was irrelevant to the firmware, and I also deleted the ASCII code information on the right. Finally, I converted the file into binary, which is the firmware I wanted to obtain.
There are many successful cases, and so far, no failures. I summarize the difficulty into three levels (I won’t discuss the third level). Level 1 is the simplest, which can be handled using md, only using Nor flash for storage.
Level 2 involves Nandflash with other file systems. Here, one command reads the nand flash into memory, and then md reads the memory, making it simple to resolve.
This was my debugging experience, and I found that using the commands above can retrieve information. Another method requires a restart; sometimes a restart can print critical address information.
There are also reading commands that read md directly from flash. If it cannot be read, other methods can be employed to read the storage in the SD card or other file systems into memory and then read it back from memory. This is the general idea.
Extracting firmware from Uboot is not limited to any specific manufacturer; nearly all manufacturers have Uboot security issues, as 95% of embedded devices use Uboot for booting, making it a standard for embedded Linux. At the developer conference, I hope to communicate with developers and also hope that security researchers can provide new methods for firmware extraction or vulnerabilities for developers to know, which will help manufacturers quickly patch vulnerabilities and contribute to security. This method may become a relatively popular way to extract firmware in the future, as I have found that it can generally extract firmware.
7. Obtain sensitive information about firmware through web and communication vulnerabilities. I won’t demonstrate this; as long as you do security, these are minor issues. Sensitive information can be extracted via the web, and passwords can be brute-forced using dictionaries to crack device passwords.
8. Use a logic analyzer to listen to flash and RAM for information. The advantage is that you don’t have to disassemble anything; just connect it. However, the downside is that the cheaper logic analyzers have low frequencies, while flash typically operates at higher frequencies, usually 100MHz or 200MHz. I conducted an experiment and found that it is indeed feasible. You need to extract these pins and capture data when it starts. Clip it on the chip; the red-marked pin is the first pin. After doing this, the data transmitted through the SPI interface can be saved.
The previous SPI command outputs data. The first one indicates reading, and the three should be addresses, followed by data. Here are the commands sent and received, and the results from the logic analyzer matched the binary of the firmware in flash, indicating that this approach is viable. I also tested it with a slightly slower device, and it worked as well.
9. Use tar/dd, nc to extract firmware. By utilizing the original hardware interface with its original software, you can achieve efficiency without soldering the board. Here, I first use the serial port to gain system privileges, and then use nc to transmit the firmware over the network.
10. Comprehensive application
The tenth method for extracting firmware is a synthesis of the previous methods, a comprehensive application. It is akin to the final move of the “Eighteen Dragon-Subduing Palms.”
If you obtain access through the serial port and are lucky enough not to have a login password, you can use this method: through USB or SD card, execute copy and packing commands to transfer this to a USB drive or large memory file system, and then copy it to an external file system (USB or TF card).
The second method is to discover that the echo command can write “1234567” in binary, sending the binary file through command input, transmitting a trojan that outputs the firmware through the serial port, which is also a thought process.
In summary, methods for extracting firmware: Smart devices generally have Uboot, and if commands similar to md are not blocked, firmware can be easily extracted. The firmware of smart devices exists in flash, and if flash lacks protective measures against unauthorized reading, it can be extracted easily. However, if FLASH and RAM are integrated into the CPU and encryption is enabled, extracting the firmware becomes quite challenging. But if they are independent (FLASH, RAM, CPU), extracting the firmware is particularly easy because you can directly read the contents of the flash using a programmer.
The hardware debugging interface method facilitates customer maintenance but may also ease unauthorized access. If software has a backdoor, it makes maintenance easier, but if the password is too simple, it can be easily cracked or guessed, or others may analyze it from the firmware. The conclusion is that the architecture of some smart devices currently has certain security vulnerabilities.
3. Approaches to Discovering Vulnerabilities from Firmware
Use binwalk to dissect firmware. Some colleagues cannot extract with binwalk, but I can. Why? Because they do not master the installation method for binwalk, which has some dependencies that need to be installed. There is one command (./deps.sh) that needs to be executed to install all related dependencies, making extraction much easier. I often use the -Me option to extract firmware.
Here is an example of cracking Linux login passwords using dictionary brute force. If the password is relatively simple, it can be cracked quickly.
Previously, I worked in development, and whenever I lack ideas, I think about what the developers would do. What debugging backdoors might they leave? What interfaces are available? After extracting the firmware, can I check for sensitive information in the system? Check if the manufacturer left maintenance backdoor passwords? If there are backdoor passwords, it becomes much easier. After examining these, check how many network services and ports are running, and if any binary running the network service has overflow points, debug the web calling interface to see if there are issues. These are the general thoughts; it’s very simple. Due to time constraints, I can only touch on a few points.
4. Recommendations for Hardening Smart Devices
Information hiding is very important. For example, Uboot allows firmware extraction because the chip information is known. If the chip is polished, it might be better, or if the packaging is modified, it would also help. Important information should be kept hidden.
Can debugging interfaces be shielded? For instance, can software and hardware be shielded, or can unused features in Busybox be removed or trimmed? For example, if smart devices enter the system with root privileges, the permissions are too high, allowing too many actions to be performed; these can be avoided, and root privileges do not necessarily need to be granted.
Other hardening suggestions are in the PPT.
Thank you all!
*Please indicate the source from the KSY community when reprinting
Series of Articles Reviewing the Summit:
-
Review of the KSY 2018 Security Developer Summit, successfully concluded!
-
Highlights from the KSY 2018 Security Developer Summit Exhibition
-
Review of the KSY 2018 Summit | Observing KRACK Key Reinstallation Attacks from the WPA2 Four-Way Handshake
