Technical experience sharing, welcome to follow and provide guidance
In the process of system development, the version deployed on the device is usually the release version, which generally does not contain debugging information. When issues arise, how do we troubleshoot? In fact, Linux provides the core dump feature, which generates a core file when a program crashes. As long as we obtain the core file, it is equivalent to having the scene of the problem, and the next step is to use GDB to analyze the issue.
Where is the core file
For Linux systems, we first need to check if systemd-coredump is installed. If systemd-coredump is not installed, the core file is stored in the default kernel location, and the kernel will write it as “core”. Therefore, the core file will exist in the root directory of the starting user. Assuming the current starting user is “kylin”, the core file will be located as follows:
# file /home/kylin/core
/home/kylin/core: ELF 64-bit LSB core file, ARM aarch64, version 1 (SYSV), SVR4-style, from '/usr/bin/ukui-panel', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/ukui-panel', platform: 'aarch64'Similarly, we can check the core location through the proc file provided by the kernel, as follows:
# cat /proc/sys/kernel/core_pattern
coreIf we have installed the systemd-coredump package, then the core file is managed by systemd, and the actual location is as follows:
# cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %hAt this point, the core file is located at<span>/var/lib/systemd/coredump/</span>
Practical Example
When a binary crashes in the system, a core file will be generated. Next, I will introduce how to debug the core file with a practical example. This experience is very useful for debugging operating systems.
I manually created a crash example, and at this point, we can see the following file in the coredump directory:
core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000.lz4From this information, we can see that “kylin-vpn” has crashed, and we should debug “kylin-vpn”. First, we need to decompress this lz4 file as follows:
# lz4 core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000.lz4
# file core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000
core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000: ELF 64-bit LSB core file, ARM aarch64, version 1 (SYSV), SVR4-style, from '/usr/bin/kylin-vpn', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/kylin-vpn', platform: 'aarch64'Now, let’s assume we directly debug with GDB to see what happens:
# gdb -c core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000 /usr/bin/kylin-vpn
Reading symbols from /usr/bin/kylin-vpn...
(No debugging symbols found in /usr/bin/kylin-vpn)
Core was generated by `/usr/bin/kylin-vpn'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000007fa009f5a0 in QDBusMetaType::typeToSignature(int) () from /lib/aarch64-linux-gnu/libQt5DBus.so.5I have pasted the key information above. We can see that the code encountered a segmentation fault in the function <span>QDBusMetaType::typeToSignature(int)</span> in <span>libQt5DBus.so.5</span>.
At this point, let’s try to disassemble the <span>QDBusMetaType::typeToSignature</span> function, but we find that we cannot disassemble it properly, as follows:
(gdb) disassemble QDBusMetaType::typeToSignature
No symbol "QDBusMetaType" in current context.What is the reason for this? In fact, it was mentioned in “GDB Debugging Method (4) – Debugging Information” that this dynamic library is a distribution dynamic library, which has naturally been stripped, and stripped dynamic libraries cannot find corresponding symbols. So this situation is quite normal, and there is no need to panic. Next, we will gradually debug the issue.
Although we know that the problem occurs in <span>QDBusMetaType::typeToSignature</span>, we still need to judge and debug the issue step by step. First, we notice that “kylin-vpn” does not have symbols, so we need to add the corresponding symbols as follows:
# dpkg -S /usr/bin/kylin-vpn
kylin-nm: /usr/bin/kylin-vpn
# dpkg -i kylin-nm-dbgsym_3.20.1.7-0k0.1tablet8.egf0.6_arm64.ddebThe above installation actually adds the symbols of “kylin-vpn” into the system, allowing GDB to load the corresponding debug information based on the build-id. The contents of the ddeb are as follows:
# dpkg -L kylin-nm-dbgsym
/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/0f
/usr/lib/debug/.build-id/0f/470de270e27775339fd5c2cc71f2f5610d10c2.debug
/usr/lib/debug/.build-id/0f/afcd08ad5eb832b1efb97a785767c90458ef82.debug
/usr/lib/debug/.build-id/1f
/usr/lib/debug/.build-id/1f/42ddf71452964a5f507f24d4db18507046b9c7.debug
/usr/lib/debug/.build-id/41
/usr/lib/debug/.build-id/41/36dbbc90afa8a433f445f7bb964f9beea34fe2.debug
/usr/lib/debug/.build-id/63
/usr/lib/debug/.build-id/63/a064e1ef52b9ab4039a21bf4093082f844ac9f.debug
/usr/lib/debug/.build-id/83
/usr/lib/debug/.build-id/83/90208727fef2f884cf665bf868d46469056e3e.debug
/usr/lib/debug/.build-id/da
/usr/lib/debug/.build-id/da/f42bf2c9e3f18cf6716cdc49ad1811e130ee46.debugAt this point, we can remount GDB and see that the binary symbols have been added, as follows:
Reading symbols from /usr/bin/kylin-vpn...
Reading symbols from /usr/lib/debug/.build-id/0f/afcd08ad5eb832b1efb97a785767c90458ef82.debug...However, we still cannot see the line numbers of the code, as follows:
(gdb) l
66 main.cpp: No such file or directory.So we need to bring in the corresponding code as follows:
scp -r source/src-vpn dest@kylin:~/Then use GDB to load the required source code snippets as follows:
# gdb -c core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000 /usr/bin/kylin-vpn -d ~/src-vpn/At this point, GDB can load all the code of “kylin-vpn” normally, as follows:
(gdb) l
66 break;
67 case QtCriticalMsg:
68 fprintf(log_file? log_file: stderr, "Critical: %s: %s (%s:%u, %s)\n", currentDateTime.constData(), localMsg.constData(), file, context.line, function);
69 break;
70 case QtFatalMsg:
71 fprintf(log_file? log_file: stderr, "Fatal: %s: %s (%s:%u, %s)\n", currentDateTime.constData(), localMsg.constData(), file, context.line, function);
72 break;
73 }
74
75 if (log_file)We can now perform source code analysis based on “kylin-vpn”.
However, we have just discovered that the actual problem is in the <span>QDBusMetaType::typeToSignature</span> function. Let’s disassemble it again as follows:
(gdb) disassemble QDBusMetaType::typeToSignature
There is no field named typeToSignature.As we can see, because we loaded the binary source code and symbols, the problematic area is not in our “kylin-vpn” binary, so this error changed from “No symbol” to “no field named”. This means we need to further debug the <span>typeToSignature</span> function.
From this information, we know that the problematic dynamic library is <span>libQt5Dbus.so.5</span>
#0 0x0000007fa009f5a0 in QDBusMetaType::typeToSignature(int) () from /lib/aarch64-linux-gnu/libQt5DBus.so.5At this point, we can install the corresponding ddeb as follows:
# dpkg -S /usr/lib/aarch64-linux-gnu/libQt5DBus.so.5
libqt5dbus5:arm64: /usr/lib/aarch64-linux-gnu/libQt5DBus.so.5
# dpkg -i libqt5dbus5-dbgsym_5.12.8+dfsg_arm64.ddebHere is a small note: by default, some systems, such as Kylin, are user-merged, meaning that the lib directory is migrated to /usr/lib, so when we use <span>dpkg -S</span> to find the package, we need to append the /usr directory.
The topic of user-merged is not the focus of this article; if interested, you can refer to the following article:
https://wiki.gentoo.org/wiki/Merge-usr
Returning to the issue, let’s check again with GDB to see what the situation is:
# gdb -c core.kylin-vpn.1000.930a6eece73248149cd465ddbe02d695.17632.1757114505000000000000 /usr/bin/kylin-vpn -d src-vpn/
Core was generated by `/usr/bin/kylin-vpn'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 QDBusMetaType::typeToSignature (type=8) at qdbusmetatype.cpp:462
462 qdbusmetatype.cpp: No such file or directory.
[Current thread is 1 (Thread 0x7f812179b0 (LWP 17851))]
(gdb) list qdbusmetatype.cpp:1
1 in qdbusmetatype.cppWe can see that at this point, the symbols of <span>libQt5DBus.so.5</span> have been added, and it is attempting to find the source location of the DWARF, but the source code <span>qdbusmetatype.cpp</span> cannot be found.
Therefore, disassembly cannot correspond to the source code, as follows:
(gdb) disassemble /m QDBusMetaType::typeToSignature
Dump of assembler code for function QDBusMetaType::typeToSignature(int):
152 in qdbusmetatype.cpp
0x0000007fa009f4f4 <+420>: add x25, x22, #0x20
0x0000007fa009f4f8 <+424>: ldarb w0, [x25]
0x0000007fa009f4fc <+428>: tbz w0, #0, 0x7fa009f5b4 <QDBusMetaType::typeToSignature(int)+612>
0x0000007fa009f500 <+432>: add x19, x19, #0x20
0x0000007fa009f504 <+436>: add x23, x19, #0x28
153 in qdbusmetatype.cpp
154 in qdbusmetatype.cpp
155 in qdbusmetatype.cpp
156 in qdbusmetatype.cpp
157 in qdbusmetatype.cpp
158 in qdbusmetatype.cpp
159 in qdbusmetatype.cpp
160 in qdbusmetatype.cppSo we need to find this file from the code repository and SCP it to the machine as follows:
scp src/dbus/qdbusmetatype.cpp root@91:~/core/If we are in the GDB running directory, there is no need to load manually. If not, we need to load manually as follows:
(gdb) directory .
Source directories searched: ~/core:~/core/src-vpn:$cdir:$cwdAt this point, we can see the source code:
(gdb) list qdbusmetatype.cpp:1
1 /****************************************************************************
2 **
3 ** Copyright (C) 2016 The Qt Company Ltd.
4 ** Contact: https://www.qt.io/licensing/
5 **
6 ** This file is part of the QtDBus module of the Qt Toolkit.
7 **
8 ** $QT_BEGIN_LICENSE:LGPL$
9 ** Commercial License Usage
10 ** Licensees holding valid commercial Qt licenses may use this file inThen we can disassemble and find the source code. We find the problematic line as follows:
462 if (type >= ct->size())
0x0000007fa009f51c <+460>: ldr x24, [x23]
=> 0x0000007fa009f5a0 <+592>: ldr x24, [x23]
463 return 0; // type not registered with usAs we can see, this assembly loads the content of <span>x23</span> into the <span>x24</span> register. The source logic is <span>type >= ct->size()</span>. Let’s analyze the context first:
=> 0x0000007fa009f5a0 <+592>: ldr x24, [x23]
0x0000007fa009f5a4 <+596>: ldr w0, [x24, #4]
0x0000007fa009f5a8 <+600>: cmp w20, w0Here, <span>w20</span> is the value of <span>type</span>, and <span>w0</span> is the value of <span>ct->size()</span>. To obtain <span>ct->size()</span>, we need to first retrieve the pointer to <span>ct</span>, which is <span>ldr x24, [x23]</span>, and then use the pointer of <span>ct</span> to get the value at the size position, which is at offset 4 <span>ldr w0, [x24, #4]</span>.
If a segmentation fault occurs here, it means that the value of <span>x23</span> could not be retrieved. In Linux systems, the default 0 address is treated as an inaccessible segmentation fault address to effectively alert the user. So we suspect that the value of <span>x23</span> is 0. Let’s verify:
(gdb) p $x23
$2 = 0
(gdb) p $w20
$1 = 8
(gdb) p type
$3 = 8As we can see, since <span>ct</span> is already nullptr, trying to get the size naturally results in a segmentation fault. We check <span>qdbusmetatype.cpp</span> for the initialization of <span>ct</span> as follows:
Q_GLOBAL_STATIC(QVector<QDBusCustomTypeInfo>, customTypes)
QVector<QDBusCustomTypeInfo> *ct = customTypes();As we can see, <span>customTypes</span> is actually a static global variable vector. Why would it be set to nullptr?
The problem is quite clear: “kylin-vpn” will call D-Bus, which in turn calls QtBase. During the system shutdown process, Qt will destruct, and the order of static variable destruction is related to the linking order (refer to the article “Using ASAN to Locate Global Variable Construction Order Issues”). This means that before “kylin-vpn” is destructed, the static variable <span>qdbusmetatype.o</span> has already been destructed. Therefore, when “kylin-vpn” accesses D-Bus and subsequently accesses the static variable of <span>qdbusmetatype.o</span>, it results in a null pointer reference issue.
To address this issue, we need to consider the timing issues between multiple processes during system design to avoid unnecessary core dumps.
At this point, the practical analysis of the core using GDB has been completed. Almost all encountered core dump files can be quickly and accurately diagnosed using the above method.
Using Pwndbg
The above method using GDB has already achieved the debugging purpose. Here, I will add how Pwndbg can quickly find the issue.
First, enable Pwndbg as follows:
# cat ~/.gdbinit
source ~/pwndbg/gdbinit.pyThen load the core file with GDB. At this point, Pwndbg will print the context at the crash point as follows:
► 0x7fa009f5a0 <QDBusMetaType::typeToSignature(int)+592> ldr x24, [x23] <Cannot dereference [0]>
0x7fa009f5a4 <QDBusMetaType::typeToSignature(int)+596> ldr w0, [x24, #4] ► 462 if (type >= ct->size())
463 return 0; // type not registered with us X23 0The reason is presented clearly, without the need for analysis. Pwndbg directly tells you that the issue is <span>Cannot dereference [0]</span>. This quickly locates the problem.
Conclusion
This article shares a very common method for debugging core dumps. With this method, any core dump issues encountered in the future can be easily resolved. If using GDB for analysis and calculation is still quite tiring, you can try deploying Pwndbg on the device, and it will directly inform you of the issues during loading, eliminating the need for analysis.