Recently, a new feature was added to NDB to allow it to access user space. However, on the ARM platform, a problem was encountered: if the CPU interrupt occurs in kernel space, any access to user space addresses fails.
The basic symptom of failure is a series of question marks printed in the debugger.
Meanwhile, the JTAG library prints the following error message: abort occurred – dscr = 0x03047d47
The dscr mentioned is an external debug register defined in ARM CoreSight technology, known as EDSCR, or External Debug Status and Control Register. Its detailed definition can be found in the ARMv8 architecture manual, and combined with the error code above, it can be interpreted as shown in the figure below.
I initially thought that the lower 6 bits of the status value could provide the cause of the error, but it consistently indicates a “breakpoint,” meaning that this interrupt to the debugger occurred due to hitting a breakpoint.
Old friends are asked to use ARM’s official DTRACE tool for testing, and they also found similar situations where, in some cases, user space could not be read. For example, in the figure below, after switching to CPU 5, accessing an address space that is accessible by other cores fails, and the memory window shows a red background with no data.
The following image shows a situation where access is successful.
Who is preventing the powerful hardware debugger from accessing memory? Based on years of experience, it might be a security mechanism at play. But where is this ‘ghost’ hiding?
Today is the weekend, and I recalled this issue. I thought of the Linux kernel uaccess code I read while writing the “Cat-Snake Battle” series.
Although kernel space has high privileges and can access user space from a privilege level perspective, accessing it is actually quite complex. First, there are many user spaces, and user space memory often does not reside in physical memory. Additionally, the kernel must have a valid reason to access user space; it cannot just “intrude into a private residence.” Therefore, accessing user space from the kernel can be quite precarious.
Moreover, since this logic is related to CPU hardware, the code is also quite dispersed. Implementations often use macros or embedded assembly, making it harder to read.
Thinking of this, I opened the kernel code and searched for uaccess. The results from searching the entire kernel code tree were too numerous, so I only searched in arch/arm64 used by GDK8. Sure enough, there were findings. A line of printk handling page faults caught my attention.
die_kernel_fault(“access to user memory outside uaccess routines”,
addr, esr, regs);
This die series print is a spectacle in the kernel. Once this function is executed, the system enters panic. The meaning of this error message: “Accessing user memory outside of uaccess routines” is the reason for calling die, effectively sentencing the system to death.
Following this message: access to user memory outside uaccess is not allowed. NDB clearly belongs to this situation. Because NDB accesses memory using JTAG, not uaccess.
So why can uaccess access it?
Opening the uaccess code under arm64, I quickly found a key function.
__uaccess_ttbr0_disable, from the function name, it appears to disable access; it is likely the culprit.
This function has a conditional compilation option, and upon checking this option, it is indeed enabled.
geduer@gdk8:~$ zcat /proc/config.gz | grep TTBR0
CONFIG_ARM64_SW_TTBR0_PAN=y
It seems that it is indeed causing trouble. Searching for the documentation of this macro:
Emulate Privileged Access Never using TTBR0_EL1 switching
configname: CONFIG_ARM64_SW_TTBR0_PAN
Linux Kernel Configuration
└─> Kernel Features
└─> Emulate Privileged Access Never using TTBR0_EL1 switching
Enabling this option prevents the kernel from accessing
user-space memory directly by pointing TTBR0_EL1 to a reserved
zeroed area and reserved ASID. The user access routines
restore the valid TTBR0_EL1 temporarily.
It is clearly stated that to prevent the kernel space from accessing user space arbitrarily, the ttbr0 register, which records the user space page directory, is deliberately swapped. This trick is truly insidious. ^-^
Using NDB to read the ttbr0 register, I see the following value:
rdmsr ttbr0_el1
msr[182000] = 00000000`02503000
To spoil the surprise, this value is a fake and differs greatly from the effective value that will be seen later.
Since the uaccess function has imposed this restriction, it should also have a way to enable it, and indeed it does.
Reading the code of this function, it reads the saved ttbr0 from the current thread information and writes it back to the physical CPU.
How to find the saved ttbr value?
There is a way. In NDB, first execute the !ps command to display the address of the current thread’s task_struct.
task_struct:0xffffffc0f409e740 pid: 179 comm:avahi-daemon
PGD:0xffffffc0f0d2b000 CR3=0x0
state 0 flags:0x404100 stack:0xffffff800b8a8000
The address above points to the task_struct structure that every thread has in Linux, which is frequently accessed in the kernel source code using the famous current macro (I call it the first tyrant of the Linux kernel). This structure is extremely large, and its starting part is the architecture-related thread_info substructure.
dt lk!task_struct
+0x000 thread_info : thread_info
+0x020 state : Int8B
+0x028 stack : Ptr64 Void
+0x030 usage :
+0x034 flags : Uint4B
+0x038 ptrace : Uint4B
+0x040 wake_entry : llist_node
+0x048 on_cpu : Int4B
+0x04c cpu : Uint4B
+0x050 wakee_flips : Uint4B
+0x058 wakee_flip_decay_ts : Int8B
+0x060 last_wakee : Ptr64 task_struct
[This section omits hundreds of lines]
Since thread_info is at the beginning of task_struct, the address of task_struct is the address of thread_info. Using the dt command to observe:
dt lk!thread_info 0xffffffc0f409e740
+0x000 flags : 0
+0x008 addr_limit : 549755813887
+0x010 ttbr0 : 0xf80000`f0d2b000
+0x018 preempt_count : 0
Indeed, the real ttbr0 has appeared.
Next, use the NDB write register command to write this saved ttbr0 back to the CPU:
wrmsr ttbr0_el1 0xf80000f0d2b000
msr[182000] = 00f80000`f0d2b000
After confirming that ttbr0 has been successfully written, try accessing user space again:
dd 0000007f`82809000
0000007f`82809000 464c457f 00010102 00000000 00000000
0000007f`82809010 00b70003 00000001 000011c0 00000000
0000007f`82809020 00000040 00000000 0001e548 00000000
0000007f`82809030 00000000 00380040 00400007 0019001a
0000007f`82809040 00000001 00000005 00000000 00000000
0000007f`82809050 00000000 00000000 00000000 00000000
0000007f`82809060 0001c4d4 00000000 0001c4d4 00000000
0000007f`82809070 00010000 00000000 00000001 00000006
It actually succeeded! The problem that had troubled me for days was resolved just like that, with the young NDB debugger and the code gun playing a positive role. Let’s take a group photo of them.
I wrote this article while traveling. Both GDK8 and the code gun are very convenient to carry, so I put them in my backpack, ready to quickly set up a powerful debugging environment at any time.
The blue accessory in the picture is called Nano Display, which can convert the HDMI output of GDK8 to USB signals for the laptop. Nano Code integrates a video playback function that can display the GDK8 desktop on the host.
The desktop of GDK8 is my beloved Lushan Xiufeng Longtan. Many friends have traveled with me there.
(Writing articles is tough, I kindly ask all readers to click “See” and feel free to share)
*************************************************
With sincerity and dedication, understanding the world through observation, viewing software with humanistic feelings, and changing lives through software technology.
Scan the QR code below or search for the “Shengge Shu” mini-program on WeChat to read more articles and audiobooks.
Also welcome to follow the public account of Ge You.
