01Test Environment
Xilinx ZCU106 Single BoardXilinx VCU TRD2020.1
02Introduction
In embedded Linux systems, Linux directly manages all CPUs. By default, the system aims to maximize throughput rather than real-time performance. To ensure real-time performance, more precise control over the CPU can be implemented based on the application scenario. Common methods include process CPU isolation, CPU affinity, interrupt CPU affinity, and process priority.
03Tools
In embedded Linux systems, tools such as ps and top from busybox are commonly used. They are lightweight but have limited functionality. For more powerful tools, they can be extracted from the Ubuntu filesystem ubuntu-base-20.04.1-base-arm64.tar.gz (Click “Read the original text” to view the link).
In this test, tools like ps and top were extracted from the Ubuntu filesystem and renamed to u-ps and u-top to distinguish them from the busybox versions.
04CPU Isolation
Linux may still schedule a process on any CPU, which can lead to ordinary processes affecting the performance of real-time processes. The Linux kernel command line parameter isolcpus can be used to achieve CPU isolation, completely preventing Linux from scheduling processes on certain CPUs, thus ensuring the response time of real-time processes.
In U-Boot, executing the following command can prevent Linux from scheduling processes on CPU2 and CPU3.
setenv bootargs "earlycon clk_ignore_unused consoleblank=0 cma=1700M uio_pdrv_genirq.of_id=generic-uio isolcpus=2,3"After Linux starts, you can check the Linux kernel command line parameters using the command “cat /proc/cmdline”.
root@vcu_trd:~# cat /proc/cmdline
earlycon clk_ignore_unused consoleblank=0 cma=1700M uio_pdrv_genirq.of_id=generic-uio isolcpus=2,3Then, using the ps tool from the Ubuntu filesystem with the psr option, you can view the CPU on which all system processes are running. The second column of the output indicates the CPU number. It can be seen that most processes run on CPU0 and CPU1. The processes running on CPU2 and CPU3 are those that have their CPU affinity set via taskset. The u-ps tool is from the package [ubuntu-base-20.04.1-base-arm64.tar.gz].
root@vcu_trd:~# u-ps -axo pid,psr,cmd,ni
PID PSR CMD NI
1 1 init 0
2 1 [kthreadd] 0
3 0 [rcu_gp] -20
4 0 [rcu_par_gp] -20
6 0 [kworker/0:0H-kblockd] -20
8 0 [mm_percpu_wq] -20
9 0 [ksoftirqd/0] 0
10 1 [rcu_sched] 0
11 0 [migration/0] -
12 0 [cpuhp/0] 0
13 1 [cpuhp/1] 0
14 1 [migration/1] -
15 1 [ksoftirqd/1] 0
17 1 [kworker/1:0H-kblockd] -20
18 2 [cpuhp/2] 0
19 2 [migration/2] -
20 2 [ksoftirqd/2] 0
21 2 [kworker/2:0-events] 0
22 2 [kworker/2:0H] -20
23 3 [cpuhp/3] 0
24 3 [migration/3] -
25 3 [ksoftirqd/3] 0
26 3 [kworker/3:0-events] 0
27 3 [kworker/3:0H] -20
28 0 [kdevtmpfs] 0
29 1 [netns] -20
30 1 [kauditd] 0
32 1 [oom_reaper] 0
33 0 [writeback] -20
34 1 [kcompactd0] 0
35 1 [khugepaged] 19
37 0 [kworker/0:1-events] 0
38 1 [kworker/u8:1-events_unboun 0
87 1 [kblockd] -20
88 0 [blkcg_punt_bio] -20
89 0 [edac-poller] -20
90 1 [watchdogd] -
91 1 [rpciod] -20
92 0 [kworker/u9:0] -20
93 1 [xprtiod] -20
94 1 [cfg80211] -20
95 1 [kswapd0] 0
96 1 [ecryptfs-kthrea] 0
97 0 [nfsiod] -20
100 0 [irq/60-a00d0000] -
107 1 [ion_system_heap] -
108 2 [irq/61-a00d1000] -
109 0 [kpktgend_0] 0
110 1 [kpktgend_1] 0
111 2 [kpktgend_2] 0
112 3 [kpktgend_3] 0
113 1 [ipv6_addrconf] -20
114 0 [krfcommd] -10
115 2 [kworker/2:1-events] 0
116 3 [kworker/3:1-events] 0
117 0 [kworker/0:2-events_power_e 0
118 0 [irq/47-fd4a0000] -
120 1 [scsi_eh_0] 0
121 1 [scsi_tmf_0] -20
122 1 [scsi_eh_1] 0
123 1 [scsi_tmf_1] -20
125 1 [spi0] 0
128 1 [sdhci] -20
129 0 [irq/41-mmc0] -
134 0 [mmc_complete] -20
135 0 [kworker/0:1H-mmc_complete] -20
165 1 [kworker/1:1H-kblockd] -20
181 0 /sbin/udevd -d 0
231 1 [irq/63-xilinx-v] -
238 1 [xilinx-hdmi-rx] -20
242 1 [irq/54-xilinx-h] -
246 1 [irq/52-xilinx-h] -
419 0 [irq/62-a0220000] -
420 2 [irq/62-a0200000] -
808 0 udhcpc -R -b -p /var/run/ud 0
815 1 /usr/bin/dbus-daemon --syst 0
818 0 /usr/sbin/haveged -w 1024 - 0
827 1 xinit /etc/X11/Xsession -- 0
831 0 /usr/bin/Xorg :0 -br -pn -1
836 1 matchbox-window-manager -th 0
841 1 dbus-launch --sh-syntax --e 0
842 1 /usr/bin/dbus-daemon --sysl 0
862 0 /usr/sbin/dropbear -r /etc/ 0
864 0 /usr/libexec/at-spi-bus-lau 0
874 0 /usr/libexec/gconfd-2 0
882 0 /usr/bin/dbus-daemon --conf 0
885 1 /usr/sbin/inetd 0
888 0 /usr/bin/settings-daemon 0
898 0 /sbin/syslogd -n -O /var/lo 0
901 0 /sbin/klogd -n 0
910 0 matchbox-desktop 0
911 0 matchbox-panel --start-appl 0
921 0 /usr/sbin/tcf-agent -d -L- 0
929 0 /bin/sh /bin/start_getty 11 0
930 0 /sbin/getty 38400 tty1 0
934 1 /usr/libexec/at-spi2-regist 0
940 1 /usr/sbin/console-kit-daemo 0
1025 0 /bin/login -- 0
1050 1 -sh 0
1055 0 /usr/sbin/dropbear -r /etc/ 0
1057 0 -sh 0
1063 0 /usr/sbin/dropbear -r /etc/ 0
1065 0 -sh 0
1071 0 top 0
7174 0 /usr/sbin/dropbear -r /etc/ 0
7176 0 -sh 0
22378 0 /usr/sbin/dropbear -r /etc/ 0
22380 0 -sh 0
22588 0 /usr/sbin/dropbear -r /etc/ 0
22590 0 -sh 0
22600 1 [kworker/1:0-events] 0
22601 1 [kworker/u8:0-events_unboun 0
22602 1 [kworker/1:2-events_power_e 0
22603 1 [kworker/u8:2-events_unboun 0
22606 0 u-ps -axo pid,psr,cmd,ni 005Process CPU Affinity
When setting process CPU affinity, it is necessary to know the process ID (PID). Tools like ps and top can be used to view the process ID (PID). The taskset tool can be used to view and control the CPU affinity of processes. By using the ‘-p’ option, you can specify the process ID (PID) to view the corresponding process’s CPU affinity.
root@vcu_trd:~# taskset -p 815
pid 815's current affinity mask: 1The following script can check the CPU affinity of all processes.
#!/bin/sh
u-ps -axo pid,psr,cmd,ni | grep -v "gst" | grep -v "xilinx" | grep -v "irq" | grep -v "kworker" | grep -v "grep" | grep -v "awk" | awk '{print $1}' > process_list.txt
echo -e "\nRead process list file:" ;
cat process_list.txt | while read line
do
# echo "CPU affinity for process ID: $line"
taskset -p $line
doneIn Linux systems with a large number of processes, the following script can be used to set the CPU affinity for all processes.
#!/bin/sh
u-ps -axo pid,psr,cmd,ni | grep -v "grep" | grep -v "awk" | awk '{print $1}' > process_list.txt
cat process_list.txt | while read line
do
echo -e "\nCheck process ID: $line"
if [ $line -gt 500 ]; then
# echo "Original CPU affinity for process ID: $line"
# taskset -p $line
echo "Set priority for process ID: $line"
taskset -a -p 1 $line
# echo "New CPU affinity for process ID: $line"
# taskset -p $line
fi
doneFor new tasks, you can specify the process CPU affinity at startup. The help information for taskset is as follows:
taskset [options] [mask | cpu-list] [pid|cmd [args...]]To specify the process CPU affinity, you can use the following command to start a new task.
taskset -a cpu-list cmdFor example, executing the command “taskset -a 8 top” will show that it indeed runs on CPU-3.
root@vcu_trd:~# u-ps -axo pid,psr,cmd,ni | grep top | grep -v grep | grep -v match
22629 3 top06Interrupt CPU Affinity
By default, Linux uses CPU0 to handle interrupts from ordinary peripherals. By changing /proc/irq/irq_number/smp_affinity, you can change the CPU that handles the interrupts. You can also view /proc/interrupts to display the number of interrupts handled by each CPU in the system.
There are also many interrupts in Linux systems, and the following script can be used to set the CPU affinity for all interrupts. The mapping of interrupts to CPUs can be changed based on the scenario.
#!/bin/sh
cat /proc/interrupts > interrupts_list_all.txt
cat /proc/interrupts | grep -v "CPU" | grep -v "IPI" | grep -v "Err" | awk '{print $1}' > interrupts_list.txt
echo -e "\nRead interrupts list file:" ;
cat interrupts_list.txt | while read line
do
# remove colon :
line_new=${line/:/}
echo -e "\nCheck interrupt: $line_new"
ls -l -h /proc/irq/$line_new/smp_affinity
# 48: GICv2 122 Level xilinx_framebuffer
# 52: GICv2 123 Level xilinx-hdmi-rx
# 54: GICv2 125 Level xilinx-hdmitxss
# 55: GICv2 127 Level xlnx-mixer
# 61: GICv2 139 Level a00d1000.sync_ip
# 62: GICv2 128 Level a0200000.al5e, a0220000.al5d
# 63: GICv2 124 Level xilinx-vphy
if [ $line_new -eq 48 ]; then
echo -e "\nSet CPU:1 affinity for interrupt: $line_new"
echo 2 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 52 ]; then
echo -e "\nSet CPU:1 affinity for interrupt: $line_new"
echo 2 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 54 ]; then
echo -e "\nSet CPU:1 affinity for interrupt: $line_new"
echo 2 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 55 ]; then
echo -e "\nSet CPU:1 affinity for interrupt: $line_new"
echo 2 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 61 ]; then
echo -e "\nSet CPU:2 affinity for interrupt: $line_new"
echo 4 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 62 ]; then
echo -e "\nSet CPU:2 affinity for interrupt: $line_new"
echo 4 > /proc/irq/$line_new/smp_affinity
elif [ $line_new -eq 63 ]; then
echo -e "\nSet CPU:2 affinity for interrupt: $line_new"
echo 2 > /proc/irq/$line_new/smp_affinity
else
echo -e "\nSet CPU:0 affinity for interrupt: $line_new"
echo 1 > /proc/irq/$line_new/smp_affinity
fi
echo -e "\nNew CPU affinity for interrupt: $line_new"
cat /proc/irq/$line_new/smp_affinity
doneAfter setting the interrupts, you can check /proc/interrupts to see that CPU2/CPU3 handled interrupts 48, 52, 54, 55, 61, and 62.
root@vcu_trd:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
3: 115462 135783 31811 204151 GICv2 30 Level arch_timer
6: 0 0 0 0 GICv2 67 Level zynqmp_ipi
7: 0 0 0 0 GICv2 175 Level arm-pmu
8: 0 0 0 0 GICv2 176 Level arm-pmu
9: 0 0 0 0 GICv2 177 Level arm-pmu
10: 0 0 0 0 GICv2 178 Level arm-pmu
12: 349750 0 0 0 GICv2 156 Level zynqmp-dma
13: 0 0 0 0 GICv2 157 Level zynqmp-dma
14: 0 0 0 0 GICv2 158 Level zynqmp-dma
15: 0 0 0 0 GICv2 159 Level zynqmp-dma
16: 0 0 0 0 GICv2 160 Level zynqmp-dma
17: 0 0 0 0 GICv2 161 Level zynqmp-dma
18: 0 0 0 0 GICv2 162 Level zynqmp-dma
19: 0 0 0 0 GICv2 163 Level zynqmp-dma
20: 0 0 0 0 GICv2 164 Level Mali_GP_MMU, Mali_GP, Mali_PP0_MMU, Mali_PP0, Mali_PP1_MMU, Mali_PP1
21: 0 0 0 0 GICv2 109 Level zynqmp-dma
22: 0 0 0 0 GICv2 110 Level zynqmp-dma
23: 0 0 0 0 GICv2 111 Level zynqmp-dma
24: 0 0 0 0 GICv2 112 Level zynqmp-dma
25: 0 0 0 0 GICv2 113 Level zynqmp-dma
26: 0 0 0 0 GICv2 114 Level zynqmp-dma
27: 0 0 0 0 GICv2 115 Level zynqmp-dma
28: 0 0 0 0 GICv2 116 Level zynqmp-dma
30: 463312 0 0 0 GICv2 95 Level eth0, eth0
32: 525 0 0 0 GICv2 49 Level cdns-i2c
33: 113 0 0 0 GICv2 50 Level cdns-i2c
34: 0 0 0 0 GICv2 42 Level ff960000.memory-controller
35: 0 0 0 0 GICv2 57 Level axi-pmon, axi-pmon
36: 181 0 0 0 GICv2 155 Level axi-pmon, axi-pmon
37: 28 0 0 0 GICv2 47 Level ff0f0000.spi
38: 0 0 0 0 GICv2 58 Level ffa60000.rtc
39: 0 0 0 0 GICv2 59 Level ffa60000.rtc
40: 0 0 0 0 GICv2 165 Level ahci-ceva[fd0c0000.ahci]
41: 233 0 0 0 GICv2 81 Level mmc0
42: 133 0 0 0 GICv2 53 Level xuartps
44: 0 0 0 0 GICv2 84 Edge ff150000.watchdog
45: 0 0 0 0 GICv2 88 Level ams-irq
46: 12 0 0 0 GICv2 154 Level fd4c0000.dma
47: 0 0 0 0 GICv2 151 Level fd4a0000.zynqmp-display
48: 0 34920 0 0 GICv2 122 Level xilinx_framebuffer
49: 0 0 0 0 GICv2 141 Level xilinx_framebuffer
50: 0 0 0 0 GICv2 142 Level xilinx_framebuffer
51: 0 0 0 0 GICv2 143 Level xilinx_framebuffer
52: 0 1142094 0 0 GICv2 123 Level xilinx-hdmi-rx
53: 0 0 0 0 GICv2 121 Level xilinx_framebuffer
54: 17669 151552 0 0 GICv2 125 Level xilinx-hdmitxss
55: 17672 151552 0 0 GICv2 127 Level xlnx-mixer
56: 0 0 0 0 GICv2 136 Level xilinx-dma-controller
57: 0 0 0 0 GICv2 137 Level xilinx-dma-controller
58: 0 0 0 0 GICv2 138 Level xilinx-dma-controller
59: 0 0 0 0 GICv2 140 Level xilinx-dma-controller
60: 81 0 0 0 GICv2 126 Level a00d0000.i2c
61: 0 0 69841 0 GICv2 139 Level a00d1000.sync_ip
62: 4 0 279353 0 GICv2 128 Level a0220000.al5d, a0200000.al5e
63: 1184 163 0 0 GICv2 124 Level xilinx-vphy
64: 0 0 0 0 GICv2 97 Level xhci-hcd:usb1
67: 0 0 0 0 zynq-gpio 22 Edge sw19
IPI0: 64845 46081 35 663483 Rescheduling interrupts
IPI1: 19 58 29 29 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 0 0 Timer broadcast interrupts
IPI5: 0 0 0 0 IRQ work interrupts
IPI6: 0 0 0 0 CPU wake-up interrupts
Err: 007Process Priority
The concept of process priority in Linux is quite complex. Generally, you can change a process’s priority by setting its Nice value using the renice tool. A higher Nice value means a lower priority for the process. The common format for renice is renice PRIORITY -p pid, where PRIORITY is the Nice value and pid is the process ID.
The following script can set the priority of all processes containing a keyword (the first parameter of the script, $1) to the value of the second parameter ($2).
#!/bin/sh
u-ps -axo pid,psr,cmd,ni
u-ps -axo pid,psr,cmd,ni | grep $1 | grep -v "grep" | grep -v "awk" | awk '{print $1}' > process_list.txt
echo -e "\nRead process list file:" ;
cat process_list.txt | while read line
do
echo -e "\nSet PID: $line to priority-nice value: $2\n\n" renice $2 -p $line
done08Others
To further improve real-time performance, you may consider adding the Linux RT Patch (Click “Read the original text” to view the link).
Follow Us
