Technical experience sharing, welcome to follow and provide guidance.

When discussing real-time capabilities of Linux, we often compare it with RTOS. Based on recent understanding of RTOS, I summarize several points on why Linux real-time performance is inferior to RTOS. These are the risk points that must be considered when using the Linux RT patch.

Uncertainty of Interrupts

Interrupts Cannot be Nested

In RTEMS, we see that interrupts support nesting in special cases such as kernel mode, but Linux does not allow interrupt nesting. This means that in Linux, when an interrupt is triggered, it must wait for the previous interrupt on the same CPU to finish processing. At this point, the interrupt is uncertain.

Softirq

Linux supports interrupt contexts (hardirq and softirq). We know that the priority of softirq is higher than that of thread tasks like kthread (tasklet, timer, netrx, etc.).

If a softirq from a peripheral driver continuously occupies a core, then all threads on that core may not get a chance to run.

I encountered this situation when handling CAN message reception (netrx), which directly caused one CPU core to stop working.

This is another uncertainty; softirq is peripheral driver code written by the customer. If the execution time of softirq is uncertain, then the interrupt is also uncertain.

Impact of Spinlock on High-Priority Task Uncertainty

For example, if there is a normal task holding a spin lock on a CPU core, then tasks on that core must wait until the spin lock is released before they can be scheduled.

At this point, the task is uncertain because no one can know how long it will take to unlock. Thus, high-priority tasks do not know when they will be able to run.

Memory Lazy Mode Causes Process Uncertainty

In Linux, memory is allocated only when it is used. When a high-priority task uses memory, the memory allocation process can only guarantee that you will get the allocation, but it cannot guarantee that you will receive it within a certain time. It may even lead to an out-of-memory (OOM) situation.

Role of PREEMPT_RT

Based on the points mentioned above, PREEMPT_RT mainly addresses the following three issues:

1. Interrupts can be preempted and nested.
2. Interrupts are threadified.
3. Softirq can be preempted.
4. Mutex replaces spin_lock.

For interrupt storms, the approach is to disable IRQ within the thread, and after threadification is complete, automatically enable IRQ.

For softirq threadification, the approach is to add all of them to ksoftirqd (tasklet, timer, netrx, etc.) for execution.

The approach for spinlock is to replace raw_spin_lock with rt_mutex (a sleepable mutex that supports priority inheritance to avoid priority inversion).

Issues with PREEMPT_RT

From the above, we can see that:

• For interrupts, the response to normal interrupts is worse.
• For softirq, the running path is longer compared to before, and it is easily preempted by regular tasks.
• For locks, being sleepable can lead to process switching, and the increase in context switches needs to be discussed separately. Multiple process switches are too costly, and performance is far inferior to spin_lock.

Therefore, we can conclude that adding the PREEMPT_RT kernel can guarantee a certain real-time performance for a high-priority task, but its negative effects will lead to poor performance for all other regular tasks. Thus, it is only suitable for customized special scenarios.

How to Optimize Real-Time Tasks Required in Special Scenarios

Based on the above, using the PREEMPT-RT kernel maintains the real-time performance of high-priority tasks to a certain extent. What else do we need to do to customize the operating system to meet the deterministic requirements of this real-time task?

DEADLINE Scheduling

The kernel actually provides EDF scheduling, which is SCHED_DEADLINE. We let this real-time task run in this special scheduling class.

The chrt tool can dynamically set the scheduling policy of a process, such as chrt –deadline / –fifo. Of course, pthread_setschedparam can also set the type of scheduler, such as SCHED_DEADLINE/SCHED_FIFO parameters.

Isolate and Taskset

By passing parameters through bootargs to isolate a certain CPU, and then using taskset to bind the real-time task to run on that CPU.

Timer

If high real-time performance is required, it is recommended to directly use assembly to call the ARM64 high-precision timer to encapsulate specific code.

Priority Inheritance

For multithreaded real-time tasks, it is recommended to set priority inheritance before creating pthread threads to avoid priority inversion. This means passing PTHREAD_PRIO_INHERIT in pthread_mutexattr_setprotocol. The default is NONE.

Locking the Heap

Memory allocated from the kernel can be swapped out by default, so we can lock the allocated memory using mlock. This way, this memory will not be swapped out. Additionally, memory with the mlock flag will automatically trigger a page fault when allocated, allocating physical pages. This effectively escapes the lazy mechanism.

Locking the Stack

Both heap and stack are memory. For the heap, we can simply use mlock. For the stack, since it is allocated by the operating system, after setting the maximum stack size, for example, 8M, if it is a thread, we can set pthread_attr_setstacksize.

Before all function calls, call a function that allocates an 8M space (or the stack size you set) using local variables, and then perform a memset call. At this point, the stack will request memory through page faults, and subsequent calls will not trigger page faults. The expression is not strong, as shown in the code below:

void stack_prefetch() {
    const size_t size = 8 * 1024 * 1024 - 4096;
    char buffer[size];
    memset(buffer, 0, sizeof(buffer));
}

Avoiding False Sharing

Programs running on multiple cores need to utilize cache coherence, such as the principle of locality. However, it is important to avoid false sharing of data. Therefore, when multiple threads access the same structure, it is best to align them to cache lines.

__attribute__((aligned(cachesize)))

Others

Other optimizations are related to performance. Through performance optimization, we can ensure low latency for tasks.

Conclusion

PREEMPT-RT is an optimization made on the Linux kernel to ensure the determinism of certain real-time tasks. It needs to be used in conjunction with many other measures for that real-time task; otherwise, it cannot reflect real-time determinism. In other words, when we discuss real-time tasks, we should discuss the real-time performance of the entire chain, not just one aspect of real-time determinism, as that would be meaningless.

Moreover, PREEMPT-RT is provided at the cost of sacrificing overall system performance, so it is important to recognize the pros and cons when using it.

PREEMPT-RT cannot achieve complete hard real-time determinism, so if facing product-level development, it is recommended to use a dual OS strategy of Linux + RTOS, where RTOS focuses on deterministic tasks and Linux focuses on complex computations (such as NN computations).