Achieving Sub-Microsecond Inter-Core Communication with Wanghu Real-Time Linux

This issue focuses on the sub-microsecond inter-core communication solution of Wanghu Real-Time Linux, which stabilizes inter-core communication latency between 120 nanoseconds and 180 nanoseconds, with jitter not exceeding 60 nanoseconds, supporting sub-microsecond real-time control tasks in more complex multi-core architectures.

Next, we will present the complete testing process and analyze how this solution achieves extremely low latency inter-core communication.

Testing Video

Technical Background

In the embedded and industrial fields, multi-core CPUs are becoming mainstream hardware platforms, providing more task concurrency and higher computational performance. Real-time operating systems typically perform CPU isolation and binding, and programs running on different CPU cores often require inter-core communication.

Linux systems have various mechanisms for Inter-Process Communication (IPC), and when it comes to communication and synchronization between different cores, the main methods are as follows:

• Shared Memory

Different CPU cores communicate and synchronize by accessing a segment of memory mapped to the same physical address, which is the mainstream method adopted by many high-performance middleware. Although shared memory is efficient, it may encounter cache contention, leading to cache misses, resulting in additional hundreds of nanoseconds of latency when accessing physical memory.

• Inter-Core Interrupts

Inter-core interrupts are primarily used by the Linux kernel as a low-level multi-core synchronization mechanism, which is then encapsulated into more forms for user use. Like other interrupts, inter-core interrupts involve CPU context switching, resulting in microsecond-level latency.

Design and Implementation Methods

As mentioned in previous articles in this series, based on isolator technology, an environment can be created on a specific CPU core without any context switching. We can use the same idea to implement a lower-latency inter-core communication method based on inter-core interrupts.

• Isolator Environment

No context switching occurs, and by default, no interrupts are triggered.

• Polling Inter-Core Interrupts

Mask CPU core interrupt responses, then poll the interrupt controller to determine if there is inter-core communication, assigning different meanings to different inter-core communication identifiers.

• Core Code Implemented in Assembly

The core code avoids accessing memory, primarily using CPU general-purpose registers to reduce cache miss issues.

Test Program Analysis

• The following Wanghu Real-Time Linux isolator APIs were used:

int isolator_start(unsigned affinity_cpu);// Start isolator function on specified CPU

uint64_t isolator_gettime(void);// Get current time (linear increase)

uint64_t isolator_spinto(uint64_t base, uint64_t wait_time);// Block in a loop until base+wait_time (ns), return end time

int isolator_rt_irq_pmd_run(unsigned irqnum, unsigned long flags, void (*func)(void));// Start blocking polling interrupt

• Code Analysis (gpio_test3_receive.c)

After the user loads insmod gpio_test3_receive.ko, it will call ipi_module_init(), which executes the following steps:

Step 1: Call API to start isolator function on cpu3

Step 2: Configure GPIO, including GPIO address mapping, multiplexing, and direction

Step 3: Call API to block and run the function to toggle GPIO

Among these, the user needs to implement the GPIO configuration in Step 2 and the GPIO toggle function in Step 3, minimizing memory access and other time-consuming operations.

• ConfigurationGPIO

The gpio_config function is used to configure GPIO pins and related registers, mainly completing IO mapping, multiplexing function settings, and configuring output modes.

• Map GPIO0 and PAD’s base address to the kernel virtual space using ioremap_np, which is a non-cached method.

• Configure the pin’s multiplexing function so that GPIO0_0 can switch to the corresponding GPIO mode.
• Set GPIO0_0 and GPIO1_12 as output pins.

• PeriodicTogglingGPIO

The gpio_toggle function is the callback for the inter-core interrupt, used to periodically output a square wave signal on GPIO0_0.

First, pull GPIO0_0 high to output a high signal. During the high signal period, insert a delay of 500 cycles using the NOP instruction. Then pull GPIO0_0 low, restoring the signal to low, while again inserting a delay of 500 cycles using the NOP instruction.

• Code Analysis (gpio_test3_send.c)

After the user loads insmod gpio_test3_send.ko, it will call ipi_module_init(), which executes the following steps:

Step 1: Call API to start isolator function on cpu2

Step 2: Initialize GPIO register address mapping

Step 3: Create a kernel thread to periodically toggle GPIO1_12 and send inter-core communication messages to cpu3

Among these, the user needs to implement the GPIO register address mapping in Step 2 and the periodic toggling of GPIO1_12 and sending inter-core communication messages to cpu3 in Step 3, minimizing memory access and other time-consuming operations.

• Initialize GPIO Address Mapping

The gpio_ioremap_init() function maps the physical base addresses of GPIO0 and GPIO1 to the kernel’s virtual address space using ioremap_np, disabling caching to ensure safe and accurate direct access to hardware registers.

• Periodic Toggling of GPIO and Sending Inter-Core Communication

• Set initialization parameters: inter-core communication register value val (SGI_ID=6, Target_cpu=3), wait event for 1000ns;

• Use API isolator_gettime() to get initialization time t0;

• Enter an infinite loop;

• Within the loop body, first pull GPIO1_12 high, then send an inter-core communication message to cpu3, followed by calling API isolator_spinto() to block until t0+wait_t, returning t1; then pull GPIO1_12 low, and again use API isolator_spinto() to block until t1+wait_t, returning t2. Finally, update t0 with t2 and enter the next loop iteration.

TestingResults

24-Hour Testing Results

In a 24-hour continuous operation test, the inter-core communication response latency of Wanghu Real-Time Linux remained stable between 120 nanoseconds and 180 nanoseconds, with latency jitter only 60 nanoseconds, successfully controlling the response latency at sub-microsecond levels.

Careful readers may notice that this result is slightly lower than the test data in “Linux Extreme Exploration IV.” The main reason is that the testing code was designed with a focus on usability and flexibility, intentionally sacrificing some real-time performance in the implementation.

Note:During the testing code or process, avoid frequent I/O operations on peripherals, as this can cause bus contention, thereby reducing real-time performance.

For the testing image and code download, please click“Read the Original”or

▼Openthe link below▼

https://www.onewos.com/resource-center/iso

We will continue to publish content on “Linux Extreme Exploration,” so please stay tuned. For more information, you can search for “Wanghu Real-Time Linux”

Wanghu Official Website

About Wanghu Real-Time Linux System

The Wanghu Real-Time Linux system is an embedded real-time operating system developed by Guoke Huanyu, characterized by strong real-time performance, high reliability, functional safety, intelligence, strong compatibility, and a domestic ecosystem. Unlike traditional microkernel real-time operating systems, it fully supports the Linux ecosystem while providing robust industry middleware support, helping customers quickly build industry solutions. The Wanghu Real-Time Linux system is committed to building a domestic industrial chain for basic software and hardware, adapting to mainstream domestic chips and hardware platforms, providing complete domestic solutions, ensuring functionality and performance while avoiding bottlenecks in critical areas.

Click the “Read the Original”below to download the testing image and testing code