In-Depth Understanding of Linux Soft Interrupts

1. Why Do We Need Soft Interrupts? — The “Efficiency Balancing Act” of Interrupt Handling

In Linux systems, hardware devices (such as network cards, hard drives, keyboards) send interrupt signals to the CPU to request attention, for example, “a network card has received new data” or “a key has been pressed on the keyboard.” When the CPU responds to an interrupt, it pauses the current task and executes the corresponding interrupt handler.

However, there is a critical contradiction here:

The priority of hardware interrupts is extremely high and must be responded to quickly (otherwise, data may be lost, such as buffer overflow in the network card);
Some interrupt handling logic is complex (such as network data parsing, writing data to disk), and if executed entirely within the hardware interrupt, it can lead to the CPU being unable to return to the original task for an extended period, slowing down system response (i.e., the risk of an “interrupt storm”).

To resolve this contradiction, Linux splits interrupt handling into Top Half and Bottom Half, with soft interrupts (Softirq) being the core implementation mechanism of the bottom half — the top half handles critical steps quickly, while the bottom half completes subsequent work more leisurely, achieving a balance of “efficient response + low latency” through collaboration.

2. Top Half vs Bottom Half: The “Division of Labor” in Interrupt Handling

To understand soft interrupts, it is essential to clarify their core differences from the top half, which can be summarized in a table:

Feature	Top Half (Hardware Interrupt)	Bottom Half (Soft Interrupt)
Trigger Method	Hardware signal (e.g., network card, timer)	Triggered by the top half (software-level scheduling)
Execution Timing	Executed immediately, interrupting the current task	Executed with delay, scheduled when the CPU is idle
Execution Environment	Interrupts are disabled (to avoid nesting)	Interrupts are allowed (does not block other requests)
Core Objective	Quickly handle critical logic (e.g., saving hardware state, clearing interrupt flags)	Asynchronously handle time-consuming operations (e.g., data parsing, memory copying)
Typical Example	Interrupt response when the network card receives data	Parsing data in the network protocol stack, handling timer expirations

In simple terms: The top half is responsible for “receiving the call,” while the bottom half is responsible for “digesting” the data, with soft interrupts being the core vehicle for the “digestion” process.

3. Core Features and Types of Linux Soft Interrupts

1. Core Features of Soft Interrupts

Asynchronous Execution: After being triggered by the top half, soft interrupts do not execute immediately but wait for the CPU’s “soft interrupt scheduling opportunity” (e.g., after the current process scheduling is complete, or after hardware interrupt handling ends);
Reentrant: Soft interrupt handlers need to support concurrent execution (the same type of soft interrupt may run simultaneously on multiple CPUs), thus ensuring thread safety (e.g., using atomic operations, spinlocks);
Higher Priority than Processes: The priority of soft interrupts is between hardware interrupts and user processes, preempting user processes but not preempting hardware interrupts;
Static Registration: Soft interrupt types are statically defined at kernel compile time and cannot be dynamically added (unlike work queues, which support dynamic creation).

2. Common Types of Soft Interrupts in Linux

The Linux kernel defines fixed types of soft interrupts (viewable in include/linux/interrupt.h), with core types including:

HI_SOFTIRQ: High-priority soft interrupt (the highest priority soft interrupt, used for handling latency-sensitive tasks);
TIMER_SOFTIRQ: Timer soft interrupt (handles timer expiration events, such as process scheduling timers, delayed tasks);
NET_TX_SOFTIRQ: Network transmit soft interrupt (handles the logic for sending network data, such as encapsulating application layer data before sending it to the network card);
NET_RX_SOFTIRQ: Network receive soft interrupt (handles data received by the network card, such as parsing network protocols and delivering data to the application layer);
SCHED_SOFTIRQ: Scheduling soft interrupt (handles tasks related to process scheduling, such as updating process priorities and triggering the scheduler);
RCU_SOFTIRQ: RCU lock soft interrupt (handles garbage collection for the RCU (Read-Copy-Update) mechanism, such as releasing memory protected by RCU);
TASKLET_SOFTIRQ: Tasklet soft interrupt (a lightweight bottom half mechanism implemented based on soft interrupts, commonly used for simple asynchronous tasks).

Among these, network transmission and reception (NET_TX/RX) and timers (TIMER) are the most commonly used types of soft interrupts, and they are also the main sources of soft interrupt load in the system.

4. How to Observe Soft Interrupts? — Practical Tools and Analysis Methods

Linux provides intuitive tools to view the running status of soft interrupts, primarily through the /proc/softirqs file, which can be quickly analyzed in conjunction with tools like top to locate soft interrupt issues.

1. View Soft Interrupt Statistics: /proc/softirqs

The /proc/softirqs file records the number of triggers for each type of soft interrupt on each CPU core since the system was booted, with example output as follows:

                    CPU0       CPU1       CPU2       CPU3
          HI:          0          0          0          0
       TIMER:    1234567     987654     876543     765432
      NET_TX:      12345       67890      54321      43210
      NET_RX:    2345678    3456789    4567890    5678901
       SCHED:     654321     543210     432109     321098
         RCU:    9876543    8765432    7654321    6543210

Each column corresponds to a CPU core (e.g., CPU0, CPU1), and each row corresponds to a type of soft interrupt;
The values indicate the number of triggers for that type of soft interrupt on the corresponding CPU, with a rapid increase in counts possibly indicating a soft interrupt storm (e.g., a network card receiving a large amount of data causing a spike in NET_RX_SOFTIRQ).

2. Real-time Monitoring of Soft Interrupts: top Command

In the top command, pressing 1 displays the load of all CPU cores, where the si column indicates the percentage of CPU occupied by soft interrupts:

%Cpu0  :  2.0 us,  1.0 sy,  0.0 ni, 95.0 id,  0.0 wa,  0.0 hi,  2.0 si,  0.0 st
%Cpu1  :  3.0 us,  1.5 sy,  0.0 ni, 93.5 id,  0.0 wa,  0.0 hi,  2.0 si,  0.0 st

A high si value (e.g., over 50%) indicates that soft interrupts are consuming a significant amount of CPU resources, potentially causing user processes to respond slowly;
Combining this with /proc/softirqs can further pinpoint which type of soft interrupt (e.g., NET_RX) is being triggered excessively on which CPU.

5. Practical Applications and Troubleshooting of Soft Interrupts

1. Typical Application Scenario: Network Card Receiving Data

Taking the example of a network card receiving data, the complete interrupt handling process is as follows:

When the network card receives data, it sends a hardware interrupt to the CPU;
The top half (hardware interrupt handler) executes:

Save the address of the network card’s receive buffer, data length, and other states;
Clear the interrupt flag of the network card (to avoid repeated triggering);
Trigger the NET_RX_SOFTIRQ soft interrupt;

The top half quickly returns, and the CPU goes back to the original task;
When the CPU schedules the soft interrupt, it executes the handler for NET_RX_SOFTIRQ:

Copy data from the network card buffer to kernel space;
Parse the network protocol (e.g., IP, TCP/UDP);
Deliver the data to the application layer (e.g., through socket interfaces);

The soft interrupt handling is complete, and the CPU continues executing user processes.

In this process, the top half completes critical operations in microseconds, while time-consuming data copying and protocol parsing are handled asynchronously by the soft interrupt, ensuring that network card data is not lost and does not block other tasks.

2. Common Issues: Soft Interrupt Storms and Troubleshooting

When a certain type of soft interrupt is triggered too frequently (e.g., a network card receiving millions of data packets per second), it can cause the si value to spike, with the CPU being occupied by soft interrupts for extended periods, preventing user processes from obtaining sufficient CPU time, leading to slow system response — this is known as a soft interrupt storm.

Troubleshooting steps:

Use the top command to check the si column and confirm that soft interrupts are consuming too much;
Check /proc/softirqs to locate the type of soft interrupt that is increasing the fastest (e.g., NET_RX) and the corresponding CPU core;
Further investigate the cause:

If it is NET_RX_SOFTIRQ: Check if the network card is receiving a large amount of abnormal traffic (e.g., DDoS attack), which can be analyzed through tcpdump packet capture;
If it is TIMER_SOFTIRQ: Check if there are a large number of timer tasks (e.g., if the application layer has created too many timer tasks);

Optimization solutions:

Disable unnecessary interrupts (e.g., disable unused network card interrupts);
Adjust CPU affinity (bind soft interrupts to specific CPU cores to avoid cross-core scheduling overhead);
Optimize applications (e.g., reduce unnecessary network requests, set timers reasonably).

6. Differences Between Soft Interrupts and Other Bottom Half Mechanisms

In addition to soft interrupts, Linux has two other commonly used bottom half mechanisms: tasklets and workqueues. The core differences among the three are as follows:

Mechanism	Underlying Implementation	Concurrency Features	Applicable Scenarios
Soft Interrupt	Statically registered, executed directly	Same type can be executed concurrently on multiple CPUs	High-performance, high-concurrency scenarios (e.g., network transmission and reception)
Tasklet	Encapsulated based on soft interrupts	Same tasklet cannot be executed concurrently	Simple asynchronous tasks (e.g., delayed processing in drivers)
Workqueue	Based on kernel threads	Serial execution (default)	Long-running tasks (e.g., disk IO, sleep operations)

In summary:

Use soft interrupts for extreme performance;
Use tasklets for simple asynchronous tasks;
Use workqueues for tasks that require sleeping or are time-consuming.

7. Simplified Kernel Implementation Details of Soft Interrupts

For readers who want to delve deeper into kernel mechanisms, here is a brief breakdown of the core execution flow of soft interrupts (based on the Linux 5.x kernel), without needing to dive into assembly level, focusing on understanding the scheduling logic:

1. Triggering and Scheduling Timing of Soft Interrupts

Trigger Method: The top half triggers a specific type of soft interrupt by calling the raise_softirq(unsigned int nr) function (where nr is the soft interrupt type number, such as NET_RX_SOFTIRQ);
Scheduling Trigger Points: The kernel checks and executes soft interrupts at the following times:

After the hardware interrupt handler ends (by calling do_IRQ() which invokes invoke_softirq());
During process scheduling (before and after executing the schedule() function);
When a kernel thread actively triggers it (e.g., the ksoftirqd kernel thread, which handles backlogged soft interrupts);
After a user process transitions to kernel mode and before returning to user mode (syscall_exit_to_user_mode()).

2. Execution Flow of Soft Interrupts

// Simplified soft interrupt execution logic
void invoke_softirq(void) {
    __u32 pending;
    // 1. Get the pending soft interrupts for the current CPU (using atomic operations to avoid contention)
    pending = local_softirq_pending();
    if (!pending) return;
    
    // 2. Execute the soft interrupt handlers (loop through all pending soft interrupt types)
    do_softirq();
}

void do_softirq(void) {
    __u32 pending;
    int max_restart = MAX_SOFTIRQ_RESTART; // Default 10 times
    
    while ((pending = local_softirq_pending()) && max_restart--) {
        // 3. Clear the pending flag (to avoid re-execution)
        local_bh_disable();
        // 4. Iterate through all soft interrupt types and execute the corresponding handler
        trace_softirq_entry(pending);
        hrtimer_run_queues(); // Handle high-resolution timers
        softirq_handle_pending(); // Execute soft interrupt handlers
        local_bh_enable();
    }
    
    // 5. If too many soft interrupts are pending, wake up the ksoftirqd kernel thread to handle them
    if (pending && !ksoftirqd_running())
        wakeup_softirqd();
}

Key points:

Soft interrupts obtain the current CPU’s pending queue through the local_softirq_pending() function (each CPU has an independent queue to avoid cross-CPU contention);
A maximum of 10 iterations of soft interrupt handlers are executed (MAX_SOFTIRQ_RESTART) to prevent soft interrupts from occupying the CPU indefinitely;
If there are still pending interrupts after 10 iterations, the ksoftirqd kernel thread (with a lower priority) is woken up to handle them in the background, ensuring that user processes can obtain CPU time.

3. Role of the ksoftirqd Kernel Thread

ksoftirqd is a kernel thread corresponding to each CPU core (named ksoftirqd/N, where N is the CPU number), with the primary responsibilities:

Handle backlogged soft interrupts (when do_softirq() loops 10 times and still has unexecuted soft interrupts);
Avoid soft interrupts from preempting user processes for extended periods (ksoftirqd has a priority of MAX_RT_PRIO-20, lower than ordinary real-time processes but higher than ordinary user processes);
You can check the status of this thread using ps aux | grep ksoftirqd; if its CPU usage is too high, it indicates a serious backlog of soft interrupts in the system.

8. Advanced Optimization Practices for Soft Interrupts

In high-concurrency scenarios (such as high-traffic web servers, database servers), optimizing soft interrupts can significantly enhance system throughput and response speed. Here are practical optimization solutions:

1. Interrupt Balancing and CPU Affinity Configuration

By default, Linux uses the irqbalance service to evenly distribute hardware interrupts across CPU cores, but soft interrupts inherit the CPU affinity of hardware interrupts (e.g., if the network card interrupt is bound to CPU0, the corresponding NET_RX_SOFTIRQ will primarily execute on CPU0), which may lead to excessive load on a single CPU.

Optimization steps:

Stop the irqbalance service (when manually configuring affinity): systemctl stop irqbalance && systemctl disable irqbalance;
Check the network card interrupt number: cat /proc/interrupts | grep eth0 (assuming the network card is eth0), example output:

123: 1234567 0 0 0  IR-PCI-MSI 1048576-edge  eth0

Where 123 is the interrupt number;

Bind the network card interrupt to specific CPU cores: echo 0-3 > /proc/irq/123/smp_affinity_list (bind the interrupt to CPU0-CPU3, supporting multi-CPU concurrent processing);
Verify soft interrupt distribution: Check /proc/softirqs to confirm that NET_RX_SOFTIRQ is growing evenly across multiple CPUs.

2. Disable Unnecessary Types of Soft Interrupts

For unnecessary types of soft interrupts (e.g., in embedded systems where network functionality is not required), they can be disabled through kernel configuration to reduce scheduling overhead:

When compiling the kernel, disable the corresponding options in make menuconfig:
Network-related soft interrupts: Networking support -> Networking options -> Remove NET_RX/SOFTIRQ (adjust according to kernel version);
RCU soft interrupt: Kernel configuration -> RCU Subsystem -> Disable RCU_SOFTIRQ (only applicable in scenarios without RCU dependencies);
Note: This optimization is only suitable for customized kernel scenarios; it is not recommended to disable arbitrarily on general servers.

3. Adjust Soft Interrupt Trigger Thresholds

For high-traffic network scenarios, kernel parameters can be adjusted to optimize soft interrupt trigger logic:

net.core.netdev_budget: The maximum number of packets processed per NET_RX_SOFTIRQ (default 300); increasing this value (e.g., adjusting to 1000) can reduce the number of soft interrupt triggers but will increase the execution time of each soft interrupt;
net.core.netdev_budget_usecs: The maximum execution time for each NET_RX_SOFTIRQ (in microseconds, default 200), to avoid a single soft interrupt execution taking too long;
Configuration method: echo “net.core.netdev_budget=1000” >> /etc/sysctl.conf && sysctl -p.

4. Use XDP (eXpress Data Path) to Bypass Soft Interrupts

XDP is a high-performance packet processing framework provided by the Linux kernel, allowing data packets to be processed directly at the network card driver level, bypassing NET_RX_SOFTIRQ and the network protocol stack, suitable for scenarios requiring extreme performance (e.g., DDoS protection, traffic forwarding):

Advantages: Low latency (microsecond level), high throughput (supports tens of millions of packets per second);
Applicable scenarios: Scenarios that do not require complex protocol parsing (e.g., simple packet filtering, forwarding);
Tools: XDP programs can be configured using the iproute2 tool or custom processing logic can be implemented using bpftool to load BPF programs.

9. Common Misconceptions and Clarifications

1. Misconception 1: Soft interrupts are “software-triggered interrupts” and have nothing to do with hardware

Clarification: The source of soft interrupts is usually hardware interrupts (triggered by the top half), but they can also be triggered actively by software (e.g., kernel threads calling raise_softirq()). The core difference lies in execution timing and priority, not the source of the trigger.

2. Misconception 2: The higher the priority of soft interrupts, the faster they are processed

Clarification: The priority of soft interrupts only determines the scheduling order (e.g., HI_SOFTIRQ executes before NET_RX_SOFTIRQ), but processing speed depends on the complexity of the handler logic and system load. A high-priority soft interrupt with complex logic may still lead to backlogs of lower-priority soft interrupts.

3. Misconception 3: Soft interrupts can sleep

Clarification: Soft interrupt handlers run in atomic context and are not allowed to sleep (e.g., calling the schedule() function or using mutex locks), as this would lead to kernel deadlock. If sleeping is required, the workqueue mechanism should be used.

4. Misconception 4: High values in /proc/softirqs indicate problems

Clarification: High values in /proc/softirqs only indicate frequent triggering of soft interrupts; whether there is a problem should be judged in conjunction with the si column in the top command (CPU usage by soft interrupts). For example, a high NET_RX_SOFTIRQ value on a high-traffic server is expected, but as long as the si usage is below 30% and the system response is normal, it is considered normal.

10. Conclusion

Soft interrupts are the “efficiency balancer” in Linux interrupt handling, splitting the top half and bottom half to ensure rapid response to hardware interrupts while avoiding complex logic from blocking the system. Its design philosophy (asynchronous collaboration, priority scheduling, concurrency safety) is also applicable to other operating systems and high-performance program design.