Abstract

Context switching is one of the most critical and sensitive performance paths in the Linux kernel, especially as modern CPUs introduce more security mitigations (such as Spectre V2, L1D Flush, and Branch Prediction Hardening), which further amplify the costs. SUSE’s Xie Yuanbin submitted a set of patches that optimize the code generation in the context switch path without altering the logic, significantly reducing the overhead of function calls and branch jumps. The main approach is to change key functions like <span><span>finish_task_switch()</span></span>, <span><span>enter_lazy_tlb()</span></span>, and <span><span>mmdrop_sched()</span></span> to <span><span>__always_inline</span></span>, ensuring they are inlined in the hot path. Empirical data shows that this optimization can achieve a 11% to 44% improvement in context switch speed on Intel platforms (depending on whether Spectre V2 mitigations are enabled). This article will analyze the principles of the patch, core code, applicable scenarios, and deep design logic.

Background: Why Can “Inlining” Speed Up Context Switching So Much?

Context switching is one of the most frequent and core system paths: when the scheduler switches from one task to another, it needs to perform MM switching, TLB handling, task time statistics, lock operations, and other functions.

However, modern CPU security mitigations have made this path more “fragile”:

• switch_mm_irq_off() may execute L1D Flush / branch prediction mitigations
• Instruction pipelines and caches may be flushed
• The first batch of instructions after switching is prone to “cold start”

In other words:the cost of executing each instruction after switching is higher than usual.Function calls and branch jumps are particularly “disastrous” performance killers.

Functions like <span><span>finish_task_switch()</span></span>, <span><span>enter_lazy_tlb()</span></span>, and <span><span>mmdrop()</span></span> are precisely located in this hottest path.

Therefore, the patch author proposed:since these functions are short, wouldn’t it be better to make them “always inline (__always_inline)”?

This is the core idea of the patch.

Core Idea of the Patch: Change Only the Compiler-Generated Code, Not the Logic

The author emphasizes in the patch description:

“The patch does not change the logic, but optimizes the compiler-generated code by adding inline attributes.”

There are three core reasons:

finish_task_switch() is in the hotspot path of context switching

Even at the <span><span>-O2</span></span> optimization level, it is still not inlined.

Pipeline/cache is in the worst state after context switching

At this point, any function call incurs a cost far higher than usual.

schedule() has the <span><span>__sched</span></span> attribute and is placed in the <span><span>.sched.text</span></span> section, while finish_task_switch() is not in that section

The binary distance generated by the two is quite far, leading to expensive jumps.

Therefore, the patch changes:

• <span><span>finish_task_switch</span></span>
• <span><span>enter_lazy_tlb</span></span>
• <span><span>mmdrop_sched</span></span>
• <span><span>tick_nohz_task_switch</span></span>
• <span><span>vtime_task_switch</span></span>

and several other functions to:

static __always_inline void finish_task_switch(struct task_struct *prev)

and also make several inline function sub-functions inline to avoid performance degradation caused by “parent function inlining while child functions are de-inlined.”

Key Patch Segments

enter_lazy_tlb(): From External Function → Always Inline

/* arch/x86/include/asm/mmu_context.h */

#ifndef MODULE
static __always_inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
{
    if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm)
        return;

    this_cpu_write(cpu_tlbstate_shared.is_lazy, true);
}
#endif

This function was originally in <span><span>arch/x86/mm/tlb.c</span></span> and was a non-inline function. It is now placed in the header file and forced to inline.

finish_task_switch(): The Most Critical Performance Hot Function → Forced Inline

/* kernel/sched/core.c */

static __always_inline struct rq *finish_task_switch(struct task_struct *prev)
{
    struct rq *rq = this_rq();
    ...
    return rq;
}

mmdrop_sched(): Avoiding Inline Relationship Disruption Leading to Performance Decline

/* include/linux/sched/mm.h */

static __always_inline void mmdrop_sched(struct mm_struct *mm)
{
    if (atomic_dec_and_test(&mm->mm_count))
        call_rcu(&mm->delayed_drop, __mmdrop_delayed);
}

All sched core and tick/vtime related functions are also inlined

For example:

static __always_inline void tick_nohz_task_switch(void)

and

static __always_inline bool vtime_accounting_enabled_this_cpu(void)

to minimize jump overhead in the context switch path.

Performance Data: Real Improvement of 11% to 44%

Measurement data from the author:

Time Taken by finish_task_switch() (measured using rdtsc)

Compiler / Configuration	Without Patch	With Patch
gcc	13.93 – 13.94	12.39 – 12.44
gcc + Spectre V2	24.69 – 24.85	13.68 – 13.73
clang	13.89 – 13.90	12.70 – 12.73
clang + Spectre V2	29.00 – 29.02	18.88 – 18.97

It can be seen that:

• Under normal circumstances: approximately 11% improvement
• When Spectre V2 mitigations are enabled: up to 44% improvement

This is a remarkable improvement.

It is worth noting that:

The patch does not change the kernel size (bzImage) and GCC and Clang builds are completely consistent before and after.

This indicates that the patch is “high yield, zero cost.”

Application Scenarios: Who Will Benefit the Most?

Systems with Frequent Context Switching

• High-load servers
• Container environments (with many tasks)
• VMs
• CI/CD build nodes

CPUs with Many Security Mitigations

For example, Intel server CPUs show significant performance degradation when Spectre V2 is enabled; this patch can “recover the losses.”

Applications with High Frequency Scheduling

• Network stacks (softirq / ksoftirqd)
• Databases (multiple workers)
• Actor models (such as Erlang/BEAM systems)
• High-speed I/O systems

In-Depth Analysis and Insights: Why Can “Inlining Optimization” Win So Much?

The “Pipeline Cold Start Effect” After Context Switching

During the context switch process, if the CPU triggers:

• L1D flush
• Branch predictor flush
• Instruction pipeline clearing

Then the execution cost of the next several instructions is much higher than usual.

Function calls → Branch jumps → Re-fetch → Re-predict → Extremely high cost

Therefore:

Reducing function calls can directly lower the real overhead of context switching.

“Spatial Locality Optimization” Related to the .sched.text Section

<span><span>schedule()</span></span> is placed in the <span><span>.sched.text</span></span> section, while other functions are placed in the regular section.

By inlining <span><span>finish_task_switch()</span></span>:

• It aggregates with <span><span>schedule()</span></span> in the same section
• Improves spatial locality
• Instruction prefetching is easier to hit

Reduces Compilation Complexity and Increases Compiler Global Optimization Space

After inlining, the compiler can perform:

• Constant propagation
• Function body merging
• Branch elimination
• Better register allocation

Thus optimizing to shorter, more compact code.

Does Not Affect Maintainability and Does Not Change Logic

The patch simply “tells the compiler to optimize harder,” with almost no conflict with readability and maintainability.

This is a type of patch that is “high yield, low risk.”

Conclusion: This is a Typical Optimization of “Sharpening the Knife”

This set of patches demonstrates a commonly overlooked but highly effective approach:

In extremely hot paths, compiler code generation is more important than code logic.

The context switch path is strongly influenced by modern CPU security mitigations, and any jumps or calls amplify the costs. By marking key functions as <span><span>__always_inline</span></span>, the patch successfully reduces the number of instructions in that path, allowing the CPU to perform less work at its most vulnerable moments.

The actual improvements achieved are:

• 11% (normal)
• 44% (with Spectre V2 enabled)

While the code logic remains completely unchanged.

In the future, this type of “compiler-level path optimization” may continue to expand to:

• sysenter related paths
• softirq entry
• TLB shootdown paths
• page fault hot paths

For kernel performance engineers, this is a category of optimization that is very worth paying attention to.

Reference link:

https://lore.kernel.org/all/[email protected]/