Learning the Linux Kernel – System Calls

Info

❓ How can user applications safely and stably access underlying hardware and system resources in a multitasking and virtual memory operating system while preventing illegal operations?

Main Idea

By introducing system calls as the only legitimate intermediary between user processes and the kernel, we can abstract hardware, ensure system security, and achieve virtualization.

Core Analogy: A system call is like a government service hall.

• Users (applications) cannot directly enter the government building (kernel) and control specific departments (hardware), but must submit requests through the service hall window (system calls).
• The hall staff (kernel) will review your application (permissions) and help you process it, ensuring the entire government system (operating system) operates safely and orderly.

"Hardware Device" "Kernel" "Application (User Space)" "Hardware Device" "Kernel" "Application (User Space)" "1. Issue system call request" "2. Permission and rule check" "3. Access hardware or allocate resources" "4. Return operation result" "5. Return result or error"

User Layer API

Info

How can we provide a stable, cross-platform programming interface for applications in an operating system while allowing the underlying kernel to implement differences and flexibility, avoiding programmers from directly dealing with complex and unstable system calls?

Main Idea

By introducing APIs (Application Programming Interfaces) as an intermediary layer, we can hide the underlying details of system calls from programmers, thus achieving application portability and kernel flexibility.

Core Analogy: An API is like a car’s cockpit. The driver (programmer) only needs to learn how to use the steering wheel, accelerator, and brake (API) as standard interfaces to drive any car, without needing to worry about whether the engine under the hood is gasoline, diesel, or electric (different operating system kernel implementations and system calls); these underlying details are encapsulated by the cockpit (API).

Key Concepts

• System Call: The low-level functions provided by the kernel, which are the true entry points for applications to interact with the kernel.
• POSIX: A set of portable standards for Unix-like operating systems that define many commonly used APIs.
• C Library: Implements most standard APIs, including POSIX APIs, and encapsulates low-level system calls.

Kernel Space
User Space
Call API Function
Implemented by C Library
Encapsulate and Trigger
Execute Function
Application
C Library (glibc)
POSIX API
System Call
Kernel

System Calls

System Call Numbers

User mode and kernel mode are completely isolated. User mode processes cannot directly call kernel functions and need to translate through system calls. How can we ensure this translation is fast?

Main Idea

By assigning a unique system call number to each system call and using a system call table to associate this number with the actual function in the kernel, we provide a stable, indexable system call interface.

Key Steps:

1. Assign Numbers: During kernel compilation, each system call function (e.g., sys_getpid()) is assigned a unique system call number.
2. Build Index: All system call function pointers are organized in an array (table) named sys_call_table, where the index is the system call number.
3. User Call: When a user process initiates a system call through the C library, it passes the system call number to the kernel instead of the function name.
4. Kernel Lookup: After receiving the system call number, the kernel uses it as an index into the sys_call_table to directly find the corresponding function pointer and execute it.

Example

Using getpid as an example:

Kernel Space
User Space
Call getpid()
Pass system call number 39
As index
Index[39]
Invalid index
Execute and return
Return -ENOSYS
Application
C Library
System call number: 39
sys_call_table
sys_getpid() function
sys_ni_syscall()(error handling)

About Call Numbers

Why use call numbers instead of directly using function names?

Performance Overhead: If function names are used, the kernel must map the function name to the corresponding kernel function address through a complex hash table or string lookup when it receives a request. This is much slower than directly accessing the array via a numeric index (sys_call_table), which is a very fast O(1) time complexity.

Security Issues: User space can directly reference kernel function names, which may create opportunities for security vulnerabilities.

Compatibility and Stability

• Compile Compatibility: Applications are linked at compile time. If function names are used, any change in the name of a system call function may cause previously compiled programs to fail to run.
• Version Compatibility: The specific implementation of a system call may change in different versions of the kernel, but as long as its system call number remains unchanged, all old programs using it can continue to work normally.

System Call Handlers

How can user space safely and efficiently switch to kernel mode? And how can it pass the call number and parameters to the kernel?

Main Idea

By triggering an interrupt or special instruction to cause an exception, allowing the system to switch from user mode to kernel mode. Use registers to pass the system call number and parameters and use a lookup table to find the kernel function.

Key Steps

• Prepare Parameters: In user space, the application places the system call number and all parameters into specified CPU registers.
• Trigger Trap: The application executes a special instruction (e.g., int $0x80), triggering a soft interrupt that switches the CPU from user mode to kernel mode.
• Kernel Processing: The kernel’s exception handler (system_call()) begins execution, reading the system call number and parameters from the registers.
• Lookup and Execute: The kernel uses the system call number as an index to look up and call the corresponding kernel function in the sys_call_table.
• Return Result: After executing the system call, the kernel places the return value into a register (e.g., eax), then switches back to user mode, resuming the application’s execution.

Example

Preparing Parameters:

• The C library of the user program prepares parameters: file descriptor, buffer address, write length.
• The system call number for write() is placed in the eax register.
• The first five parameters (file descriptor, buffer, length) are placed in the ebx, ecx, edx registers.Trigger Trap:
• The C library executes the int $0x80 instruction. Soft InterruptKernel Processing:
• The kernel’s system_call() function is called, reading the system call number from eax and parameters from ebx, ecx, edx.Lookup and Execute:
• system_call() looks up <code>sys_call_table and calls the <code>sys_write() function.
• sys_write() performs the file write operation and returns the number of bytes written.Return Result:
• The kernel places the return value in the eax register, then returns to user mode.

Related Concepts

Soft Interrupt/Exception: A special event triggered by software to interrupt the current program execution flow and switch to kernel mode.int $0x80: The instruction used to trigger a soft interrupt on x86 systems, typically used for system calls.

Two Core Concepts

• system_call(): The system call handler in the kernel, which is the unified entry point for all system calls.
• sys_call_table: An array that stores all system call function pointers for quick lookup and execution.

Info

Since it is a soft interrupt, does it first go through the soft interrupt handler, enter the interrupt handling task, and then call system_callNot entirely correct

Because: The handler for interrupt vector number 128 is system_call() (in newer kernels, it is <code>entry_SYSCALL_64).

Flowchart (Important)

"sys_call_table" "Kernel (system_call())" "CPU Registers" "User Space (C Library)" "sys_call_table" "Kernel (system_call())" "CPU Registers" "User Space (C Library)" "1. Store system call number (eax)" "2. Store parameters (ebx, ecx...)" "3. int $0x80 or sysenter" "4. Read register contents" "5. Lookup by system call number" "6. Return function address" "7. Execute corresponding kernel function" "8. Store return value (eax)" "9. Switch back to user space"

Related Source Code

ENTRY(entry_SYSCALL_64)
    UNWIND_HINT_EMPTY
    /*
     * Interrupts are off on entry.
     * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON,
     * it is too small to ever cause noticeable irq latency.
     */

    swapgs
    /* tss.sp2 is scratch space. */
    movq    %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
    SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
    movq    PER_CPU_VAR(cpu_current_top_of_stack), %rsp

    /* Construct struct pt_regs on stack */
    pushq    $__USER_DS                /* pt_regs-&gt;ss */
    pushq    PER_CPU_VAR(cpu_tss_rw + TSS_sp2)    /* pt_regs-&gt;sp */
    pushq    %r11                    /* pt_regs-&gt;flags */
    pushq    $__USER_CS                /* pt_regs-&gt;cs */
    pushq    %rcx                    /* pt_regs-&gt;ip */

How to Design a System Call Interface

How to design a new Linux system call interface that has good backward compatibility and excellent extensibility?

Main Idea

==Provide mechanisms rather than policies==

• Each system call has only one clear purpose
• Use flag parameters to handle functional extensions

Example

A clear purpose

• Create a separate chown() system call specifically for changing ownership.
• Create a separate chmod() system call specifically for changing permissions.Flag parameters for extension
• If in the future an option is needed, such as “recursively change permissions”, a flag parameter (e.g., <code>CHMOD_RECURSIVE) can be added to <code>chmod() instead of creating a new <code>chmod_recursive() system call.
• open() system call is a best practice: Its flags<code> parameter (e.g., <code>O_RDONLY, <code>O_CREAT, <code>O_EXCL) provides rich functionality through flags while maintaining the simplicity and stability of the system call interface.

Parameter Protection

When a system call is invoked, how can we ensure that the parameters passed are valid and safe, preventing malicious calls from damaging the kernel process?

Main Idea

==Strict Parameter Validation and Permission Checks==

System calls ensure safety through parameter validation, data copying, and permission checks.

• copy_from_user and copy_to_user are responsible for safely moving data between user space and kernel space, performing legality checks on pointers internally.
• capable() function checks whether the process has the necessary capabilities to perform the operation, completing the final permission validation. If the system call’s parameter checks fail, it returns -EFAULT error, which is equivalent to a validation error and will not cause a SIGSEGV segmentation fault.

Example

#include &lt;stdio.h&gt;
#include &lt;unistd.h&gt;

int main() {
    // This is an illegal address that does not belong to any valid memory area of the process.
    char *illegal_ptr = (char *)0x12345678;

    // Attempting to directly access this illegal address, such as writing a character to it.
    // This line of code will cause the CPU to attempt to write 'H' to address 0x12345678.
    // The Memory Management Unit (MMU) will find that this address has no corresponding page table mapping,
    // thus triggering a hardware exception, causing the operating system to send a SIGSEGV signal to the process.
    //*illegal_ptr = 'H';

    // If the program does not crash on the previous line, it will attempt to call write(). write() will fail and return -1, setting errno to EFAULT
    write(1, illegal_ptr, 10);

    return 0;
}

Warning

Avoid unnecessary new system calls Although creating system calls is relatively easy, due to the maintenance costs, interface solidification, and compatibility issues it brings, developers should avoid unnecessary new system calls and choose more appropriate and flexible alternatives.

New system calls should not be created lightly; instead, priority should be given to using existing, general kernel interfaces (such as device nodes, file system interfaces, or ioctl()) to maintain the simplicity and stability of the system call interface.

Learning the Linux Kernel – System Calls

Main Idea

User Layer API

Main Idea

Key Concepts

System Calls

System Call Numbers

Main Idea

Example

About Call Numbers

System Call Handlers

Main Idea

Key Steps

Example

Related Concepts

Flowchart (Important)

Related Source Code

How to Design a System Call Interface

Main Idea

Example

Parameter Protection

Main Idea

Example

Leave a Comment Cancel reply