In-Depth Analysis of Memory Mapping in Linux Device Drivers

1. In-Depth Analysis of Working Principles

Memory mapping (Memory Mapping, <span>mmap</span>) is a method that directly maps device memory or driver kernel buffers into the user process address space. The core idea is: to establish a direct mapping from a virtual memory area (VMA) to physical memory (or device memory) by manipulating the process’s page table.

1.1 Core Ideas and Advantages

• • Zero-Copy: Avoids explicit data copying between user space and kernel space (such as <span>read</span>/<span>write</span> system calls). User processes can directly access device memory via pointers, significantly improving the efficiency of operations involving large amounts of data.
• • Direct Access: Performance-sensitive applications (such as video processing and network packet processing) can directly manipulate device buffers.
• • Reduced System Call Overhead: Once the mapping is established, accessing data does not require entering the kernel.

1.2 mmap System Call Process

Device/Physical Memory Device Driver Kernel User Process Device/Physical Memory Device Driver Kernel User Process 1. Establish page table mapping 2. Possibly use remap_pfn_range to complete mapping CPU generates page fault (Page Fault) loop [subsequent access] mmap(..., fd, offset, ...) calls driver file's .mmap operation returns VMA configuration status returns user space virtual address read/write mapped memory page fault handler checks VMA validity completed by VMA operation function (e.g., page fault handling) returns data/writes data

The above diagram illustrates the core process of <span>mmap</span>:

1. 1. The user calls <span>mmap</span>, passing a file descriptor <span>fd</span>.
2. 2. The kernel finds the corresponding <span>file_operations</span> structure based on <span>fd</span> and calls its <span>.mmap</span> method.
3. 3. The driver’s <span>.mmap</span> method is responsible for establishing the page table mapping, usually using <span>remap_pfn_range</span> or <span>io_remap_pfn_range</span> functions.
4. 4. After successful mapping, access to that segment of virtual address in user space will be directly converted by the MMU through the page table to access device memory/physical memory.

1.3 Address Space and Page Table Management

Linux uses a hierarchical page table structure (PGD -> P4D -> PUD -> PMD -> PTE) to manage the conversion from virtual addresses to physical addresses. <span>mmap</span><span>'s essence is to create a PTE entry in the current process's page table for the specified VMA, pointing it to the device-specific physical address (PFN).</span>

2. Implementation Mechanism and Code Framework

2.1 Core Roles of the Driver

Driver developers need to implement two core parts:

1. 1. <span>file_operations</span> structure’s <span>.mmap</span> method.
2. 2. Provide and manage physical memory available for mapping (which may be device RAM, DMA buffers, or pages allocated by the kernel).

2.2 Core Data Structures

Data Structure	Description	Role in mmap
<span>struct vm_area_struct</span> (VMA)	Describes the attributes of a segment of virtual memory area for a process (such as starting address, length, permissions, etc.).	Parameter of the driver’s <span>.mmap</span> method, allowing the driver to understand the area the user wants to map and set its attributes.
<span>struct page</span>	Represents a physical page frame kernel abstraction.	Basic unit of memory management. <span>remap_pfn_range</span><span> internally handles the conversion from </span><code><span>page</span> to physical address.
<span>struct file_operations</span>	Structure containing function pointers for all file operations provided by the driver.	The <span>.mmap</span> member must point to the driver’s implementation of the <span>my_mmap</span> function.

2.3 Core API Functions

Function	Description	Usage Scenario
<span>int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, unsigned long size, pgprot_t prot);</span>	The most critical function. Used to establish a page table mapping from a contiguous physical address (specified by PFN) to the user space VMA.	Mapping contiguous physical addresses of memory, such as RAM on the device board, reserved physical memory, or large pages allocated by the kernel’s <span>alloc_pages</span>.
<span>void *vmalloc(unsigned long size);</span><span>void vfree(void *addr);</span>	Allocates virtually contiguous addresses but possibly non-contiguous physical addresses of memory.	When large blocks of memory are needed but physical continuity cannot be guaranteed. Cannot be directly mapped using <span>remap_pfn_range</span>, requires page-by-page mapping.
<span>struct page *alloc_pages(gfp_t gfp_mask, unsigned int order);</span><span>void __free_pages(struct page *page, unsigned int order);</span>	Allocates physically contiguous pages. Returns a pointer to <span>struct page</span>.	Allocating physically contiguous memory for mapping. PFN can be obtained via <span>page_to_pfn</span> for use in <span>remap_pfn_range</span>.
<span>int io_remap_pfn_range(struct vm_area_struct *vma, ...);</span>	<span>remap_pfn_range</span>‘s variant for mapping I/O memory (i.e., regions mapped via <span>ioremap</span>).	Mapping device MMIO (Memory-Mapped I/O) regions. Ensure to use the correct memory attributes (e.g., uncached).

3. Simplest Complete Example Source Code

Below is the simplest working character device driver example that allocates a page of physical memory and maps it to user space.

3.1 Driver Code (<span>mmap_demo.c</span>)

#include <linux/module.h>
#include <linux/fs.h>        // file_operations, alloc_chrdev_region
#include <linux/cdev.h>      // cdev
#include <linux/slab.h>      // kmalloc, kfree
#include <linux/mm.h>        // remap_pfn_range, vm_area_struct
#include <linux/version.h>

#define DEVICE_NAME "mmap_demo"

static int major;
static struct cdev my_cdev;
static char *kernel_buffer;  // Pointer to kernel buffer
static dma_addr_t phys_addr; // Physical address of the buffer (actually the physical address corresponding to the kernel logical address)
static size_t buffer_size = PAGE_SIZE; // Buffer size is one page

static int my_mmap(struct file *filp, struct vm_area_struct *vma)
{
    unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
    unsigned long pfn_start;
    unsigned long size = vma->vm_end - vma->vm_start;
    int ret;

    // 1. Simple boundary and offset check
    if (offset + size > buffer_size) {
        return -EINVAL;
    }

    // 2. Prevent caching (as needed). Usually not required for normal memory.
    // vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

    // 3. Calculate the starting physical page frame number (PFN) to be mapped
    // virt_to_phys converts the kernel logical address to a physical address
    pfn_start = (virt_to_phys(kernel_buffer) + offset) >> PAGE_SHIFT;

    // 4. Establish mapping
    ret = remap_pfn_range(vma, vma->vm_start, pfn_start, size, vma->vm_page_prot);
    if (ret) {
        dev_err("mmap failed: %d\n", ret);
        return -EAGAIN;
    }

    printk(KERN_INFO "mmap called: vma=0x%lx-%lx, pfn=0x%lx\n",
           vma->vm_start, vma->vm_end, pfn_start);
    return 0;
}

static struct file_operations fops = {
    .owner = THIS_MODULE,
    .mmap = my_mmap,
};

static int __init my_init(void)
{
    dev_t dev;

    // 1. Dynamically allocate major device number
    if (alloc_chrdev_region(&dev, 0, 1, DEVICE_NAME) < 0) {
        return -1;
    }
    major = MAJOR(dev);

    // 2. Allocate one page of physical memory (GFP_KERNEL)
    kernel_buffer = (char *)__get_free_pages(GFP_KERNEL, 0); // order=0 (2^0=1 page)
    if (!kernel_buffer) {
        goto fail_no_mem;
    }
    // Initialize memory content
    sprintf(kernel_buffer, "Hello from kernel space! PID: %d\n", current->pid);
    // Get its physical address (for demonstration only, PFN is calculated during actual mapping)
    phys_addr = virt_to_phys(kernel_buffer);

    printk(KERN_INFO "Allocated kernel buffer: virt=%p, phys=0x%llx\n",
           kernel_buffer, (unsigned long long)phys_addr);

    // 3. Initialize and register character device
    cdev_init(&my_cdev, &fops);
    my_cdev.owner = THIS_MODULE;
    if (cdev_add(&my_cdev, dev, 1) < 0) {
        goto fail_cdev_add;
    }

    printk(KERN_INFO "MMAP demo module loaded with major %d\n", major);
    return 0;

fail_cdev_add:
    free_pages((unsigned long)kernel_buffer, 0);
fail_no_mem:
    unregister_chrdev_region(MKDEV(major, 0), 1);
    return -1;
}

static void __exit my_exit(void)
{
    dev_t dev = MKDEV(major, 0);

    cdev_del(&my_cdev);
    unregister_chrdev_region(dev, 1);
    free_pages((unsigned long)kernel_buffer, 0);

    printk(KERN_INFO "MMAP demo module unloaded\n");
}

module_init(my_init);
module_exit(my_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");

3.2 User Space Test Program (<span>test_mmap.c</span>)

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>

#define DEVICE_PATH "/dev/mmap_demo" // Must match the device node created by the driver

int main()
{
    int fd;
    char *mapped_buffer;
    char new_data[] = "Message from user space via MMAP!";

    // 1. Open device file
    fd = open(DEVICE_PATH, O_RDWR);
    if (fd < 0) {
        perror("Failed to open device");
        exit(EXIT_FAILURE);
    }

    // 2. Call mmap
    // MAP_SHARED: Modifications to the mapping will be written back to the device/file
    mapped_buffer = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (mapped_buffer == MAP_FAILED) {
        perror("mmap failed");
        close(fd);
        exit(EXIT_FAILURE);
    }

    printf("Device mapped at address: %p\n", mapped_buffer);

    // 3. Directly read the mapped memory (data from driver initialization)
    printf("Read from mmap: %s", mapped_buffer);

    // 4. Directly write to the mapped memory (modify device memory)
    strncpy(mapped_buffer, new_data, sizeof(new_data));
    printf("Wrote to mmap: %s\n", mapped_buffer);

    // 5. Read again to confirm successful write
    printf("Read again from mmap: %s", mapped_buffer);

    // 6. Cleanup
    munmap(mapped_buffer, getpagesize());
    close(fd);

    return 0;
}

3.3 Compilation and Testing Steps

1. 1. Compile the driver (requires kernel header files)

   obj-m += mmap_demo.o
   all:
       make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

   make

2. 2. Compile the test program

   gcc -o test_mmap test_mmap.c

3. 3. Load the driver and create the device node

   sudo insmod mmap_demo.ko
   # View dynamically allocated major device number
   dmesg | tail
   # Assuming major device number is 250, create device node
   sudo mknod /dev/mmap_demo c 250 0
   sudo chmod 666 /dev/mmap_demo # For testing convenience

4. 4. Run the test program

   ./test_mmap

   The output should be similar to:

   Device mapped at address: 0x7f8a5c000000
   Read from mmap: Hello from kernel space! PID: 1234
   Wrote to mmap: Message from user space via MMAP!
   Read again from mmap: Message from user space via MMAP!

5. 5. Check kernel logs

   dmesg | tail

   You can see the allocated address and <span>mmap</span><span> call information printed by the driver.</span>

4. Common Tools, Commands, and Debugging Methods

Tool/Command	Description	Example/Usage
<span>dmesg</span>	View kernel ring buffer logs.	`dmesg`
<span>mmap</span> related Tracepoints	Dynamic tracing of <span>mmap</span> related kernel functions.	<span>sudo perf probe --add 'do_mmap’</span><span>sudo perf record -e probe:do_mmap -a</span>
<span>/proc/<pid>/maps</span>	View the virtual memory mapping layout of a specified process.	<span>cat /proc/$$/maps</span> View the memory mapping of the current shell. Print its PID in the test program, then check the maps file of that PID to confirm if the mapping area exists.
<span>/proc/<pid>/pagemap</span>	(requires root) View the mapping relationship from process virtual addresses to physical addresses.	Scripts can be written to verify whether user space addresses are indeed mapped to the physical addresses specified by the driver.
<span>devmem2</span>	Tool for directly reading and writing physical addresses (dangerous! Use with caution).	<span>sudo devmem2 0xPhysicalAddress</span> Can read the contents of physical memory allocated by the driver, comparing it with what user space sees.
<span>strace</span>	Trace system calls of a process.	<span>strace ./test_mmap</span> Can see the parameters and return values of the <span>mmap</span> system call.
<span>GDB</span> with <span>vmlinux</span>	Debugging the kernel with kernel symbol files (requires complex configurations like KGDB).	Set breakpoints in the <span>.mmap</span> function for advanced debugging.
Concurrency & Race Conditions	The mapped memory may be accessed simultaneously by user and kernel space.	Synchronization issues must be considered! Use mutexes or RCU to protect access to shared buffers.
Data Cache Coherence	For mapped device memory (such as FPGA registers), it is usually necessary to set it as uncached.	In the <span>.mmap</span> function, use <span>vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);</span>
IOMMU	On systems with IOMMU, device addresses (IOVA) and physical addresses (PA) may differ.	Use DMA API (such as <span>dma_alloc_coherent</span>) to allocate memory, and pass its IOVA (equivalent to PA) to <span>remap_pfn_range</span>.

Debugging Process Flowchart

Is
No
User program access to mmap address fails (segmentation fault/data error)
Check kernel dmesg
Driver error printed
Check driver code: 1. Boundary checks 2. remap_pfn_range return value 3. Memory allocation success
No driver error printed
Use strace to confirm if mmap system call succeeded and parameters
Check /proc/pid/maps to confirm if mapping area exists and attributes are correct
Write a simple test: only mmap, read-only, verify basic functionality
Still having issues?
Use tools like devmem2 to check physical memory contents, or use KGDB for kernel debugging
The problem may be in the user program's access method

Conclusion

Memory mapping in Linux device drivers is a powerful and efficient mechanism, with the core being the driver calling the <span>.mmap</span> method to invoke functions like <span>remap_pfn_range</span> to manipulate page tables.

Key Points	Description
Core Functions	<span>remap_pfn_range</span>, <span>io_remap_pfn_range</span>
Core Structures	<span>struct vm_area_struct</span> (VMA)
Memory Preparation	Use <span>alloc_pages</span>, <span>dma_alloc_coherent</span> etc. to allocate physically contiguous memory for mapping.
Considerations	Synchronization (concurrent access), cache coherence (device memory should be set to uncached), security (boundary checks).
Debugging	<span>dmesg</span>, <span>/proc/pid/maps</span>, <span>strace</span>, <span>devmem2</span>.