Linux Physical Memory Models

In general, the kernel manages physical memory in pages, with each page being 4K in size, described using the struct page structure, which stores various information about the physical page.

To quickly index specific physical memory pages, the kernel defines an index number for each physical page struct page: PFN (Page Frame Number), where PFN corresponds one-to-one with struct page.

The kernel provides two macros to convert between PFN and the physical memory page struct page: page_to_pfn and pfn_to_page.

The way the kernel manages physical pages is called the physical memory model. Different physical memory models have different scenarios and calculation logic for page_to_pfn and pfn_to_page. This section introduces two common physical memory models.

1. FLATMEM Flat Memory Model

First, let’s imagine physical memory as a contiguous block of storage space. In this large block of contiguous memory, the kernel divides it into physical memory blocks struct page page by page.

Since this block of physical memory is contiguous, the physical addresses are also contiguous. The divided physical memory pages must also be contiguous, and since each page size is fixed, it is easy to think of organizing these contiguous physical memory pages using an array, where the index in the array is the PFN. This memory model is called the Flat Memory Model (FLATMEM).

The kernel uses a global array mem_map to organize all the divided physical memory pages. The index of the mem_map global array corresponds to the PFN of the respective physical page.

In the flat memory model, the calculation logic for page_to_pfn and pfn_to_page is very simple, essentially based on offset operations on the mem_map array.

// include/asm-generic/memory_model.h
#if defined(CONFIG_FLATMEM)

#ifndef ARCH_PFN_OFFSET
#define ARCH_PFN_OFFSET     (0UL)
#endif

#define __pfn_to_page(pfn)  (mem_map + ((pfn) - ARCH_PFN_OFFSET))
#define __page_to_pfn(page) ((unsigned long)((page) - mem_map) +  ARCH_PFN_OFFSET)
#endif

Linux initially used this memory model because, during the early development of Linux, the physical memory that needed to be managed was usually not large (for example, tens of MB). At that time, using the flat memory model (FLATMEM) to manage physical memory was efficient enough.

The FLATMEM flat memory model is only suitable for managing a whole block of contiguous physical memory. Using the FLATMEM flat memory model to manage multiple non-contiguous physical memory blocks would result in significant memory space waste.

Later, Linux introduced the Discontiguous Memory Model (DISCONTIGMEM), but this model was short-lived and was completely replaced by the Sparse Memory Model. The latest kernels have removed this part, so we will not elaborate further here.

2. SPARSEMEM Sparse Memory Model

The core idea of the SPARSEMEM sparse memory model is to manage smaller contiguous memory blocks more finely. The units used to manage contiguous memory blocks are called sections, represented in the kernel by the struct mem_section structure. Each struct mem_section structure contains a pointer section_mem_map that points to the array of pages managing the contiguous memory within the section.

// include/linux/mmzone.h
struct mem_section {
    unsigned long section_mem_map;
};

All mem_section structures in the SPARSEMEM memory model are stored in a global array:

#ifdef CONFIG_SPARSEMEM
#ifdef CONFIG_SPARSEMEM_EXTREME
extern struct mem_section **mem_section;
#else
extern struct mem_section mem_section[NR_SECTION_ROOTS][SECTIONS_PER_ROOT];
#endif
#endif

The organization and management of physical memory in the SPARSEMEM memory model is illustrated in the following diagram:

Where NR_SECTION_ROOTS and SECTIONS_PER_ROOT are both configurable, with specific configuration methods as follows:

#define SECTIONS_PER_ROOT   (PAGE_SIZE / sizeof (struct mem_section))
#define NR_SECTION_ROOTS    DIV_ROUND_UP(NR_MEM_SECTIONS, SECTIONS_PER_ROOT)

In the SPARSEMEM sparse memory model, the calculation logic for page_to_pfn and pfn_to_page also changes, with the overall logic as follows:

// include/asm-generic/memory_model.h
#if defined(CONFIG_SPARSEMEM)
/*
 * Note: section's mem_map is encoded to reflect its start_pfn.
 * section[i].section_mem_map == mem_map's address - start_pfn;
 */
#define __page_to_pfn(pg)                   \
({  const struct page *__pg = (pg);             \
    int __sec = page_to_section(__pg);          \
    (unsigned long)(__pg - __section_mem_map_addr(__nr_to_section(__sec))); \
})

#define __pfn_to_page(pfn)              \
({  unsigned long __pfn = (pfn);            \
    struct mem_section *__sec = __pfn_to_section(__pfn);    \
    __section_mem_map_addr(__sec) + __pfn;      \
})
#endif

• In the conversion of page_to_pfn, it is first necessary to locate the specific section structure in the mem_section array using page_to_section based on the struct page structure. Then, the section_mem_map is used to locate the specific PFN.

static inline unsigned long page_to_section(const struct page *page)
{
    return (page->flags >> SECTIONS_PGSHIFT) & SECTIONS_MASK;
}

• In the conversion of pfn_to_page, it is first necessary to locate the specific section structure in the mem_section array using __pfn_to_section based on the PFN. Then, the PFN is used to locate the specific physical page in the section_mem_map array.

static inline struct mem_section *__pfn_to_section(unsigned long pfn)
{
    return __nr_to_section(pfn_to_section_nr(pfn));
}

3. Conclusion

This article mainly introduces two models for the kernel to manage physical pages:

One is the Flat Memory Model (FLATMEM), which is suitable for managing a whole block of contiguous physical memory. However, using the FLATMEM flat memory model to manage multiple non-contiguous physical memory blocks would result in significant memory space waste.

The other is the Sparse Memory Model (SPARSEMEM), which focuses on finely managing smaller contiguous memory blocks, effectively supporting memory holes and hot-plugging mentioned later. Currently, this is the default configuration in the kernel.

Technical developers focused on sharing Linux knowledge, follow me to learn programming knowledge together.