1. Background
In early memory management modules, the concept of compound pages was introduced to describe Linux Compound Pages; however, there are design flaws inherent in compound pages themselves.
In practical applications, the compound_head of compound pages presents issues in distinguishing between head pages and tail pages. Improper usage can easily lead to system exceptions. Additionally, there are numerous checks for compound_page in the system. Although they are large pages, they conceptually result from page concatenation. Therefore, it is necessary to obtain the head page of the compound page from the current page, which incurs performance overhead, and compound pages cannot be used in page cache management. In this context, we introduce the topic of Folios.
2. Theory
Illustration of the structure of a compound page: it can be seen that the composition is complex. In practical applications, it is necessary to determine whether the page is a compound page and whether it is a head page. If it is not a head page, the compound_head must be used to obtain the head page of the compound page.
Due to the random read characteristics of file access, compound pages cannot be effectively applied in memory management of page cache:
During file page access, different processes access the pages differently. Some functions need to access the head page, while others expect to access other pages. Therefore, the consistency within the compound page cannot be guaranteed in this design. Thus, compound pages are not suitable for file access functionality. To resolve this issue, a new, integrated concept must be introduced. This leads us to the question: What are Folios?
The introduction of the concept of folios effectively resolves the head page or tail page issues of compound pages, providing a theoretical basis for the application of large pages in file systems and bringing the following benefits to the system.
3. Source Code AnalysisStruct
struct folio {/* private: don't document the anon union */union {struct {/* public: */unsigned long flags;union {struct list_head lru;struct {void *__filler;unsigned int mlock_count;};/* public: */};struct address_space *mapping;pgoff_t index;union {void *private;swp_entry_t swap;};atomic_t _mapcount;atomic_t _refcount;#ifdef CONFIG_MEMCGunsigned long memcg_data;#endif/* private: the union with struct page is transitional */};struct page page;};union {struct {unsigned long _flags_1;unsigned long _head_1;unsigned long _folio_avail;/* public: */atomic_t _entire_mapcount;atomic_t _nr_pages_mapped;atomic_t _pincount;#ifdef CONFIG_64BITunsigned int _folio_nr_pages;#endif/* private: the union with struct page is transitional */};struct page __page_1;};union {struct {unsigned long _flags_2;unsigned long _head_2;/* public: */void *_hugetlb_subpool;void *_hugetlb_cgroup;void *_hugetlb_cgroup_rsvd;void *_hugetlb_hwpoison;/* private: the union with struct page is transitional */};struct {unsigned long _flags_2a;unsigned long _head_2a;/* public: */struct list_head _deferred_list;/* private: the union with struct page is transitional */};struct page __page_2;};};The traditional struct page structure shows that the information describing page attributes is contained within the page itself. In conventional compound pages, checking page attributes requires traversing the compound page to complete the status check. For example, the implementation of the page_mapped function shows that it needs to traverse each individual page in the compound page to complete the status detection.
bool page_mapped(struct page *page){int i;/* Check if it is a compound page; if not, return atomic_read(&page->_mapcount) >= 0 */if (likely(!PageCompound(page)))return atomic_read(&page->_mapcount) >= 0;/* If it is a compound page, get the corresponding head page */page = compound_head(page);/* Check if the compound_mapcount is >= 0, indicating that the mapping relationship has been established */if (atomic_read(compound_mapcount_ptr(page)) >= 0)return true;.../* Get the number of individual pages in the compound page, traverse the mapcount values of each individual page; if greater than 0, the compound page has established a mapping */for (i = 0; i < compound_nr(page); i++) {if (atomic_read(&page[i]._mapcount) >= 0)return true;}return false;}However, from the design of folios, it can be seen that the entire large page is described through the _mapcount, _refcount, flags, etc., in the struct folio. The _mapcount clearly indicates the page status of the entire compound page, and the flags can be obtained directly from folio->flags without needing to access folio->page->flags. The same applies to other page attributes. Similarly, the folio_mapped function shows significant differences, reflecting the integrated design concept.
/* Check if it is a large page (compound page) * If it is a single page, directly use folio->_mapcount without needing folio->page->_mapcount * If it is not a single page, meaning it is a compound large page, then folio_large_is_mapped calls folio->_entire_mapcount * Describes the entire large compound page without needing to traverse each page's status in the compound page, making this strategy efficient */static inline bool folio_mapped(struct folio *folio){if (likely(!folio_test_large(folio)))return atomic_read(&folio->_mapcount) >= 0;return folio_large_is_mapped(folio);}static inline bool folio_large_is_mapped(struct folio *folio){/* Reading _entire_mapcount below could be omitted if hugetlb * participated in incrementing nr_pages_mapped when compound mapped. */return atomic_read(&folio->_nr_pages_mapped) > 0 || atomic_read(&folio->_entire_mapcount) >= 0;}ApplicationIn the kernel, the compound page has issues with confusion between head page and tail page. The design of folios avoids this problem; after allocating a compound page through __alloc_pages, it is forcibly converted to folio, so regardless of whether it is a compound page, it can be considered a “head page”.
/* alloc_page allocates a page and is forcibly converted to folios, folio describes the entire compound page */struct folio *__folio_alloc(gfp_t gfp, unsigned int order, int preferred_nid,nodemask_t *nodemask){struct page *page = __alloc_pages(gfp | __GFP_COMP, order, preferred_nid, nodemask);return page_rmappable_folio(page);}EXPORT_SYMBOL(__folio_alloc);/* Convert the pages allocated by alloc_page to folio for management */static inline struct folio *page_rmappable_folio(struct page *page){struct folio *folio = (struct folio *)page;folio_prep_large_rmappable(folio);return folio;}By using page_folio to obtain the folio corresponding to the page, a C11_Generic generic code design approach is adopted to obtain the corresponding folio based on the different types of pages.
#define page_folio(p) (_Generic((p), \ const struct page *: (const struct folio *)_compound_head(p), \ struct page *: (struct folio *)_compound_head(p)))To facilitate understanding, we can abstract the logic:
switch (typeof(p)) { case const struct page *: return (const struct folio *)_compound_head(p); case struct page *: return (struct folio *)_compound_head(p);}By organizing the folio, we address two issues brought by compound pages:
-
Reduce redundant calls to compound_head
-
Regardless of whether the folio is a compound page, it is treated as a head page, fixing potential bugs caused by tail pages in compound pages.
4. Think and Verify
How to use foliosThe purpose of understanding folios is to clarify when to use folios and when to treat folios as folios in applications.
Folios are essentially a collection of physically contiguous and virtually contiguous pages of size 2^n, where if n = 0, the folio is a single page, and if n > 0, the folio is a compound page. However, whether it is a single page or a compound page, they are collectively referred to as a folio, which is a holistic concept. Additionally, one certainty about folios is that they will never be a tail page, thus avoiding the semantic confusion of compound pages. After understanding folios, the key lies in how to determine the current scenario as a whole rather than as individual units, which needs further clarification.
-
In the memory reclamation process, shrink_folio_list isolates the old generation folio corresponding to min_seq. In the reclamation process, this folio is a holistic concept, and a typical scenario is the folio_check_references process, which improves the efficiency of folio checks as a whole.
-
The state change process of folios is also a holistic concept: PG_locked is targeted at the entire folio, not at a specific page within the compound page.
static inline void folio_lock(struct folio *folio){might_sleep();if (!folio_trylock(folio))__folio_lock(folio);}void __folio_lock(struct folio *folio){folio_wait_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE,EXCLUSIVE);}Folios are widely applied as a whole in Linux kernel memory management, and understanding the folio mechanism is beneficial for promoting an understanding of trends in Linux memory management.
Deep Think of Folios
Folios address the ambiguities and performance degradation issues introduced by compound pages:
-
Reduce redundant calls to compound_head;
-
Regardless of whether the folio is a compound page, it is treated as a head page, fixing potential bugs caused by tail pages in compound pages.
However, we cannot help but ask: Is the role of folios limited to this? The answer is no. Folios are not merely confined to solving compound_page issues; more profoundly, the application of large pages in different designs is significant. Reading LWN, we see that (https://lwn.net/Articles/869942/) Matthew Wilcox’s talk on “Efficient Buffered I/O” highlights that the concept of folios is crucial for improving I/O performance. This strategy is also vital for file systems, as the first merging of folios in the kernel was achieved through file systems. Folios will have a positive impact on memory management, file systems, and I/O management systems.
5. References
https://lwn.net/Articles/932474/
https://lwn.net/Articles/931794/
https://lkml.org/lkml/2021/10/30/263
https://lwn.net/Articles/849538/
https://lwn.net/Articles/869942/
https://lwn.net/Articles/856016/
https://lwn.net/Articles/937239/
https://lwn.net/Articles/964097/
https://lwn.net/Articles/686690/
https://kernelnewbies.org/MatthewWilcox/Folios
https://github.com/torvalds/linux/commit/49f8275c7d92
https://lore.kernel.org/all/[email protected]/T/#u
https://lore.kernel.org/all/[email protected]/#r
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/linux-mm/[email protected]/#r
https://lore.kernel.org/all/[email protected]/
https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=49f8275c7d9247cf1dd4440fc8162f784252c849
https://lwn.net/ml/linux-kernel/[email protected]/