🎪 The Linux Memory Circus: The Dual Life of File Caching

Author: A Linux veteran once chased by the OOM killerSlogan: Memory management is not magic, but sometimes it’s more magical than magic!

🎯 Introduction: Why Should You Care?

Imagine this scenario: you carefully configured a container with 32GB of memory, started a database that is “just right” at 32GB, and then…Bang! The container was taken down by the OOM killer! 😱

At this moment, you might think: “I calculated everything correctly!” Don’t worry, today I will take you into the circus of Linux memory management to see what those “inactive files that are not released” are really up to!

🎪 Act One: The Great Classification of Memory – The Feud Between Two Families

💡 The two great families of the memory world:  
   👤 Anonymous Memory Family - "I am a temp worker, throw me away when done"  
   📁 File Memory Family - "I have a background, the disk is my father"  

👤 Anonymous Memory – The “Temp Worker” of the Memory World

# This is the memory dynamically allocated by your application  
malloc(1024);  # "Boss, give me a piece of land to put something"  

Characteristics:

• 🚫 No disk backup (like your draft paper in memory)
• 💸 Relies on swap for survival when memory is tight
• 🔄 Includes heap, stack, and dynamically allocated memory

📁 File Memory – The “Insider” of the Memory World

# When reading a file, the kernel secretly helps you cache it  
cat large_file.txt > /dev/null  # "This file is nice, I will cache it for later"  

Characteristics:

• 💾 Has disk files as a backup
• 🚀 A tool to enhance I/O performance
• 🗑️ Can be discarded directly when memory is tight (clean pages) or written back and then discarded (dirty pages)

🎪 Act Two: The Dual Life of File Caching – Active vs Inactive

🎭 The two actors of the file caching family:  
   🔥 Active File - "The Hot Star"  
   ❄️ Inactive File - "The Has-Been"  

The LRU Linked List Stage of the Memory Circus

// The "audition scene" in the kernel source - mm/vmscan.c  
// Let's see how the pages compete:  

static void shrink_active_list(...) {  
    // Judge kswapd appears: "Let me see which pages are still active!"  
    
    if (page_referenced(page, ...)) {  
        // "Wow, this page has been accessed recently, promoted to Active queue!"  
        list_add(&page->lru, &l_active);  
    } else {  
        // "Sorry, you haven't been accessed for too long, demoted to Inactive queue"  
        list_add(&page->lru, &l_inactive);  
    }  
}  

The rules of this audition show are simple:

• 🎯 Recently accessed → Enter Active queue (debut in the C position)
• 😴 Not accessed for a long time → Demoted to Inactive queue (pending area)
• 🗑️ When memory is tight → Prioritize eliminating pages from the Inactive queue

The Real-Life Performance of the Memory Circus

# Let's see the current situation of your memory circus  
$ cat /proc/meminfo | grep -E "(Active|Inactive).*file"  
Active(file):   1245000 kB    # 🔥 The Hot Stars  
Inactive(file): 560000 kB     # ❄️ The Has-Beens  

🎪 Act Three: The Tragedy Replayed – The OOM Disaster of the 32GB Container

Now let’s return to the tragedy at the beginning: 32GB container + 32GB database = OOM 💥

The “Death Calculation” of Memory Allocation

📊 Let's calculate this bill:  

Total memory of the container: 32GB  
───────────────────────────────  
Database working set:   28GB  (Anonymous Memory)  
File cache:       3GB   (at the start)  
Other overhead:       1GB  
───────────────────────────────  
Used:        32GB  (Perfect! Wait...)  

But the file cache will grow!  
───────────────────────────────   
File cache grows to: 4GB  
Database working set:   28GB  
Other overhead:       1GB  
───────────────────────────────  
Total:          33GB  (Exceeds 1GB → OOM!)  

Why does the inactive file not come to the rescue?

// mm/vmscan.c - The "rescue team" working scene of the kernel  
static void shrink_node_memcgs(...) {  
    // Rescue team kswapd: "Report! Memory is tight!"  
    // But..."under cgroup limits, the rescue operation is delayed!"  
    
    // Inactive File: "I also want to be reclaimed, but it takes time to reclaim!"  
    // Database: "I can't wait! I need memory! Right now!"  
}  

Three Major Reasons for Rescue Failure:

1. ⏰ cgroup statistics delay: “I didn’t react in time, and the memory exploded!”
2. 🐌 Slow reclamation speed: “Cleaning up inactive files takes time, especially dirty pages need to be written back to disk”
3. 🎪 Database is too demanding: Prefetching, logging, temporary tables… constantly creating new file caches

🎪 Act Four: Rescue Solutions – Setting New Rules for the Memory Circus

🎯 Solution 1: “Slimming Down” the Container – Reserve Buffer Space

# Wrong approach: Stuffing the container like a train during Spring Festival  
docker run --memory=32g database:latest  

# Correct approach: Leave some breathing space for memory  
docker run \  
  --memory=32g \  
  --memory-reservation=28g \  # Soft limit, early warning  
database:latest  

Metaphor: Don’t stuff the elevator to the brim, leave some space for emergencies!

🎯 Solution 2: Adjust Kernel Parameters – Change the Rules of the Game

# Make the kernel more proactive in cleaning file caches  
echo 100 > /proc/sys/vm/vfs_cache_pressure    # "Cleaner, work more actively!"  

# Reduce swap tendency, prioritize reclaiming file caches  
echo 50 > /proc/sys/vm/swappiness             # "Don't always think about swap, clean file caches first!"  

# Prevent overcommitment  
echo 1 > /proc/sys/vm/overcommit_memory       # "Act within your means, don't make promises recklessly!"  

🎯 Solution 3: Database Optimization – Make the Actors Follow the Rules

-- MySQL: Give file caches some respect  
SET GLOBAL innodb_buffer_pool_size = 26G;     -- "Leave 2GB buffer space for yourself"  
SET GLOBAL innodb_flush_method = O_DIRECT;    -- "Avoid double caching, save some memory"  

-- PostgreSQL: Same principle  
ALTER SYSTEM SET shared_buffers = '26GB';     -- "Don't be too greedy!"  

🎯 Solution 4: Proactive Cleanup – Regular Big Cleanups

#!/bin/bash  
# "Big cleanup script" before starting the database  

echo "🧹 Starting to clean memory cache..."  
sync                                   # First, write dirty data back to disk  
echo 3 > /proc/sys/vm/drop_caches     # Clean pagecache, dentries, inodes  
echo "✨ Cleanup completed! Starting database..."  
/usr/bin/mysqld  

🎯 Solution 5: Intelligent Monitoring – Hire a Memory Security Guard

#!/usr/bin/env python3  
# Daily work of the memory security guard  
def check_memory_usage():  
    """Security patrol: Check memory usage"""  
    if memory_usage_rate > 85%:  
        print("🚨 Alert! Memory usage too high!")  
        os.system('sync && echo 1 > /proc/sys/vm/drop_caches')  
        print("✅ Emergency cleanup executed")  

🎪 Act Five: Best Practices – Wisdom from the Veteran

💡 Golden Rule: Never Overeat

# Wrong: Database working set = Container memory  
Container memory = 32GB  
Database memory = 32GB    # 💀 Death configuration!  

# Correct: Leave 20% buffer space  
Container memory = 32GB  
Database memory = 26GB    # ✅ Safe configuration!  
Buffer space = 6GB       # For file caches and other overhead  

📊 Monitoring Metrics – Health Check of the Circus

# Daily inspection command  
cat /sys/fs/cgroup/memory/memory.usage_in_bytes    # Total usage  
cat /sys/fs/cgroup/memory/memory.stat | grep file  # File cache status  

# Set alert threshold  
ALERT_THRESHOLD=0.9  # Alert at 90% usage  

🎭 Kubernetes Configuration – Professional Circus Management

apiVersion: v1  
kind: Pod  
spec:  
  containers:  
  - name: database  
    resources:  
      requests:  
        memory: "30Gi"    # I usually need this much  
      limits:  
        memory: "34Gi"    # But give me this much at most, leave some buffer  

🎪 Conclusion: Summary of Experiences

Through this performance of the memory circus, we learned:

1. 🎭 File caching has a dual life: Active is the hot star, Inactive is the has-been
2. 💥 The root cause of OOM tragedy: Under cgroup limits, file cache reclamation is not timely enough
3. 🛠️ Solutions: Reserve buffer, adjust parameters, optimize configurations, proactive monitoring
4. 🎯 Core Idea: Don’t calculate memory too rigidly, leave some leeway for system management

Remember, **a good operator does not use memory to 100%, but knows when to use 80%!**

Now, go tame your memory circus! 🎪✨

Appendix: The Debugging Command Toolbox of the Veteran 🧰

# View detailed memory information  
cat /proc/meminfo  

# View cgroup memory statistics  
cat /sys/fs/cgroup/memory/memory.stat  

# Real-time monitoring of memory reclamation  
watch -n 1 'cat /proc/vmstat | grep scan'  

# View process memory details  
cat /proc/$(pgrep mysqld)/smaps | grep -i anon  

Wish you happy debugging, stay away from the OOM killer! 🔫💀→😊