Linux Memory Leak Analysis

In Linux system administration, a memory leak is a common performance issue that can lead to resource exhaustion, performance degradation, or crashes. A memory leak refers to memory allocated by a program that is not properly released, gradually accumulating and occupying memory. According to the official Valgrind report, memory leaks account for over 20% of bugs in C/C++ programs. In high-load servers or long-running applications, memory leaks can trigger the OOM (Out of Memory) killer, terminating critical processes. Timely diagnosis and repair of memory leaks can significantly enhance system stability and efficiency.

1. Basic Knowledge of Memory Leaks

1.1 What is a Memory Leak?

A memory leak refers to memory that is dynamically allocated by a program (such as using malloc or new) but is not released (using free or delete), leading to a continuous increase in memory usage. Leaked memory cannot be reused, ultimately exhausting available memory. In Linux, process memory leaks can lead to increased Swap usage, performance degradation, or system crashes.

Categories:

• Explicit Leak: Not calling free/delete.
• Implicit Leak: Lost pointers that cannot be released.
• Resource Leak: File descriptors or sockets not closed.

Symptoms:

• Continuous increase in memory usage.
• Increased Swap usage.
• Performance degradation.
• OOM killer activation.

Example (C Language):

#include <stdlib.h>

void leak() {
    int *ptr = malloc(100 * sizeof(int));  // Allocated but not freed
}

int main() {
    while (1) {
        leak();
    }
    return 0;
}

Memory continuously increases after running.

1.2 Importance of Memory Leaks

Memory leaks are performance killers in Linux:

• Stability: Long-running applications may crash.
• Resource Waste: Occupied memory that cannot be reclaimed.
• Security Risks: Leaks may expose sensitive data.
• Economic Loss: Server restarts can lead to business interruptions.
• Difficult Diagnosis: Implicit leaks are hard to detect.

For example, in 2023, a cloud service experienced node crashes due to memory leaks, affecting users globally.

1.3 Typical Scenarios of Memory Leaks

• Web Servers: Plugin leaks in Nginx or Apache.
• Databases: MySQL not releasing query buffers.
• Applications: C/C++ programs using malloc without free.
• Scripts: Python not closing files.
• Kernel: Driver module leaks.

1.4 Challenges in Analyzing Memory Leaks

• Concealment: Small leaks accumulate over time.
• Tool Complexity: Tools like Valgrind require compilation support.
• Performance Overhead: Analysis tools increase load.
• Multithreading: Thread safety issues complicate matters.
• Kernel Leaks: Require specialized tools like kmemleak.

1.5 Goals of Analyzing Memory Leaks

• Quick Localization: Identify leaking code.
• Repair: Free allocated memory.
• Prevention: Code reviews and testing.
• Monitoring: Real-time leak detection.
• Automation: Integrate CI/CD testing.

2. Principles of Linux Memory Leaks

2.1 Memory Allocation Mechanism

Linux memory allocation is divided into user space and kernel space:

• User Space: malloc/free (glibc implementation).
• Kernel Space: kmalloc/kfree.

malloc Principle: Uses the ptmalloc allocator, based on dlmalloc and sbrk/mmap.

Leak Principle: Allocation pointer lost, cannot be freed.

2.2 Memory Management Model

• Virtual Memory: Process address space isolation.
• Page Faults: Demand paging allocates memory.
• Swap: Pages swapped out when memory is insufficient.

OOM: Memory exhaustion triggers the OOM killer.

2.3 Types of Leaks

• Not Released: No free after malloc.
• Double Free: Double free leads to crashes.
• Dangling Pointers: Used after free.

Kernel Leak: kmalloc not kfree.

2.4 Leak Detection Principles

• Reference Counting: Tracks allocated references.
• Mark and Sweep: Garbage collection algorithm.
• Instrumentation: Tools like Valgrind intercept malloc/free.

2.5 Summary of Principles

Memory leaks occur when allocation and release do not match, and tools intercept and analyze to locate them.

3. Linux Memory Leak Diagnosis Tools

3.1 Valgrind

Purpose: Memory leak detection.

Installation:

sudo apt install valgrind

Usage:

valgrind --leak-check=full ./myapp

Output:

==1234== Memcheck, a memory error detector
==1234== Using Valgrind-3.15.0
==1234== HEAP SUMMARY:
==1234==     in use at exit: 100 bytes in 1 blocks
==1234==   total heap usage: 1 allocs, 0 frees, 100 bytes allocated
==1234== 
==1234== 100 bytes in 1 blocks are definitely lost in loss record 1 of 1
==1234==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==1234==    by 0x40053B: leak (leak.c:3)
==1234==    by 0x40054B: main (leak.c:7)
==1234== 
==1234== LEAK SUMMARY:
==1234==    definitely lost: 100 bytes in 1 blocks

Advanced:

• –show-leak-kinds=all: Detailed leak types.
• –track-origins=yes: Trace origins.

Advantages: Detailed reports. Disadvantages: High overhead, 10-30 times slower.

3.2 ASan (Address Sanitizer)

Purpose: Built into GCC/Clang, detects leaks.

Compilation:

gcc -fsanitize=address -g leak.c -o leak_asan

Run:

./leak_asan

Output:

=================================================================
==1234==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010 at pc 0x7f8b9c0b2b97 bp 0x7ffc1b2b2b80 sp 0x7ffc1b2b2b78
READ of size 4 at 0x602000000010 thread T0
    #0 0x7f8b9c0b2b96 in main leak.c:8
    #1 0x7f8b9b8b0830 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)

Advantages: Runtime detection, no tools required. Disadvantages: Requires recompilation.

3.3 kmemleak (Kernel Leak)

Enable:

sudo echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

Principle: Scans for unreferenced kernel memory.

Advantages: Kernel-level detection. Disadvantages: Only kernel memory.

3.4 tracemalloc (Python)

Purpose: Python memory tracking.

Example:

import tracemalloc

tracemalloc.start()
# Code
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
    print(stat)

Advantages: Built-in, easy to use. Disadvantages: Only for Python.

3.5 pprof (Go)

Purpose: Go memory profiling.

Example:

import "net/http"
import _ "net/http/pprof"

go func() {
    http.ListenAndServe("localhost:6060", nil)
}()

Access http://localhost:6060/debug/pprof/heap.

Advantages: Built-in in Go. Disadvantages: Only for Go.

3.6 htop/top/smep

Purpose: Monitor process memory.

htop:

htop

smem:

sudo apt install smem
smem -k

Advantages: Real-time monitoring. Disadvantages: Does not locate leak code.

3.7 gdb

Purpose: Debug leaks.

Example:

gdb ./myapp
(gdb) run
(gdb) bt  # Stack trace

Advanced: Use heap plugin.

4. Memory Leak Investigation Process

4.1 Preliminary Monitoring

1. Check memory usage:

free -h
vmstat 1 10

1. Locate high memory processes:

top -o %MEM
ps aux --sort=-%mem | head

4.2 Process-Level Analysis

1. View process memory mapping:

pmap -x <pid>
cat /proc/<pid>/smaps

1. Compare memory changes:

while true; do
    ps -p <pid> -o rss,vsz >> memory_log.txt
    sleep 60
done

4.3 Tool Diagnosis

1. Valgrind:

valgrind --leak-check=full ./myapp

1. ASan: Compile with -fsanitize=address.
2. gdb:

gdb --args ./myapp
(gdb) run
(gdb) heap

4.4 Kernel Memory Leaks

1. Enable kmemleak: Edit /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="kmemleak=on"
sudo update-grub
sudo reboot
sudo echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

4.5 Automation Script

#!/bin/bash
PID=$1
while true; do
    ps -p $PID -o rss,vsz >> memory_$PID.txt
    sleep 60
done

5. Methods for Fixing Memory Leaks

5.1 Code-Level Fixes

• C/C++: Match malloc/free, new/delete.
• Python: Use gc.collect() or weakref.
• Go: Use defer runtime.GC().

Example: Fixing C leak:

int *ptr = malloc(100 * sizeof(int));
if (ptr) {
    // Use
    free(ptr);
}

5.2 System-Level Fixes

• Kill process:

  sudo kill -9 <pid>
  

• Adjust oom_adj:

  echo -17 | sudo tee /proc/<pid>/oom_adj
  

5.3 Preventive Measures

• Code Review: Use static analysis like cppcheck.

  cppcheck myapp.c
  

• CI/CD Testing: Integrate Valgrind.

• Monitoring Alerts: Use Prometheus to monitor process memory.

• Resource Limits: cgroups.

  sudo cgcreate -g memory:/myapp
  sudo cgset -r memory.limit_in_bytes=512M myapp
  sudo cgexec -g memory:myapp ./myapp
  

6. Case Studies of Memory Leaks

6.1 Case 1: C Program Memory Leak

Scenario: Looping malloc without free.

Diagnosis:

valgrind --leak-check=full ./leak_app

Fix: Add free(ptr).

Result: Memory stabilized.

6.2 Case 2: Python Script Leak

Scenario: File not closed.

Diagnosis:

import tracemalloc
tracemalloc.start()
# Code
snapshot = tracemalloc.take_snapshot()
print([stat for stat in snapshot.statistics('lineno') if stat.size > 1000000])

Fix: Use with open().

Result: Leak eliminated.

6.3 Case 3: Kernel Driver Leak

Scenario: Custom driver kmalloc not kfree.

Diagnosis: kmemleak scan.

Fix: Add kfree.

Result: Kernel memory stabilized.

7. Future Trends in Memory Leak Analysis

• AI Tools: Automatic leak detection.
• eBPF: Kernel-level tracing.
• Rust: Memory-safe language reduces leaks.

8. Conclusion

Linux memory leak analysis is an essential skill for development and operations. By using tools like Valgrind and ASan, efficient localization and repair can be achieved.