How to Diagnose Memory Leak Issues in Programs Written in C and Go

Performance bottleneck analysis and flame graph generation based on pprof

A comprehensive guide on using Golang pprof

How to diagnose memory leak issues in programs written in C and Go. This article provides specific commands and practical examples for diagnosing memory leaks in C and Go programs, using system tools and language features for layered detection:

—

1. Memory Leak Diagnosis in C

1. Valgrind Tool Detection

# Add debug symbols during compilation
gcc -g -o program program.c
# Run detection (outputs locations of unreleased memory)
valgrind --leak-check=full --show-leak-kinds=all ./program

Output example:

==12345== 1,024 bytes in 1 blocks are definitely lost
==12345==    at 0x4C2BBAF: malloc (vg_replace_malloc.c:381)
==12345==    by 0x108ABC: create_object (program.c:42)  # Locate the leak code line

2. AddressSanitizer (ASAN)

# Enable ASAN during compilation
gcc -fsanitize=address -g -o program program.c
./program  # Outputs leak stack when the program crashes

Output characteristics:

===================================================================
45678==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1024 byte(s) in 1 object(s) allocated from:
    #0 0x7f2a5b in malloc (/lib64/libasan.so.6+0xab5b)
    #1 0x55d678 in main program.c:42

3. Core Dump Analysis

ulimit -c unlimited     # Allow core file generation
./program &             # Run the program
kill -SIGSEGV $(pgrep program)  # Trigger core dump
gdb ./program core      # Analyze core file
(gdb) info malloc-history  # View memory allocation history (must compile with -g)

—

2. Memory Leak Diagnosis in Go

1. pprof Memory Analysis Steps:
Integrate pprof in the code:

import _ "net/http/pprof"
func main() {
    go func() { http.ListenAndServe(":6060", nil) }()
    // Business logic...
}

2. Obtain Memory Snapshot:

# Real-time analysis (replace PORT)
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/heap

Web interface operations:
– Click Top to view the functions with the highest memory usage
– Use Flame Graph to locate the call chain (example: `main.processData` occupies 70% of memory)

2. Goroutine Leak Detection

# Check if the number of goroutines is continuously increasing
curl http://localhost:6060/debug/pprof/goroutine?debug=2 > goroutines.txt
# Analyze blocked goroutines
go tool pprof http://localhost:6060/debug/pprof/goroutine
(pprof) top -cum

Common leak scenarios:

// Example: Goroutine without exit condition
go func() {
    for { /* No exit condition */ }
    // Leak!
}()

3. Runtime Monitoring

# Observe GC behavior (whether memory is reclaimed)
GODEBUG='gctrace=1' ./program

Output interpretation:

gc 1 @0.023s: 4->4->0 MB, 0.5 ms  # Memory did not decrease after reclamation → Possible leak

—

3. Key Points for Mixed Programming (CGO) Leak Diagnosis:

– Memory allocated in C must be manually released:

/*  #include <stdlib.h>  */
import "C"
func main() {
    ptr := C.malloc(1024)
    defer C.free(ptr)  // Must be explicitly released!
}

– Cross-language toolchain:

# Use eBPF to detect cgo leaks (requires Linux 4.15+)
sudo memleak-bpfcc -p $(pgrep program)

Output:

Top 5 stacks with outstanding allocations:
    1024 bytes in 1 allocations from
        malloc+0x10 [libc.so.6]
        main._Cfunc_createObject+0x20 [program]

—

4. Practical Recommendations:

1. Layered Diagnosis:
– First confirm the memory growth trend of the process using `top/htop`
– C: Prioritize using `valgrind` or `ASAN`
– Go: Combine `pprof` + `GODEBUG=gctrace=1`

2. Preventive Measures:
– C: Add comments to each `malloc` indicating the responsibility for release
– Go: Use `defer` to release resources (files, locks, network connections)

3. Container Environment:

# Obtain pprof data inside the container
docker exec -it <container> curl http://localhost:6060/debug/pprof/heap > heap.out
go tool pprof heap.out  # Local analysis

Through the combination of the above tools, 90% of memory leak scenarios can be covered. If kernel-level leaks are involved (such as unclosed sockets), deep analysis with `lsof -p PID` or `bpftrace` is required.

For the issue of continuous memory growth in the `stmanager` process written in C and Go, a combination of system tools, language features, and code logic should be used for diagnosis. Below is a step-by-step diagnosis plan:

—

1. Preliminary Diagnosis: System-level Monitoring and Process Analysis

1. Monitor Memory Changes
Use `top` or `htop` to observe in real-time whether the memory usage of `stmanager` (RES/physical memory) is continuously increasing:

top -p $(pgrep stmanager)

– If memory fluctuates periodically, it may be normal GC or caching mechanism; if it increases unidirectionally, there is a risk of leakage.

2. Locate Memory Allocation Sources
Use `pmap` to view the process memory mapping and identify memory segments with abnormal growth:

pmap -x $(pgrep stmanager)

– Focus on the growth trend of `[heap]` (heap memory) or `[anon]` (anonymous memory).

—

2. C Language Part: Manual Memory Management Diagnosis

1. Check for Memory Leaks
– Tool Detection:
Use `valgrind` to detect unreleased memory blocks:

valgrind --leak-check=full --show-leak-kinds=all ./stmanager

– Output example:

==1234== 1,024 bytes in 1 blocks are definitely lost in loss record 1 of 1
==1234==    at 0x4C2BBAF: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==1234==    by 0x108745: create_struct (example.c:42)

– Focus on checking `malloc`/`calloc` locations that are not matched with `free`.

– AddressSanitizer:
Enable `-fsanitize=address` during compilation, run the program to directly locate leak points:

gcc -fsanitize=address -g structmanager.c -o stmanager
./stmanager

2. Analyze Core Dump Files
– Generate core dump:

ulimit -c unlimited  # Allow large file generation
kill -SIGSEGV $(pgrep stmanager)  # Trigger crash to generate core file

– Use `gdb` for analysis:

gdb ./stmanager core
(gdb) info malloc-history  # View memory allocation history (must compile with -g)

—

3. Go Language Part: Garbage Collection and Object Management

1. Enable pprof Performance Analysis
– Integrate `net/http/pprof` in Go code:

import _ "net/http/pprof"
func main() {
    go func() {
        http.ListenAndServe(":6060", nil)  // Start pprof HTTP service
    }()
    // Other logic...
}

– Obtain memory snapshot through `pprof`:

go tool pprof http://localhost:6060/debug/pprof/heap

– Analyze memory hotspots:
In the `pprof` interactive interface, enter `top` to view the function call stacks that occupy the most memory:

(pprof) top
1.23GB of 1.50GB total (82%)
Dropped 12 nodes (cum <= 0.01GB)
flat  flat%   sum%        cum   cum%
0.80GB 53.33% 53.33%     0.80GB 53.33%  main.processStruct (example.go:30)
0.43GB 28.67% 82.00%     0.43GB 28.67%  runtime.mallocgc

2. Check for Goroutine Leaks
– Use `pprof` to analyze goroutines:

go tool pprof http://localhost:6060/debug/pprof/goroutine

– If the number of goroutines increases abnormally, check for `goroutine`s that do not exit:

go func() {
    for {
        // Loop without exit condition may cause leak
    }
}()

3. Check Structure References and Caches
– Unreleased structure references:
Check if global or cache variables continuously hold structure pointers, for example:

var cache = make(map[string]*Struct)
// Must regularly clean up expired entries, otherwise memory will continue to grow

– Slice/Map Expansion:
When frequently adding elements to a `slice`, if the capacity is not limited or the underlying array is not reused, it may lead to memory bloat:

data := make([]byte, 0, 1024)  // Pre-allocate capacity to avoid frequent expansion

—

4. Mixed Language Interaction Diagnosis (C and Go)

1. Memory Management in CGO Calls
– If Go calls C code via `cgo`, ensure that memory allocated in C is explicitly released:

/* 
#include <stdlib.h>
*/
import "C"
func createCStruct() unsafe.Pointer {
    ptr := C.malloc(C.size_t(1024))
    return ptr
}
// Must manually release after use
C.free(ptr)

2. Cross-language Cache Sharing
– Check shared cache structures between C and Go (such as shared memory or global variables) to avoid memory accumulation due to one side not releasing.

—

5. Code-Level Optimization Recommendations

| Problem Type | C Language Solution | Go Language Solution |

| Memory Leak | Use `valgrind` or `AddressSanitizer` to locate unreleased memory | Use `pprof` to analyze memory allocation hotspots |

| Structure Not Released | Check `free()` call logic | Clean up global cache, nullify unused structure references |

| Frequent Allocation/Release | Use Memory Pool to reuse memory blocks | Pre-allocate `slice` capacity, reuse objects (sync.Pool) |

| Goroutine Leak | – | Add context cancellation mechanism (`context.Context`) |

—

6. Summary Steps

1. System Monitoring: Confirm memory growth trends and sources.

2. Language-Specific Diagnosis:

– C: Use `valgrind`/`gdb` to detect leaks.

– Go: Use `pprof` to analyze memory and goroutines.

3. Check Interaction Logic: Ensure memory from CGO calls is correctly released.

4. Code Optimization: Limit cache size, reuse memory, clean up references.

Through the above methods, the root cause of the memory growth in `stmanager` can be accurately located, whether it is a memory leak in C or an unreleased structure reference issue in Go.