Follow + Star, learn new Python skills every day

Source: Internet

Python handles “memory management” very thoughtfully, and most of the time you don’t have to manage it yourself. However, as an advanced developer, understanding its principles will give you more confidence when writing high-performance, long-running, data-intensive programs.

This article will help you thoroughly understand:

• • How memory is allocated
• • How Python determines if an object can be collected
• • Why circular references can become a pitfall
• • How to manually intervene to optimize memory

1. Overall Design of Python Memory Management

In simple terms:

Python primarily uses reference counting, supplemented by garbage collection, and has a built-in private memory pool to enhance performance.

In other words:

• • Small objects → Managed by Python itself (small object pool)
• • Large objects → Handed over to the operating system
• • When an object can be released → Mainly depends on “reference counting”
• • Circular references that cannot be resolved by reference counting are handled by the “garbage collector (gc)”

This mechanism keeps Python runtime efficient and stable.

2. Reference Counting: The Core Memory Determination Mechanism of Python

Each object has a reference count (refcount):

• • When a variable points to an object → Count +1
• • When a variable is reassigned or goes out of scope → Count -1
• • When the count reaches 0 → The object is immediately released from memory

For example:

import sys

a = [1, 2, 3]
print(sys.getrefcount(a))   # 2 (one is a, one is the parameter passed)
b = a
print(sys.getrefcount(a))   # 3
del b
print(sys.getrefcount(a))   # 2

The reference counting mechanism is simple and direct: “release when the count is 0”, so most objects in Python are released immediately.

3. Circular References: The Tough Problem for Reference Counting

The problem arises: what happens if two objects reference each other?

class Node:
    def __init__(self):
        self.next = None

a = Node()
b = Node()

a.next = b
b.next = a

Even if you execute:

del a
del b

The reference count of the objects is still not 0 because they point to each other. In this case, Python’s GC (garbage collector) needs to step in.

4. Python’s Garbage Collection: The Master of Handling Circular References

Python’s memory recovery uses generational garbage collection (Generational GC).

It divides objects into three generations:

• • Generation 0 (young generation): Newly created objects
• • Generation 1
• • Generation 2 (old generation): Long-lived objects

Why do it this way?

Because experience tells us:

“The longer an object survives, the less likely it is to be collected.”

GC workflow:

1. 1. Focus on checking young generation objects
2. 2. If the recovery effect is poor, continue to check the old generation
3. 3. The detection strategy mainly targets “container types” (list, dict, set, class, etc.)
4. 4. Discover unreachable references → Collect

5. Manually Using the Garbage Collection Module: gc

Python provides a module <span>gc</span> that allows you to:

• • View the current garbage collection status
• • Manually trigger GC
• • Debug circular reference issues

Check if GC is enabled

import gc
print(gc.isenabled())

Manually trigger garbage collection (commonly used for large data processing)

gc.collect()

Debug circular reference objects (advanced usage)

gc.set_debug(gc.DEBUG_LEAK)

6. Practical Memory Optimization Techniques

① Try to use generators (saves a lot of memory)

def gen():
    for i in range(10_000_000):
        yield i

Generators produce one element at a time, saving much more memory than lists.

② Use <span>del</span> to explicitly delete large objects

Suitable for large-scale data processing:

del big_list
gc.collect()

③ Avoid unnecessary variable references

For example, in data processing:

df2 = df1  # Shared memory! Will not copy

If you need to copy, be sure to use:

df2 = df1.copy()

④ For many small objects: use <span>slots</span> to optimize memory

class Person:
    __slots__ = ("name", "age")

Objects no longer use <span>__dict__</span> to store attributes, saving 30%-50% memory.

⑤ Try to use built-in types/libraries instead of reinventing the wheel

For example:

• • <span>list</span> append is much faster than a self-implemented linked list
• • Numpy arrays save 10 times more space than Python lists

7. Conclusion

Python’s memory management is not complicated; essentially, it is:

• • Reference counting: responsible for immediate release
• • Garbage collection: handles circular references
• • Memory pool: enhances object creation efficiency

Understanding these mechanisms will enable you to:

• • Avoid memory leaks
• • Write faster and more resource-efficient code
• • Be more stable in long-running projects (crawlers/servers)

Long press or scan the QR code below to get free access to Python public courses and hundreds of gigabytes of learning materials organized by experts, including but not limited to Python e-books, tutorials, project orders, source code, etc.



Recommended Reading
Practical Tips for Python 3.14: 10 Small Improvements to Make Your Code Clearer
10 Tricks for Processing Python Lists: How to Efficiently Handle Data Like a Big Company Engineer?
Practical Python Programming: Implementing a Batch Processing Tool for Excel (Desktop Utility Script)
Practical Python: How to Read Multiple Excel Formats and Achieve Cross-Sheet Matching and Merging (Supports XLS / XLSX)

Click to read the original text for more information