Understanding Python’s Interning Mechanism

01

Introduction

When using Python for identity checks with is, we often encounter some confusing situations. Some data compares as equal using is, while other data of the same type does not. For example:

a = "abc"
b = "abc"
print(a is b)  # True
a = "ab-cd"
b = "ab-cd"
print(a is b)  # False

What is going on here?

To answer this question, we must discuss Python’s interning mechanism. In Python, the interning mechanism refers to the caching and reuse of certain immutable objects, allowing objects with the same value to share the same memory space, thereby reducing memory allocation and improving comparison efficiency. In simple terms, it allows identical values to share a single instance.

02

Advantages

– Memory savings: Avoids repeated memory allocation for identical immutable objects.

– Comparison speedup: Compares equality through memory addresses, which is faster.

03

Data Types

Type	Rule
String	Empty strings or strings composed of letters, numbers, and underscores, with no length limit.
Bytes	Empty bytes or bytes from 0-9.
Integer	Small integers from -5 to 256.
Boolean	True, False.
None	None.
Tuple	Empty tuple.

04

Strings

Python enables interning for certain strings, with the following rules for interned strings:

– Empty strings.

– Strings composed of letters, numbers, and underscores, with no length limit.

– Some system-reserved strings.

For example:

a = ""
b = ""
print(a is b)  # True
a = "hello_1"
b = "hello_1"
print(a is b)  # True

05

Bytes

Python enables interning for certain byte types, with the following rules for interned bytes:

– Empty bytes.

– Bytes from 0-9.

For example:

a = b""
b = b""
print(a is b)  # True
a = b"9"
b = b"9"
print(a is b)  # True

06

Integers

Python enables interning for certain integers, with the following rules for interned integers:

– Small integers from -5 to 256.

For example:

a = -1
b = -1
print(a is b)  # True
a = 100
b = 100
print(a is b)  # True

07

Booleans

Python enables interning for boolean types.

For example:

a = True
b = True
print(a is b)  # True
a = False
b = False
print(a is b)  # True

08

None

Python also enables interning for the None value.

For example:

a = None
b = None
print(a is b)  # True

09

Tuples

Python also enables interning for empty tuples.

For example:

a = ()
b = ()
print(a is b)  # True

10

sys.intern

For strings that do not meet the interning rules, sys.intern can be used to force interning.

For example:

import sys
a = sys.intern("-123")
b = sys.intern("-123")
print(a is b)  # True

11

Conclusion

The interning mechanism is a method used by CPython to optimize memory and comparison performance. Currently, it only supports certain immutable data types. For strings that do not meet the rules, sys.intern can be used to force interning.

Scan to follow