Multiprocessing in Python

Introduction

Processes in Python

from multiprocessing import Process
import time

def print_numbers():
    for i in range(10):
        time.sleep(1)
        print(i)

def print_letters():
    for letter in 'abcdefghij':
        time.sleep(1)
        print(letter)

if __name__ == '__main__':
    process1 = Process(target=print_numbers)
    process2 = Process(target=print_letters)

    process1.start()
    process2.start()

    process1.join() #  join method is used to wait for the process to finish
    process2.join()


------
$ python a.py 
0
a
b
1
c
2
d
3
e
4
5
f
g
6
h
7
8
i
j
9

Inter-Process Communication

The multiprocessing module in Python provides various tools for inter-process communication, including pipes and queues. Below is an example demonstrating communication between the main process and a child process.

from multiprocessing import Process, Queue

def worker(q):
    q.put('Hello, world!')

def main():
    q = Queue()
    p = Process(target=worker, args=(q,))
    p.start()
    print(q.get())
    p.join()

if __name__ == '__main__':
    main()

---
$ python a.py 
Hello, world!

Process Synchronization

The multiprocessing module in Python provides various tools for process synchronization, including locks and semaphores.

from multiprocessing import Process, Lock, Value
import time
import random

def worker(lock, shared_counter):
    """Child process task: modify the shared counter after acquiring the lock"""
    process_name = f"Process-{Process().name}"

    # Simulate random time-consuming operation (non-critical section)
    time.sleep(random.uniform(0.1, 0.5))

    # Use with statement to automatically acquire and release the lock (recommended)
    with lock:
        print(f"{process_name} entered critical section")

        # ---- Critical section begins ----
        current_value = shared_counter.value
        time.sleep(0.1)  # Simulate time-consuming operation (amplifying race condition)
        shared_counter.value = current_value + 1
        print(f"{process_name} modified counter: {current_value} -> {shared_counter.value}")
        # ---- Critical section ends ----

if __name__ == '__main__':
    # Create a shared counter (initial value=0) and a process lock
    counter = Value('i', 0)  # 'i' indicates a signed integer
    process_lock = Lock()

    # Create and start 5 processes
    processes = []
    for i in range(5):
        p = Process(target=worker, args=(process_lock, counter))
        processes.append(p)
        p.start()

    # Wait for all processes to complete
    for p in processes:
        p.join()

    # Verify result (should be 5 in correct cases)
    print(f"\nFinal counter value: {counter.value} (expected value: 5)")

---
$ python a.py 
Process-Process-2:1 entered critical section
Process-Process-2:1 modified counter: 0 -> 1
Process-Process-1:1 entered critical section
Process-Process-1:1 modified counter: 1 -> 2
Process-Process-5:1 entered critical section
Process-Process-5:1 modified counter: 2 -> 3
Process-Process-3:1 entered critical section
Process-Process-3:1 modified counter: 3 -> 4
Process-Process-4:1 entered critical section
Process-Process-4:1 modified counter: 4 -> 5

Final counter value: 5 (expected value: 5)

Process Pool

We can use the multiprocessing.Pool class to create a pool of processes. The number of processes in the pool is fixed, and when a new task is submitted to the pool, if there are idle processes, one will be assigned to the task. If all processes are busy, the task will wait until a process becomes idle.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(square, [1, 2, 3, 4, 5]))


---
$ python a.py 
[1, 4, 9, 16, 25]

I have a bold idea: every technical article I post has been personally verified to ensure current usability. I welcome friends to practice along; if you encounter issues, feel free to leave a message; or if you have technical difficulties, let me know, and I will verify and experiment. Let’s improve together and achieve mutual success!!!

Recommended HighlightsChina’s first free one-core one-G cloud server?The secrets that Linux veterans won’t tell: why tar.gz crushes zip and 7z?Highly recommended, 5 must-have productivity software for computersTested, an incredibly useful Linux connection tool, no less than xshll magic