Mastering Parallel Tasks with Multiprocessing in Python

In Python, the `multiprocessing` module is part of the standard library, designed to support parallel computing and multi-process programming.

Unlike threading, `multiprocessing` achieves task parallelization by creating independent processes, which not only bypasses the limitations of Python’s Global Interpreter Lock (GIL) but also fully utilizes multi-core CPUs to improve program execution efficiency.Today, we will briefly introduce the basic usage of the `multiprocessing` module and demonstrate how to leverage processes to accelerate computations through multiple examples.1. Introduction to the `multiprocessing` ModuleThe `multiprocessing` module provides a process-based interface that allows us to easily create processes, use shared memory, and communicate between processes via queues.This way, programs can execute multiple tasks in parallel, thus enhancing efficiency, especially in CPU-intensive tasks.Main Features:

• Create and manage processes: New processes can be created using the `Process` class.
• Inter-process communication: Data can be passed between processes using `Queue`, `Pipe`, or `Value`.
• Shared data: Shared memory can be used to manage data accessed by multiple processes.

2. Creating Processes2.1. Creating Simple Processes with `Process`First, let’s look at how to create and start a new process using the `Process` class.

import multiprocessing
import time

def worker(num):
    print(f"Worker {num} started")
    time.sleep(2)
    print(f"Worker {num} finished")

if __name__ == "__main__":
    # Create processes
    process1 = multiprocessing.Process(target=worker, args=(1,))
    process2 = multiprocessing.Process(target=worker, args=(2,))

    # Start processes
    process1.start()
    process2.start()

    # Wait for processes to complete
    process1.join()
    process2.join()

    print("Both workers are done.")

Output:

Worker 1 started
Worker 2 started
Worker 1 finished
Worker 2 finished
Both workers are done.

Analysis:

• `multiprocessing.Process`: Used to create new processes.
• `start()`: Starts the process.
• `join()`: Waits for the process to complete, ensuring the program exits only after all processes have finished executing.

3. Inter-process CommunicationIn practical applications, we often need communication between processes.The `multiprocessing` module provides mechanisms like `Queue` and `Pipe` to facilitate data transfer between processes.3.1. Using `Queue` for Inter-process Communication

import multiprocessing

def worker(q):
    result = 5 * 5  # Perform task
    q.put(result)  # Put result into queue

if __name__ == "__main__":
    q = multiprocessing.Queue()

    process = multiprocessing.Process(target=worker, args=(q,))
    process.start()
    process.join()

    result = q.get()  # Get result from queue
    print(f"Result from worker: {result}")

Output:

Result from worker: 25

Analysis:

`Queue`: Used for inter-process communication, data is transmitted through the queue.
`put()`: Places data into the queue.
`get()`: Retrieves data from the queue.

4. Using `Pool` for Process Pool ManagementWhen we need to start a large number of processes to handle tasks in parallel, using the `Pool` class is more efficient.It allows for reusing processes in the pool, avoiding the overhead of frequently creating and destroying processes.4.1. Using `Pool.map()` for Parallel Computation

import multiprocessing

def square(x):
    return x * x

if __name__ == "__main__":
    with multiprocessing.Pool(processes=4) as pool:
        result = pool.map(square, [1, 2, 3, 4, 5])
    print(f"Squared numbers: {result}")

Output:

Squared numbers: [1, 4, 9, 16, 25]

Analysis:

• `Pool.map()`: Similar to the `map()` function, it assigns tasks to processes in the pool and returns results. Each process handles one element from the list.

4.2. Using `apply_async()` for Asynchronous Tasks

import multiprocessing
import time

def long_running_task(x):
    time.sleep(2)
    return x * x

if __name__ == "__main__":
    with multiprocessing.Pool(processes=4) as pool:
        result = pool.apply_async(long_running_task, (4,))
        print("Task is running asynchronously...")
        print(f"Result: {result.get()}")

Output:

Task is running asynchronously...
Result: 16

Analysis:

• `apply_async()`: Executes tasks asynchronously, allowing us to perform other operations while the task is still running.
• `get()`: Retrieves the result of the asynchronous task.

5. Shared MemoryThe `multiprocessing` module also provides shared memory objects, allowing different processes to directly access shared data without passing it through queues or pipes.5.1. Sharing Data with `Value` and `Array`

import multiprocessing

def increment(shared_value):
    for _ in range(1000):
        shared_value.value += 1

if __name__ == "__main__":
    shared_value = multiprocessing.Value('i', 0)

    process1 = multiprocessing.Process(target=increment, args=(shared_value,))
    process2 = multiprocessing.Process(target=increment, args=(shared_value,))

    process1.start()
    process2.start()

    process1.join()
    process2.join()

    print(f"Final value: {shared_value.value}")

Output:

Final value: 2000

Analysis:

• `Value`: Used to create a shared memory value, where `’i’` indicates that it is an integer type.
• Shared data can be accessed by multiple processes simultaneously, and `multiprocessing` automatically synchronizes data access.

6. Considerations

• Inter-process communication may be slower than communication between threads, so it is not suitable for all tasks.For I/O-bound tasks, the `threading` module may be more appropriate.
• When using multiple processes, the creation and destruction of processes incur some overhead, so choose an appropriate parallel strategy based on the scale and type of tasks.
• Python’s GIL limits the performance of multi-threading in CPU-intensive tasks, but `multiprocessing` is not subject to this limitation and can better utilize multi-core CPUs.

7. ConclusionThe `multiprocessing` module provides powerful multi-process support for Python. With tools like `Process`, `Queue`, and `Pool`, we can easily achieve parallel computation and efficiently utilize computing resources. Whether in CPU-intensive tasks or scenarios requiring concurrent execution of multiple tasks, `multiprocessing` delivers excellent performance.We hope the examples in this article will help you better understand and utilize Python’s process model, enhancing your program’s performance and efficiency.