When programming in C++ under the Linux environment, multithreading provides excellent concurrency capabilities, allowing programs to perform more efficiently when handling complex tasks. However, multithreaded programming is not without its challenges; deadlock issues lurk like hidden “killers,” potentially causing programs to become stuck. Once a deadlock occurs, the program is trapped in an unbreakable loop, with each thread waiting for the other to release resources, yet none willing to relinquish their hold, ultimately causing the entire program to stall.

This situation not only prevents the program from functioning correctly but can also impact the stability of the entire system. For example, in a network server program, if a deadlock occurs, the server may fail to respond to new client requests, leaving many users’ operations in limbo, with dire consequences. For C++ developers, mastering the skills to troubleshoot deadlocks is crucial. Today, we will delve into how to accurately locate and resolve deadlock issues using Shell commands and the powerful debugging tool GDB in the Linux system, revitalizing your program. First, let’s understand how deadlocks occur.

Part 1Deadlock — The Hidden Killer of Multithreaded Programming

In the realm of Linux C++ multithreaded programming, deadlocks are like a hidden assassin, constantly threatening the normal operation of programs. Multithreading endows programs with powerful concurrency capabilities, allowing us to fully utilize the performance of multi-core processors and improve execution efficiency. However, just as there are shadows behind the sunlight, the convenience brought by multithreading also introduces the tricky problem of deadlocks.

Imagine a narrow bridge that can only accommodate one person at a time. Two people approach the bridge from opposite ends simultaneously, and when they meet in the middle, neither is willing to back down, resulting in a stalemate. Consequently, neither can proceed, and they can only wait indefinitely; this vividly illustrates deadlock in real life. In multithreaded programming, when two or more threads are waiting for each other to release the resources they occupy, they fall into a similar impasse, and the program cannot continue, just like the two people stuck on the narrow bridge.

The dangers of deadlocks should not be underestimated, especially in systems that require high real-time performance and stability, such as server programs. In server programs, threads typically handle a large number of concurrent requests. If a deadlock occurs, some threads may become stuck and fail to respond to client requests in a timely manner, which not only reduces system throughput but can also lead to complete server failure, affecting many users’ normal usage. For instance, if a deadlock occurs on an online shopping platform’s server, users may be unable to place orders or make payments, and merchants may be unable to process orders, which would undoubtedly be a disaster for the platform’s operation and user experience.

In addition to server programs, deadlocks can also arise in scenarios that require frequent resource sharing and thread collaboration. For example, in a multithreaded file processing system, multiple threads may need to access and modify the same file simultaneously. If access control to the file resources is not handled properly, deadlocks can easily occur, leading to file processing errors, data loss, and other serious consequences.

Therefore, learning how to troubleshoot and resolve deadlock issues is crucial for Linux C++ programmers. Only by mastering effective troubleshooting methods can we quickly locate problems and find solutions when deadlocks occur, allowing programs to resume normal operation and ensuring system stability and reliability.

Part 2Exploring the Roots of Deadlocks

Deadlocks do not occur without reason; they are often the result of multiple factors acting together. In multithreaded programming, understanding the causes of deadlocks is like finding the key to solving the deadlock puzzle, helping us better prevent and troubleshoot deadlock issues. Next, let’s analyze the common causes of deadlocks in detail, along with specific code examples.

2.1 Improper Locking Order

When multiple threads need to acquire multiple locks, if they do so in an inconsistent order, it is like two intersecting tracks, making deadlocks likely to occur. Suppose there are two threads, thread1 and thread2, both needing to acquire locks mutex1 and mutex2. Thread1 first acquires mutex1 and then attempts to acquire mutex2; meanwhile, thread2 first acquires mutex2 and then attempts to acquire mutex1. When thread1 has mutex1 and thread2 has mutex2, the two threads will be stuck in an unresolvable knot, waiting for each other to release the locks they need, thus falling into a deadlock.

Here is a specific code example:

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1;
std::mutex mutex2;

void thread1Function() {
    mutex1.lock();
    std::cout << "Thread 1: Acquired mutex1" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex2.lock();
    std::cout << "Thread 1: Acquired mutex2" << std::endl;
    mutex2.unlock();
    mutex1.unlock();
}

void thread2Function() {
    mutex2.lock();
    std::cout << "Thread 2: Acquired mutex2" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex1.lock();
    std::cout << "Thread 2: Acquired mutex1" << std::endl;
    mutex1.unlock();
    mutex2.unlock();
}

int main() {
    std::thread thread1(thread1Function);
    std::thread thread2(thread2Function);

    thread1.join();
    thread2.join();

    return 0;
}

In this code, the order of acquiring locks in thread1Function and thread2Function is different, which is like planting a time bomb, creating conditions for a deadlock to occur. When both threads run simultaneously, as long as they acquire locks in an inconsistent order, a deadlock situation is highly likely to arise.

2.2 Repeated Locking

If a thread attempts to acquire a lock it already holds, and that lock does not support reentrancy (i.e., allowing the same thread to acquire the same lock multiple times), it is like setting up an obstacle for itself, inevitably leading to a deadlock. For example, in C++, if a thread calls the lock method on a std::mutex it has already locked, it will fall into a deadlock situation. This is because it cannot acquire a lock it already holds, and other threads cannot acquire that lock either, like a blocked passage where all threads cannot proceed, causing the entire program to come to a standstill.

Here is a code example demonstrating deadlock caused by repeated locking:

#include <iostream>
#include <thread>
#include <mutex>

std::mutex myMutex;

void recursiveFunction(int count) {
    myMutex.lock();
    std::cout << "Entering recursiveFunction, count: " << count << std::endl;
    if (count > 0) {
        recursiveFunction(count - 1);
    }
    myMutex.unlock();
    std::cout << "Exiting recursiveFunction, count: " << count << std::endl;
}

int main() {
    std::thread myThread(recursiveFunction, 3);
    myThread.join();
    return 0;
}

In this example, the recursiveFunction is recursive, and each call attempts to acquire the myMutex lock. During the recursive calls, since myMutex does not support reentrancy, the second attempt to acquire the lock will be blocked, leading to a deadlock. It is like a person entering a maze with only one entrance and blocking the entrance each time they enter, preventing themselves from getting out and others from entering.

2.3 Not Unlocking After Locking

After a thread acquires a lock, it should normally unlock it promptly after using the resource so that other threads can acquire the lock and access the resource. However, if a thread fails to release the lock due to an exception or logical error after acquiring it, it is like a person occupying a public resource without returning it, preventing other threads from acquiring that lock and ultimately leading to a deadlock.

Here is a code example of a thread that fails to unlock due to an exception, leading to a deadlock:

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex;

void someFunction() {
    mutex.lock();
    std::cout << "Locked the mutex" << std::endl;
    throw std::runtime_error("Something went wrong");
    mutex.unlock();
    std::cout << "Unlocked the mutex" << std::endl;
}

int main() {
    std::thread thread(someFunction);
    thread.join();
    return 0;
}

In this code, the someFunction throws an exception after acquiring the lock. Due to the exception being thrown, the mutex.unlock() statement is not executed, and the lock is not released. As a result, if other threads attempt to acquire this lock, they will wait indefinitely, leading to a deadlock. This is akin to a person borrowing something from someone else but forgetting to return it due to an unexpected situation, causing others to be unable to use it, resulting in resource waste and erroneous program operation.

Part 3Setting Up a Deadlock Experiment: Simulating Deadlock Scenarios

To better understand the phenomenon of deadlocks, let’s set up a simple deadlock experiment. By writing a piece of C++ code that deliberately creates a deadlock, we can later use Shell and GDB for troubleshooting.

3.1 Writing Deadlock Code

Here is a piece of C++ code that will cause a deadlock:

#include <iostream>
#include <thread>
#include <mutex>

std::mutex mutex1;
std::mutex mutex2;

void thread1Function() {
    mutex1.lock();
    std::cout << "Thread 1: Acquired mutex1" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex2.lock();
    std::cout << "Thread 1: Acquired mutex2" << std::endl;
    mutex2.unlock();
    mutex1.unlock();
}

void thread2Function() {
    mutex2.lock();
    std::cout << "Thread 2: Acquired mutex2" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(1));
    mutex1.lock();
    std::cout << "Thread 2: Acquired mutex1" << std::endl;
    mutex1.unlock();
    mutex2.unlock();
}

int main() {
    std::thread thread1(thread1Function);
    std::thread thread2(thread2Function);

    thread1.join();
    thread2.join();

    return 0;
}

In this code, we create two threads, thread1 and thread2, and two mutexes, mutex1 and mutex2. In the thread1Function, thread1 first acquires mutex1, then sleeps for 1 second before attempting to acquire mutex2; in the thread2Function, thread2 first acquires mutex2, also sleeps for 1 second, and then attempts to acquire mutex1. This different order of locking sets the stage for a potential deadlock.

3.2 Compiling and Running the Code

Save the above code as deadlock_example.cpp and compile it using g++:

g++ -g -o deadlock_example deadlock_example.cpp -lpthread

Here, the -g option is used to include debugging information in the executable file, facilitating later debugging with gdb. The -lpthread option links the thread library since we are using multithreaded programming.

After compilation, run the executable:

./deadlock_example

After running, you will find that the program outputs “Thread 1: Acquired mutex1” and “Thread 2: Acquired mutex2” but then becomes stagnant, failing to continue execution. This is a typical symptom of deadlock, where two threads are waiting for each other to release locks, preventing the program from progressing.

Part 4Shell Debuts: Insights into Process States

When we suspect that a program has encountered a deadlock, we can first use shell commands to observe the process state and gather key information, providing clues for further investigation into the deadlock.

4.1 Using ps aux to View Process Overview

ps aux is a very useful shell command that displays detailed information about all processes of all users in the current system. Through this command, we can obtain key data such as the CPU usage (% CPU) and memory usage (% MEM) of the processes. When troubleshooting deadlocks, this information can help us initially determine whether a process is in an abnormal state.

When we execute ps aux | grep deadlock_example (assuming the name of the executable file we compiled earlier is deadlock_example), we will get output similar to the following:

user     12345  0.0  0.1  123456 7890 pts/0    S    12:34   0:00 ./deadlock_example

In this output, %CPU indicates the percentage of CPU occupied by the process, and %MEM indicates the percentage of memory occupied. If a process is in a deadlock, it typically cannot execute tasks normally, and the CPU utilization will be very low, even close to 0. At the same time, since threads are blocked, the process may hold onto certain resources without releasing them, and memory usage may not show significant changes but will not release occupied memory. Therefore, when we see a process’s CPU utilization consistently at a low level, and memory usage shows no significant fluctuations, we need to be wary of the possibility of a deadlock.

4.2 top -Hp for In-Depth Thread Analysis

The top command is a dynamic tool for viewing process information in real-time, while top -Hp is a powerful extension of the top command that allows us to view the CPU and memory usage of each thread within a specified process. This is very helpful for troubleshooting deadlocks, as deadlocks often occur at the thread level. By examining the state of the threads, we can more accurately identify signs of deadlocks.

When we execute top -Hp <pid> (<pid> is the process ID found using the ps aux command), we will enter a real-time updating interface displaying detailed information about each thread within that process, including thread ID (PID), user (USER), CPU usage (% CPU), memory usage (% MEM), etc.

Under normal circumstances, we expect to see each thread actively working, with CPU usage showing some fluctuations, indicating that threads are executing tasks. However, if a deadlock occurs, some abnormal situations may arise. For example, some threads may have a CPU usage of 0 and be in a blocked state, while other threads are trying to acquire resources held by the blocked threads, causing those threads to be unable to continue executing, leading to a contradiction between active and blocked threads. If we observe this situation, we can further confirm the possibility of a deadlock, guiding us for deeper debugging with gdb.

Part 5GDB Takes Center Stage: Deep Debugging to Locate Deadlocks

After initially determining that a program may have encountered a deadlock using shell commands, we need to use the powerful debugging tool gdb for deeper analysis to accurately locate where the deadlock occurs.

5.1 gdb attach to Attach to a Process

The gdb attach command allows us to attach the debugger to a running process, like installing a real-time monitoring system on a moving car, enabling detailed observation and debugging of the internal running state of the process. Before using gdb attach, we need to obtain the target process’s ID (PID), which can be done using the ps aux command mentioned earlier.

Assuming we found the process ID of the deadlocked program to be 12345 using ps aux | grep deadlock_example, we can then use gdb to attach to that process:

gdb -p 12345

After executing the above command, gdb will pause the target process, allowing us to use various debugging commands to analyze the process. It is important to note that using the attach command in a production environment should be done with caution, as the attachment operation may cause the process to pause for a period, affecting its normal operation.

5.2 thread apply all bt to View Stack

Once gdb successfully attaches to the process, we can use the thread apply all bt command to view the stack information of all threads. The stack information is like the “footprints” of the program’s execution, recording the functions called by each thread during execution and their parameters. By analyzing this stack information, we can understand the execution state of each thread and find the line of code where the deadlock occurs.

After executing the thread apply all bt command in gdb, we will get output similar to the following:

Thread 1 (Thread 0x7ffff7fde700 (LWP 12345)):
#0  0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff7b2db98 in _L_lock_898 () from /lib64/libpthread.so.0
#2  0x00007ffff7b2d9d0 in __GI___pthread_mutex_lock (mutex=0x555555756040) at pthread_mutex_lock.c:64
#3  0x00005555555556d2 in thread1Function () at deadlock_example.cpp:9
#4  0x00007ffff7b27a0d in start_thread (arg=0x7ffff7fde700) at pthread_create.c:311
#5  0x00007ffff7a0c41f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7ffff77dd700 (LWP 12346)):
#0  0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff7b2db98 in _L_lock_898 () from /lib64/libpthread.so.0
#2  0x00007ffff7b2d9d0 in __GI___pthread_mutex_lock (mutex=0x555555756050) at pthread_mutex_lock.c:64
#3  0x0000555555555772 in thread2Function () at deadlock_example.cpp:16
#4  0x00007ffff7b27a0d in start_thread (arg=0x7ffff77dd700) at pthread_create.c:311
#5  0x00007ffff7a0c41f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

In this output, each line represents a function call, with #0 indicating the function currently being executed by the thread. The lines above #0 show the other functions that called the current function. By observing this stack information, we can see that both thread1 and thread2 are stuck in the __GI___pthread_mutex_lock function, waiting for different mutexes (0x555555756040 and 0x555555756050), which is the key clue to the occurrence of the deadlock. Coupled with the line numbers (deadlock_example.cpp:9 and deadlock_example.cpp:16), we can further pinpoint the specific lines of code where the deadlock occurs.

5.3 info threads for Auxiliary Analysis

In addition to the thread apply all bt command, the info threads command is also a valuable tool when debugging multithreaded programs. The info threads command lists the status and index of all threads, making it easy for us to analyze each thread’s situation one by one.

After executing the info threads command in gdb, we will get output like the following:

  Id   Target Id         Frame
  2    Thread 0x7ffff77dd700 (LWP 12346) "deadlock_example" 0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0
* 1    Thread 0x7ffff7fde700 (LWP 12345) "deadlock_example" 0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0

In this output, the Id column indicates the index of the thread, the Target Id contains the thread’s LWP (lightweight process ID) and thread name, and the Frame shows the function location where the thread currently resides. By using the info threads command, we can quickly understand the general status of each thread.

If we want to focus on a specific thread, we can use the thread <thread ID> command to switch to that thread and then use the bt command to view its specific stack information. For example, to view the stack information of thread 2, we can execute the following:

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff77dd700 (LWP 12346))]
#0  0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff7b31b97 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff7b2db98 in _L_lock_898 () from /lib64/libpthread.so.0
#2  0x00007ffff7b2d9d0 in __GI___pthread_mutex_lock (mutex=0x555555756050) at pthread_mutex_lock.c:64
#3  0x0000555555555772 in thread2Function () at deadlock_example.cpp:16
#4  0x00007ffff7b27a0d in start_thread (arg=0x7ffff77dd700) at pthread_create.c:311
#5  0x00007ffff7a0c41f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Through this method, we can analyze the execution status of each thread in more detail, further determining the code sections that triggered the deadlock.