Quickly Identify CPU-Intensive Processes in Linux and Locate Specific Code Locations

Determining which program (or thread) is consuming the most CPU in a Linux system and pinpointing the specific code location is a classic issue in system performance troubleshooting. I will provide you with a complete guide from macro to micro, from simple to in-depth.

Overview: Two Steps to Solve the Problem

1. Identify High CPU Processes: Find out which process (or thread) is the culprit.

2. Locate Hot Functions within the Process: Analyze what code the process is executing that is causing high CPU usage.

Step 1: Determine the Program (Process/Thread) with the Highest CPU Usage

Method 1: Use the <span>top</span> Command (Classic, Basic)

<span>top</span> is a built-in real-time system monitoring tool in Linux.

1. Start top:

   top

Check Sorting: After entering <span>top</span>, the process list may not be sorted by CPU usage by default.

• Press <span>P</span> (uppercase, Shift+p) to sort the process list in descending order of CPU usage. The process at the top is the one consuming the most CPU.

• Press <span>M</span> (uppercase, Shift+m) to sort by memory usage.

Key Information Interpretation:

• <span>PID</span>: Process ID, uniquely identifies a process.

• <span>USER</span>: Process owner.

• <span>%CPU</span>: CPU usage percentage.

• <span>COMMAND</span>: Process name or startup command.

Advanced Tip: View CPU Usage at the Thread Level In <span>top</span>, press <span>H</span> (uppercase, Shift+h) to toggle thread information display. Each entry in the list represents a thread, allowing you to see which thread under a process is consuming CPU. Note the PID of this high CPU thread (at this point it represents the thread ID, TID).

Method 2: Use the <span>htop</span> Command (Recommended, More Powerful)

<span>htop</span> is an enhanced version of <span>top</span>, with a more user-friendly interface and more intuitive features. It usually needs to be installed:

# Ubuntu/Debian
sudo apt install htop
# CentOS/RHEL/Fedora
sudo yum install htop    # or sudo dnf install htop

Using <span>htop</span>:

1. After starting, it is sorted by CPU% by default.

2. It uses colors to distinguish different resources, making it clear at a glance.

3. Similarly, you can enter settings by pressing <span>F2</span> to ensure that the thread view is enabled (<span>Display option</span> -> <span>Tree view</span> and <span>Show custom thread names</span> are very useful), allowing you to clearly see the tree structure of processes and threads.

Method 3: Use the <span>ps</span> Command (Script-Friendly)

<span>ps</span> can quickly take a “snapshot”, suitable for use in scripts.

# Display the top 10 processes with the highest CPU usage at the current moment
ps aux --sort=-%cpu | head -n 10
# Display thread information (TID is the LWP column)
ps -eLf --sort=-%cpu | head -20

At this point, you have found the PID (or TID) of the high CPU process. Assume the PID we found is <span>1234</span>.

Step 2: Quickly Locate the Specific Code Location

After finding the PID, the next step is to analyze what this process is “doing”, which function or system call is consuming the CPU.

Method 1: Use <span>perf</span> (Preferred Recommendation, Powerful)

<span>perf</span> is a performance analysis tool built into the Linux kernel, very powerful.

1. Install perf (if not already installed):

# Ubuntu/Debian
sudo apt install linux-tools-common linux-tools-$(uname -r)
# CentOS/RHEL
sudo yum install perf

2. Real-time Analysis of a Specific Process:

# Perform a 30-second CPU sampling analysis on the process with PID 1234
sudo perf top -p 1234
# Or, directly record analysis data and generate a report
sudo perf record -F 99 -p 1234 -g -- sleep 30  # Sample at 99Hz for 30 seconds, -g indicates to record the call stack
sudo perf report --stdio  # Display the report in text form in the terminal
sudo perf report          # Open an interactive TUI interface to view the report (recommended)

Interpretation of <span>perf report</span> Output:

• It will display a list of “hot” functions sorted by CPU usage ratio.

• Expand the <span>+</span> sign next to the function (press Enter in the TUI interface) to see the complete call stack (Call Stack), which directly leads you to the specific function or even code line that consumes the most CPU (if debug information is available).

3. Analyze the Entire System:

# Directly view the functions consuming the most CPU in the entire system
sudo perf top

Method 2: Use <span>gdb</span> (Debugger, Precise Location)

<span>gdb</span> can be directly attached to a running process to view its current execution state.

1. Attach to the Process:

sudo gdb -p 1234

2. View Thread Stack:

At the <span>(gdb)</span> prompt:

# View the stack of all threads
(gdb) thread apply all bt
# If there are many threads, you can first switch to the thread with high CPU (need to know TID first)
(gdb) thread <TID>
(gdb) bt  # View the call stack of the current thread

<span>bt</span> (backtrace) command will print out the entire function call chain from the <span>main()</span> function to the current execution point. If you find a certain function (especially one you wrote) frequently appearing at the top of the stack after executing <span>bt</span> multiple times, it is the hot code.

Note: Using <span>gdb</span> will pause (Suspended) the target process’s execution, and after analysis, you need to enter the <span>detach</span> command to detach and let the process continue running, and finally <span>quit</span> to exit. This is suitable for online problem troubleshooting but needs to be used with caution.

Method 3: Use <span>bpftrace</span>/<span>eBPF</span> (Next-Generation Advanced Tool)

This is a more modern and powerful kernel tracing technology, suitable for more complex scenarios. For example, use <span>bpftrace</span> to count the stack of a specific process:

# Count the user-space stack of the process with PID 1234, printing every 5 seconds
sudo bpftrace -e 'profile:hz:99 /pid == 1234/ { @[ustack] = count(); }'

This will produce a flame graph-friendly output, showing which code paths are executed the most.

Summary and Best Practice Process

1. Quick Confirmation: Use <span>top</span> or <span>htop</span>, press <span>P</span> to sort, and find the high CPU PID.

• Advanced: In <span>htop/top</span>, press <span>H</span> to find the high CPU TID.

In-Depth Analysis:

• Preferred <span>perf</span>: Use <span>sudo perf top -p <PID></span> for real-time viewing, or <span>perf record</span> + <span>perf report</span> for detailed analysis. This is the most universal and effective method.

• Backup <span>gdb</span>: Use when you need to precisely view the process state at a certain moment,<span>sudo gdb -p <PID></span>, then <span>bt</span> to view the stack. Remember it will interrupt the process.

• Advanced <span>bpftrace</span>: If you need more complex filtering and statistics, or to generate flame graphs.

Key Prerequisite: Debugging Information (Debug Symbols) To see meaningful function names (instead of a bunch of hexadecimal addresses), you need to ensure your program is compiled with debugging information. Add the <span>-g</span> option during compilation (e.g., GCC):

gcc -g -o my_program my_program.c

1. For system libraries, you may need to install <span>-dbgsym</span> or <span>-debuginfo</span> packages.

With the above combination of techniques, you will surely be able to pinpoint the code line from the process and accurately identify the performance bottleneck.