🌟 The 24 Most Common Performance Tools for Linux Developers (In-Depth Guide)

—— Mastering everything from system bottlenecks to DPDK tuning in one article

In the world of Linux, performance analysis is a “core skill” for developers, operations engineers, and architects. Whether you are tuning large-scale servers, optimizing network forwarding performance (such as DPDK), or analyzing system stuttering issues, various performance tools are your most reliable partners.

This article will take you through an in-depth understanding of the 24 most commonly used and valuable Linux performance tools — covering multiple aspects including CPU, memory, disk, network, kernel, and containers. Additionally, it includes: Best Practices for DPDK System Tuning.

🧭 Table of Contents

1. ⭐ Basic Understanding of Linux Performance Analysis

2. ⭐ Introduction to 24 Core Performance Tools (Categorized by Domain)

3. ⭐ Tool Comparison Table (CPU/Memory/Network/IO/Kernel)

4. ⭐ Performance Analysis Thinking: How to Identify Bottlenecks?

5. ⭐ Dedicated Chapter on DPDK Tuning: NUMA, Hugepage, isolcpus, RPS/XPS

6. ⭐ Summary: Build Your Own Performance Analysis Methodology

🥇 1. Basic Understanding of Linux Performance Analysis

Although there are many performance tools in Linux, they can be categorized into five main directions:

Category	Focus	Core Issues
CPU	Scheduling, context switching, hot functions	What is the CPU busy with? Who is consuming the most?
Memory	Memory usage, page faults, NUMA	Why is memory insufficient/unbalanced?
IO (Disk)	IOPS, latency, queue depth	Why is the disk slow?
Network	Bandwidth, packet loss, queues, IRQ	Where is the network going? Where is the bottleneck?
Kernel/System	Scheduling, locks, preemption, soft interrupts	Why is the system stuttering?

👇 Next, we will officially enter the explanation of all tools.

🏆 2. The 24 Most Common Performance Tools for Linux (In-Depth Analysis)

🔥 1. CPU Performance Tools (8 Tools)

① top / htop — Overall CPU Usage

🧭 Location: Is the system really “busy”?

The most commonly used resource monitoring tool, suitable for quickly assessing system status:

• load average

• CPU usage (user/system/idle)

• Process usage

• Context switching (htop can display)

👉 Applicable Scenarios

• Preliminary check for system bottlenecks

• Whether the CPU is occupied by IRQ or kernel

② mpstat — Multi-Core CPU Utilization Analysis

🧭 Location: Is the load balanced across CPU cores?

mpstat -P ALL 1

It can show:

• Proportion of user/sys/irq/softirq for each core

• Whether there is a problem of “some cores being particularly busy”

• Especially important for observing isolated cores in DPDK

③ pidstat — View CPU/Context Switching for a Specific Process

pidstat -w -p <pid>

Can locate:

• Whether the scheduling overhead for a specific thread is large

• Whether the process context switching is abnormally high (common bottleneck!!)

④ mpstat / sar (CPU Subset) — Historical CPU Analysis

<span>sar -u 1</span> can track historical CPU trends.

⑤ perf — Function-Level CPU Flame Graph, Hotspot Analysis (The Strongest Tool)

🔥 The King of Linux Performance Analysis

It can do a lot of things:

• Sampling flame graphs

• Kernel/user-space hot functions

• Analyze cache misses, branch prediction failures

• Analyze soft interrupt overhead

Example:

perf top -C 1
perf record -F 99 -g -- sleep 10
perf report

⑥ ftrace — Kernel Tracing

Can trace:

• Scheduling delays

• Kernel function execution flow

• ISR, interrupt handling flow

Suitable for analyzing:

• System stuttering

• Delays caused by kernel locks

⑦ trace-cmd

A more user-friendly wrapper tool than ftrace.

⑧ turbostat (x86 only)

View CPU C-state/P-state suitable for analyzing performance issues caused by CPU power saving.

🔥 2. Memory Analysis Tools (5 Tools)

⑨ free / /proc/meminfo

Basic memory viewing tool.

Important metrics:

• available memory

• buffers/cached

• slab

• hugepages usage (must check for DPDK)

⑩ vmstat — Memory + IO + Scheduling Combined

vmstat 1

Important fields:

• si/so → swap (system is overwhelmed)

• cs → context switching

• bi/bo → IO

⑪ slabtop — Kernel Object Memory Statistics

Can check whether the kernel is running out of memory due to a specific object (e.g., TCP skbuffer).

⑫ numastat — NUMA Imbalance Analysis

Key tool for DPDK tuning.

Can check:

• Whether memory is crossing NUMA

• Whether numa_miss / numa_foreign is too high

⑬ smem / pmap — Process Memory Analysis

🔥 3. Disk/IO Tools (4 Tools)

⑭ iostat — IO Latency/IOPS View

iostat -x 1

Focus fields:

• r/s, w/s

• await (latency)

• util (device utilization)

⑮ pidstat -d — Single Process IO

Can locate “which process is doing excessive IO”.

⑯ blktrace — Low-Level IO Events

Used for in-depth analysis of IO queue behavior.

⑰ fio — Load Testing Tool

Commonly used for SSD/HDD performance testing.

🔥 4. Network Performance Tools (5 Tools)

⑱ ss — More Advanced netstat

ss -s
ss -tulnp

Can see socket, queue, TCP state, etc.

⑲ ethtool — Network Card Capabilities, Queues, Interrupt Information

Commonly checked before disabling RPS/XPS/TSO for DPDK:

ethtool -S eth0
ethtool -l eth0
ethtool -k eth0

⑳ tcpdump — Packet Capture Troubleshooting

㉑ nstat / dropwatch — Packet Loss Analysis

Can check:

• Packet loss at the network card level

• Packet loss at the kernel level (skb failure)

• Packet loss location before DPDK driver

㉒ nicstat / iperf

Network bandwidth testing tools.

🔥 5. System and Kernel Tools (5 Tools)

㉓ dstat — Comprehensive Monitoring

All-in-one for CPU + Memory + IO + Network.

㉔ ps / pstree — Process and Thread Structure

Used in conjunction with perf, pidstat to troubleshoot issues.

㉕ strace — System Call Tracing

Applicable for:

• Why is the program stuck?

• Why is a certain line of code executing slowly?

• Which syscall is blocking the program?

㉖ lsof — File Handle Tracking

Can check:

• Whether a program is leaking file handles

• Which process is occupying a certain port

㉗ sysctl — Modify System Kernel Parameters

Used for performance optimization, such as:

• Network queue length

• Scheduler parameters

• Kernel memory management parameters

🧩 3. Tool Comparison Chart

🔶 Classified by Business Scenario (Recommended to Save)

Scenario	Most Common Tools
CPU Hotspot Analysis	perf, ftrace, mpstat
Memory Leak/Anomalies	smem, vmstat, slabtop
Network Packet Loss	ethtool, nstat, dropwatch
System Stuttering	perf, trace-cmd, vmstat
Kernel Analysis	ftrace, trace-cmd
Process Debugging	strace, lsof, pidstat

🔶 Domain Distribution Chart (Suitable for Public Account Illustration)

🟦 CPU🟥 Memory🟨 IO🟩 Network🟪 Kernel

CPU：top, mpstat, pidstat, perf, turbostat
内存：free, vmstat, numastat, smem, slabtop
IO：iostat, pidstat -d, blktrace, fio
网络：ethtool, ss, nstat, dropwatch, iperf
内核：ftrace, trace-cmd, ps, strace, sysctl

🚀 4. Performance Analysis Thinking Model (Very Important)

99% of faults can be located using the following 5 steps:

① First determine “which type of resource” is busy?

• CPU?

• Memory?

• Network?

• Disk?

Tools: top, vmstat, iostat, sar

② Find out “which process” is consuming resources?

Tools: ps, pidstat, top

③ Find out “which function/module” is the most time-consuming?

Tools: perf, ftrace, flame graph

④ Determine if it is a kernel bottleneck

Tools: trace-cmd, ftrace

⑤ Identify who is causing the business to slow down

• TCP queuing?

• Memory NUMA crossing nodes?

• Disk latency?

• CPU overwhelmed by interrupts?

• Program logic bug?

🚀 5. DPDK Dedicated Tuning Guide (Including NUMA/IRQ Optimization)

① CPU Isolation isolcpus/nohz_full

isolcpus=1-10 nohz_full=1-10 rcu_nocbs=1-10

✔ Avoid kernel scheduling interference✔ Reduce scheduling overhead✔ Bind DPDK lcore with taskset

② Hugepage (Must)

DPDK relies on large pages to reduce TLB misses:

echo 4096 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

③ Disable IRQ Balance

systemctl stop irqbalance

④ Disable RPS/XPS

echo 0 > /sys/class/net/eth0/queues/rx-0/rps_cpus

⑤ NUMA Binding

--lcores "1@(0) 2@(1) 3@(1)"

⑥ BIOS Tuning

• Disable C-State

• Disable HT

• Disable energy-saving policies

• Enable large cache mode

⑦ Adjust UIO / VFIO Drivers

In DPDK scenarios:

• vfio-pci (secure)

• igb_uio (higher performance)

🔚 6. Summary: Build Your Own Performance Analysis System

Linux performance analysis is not about “the skills of a single tool”, but a complete system.