In the previous chapter, we discussed the workflow of disks and several typical metrics for evaluating disk performance, along with a brief introduction to these metrics and the tools for obtaining them.
For detailed information, please refer to: Linux Performance Tuning: A Detailed Explanation of Disk Workflow and Performance Metrics.
Today, we will focus again on these metrics, as they do not exist in isolation; they are often interrelated. Moreover, the same phenomenon may require several metrics to be assessed together. Additionally, apart from these overall performance metrics, we sometimes need to analyze process-level I/O requests to further investigate the causes of issues.
1. Disk Performance Metrics
Let’s briefly review these metrics:
Utilization: Indicates the busyness of the disk, represented as the percentage of time spent processing I/O requests during a collection cycle.Saturation: Measures the length of the disk queue, i.e., the number of I/O requests waiting to be processed.IOPS: The number of I/O operations per second, generally used to measure the speed at which the disk processes requests, especially for workloads involving a large number of small file requests, where this value is often required to be relatively high.Throughput: The amount of data transferred per second, which is often closely related to bandwidth.Response Time: The time taken for an I/O request from initiation to completion, generally including queue transfer time, queue wait time, and disk processing time.
The values of these metrics can be obtained using the iostat tool. So, what are the relationships between these metrics? Let’s break them down.
-
Utilization and Saturation: When the disk utilization continues to rise, even reaching 100%, it indicates that the disk is very busy. At this point, new requests may enter a queue, causing the saturation value to rise sharply. Conversely, if the disk has weak processing capability or is faulty, a sudden spike in requests may lead to queue blockage, and the utilization may not necessarily be very high.
-
Utilization and IOPS, Throughput: Within the peak performance limits of the disk, as the business load increases, the values of IOPS and throughput will also rise, accompanied by an increase in utilization. When the limit is reached, if the load continues to increase, the values of IOPS and throughput may not only stop rising but may even decline due to queuing or repeated retries, leading to increased saturation.
-
Response Time and Utilization, Saturation: To ensure that a request is processed quickly enough, it is necessary to have a sufficiently fast network (short transfer time), no queuing (saturation close to 0), and fast disk processing (physical characteristics, SSD > HDD). As mentioned earlier, if utilization exceeds a certain threshold, saturation will also increase, leading to a sharp rise in response time, which directly manifests as significant lag in business operations.
-
IOPS and Throughput: For small individual I/O requests, such as 4K, the disk may reach its IOPS limit, but the throughput may not be high. Similarly, if individual I/O requests are large and reach the throughput limit, the primary limiting factor may be bandwidth.
Therefore, when considering these metrics, we need to make a comprehensive judgment based on business usage. In summary: as the load (IOPS and throughput) gradually increases, utilization increases steadily, and saturation remains relatively unchanged; however, when the load reaches a certain height, response time begins to increase, and saturation also starts to rise; when the load reaches 100%, saturation will increase rapidly, and response time will rise sharply.
Having understood the overall performance metrics, let’s look at process-level I/O and the tools used to monitor it.
2. Process-Level I/O
To monitor process I/O, we mainly use pidstat, iotop, and the /proc filesystem.
# Install pidstat and iotop on Ubuntu system
sudo apt install iotop
sudo apt install sysstat
#/proc is built-in to the system
1. Use iotop -o -d 1 to observe the read/write speed of threads and the percentage of I/O wait time in real-time.
Total DISK READ : 0.00 B/s | Total DISK WRITE : 499.55 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 436.22 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
27464 be/4 root 0.00 B/s 3.52 K/s 0.00 % 0.76 % qemu-system-x86_64 -name guest=i-oxxx5~ci.0,addr=0xd -msg timestamp=on [worker]
1792 be/3 root 0.00 B/s 66.84 K/s 0.00 % 0.01 % [jbd2/sda3-8]
913 be/4 root 0.00 B/s 182.93 K/s 0.00 % 0.00 % systemd-journaldDISK READ/DISK WRITE: The real-time read/write speed of the process, measured in B/s, K/s, M/s.IO>: The percentage of I/O wait time, which is the time spent waiting for I/O completion as a percentage of total running time; the lower this value, the better.SWAPIN: The percentage of time spent swapping from swap to physical memory.TID: Thread ID number.PRIO: I/O priority.
2. Use pidstat -d 1 to continuously observe the I/O statistics of processes, including reads per second (kB_rd/s) and writes per second (kB_wr/s).
root@node:~# pidstat -d 1
Linux 4.15.0-58-generic (cs1ahyper01n07) 11/13/2025 _x86_64_ (64 CPU)
10:12:05 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
10:12:06 PM 0 913 0.00 3.70 0.00 0 systemd-journal
10:12:06 PM 0 2598 0.00 155.56 3.70 0 cron
10:12:06 PM 0 4855 0.00 29.63 0.00 0 bkunifylogbeat
10:12:06 PM 104 11940 0.00 3.70 0.00 0 rsyslogd
10:12:06 PM 0 24244 0.00 7.41 0.00 0 qemu-system-x86
10:12:06 PM 0 48974 0.00 485.19 0.00 0 qemu-system-x86
PID: Process IDkB_rd/s: Amount of data read per second (KB)kB_wr/s: Amount of data written per second (KB)kB_ccwr/s: Amount of data canceled from writing per second (KB)
3. You can also check the filesystem /proc, specifically in /proc/<pid>/io.
# For example:
cat /proc/123/io
# Output explanation:
rchar: 145678 # Number of characters read (including cache)
wchar: 234567 # Number of characters written (including cache)
syscr: 1234 # Number of read system calls
syscw: 5678 # Number of write system calls
read_bytes: 45678 # Actual bytes read physically
write_bytes: 123456 # Actual bytes written physically
cancelled_write_bytes: 12345 # Canceled write bytes
It is important to note that this data counts the cumulative values since the process started; restarting will reset the count.
With this observational data, we can better analyze performance changes from overall disk I/O to more detailed process-level I/O. In the future, we will continue to explore the factors affecting disk latency. We welcome your attention, and if you have any related questions, feel free to share in the comments section!
Linux Performance Tuning Series
– Memory Section
Linux Performance Tuning: About Memory
Linux Performance Tuning: Why Has Swap Increased?
Linux Performance Tuning: Understanding Caches in Memory
Linux Performance Tuning: How to Quickly Locate Memory Leaks?
Linux Performance Tuning: Detailed Usage of Memory Analysis Tools memleak-bpfcc and valgrind
Linux Performance Tuning: A Comprehensive Guide to Troubleshooting Memory Issues
– Disk Section
Linux Performance Tuning: In-Depth Understanding of Filesystems and Disks
Linux Performance Tuning: Detailed Explanation of Disk Workflow and Performance Metrics
Linux Performance Tuning: Detailed Explanation of Common Scenarios for FIO Performance Testing