Hello everyone, I am Yu Yong.
Yesterday, I encountered a problem: a Linux server used for data transfer suddenly stopped being able to perform data transfers. Upon checking the logs, there were messages indicating that files could not be saved and ‘No space left on device’.
Using the command df -Th to check disk space, I found that there was still plenty of space available in each partition. I tried restarting the application, MinIO, and TCS, but the message ‘No space left on device’ persisted.
Why does the system report ‘No space left on device’ when there is clearly still space available?
After searching online and consulting with Fang from our Linux technical group, we arrived at the same solution: using the commands df -i or df -ia to check if the inodes are full.
To my surprise, I found that the inodes for the opt path were already 100% used, which is exactly where our application stores its files.
Since this machine is used for data transfer, it contains a large number of small files. I checked the system’s temporary directory and found data files stored from January 1, 2024, to the present. After using the find command to locate and delete the files from 2024, the system returned to normal.
Knowledge Expansion
1. What is an inode?
In a Linux system, the core of file storage is the inode (metadata) and block (data block). An inode is the ‘metadata manager’ of a file; it does not store the actual data of the file but records the file type, permissions, owner, timestamps, hard link count, and pointers to data blocks (which point to the block locations where the file data is stored). Each file/directory corresponds to a unique inode; even if the filenames are different (hard links), as long as they point to the same inode, they are considered the same file. A block is the ‘data carrier’ of the file, actually storing the binary data of the file (such as text, images, program code, etc.). Blocks are allocated in fixed sizes (e.g., 4 KB), and even if a file is very small (e.g., 1 KB), it will occupy a full block (which may lead to space waste, known as ‘internal fragmentation’).
2. Why can inodes run out?
The file system allocates a fixed number of inodes when formatting (the number is determined by the parameters during formatting, usually calculated automatically based on disk size), and the number of inodes cannot grow dynamically. When all inodes are occupied and there are no free ones, it leads to inode exhaustion, meaning that even if there is remaining disk space, new files or directories cannot be created.
Summary: When inodes are full, no new files can be added; when blocks are full, the disk is full, and no new files can be added.
That’s all for today’s sharing.