This article mainly explains the knowledge related to file handles in Linux (File Handle) and commands.
A file handle (FD) is a non-negative integer that serves as anindex, allowing a process to find the corresponding information of anopen file in itsfile descriptor table. Essentially, it is an abstract reference to all I/O resources, representing not only files on disk but also pipes, sockets, devices, etc.
First question: What is a file handle?
In Linux, a file handle (File Handle), more commonly referred to as a file descriptor (File Descriptor, FD), is a non-negative integer. It is an abstract concept used to represent an open file, device, network connection, pipe, or any other I/O resource. It essentially serves as an abstract reference to all I/O resources, representing not only files on disk but also pipes, sockets, devices, etc.
It can be thought of as a “ticket” or “index” between the application and kernel resources. The application does not directly operate on the file itself but instead passes this “ticket” to the kernel, which performs the actual read and write operations.
Next question: Why do we need file handles?
File handles are designed to isolate applications from the underlying hardware and the complexities of the file system. Applications only need to care about which “number” (FD) they are operating on, without needing to worry about which track or sector the data is on the disk.
They serve as a unified interface, as Linux follows the philosophy of “everything is a file.” Whether it is a real file, hardware device (like /dev/sda), standard input/output, or network sockets (Socket), they can all be operated using the same interface: open() -> read()/write() -> close(), greatly simplifying the programming model.
Understanding resource management, the kernel tracks the resources used by each process through file descriptors, ensuring that resources are correctly allocated and released.
Manifestation of “everything is a file” (the core idea of Linux systems)
|
File Descriptor (FD) |
Default Associated Object |
Description |
|
0 |
stdin (standard input) |
Typically corresponds to keyboard input |
|
1 |
stdout (standard output) |
Typically corresponds to terminal screen |
|
2 |
stderr (standard error) |
Typically corresponds to terminal screen |
|
3+ |
Regular files, sockets, pipes, etc. |
Returned by system calls such as open(), socket(), pipe(), etc. |
Next question: How are file handles managed?
Understanding these three kernel data structures and their relationships is key to a deeper understanding of file handles.
1. Per-Process File Descriptor Table (Per-Process File Descriptor Table)
Ownership: Private to each process.
Content: An array where the index is the file descriptor (FD, such as 0, 1, 2, 3…), and each element of the array is a pointer to an entry in the open file table.
2. System-wide Open File Table (Open File Table)
Ownership: Global to the kernel.
Content: Contains file status flags (such as O_RDONLY, O_APPEND), current file offset (read/write position), and a pointer to the corresponding entry in the i-node table.
Key Point: Different file descriptors from different processes can point to the same entry in the open file table (for example, through fork() inheritance in child processes or using the dup() system call). This is why they share the file offset.
3. i-node Table (i-node Table)
Ownership: Global to the kernel (caches inode information on the file system).
Content: Contains metadata of the file (such as permissions, owner, size, timestamps) and pointers to data blocks on the disk.
Key Point: Each file has a unique i-node. Multiple entries in the open file table can point to the same i-node, meaning the same file can be opened multiple times.
Related Practical Operations – Viewing and Managing File Handles
1. View Limits
System-wide maximum limit: The maximum number of file handles that can be opened by the entire system.
cat /proc/sys/fs/file-max
User-level process limit: The maximum number of file handles that a single process can open.
ulimit -n # View soft limit (current effective limit)
ulimit -Hn # View hard limit (maximum value that soft limit can be set to)
2. Adjust Limits
Temporary adjustment (will not persist after reboot):
# Temporarily adjust the user process limit for the current shell session
ulimit -n 65535
# Temporarily adjust the system-wide limit
echo 2000000 > /proc/sys/fs/file-max
# or use sysctl
sysctl -w fs.file-max=2000000
Permanent adjustment:
Adjust user-level limits: Edit/etc/security/limits.conf, and add at the end:
soft nofile 65535 # Set soft limit for all users
hard nofile 65535 # Set hard limit for all users
# Or for specific users
username soft nofile 100000
username hard nofile 100000
Effective condition: Requires the user to log in again.
Adjust system-wide limits: Edit/etc/sysctl.conf, and add:
fs.file-max = 2097152
Effective condition: Execute sysctl -p and reboot the system to ensure everything is in order.
3. View Usage
View the current number of handles used by the system:
cat /proc/sys/fs/file-nr
Outputs three numbers: number of allocated handles| number of used but unreleased handles | maximum handle limit (file-max)
Note: The first value is usually larger than the second because the kernel caches some allocated but unused handles.
View details of handles opened by a specific process:
lsof -p <PID> # List all files and information opened by a specific process
lsof -p <PID> | wc -l # Count how many file handles a specific process has opened
Find the process that opened a specific file:
lsof /path/to/file
A common classic issue: “Too many open files” error
Reason:
1. The process forgot to close files or connections after opening them (code bugs, handle leaks).
2. The system or user limits are set too low to meet the application’s concurrency requirements.
Troubleshooting Steps:
1. Confirm the error: See the error through dmesg | tail or application logs.
2. Locate the process: Use lsof -p <PID> | wc -l to confirm if the number of handles for that process is close to the limit set by ulimit -n . You can also use lsof -p <PID> to see which files are opened and determine if there are leaks (for example, log files being opened infinitely instead of being appended).
3. Check system limits: Confirm if cat /proc/sys/fs/file-max and ulimit -n are sufficient.
4. Resolution:
Short-term: Appropriately raise the limits (see Chapter 3).
Long-term: Fix the application’s code bugs, ensuring all opened resources are properly closed.
One point to note: File handles vs. process handles
In the context of Linux, file handles typically refer to file descriptors. However, in a broader context, “process handles” may refer to identifiers that operate on processes (such as HANDLE in Windows), which is a different concept from Linux’s file descriptors. In Linux, we focus solely on file descriptors.
The article ends here.