Previously, we introduced several Linux zero-copy technologies, among which sendfile and splice are very similar. Both system calls allow data to be moved/copied directly in kernel space without passing through user space. In this article, we will analyze the kernel source code of these two system calls to see how they achieve zero-copy.
1. Analysis of the splice System Call Source Code
Below is the code for the splice system call. The function name expands to __x64_sys_splice, and it can be seen that it basically calls the __do_splice function. Let’s continue to explore the implementation of the __do_splice function.
The implementation of the __do_splice function, besides checking the parameters passed from user space, is to call the do_splice function. I have trimmed the code of the do_splice function to keep only the main structure, as shown below. The main structure of the do_splice function consists of three branches, which are divided based on whether the in file and out file are pipe files. If both in file and out file are pipe files, it calls the splice_pipe_to_pipe function. If only the in file is a pipe file, it calls the do_splice_from function. If only the out file is a pipe file, it calls the splice_file_to_pipe function. Below, we will focus on the do_splice_from and splice_file_to_pipe functions.
According to the logic of the splice example program we wrote in the previous article, we first use splice to send the data from the source file to the pipe, and then use splice again to send the data from the pipe to the target file. Therefore, the first call to splice should go through the do_splice function’s splice_file_to_pipe branch. Below is the implementation of the splice_file_to_pipe function, which basically calls the do_splice_read function.
The core code of the do_splice_read function is as follows. It handles two situations: Direct IO and non-Direct IO (using Page Cache). Therefore, splice can not only copy files through Page Cache as we demonstrated earlier, but also use Direct IO to directly transfer data through DMA and hardware devices. From the comments, it can be seen that in this case, because it cannot use the existing memory in Page Cache, additional memory needs to be allocated (for DMA transfer). For the non-Direct IO branch, the do_splice_read function calls the splice_read callback function in the file operation set.
Taking the ext4 file system as an example, let’s look at the implementation of the splice_read callback function: ext4_file_splice_read.
The implementation of the ext4_file_splice_read function is as follows. It calls filemap_splice_read, and from the function name containing filemap, it can be guessed that this function reads data from the Page Cache of the file.
Next, let’s look at the code of the filemap_splice_read function. I have simplified the code of this function to keep only the core code. It essentially first calls the filemap_get_pages function to get pages from the Page Cache (if there is no Page Cache hit, it needs to perform I/O to read the file data into the Page Cache), and then calls the splice_folio_into_pipe function to send the pages obtained from the Page Cache to the pipe file. Note that this is not a copy, but rather associates the pages from the Page Cache with the pipe, so that the pipe file has the address of the Page Cache pages of this file.
This is the process of sending data from the source file to the pipe using splice. Now let’s return to the __do_splice function to see the implementation of the do_splice_from branch and how this function sends data from the pipe to the target file. This function directly calls the splice_write callback function in the file operation set. Taking the ext4 file system as an example, this function is iter_file_splice_write.
Below is the simplified core code of the iter_file_splice_write function. Its core logic is to continuously get pages from the pipe, encapsulate the pages into iov, and then call the write_iter function in the file operation set to write the data from iov to the target file.
Taking the ext4 file system as an example, the write_iter callback function is ext4_file_write_iter, and the code is as follows. This function also handles two situations: Direct IO and non-Direct IO (using Page Cache). If the target file is opened in Direct IO mode, it calls the ext4_dio_write_iter function to directly use the pages from the pipe for DMA transfer, ultimately writing the data to the target file. If the target file is opened in non-Direct IO mode, it calls the ext4_buffered_write_iter function to copy the pages from the pipe to the Page Cache of the target file.
This is the process of sending data from the pipe to the target file. Below is a summary of the two splice system call processes using a function graph.
2. Analysis of the sendfile System Call Source Code
Below is the code for the sendfile system call. The function name expands to __x64_sys_sendfile64, and it can be seen that it basically calls the do_sendfile function. Let’s continue to explore the implementation of the do_sendfile function.
The core code of the do_sendfile function is as follows. It first checks whether the file corresponding to the out fd is a pipe file. If it is, it can obtain the pipe_inode_info opipe, and then it will call the splice_file_to_pipe function, which we have analyzed above, to pass the file data to the pipe. If the file corresponding to the out fd is not a pipe file, it goes to the upper branch and calls the do_splice_direct function.
Let’s first look at the case where the out fd corresponds to a regular file: the function do_splice_direct. From the comments of this function, we can vaguely feel that the sendfile system call relies on the implementation of the splice system call. If you have this feeling, it will be much easier to analyze further.
The core code of the splice_direct_to_actor function (after trimming) is as follows. It appears that it first creates an internal pipe (not visible to the user), then calls the do_splice_read function (which we also used in the splice system call) to send data from the source file to this internal pipe, and then calls the actor, which is a function pointer passed in. Returning to the do_splice_direct function, we see that this actor points to the direct_splice_actor function.
The code of the direct_splice_actor function is as follows. It is very simple, just calls the do_splice_from function, which we also analyzed when looking at splice.
This is the implementation of the sendfile system call. The following function graph describes this process.
It can be seen that the implementations of the sendfile and splice system calls are highly overlapping. The only difference is that the implementation of sendfile turns the user-provided pipe in splice into a private internal pipe on the current process, which is created by the kernel itself. The subsequent operations are exactly the same as splice: first calling the do_splice_read function to send data from the source file to the pipe, and then copying data from the pipe to the target file.