Many people habitually type the cp command in a Linux environment, specifying the source path and destination path, pressing Enter, and waiting for the progress to finish.

For small files, this approach is almost without issue.

However, once we enter a real production environment where the objects being copied are large files of dozens or even hundreds of GB, or thousands of small files, this seemingly straightforward method reveals a significant efficiency gap.

This is because copying large files at once may occupy I/O bandwidth for too long, affecting the operation of other system services, and even causing performance fluctuations due to cache being flushed.

Therefore, quickly copying files is not as simple as typing a command; it involves factors such as command selection, parameter optimization, file system characteristics, hardware performance, and kernel I/O scheduling.

Before understanding how to make copying faster, we need to clarify the essential process of copying in Linux.

File copying is not a simple binary data copy; it is a process of alternating read and write between user space and kernel space.

The command issues system calls, such as the commonly used read() and write(), which read the data from the source file into the kernel cache and then write it to the target file’s location.

If it is a cross-file system copy, the kernel will actually write the data to the target storage device.

If it is on a file system that supports Copy-on-Write (COW), such as Btrfs or XFS, certain commands can “copy” by referencing the original data blocks, which does not actually produce a real write, and this type of copy is almost instantaneous.

Take our most common tool, cp, for example.

Its default implementation reads a certain size of data blocks and writes them to the target file. In GNU coreutils, the block size may not be optimal for all hardware.

If you want to complete the copy as quickly as possible on the same file system, you can use:

cp --reflink=auto --sparse=always source_file target_file

This way, on a COW-supported file system, the data will not be rewritten; it will only generate new references, and --sparse=always avoids writing zero-filled blocks in the case of sparse files, significantly reducing I/O volume.

On a regular file system, the speed of cp is mainly limited by hard disk performance and block size. Although GNU cp does not have a direct parameter to adjust block size, its core logic is similar to the idea of manual chunking.

When files are large and you want to monitor progress in real-time, rsync is a better choice.

Many people view rsync as a remote synchronization tool, but it can also perform well in local copying.

The following command is very useful for copying files between local disks:

rsync -ah --progress source_file target_directory

-a preserves file permissions, timestamps, and other attributes, -h makes the output more readable, and --progress provides real-time progress display.

rsync tries to utilize cache and performs differential copying whenever possible, which is especially efficient for updating files.

In high I/O pressure systems, rsync will use bandwidth more smoothly than cp, reducing interference with other processes.

So what if the goal is to pursue maximum speed?

In that case, <code>dd can be brought out.

It directly operates on the underlying block device, without relying on additional processing from the file system, making it very suitable for pure data transfer scenarios.

For example:

dd if=source.iso of=/destination/source.iso bs=64M status=progress

Here, bs indicates the block size for each read and write operation. Increasing it can reduce the number of system calls, thereby lowering CPU overhead.

On mechanical hard drives, 64M is often a balance point that enhances throughput without consuming too much memory;

on SSDs or NVMe, the optimal block size can be larger, even exceeding 256M, which needs to be determined through testing.

status=progress provides real-time progress feedback.

Then, in cross-server copying scenarios, operations and maintenance personnel may find that scp is more suitable:

scp -C source_file user@remote:/target_path

Where -C enables compression, which can significantly improve speed when network bandwidth is limited.

However, the downside of scp is that it does not support resuming interrupted transfers; if the connection drops midway, the transfer needs to be restarted.

In contrast, rsync‘s remote synchronization mode is more suitable for transferring large files:

rsync -ah --progress source_file user@remote:/target_path

It supports resuming interrupted transfers and can skip already completed parts when some data has been transferred, saving a lot of time and bandwidth.

When copying a large number of small files instead of large files, performance bottlenecks often occur in the file system’s metadata operations.

These operations involve opening, closing files, writing permissions, etc., which require higher random access to the disk.

So how can we optimize this?

A common method is to first package these small files into an archive stream and then perform the copy.

tar cf - source_directory | (cd /target_directory && tar xpf -)

This method reduces the frequency of directory traversal and metadata writing, significantly improving overall speed, especially on network file systems.

So far, I have discussed the selection of command tools, but in addition to command selection, Linux system parameters also directly affect copy speed.

Have you noticed that <code>read_ahead_kb can control the pre-read amount during sequential reads? Appropriately increasing it can enhance performance during large file transfers:

cat /sys/block/sda/queue/read_ahead_kb echo 4096 | sudo tee /sys/block/sda/queue/read_ahead_kb

Especially on NVMe devices, this value can be set higher because their latency and bandwidth are more suitable for large-scale pre-reading.

Similarly, the choice of I/O scheduler is also important. In tasks primarily involving sequential read and write, noop or deadline schedulers are usually more efficient than cfq.

At this point, some may think that in a multi-core, multi-channel storage environment, multi-threaded copying can further squeeze hardware performance to improve throughput.

It feels a bit like trading space for time!

You can use GNU parallel to run multiple copy tasks simultaneously:

parallel -j 4 cp {} /target_directory ::: file1 file2 file3 file4

-j4 indicates running four tasks simultaneously. The specific number of threads should be determined based on the device’s CPU core count, disk bandwidth, and system load.

In NVMe RAID or high-performance SAN storage, multi-threaded copying can significantly shorten the total task time.

Having discussed various copying command tools, what if the copy fails?

We can’t just start over without knowing the reason!

Of course, we can, but the cause must be addressed!

Therefore, monitoring the copy process is especially important for everyone in a production environment.

Let's look at a tool called <code>pv, which can insert real-time rate and progress information into the data stream pipeline:

pv source_file > /target_path/source_file

Then, when combined with dd, you can enjoy both block size optimization and real-time monitoring:

pv source_file | dd of=/target_path/source_file bs=64M

During long transfers, pv can help us determine if the speed is abnormal and whether we need to interrupt the task or adjust the strategy.

Overall, these command tools are not very complex since they operate on local devices. But what if they don’t?

For example, in a cloud computing environment.

To conclude, in a cloud computing environment, copy speed will also be limited by the virtualization layer and underlying network protocols.

When copying through the internal high-speed network provided by public clouds, compressing the transfer can often improve overall efficiency:

tar czf - large_directory | ssh user@remote "tar xzf - -C /target_path"

This method not only reduces the amount of data transferred but also merges many small files into a single stream, lowering the overhead caused by metadata operations.

In distributed file systems like CephFS and GlusterFS, reducing the number of small files and batch transfers is also key to optimization.

To truly achieve fast file copying in Linux, the core lies in understanding the implementation mechanisms of different tools, the working principles of operating system I/O, and the limitations of specific hardware and network conditions.

For a single large file, block size adjustment and low-overhead commands are key;

for a large number of small files, packaging for transfer can significantly enhance speed;

when crossing networks, compression and resuming capabilities determine efficiency.

In actual production environments, dynamically selecting solutions based on hardware characteristics, network conditions, and system load is the fundamental method to ensure copying is both fast and stable.

Alright, I wrote this rather hastily and briefly, but I hope it helps everyone!

As the saying goes: Keep your passion alive to enjoy it for a long time!

⌨️ If you found this helpful, remember to give it a thumbs up!⌨️ Sharing is the greatest support!Word count: 2624—Recommended Reading— How to Quickly Copy Files in Linux? Sanhua Cat Media Matrix

Unauthorized reproduction is prohibited

Welcome to share to your circle of friends

Submissions, business inquiries

[email protected]

As the saying goes: Keep your passion alive to enjoy it for a long time!

Leave a Comment Cancel reply