Daily Linux Command: Redirection

Do you need to memorize Linux commands? Daily Linux Command: man

In the previous article, we learned about pipes (|), where the concept of redirection was mentioned but not elaborated. In this article, we will delve into this concept in detail.

By default, the shell sets the standard input source of a program to the keyboard and the standard output target to the terminal (i.e., the screen). The redirection feature allows users to change these default settings, for example, to a file, enabling commands to read input directly from a file or write output to a file, greatly expanding the application scenarios of the shell.

Redirecting Standard Output

The redirection output symbol (>) can redirect the standard output of a command to a specified file. If you want to pass the output of a command to another command, you need to use the pipe we discussed in the previous article.

The command line format for redirecting output is:

command [args] > file_name

For example, using the cat command to read a file and redirect its output to another file:

>$ cat /etc/services > my_services.txt

At this point, the cat command will not output anything to the terminal but will completely copy the contents of /etc/services to the my_services.txt file. We can calculate the hash values of these two files to verify their consistency:

>$ sha256sum /etc/services my_services.txt
2d84c0e7fe5ee474f8dc0096ccce7a5b10286be3da82d276acc5e79e7b8e42c0  /etc/services
2d84c0e7fe5ee474f8dc0096ccce7a5b10286be3da82d276acc5e79e7b8e42c0  my_services.txt

The hash values of the two files are exactly the same.

⚠️ Warning: Redirection may overwrite file contents

Redirecting Standard Input

The redirection input symbol (<) allows the standard input of a command to come from a file instead of the keyboard.

The command line format for redirecting input is:

command [args] < file_name

For example, bc(1) is a command-line calculator that can read mathematical expressions from standard input and print the results to standard output. We can define complex calculations in a file and let bc read and compute from the file. First, let’s check the contents of the formula:

>$ cat calc.txt
x = 5
y = 3
x * y + 2
(x + y) / (x - y)

Then use the input redirection symbol (<) to pass the file contents to the bc command:

$ bc < calc.txt
17
4

Viewing and Saving Logs Simultaneously

If you want the output of a command to be displayed on the terminal while also being saved to a file, how can this be achieved? This requirement is common in practical work, for example, when debugging a program, you need to monitor log output in real-time while also archiving the logs for later analysis.

The solution is to use the tee(1) command. Note that since we need to use the output of one program as the input of another, we should use the pipe symbol (|) instead of the ordinary output redirection symbol (> or <).

>$ journalctl -f | tee all.log
Oct 22 15:28:13 pve-dybai-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 15:28:13 pve-dybai-fedora systemd[1]: pmie_farm_check.service: Deactivated successfully.
^C

tee will display the output of journalctl on the terminal and write it to the all.log file. Next, let’s check the size and contents of that file:

$ ls -lh all.log
-rw-r--r--. 1 dybai dybai 343 Oct 22 15:33 all.log
$ cat all.log
Oct 22 15:28:13 pve-dybai-fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=pmie_farm_check comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 22 15:28:13 pve-dybai-fedora systemd[1]: pmie_farm_check.service: Deactivated successfully.

Appending Output to a File

Using the output redirection symbol (>) will clear the target file before writing content. If you want to append content to a file without clearing the existing content, you can use the >> symbol.

The first time writing content to the file:

>$ echo 'x = 2' >> calc.txt
>$ cat calc.txt
x = 2

The second time writing another entry, the file now contains two records:

>$ echo 'y = 3' >> calc.txt
>$ cat calc.txt
x = 2
y = 3

✦•———- Advanced Usage Techniques ———-•✦

Redirecting Standard Error

Consider the following scenario: using the cat command to read the /etc/shadow file and redirecting it to the my_shadow file.

>$ cat /etc/shadow > my_shadow
cat: /etc/shadow: Permission denied

At this point, the cat command outputs Permission denied to the terminal, and checking the sizes of the two files:

>$ ls -lh /etc/shadow my_shadow
----------. 1 root  root  1.5K Apr 24 14:02 /etc/shadow
-rw-r--r--. 1 dybai dybai    0 Oct 22 16:45 my_shadow

/etc/shadow is 1.5KB in size, while my_shadow is 0, indicating that the redirection was unsuccessful. The content output to the terminal is not standard output but rather standard error. In the previous article about pipes, we mentioned three file descriptors:0 (standard input), 1 (standard output), and 2 (standard error). Standard error corresponds to file descriptor 2.

When a program uses functions like perror(3), error messages are output through standard error. The output redirection symbol (>) by default only redirects standard output and does not redirect standard error. If you want to log error messages as well, you need to redirect standard error to standard output.

Below is a complete example demonstrating this process. First, write a program that attempts to read the unauthorized /etc/shadow file and outputs error information using perror(3):

#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    char buf[32];
    int fd = open("/etc/shadow", O_RDONLY);
    if (fd < 0) {
        perror("open");
        exit(1);
    }
    read(fd, buf, sizeof(buf));
    printf("%s\n", buf);
    close(fd);
    return 0;
}

Compile and run the program while redirecting output:

>$ gcc -Wall -o demo_err main.c
>$ ./demo_err > log.txt
open: Permission denied
>$ ls -lh log.txt
-rw-r--r--. 1 dybai dybai 0 Oct 22 17:01 log.txt

At this point, log.txt is empty, and the error message has not been redirected to the file. Next, redirect standard error to standard output:

>$ ./demo_err > log.txt 2>&1
>$ cat log.txt
open: Permission denied

The terminal no longer displays the error message, while log.txt contains the error information. 2>&1 indicates that file descriptor 2 (standard error) is redirected to file descriptor 1 (standard output).

Filtering Error Messages

Sometimes we do not want to see the error messages from commands to avoid interfering with normal output. In this case, we can redirect standard error to /dev/null, which is a special “black hole” file where all written content disappears.

For example, searching for a file named shadow in the /etc directory will output a lot of error messages due to permission issues:

>$ find /etc -name shadow
find: ‘/etc/audit’: Permission denied
find: ‘/etc/credstore.encrypted’: Permission denied
find: ‘/etc/credstore’: Permission denied
find: ‘/etc/cups/ssl’: Permission denied
find: ‘/etc/dhcp’: Permission denied
find: ‘/etc/firewalld’: Permission denied
find: ‘/etc/grub.d’: Permission denied
find: ‘/etc/libvirt’: Permission denied
find: ‘/etc/lvm/archive’: Permission denied
find: ‘/etc/lvm/backup’: Permission denied
find: ‘/etc/lvm/cache’: Permission denied
find: ‘/etc/lvm/devices’: Permission denied
find: ‘/etc/nftables’: Permission denied
find: ‘/etc/openvpn/client’: Permission denied
find: ‘/etc/openvpn/server’: Permission denied
find: ‘/etc/polkit-1/localauthority’: Permission denied
find: ‘/etc/polkit-1/rules.d’: Permission denied
find: ‘/etc/sos/cleaner’: Permission denied
find: ‘/etc/ssh/sshd_config.d’: Permission denied
find: ‘/etc/sssd’: Permission denied
find: ‘/etc/sudoers.d’: Permission denied
find: ‘/etc/valkey’: Permission denied
/etc/shadow

By redirecting standard error to /dev/null, we can keep only the search results:

>$ find /etc -name shadow 2>/dev/null
/etc/shadow

Here Document and Output Redirection

Sometimes you need to append multiple lines of content to a configuration file through a script, and cannot use editors like vim. In this case, you can combine here documents and output redirection.

For example, to append configuration to the /etc/apt/sources.list file:

>$ cat >> /etc/apt/sources.list << EOF
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ trixie main contrib non-free non-free-firmware
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ trixie main contrib non-free non-free-firmware
EOF

Check the /etc/apt/sources.list, and you will see that the following two lines have been added to the bottom of the file:

deb https://mirrors.tuna.tsinghua.edu.cn/debian/ trixie main contrib non-free non-free-firmware
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ trixie main contrib non-free non-free-firmware

This feature is convenient for writing multiple lines of content to a file in scripts.Special Note:cat should use the append redirection symbol (>>) instead of the redirection symbol (>), otherwise, the original file will be cleared! The EOF can be replaced with any character (like ABC), and the effect will be the same.

Congratulations, you have gained useful knowledge again✨