Redirecting streams in linux

Explained: Input, Output and Error Redirection in Linux

If you are familiar with the basic Linux commands, you should also learn the concept of input output redirection.

You already know how a Linux command functions. It takes an input and gives you an output. There are a few players in the scene here. Let me tell you about them.

Stdin, stdout and stderr

When you run a Linux command, there are three data stream that play a part in it:

• Standard input (stdin) is the source of input data. By default, stdin is any text entered from the keyboard. It’s stream ID is 0.
• Standard output (stdout) is the outcome of command. By default, it is displayed on the screen. It’s stream ID is 1.
• Standard error (stderr) is the error message (if any) produced by the commands. By default, stderr is also displayed on the screen. It’s stream ID is 2.

These streams contain the data in plain text in what’s called buffer memory.

Think of it as water stream. You need a source for water, a tap for example. You connect a pipe to it and you can either store it in a bucket (file) or water the plants (print it). You can also connect it to another tap, if needed. Basically, you are redirecting the water.

Linux also has this concept of redirection, where you can redirect the stdin, stdout and stderr from its usual destination to another file or command (or even peripheral devices like printers).

Let me show how redirection works and how you can use it.

The output redirection

The first and simplest form of redirection is output redirection also called stdout redirection.

You already know that by default, the output of a command is displayed on the screen. For example, I use the ls command to list all the files and this is the output I get:

With output redirection, you can redirect the output to a file. If this output files doesn’t exist, the shell will create it.

For example, let me save the output of the ls command to a file named output.txt:

The output file is created beforehand

What do you think the content of this output file should be? Let me use the cat command to show you a surprise:

Did you notice that the inclusion of output.txt there? I deliberately chose this example to show you this.

The output file to which the stdout is redirected is created before the intended command is run. Why? Because it needs to have the output destination ready to which the output will be sent.

Append instead of clobber

One often ignored problem is that if you redirect to a file that already exists, the shell will erase (clobber) the file first. This means the existing content of the output file will be removed and replaced by the output of the command.

You can append, instead of overwriting it, using the >> redirection syntax.

Tip: You can forbid clobbering in current shell session using: set -C

Why would you redirect stdout? You can store the output for future reference and analyze it later. It is specially helpful when the command output is way too big and it takes up all of your screen. It’s like collecting logs.

Pipe redirection

Before you see the stdin redirection, you should learn about pipe redirection. This is more common and probably you’ll be using it a lot.

With pipe redirection, you send the standard output of a command to standard input of another command.

Let me show you a practical example. Say, you want to count the number of visible files in the current directory. You can use ls -1 (it’s numeral one, not letter L) to display the files in the current directory:

You probably already know that wc command is used for counting number of lines in a file. If you combine both of these commands with pipe, here’s what you get:

With pipe, both commands share the same memory buffer. The output of the first command is stored in the buffer and the same buffer is then used as input for the next command.

You’ll see the result of the last command in the pipeline. That’s obvious because the stdout of earlier command(s) is fed to the next command(s) instead of going to the screen.

Pipe redirection or piping is not limited to connecting just two commands. You may connect more commands as long as the output of one command can be used as the input of the next command.

Remember the stdout/stdin is a chunk of data, not filenames

Some new Linux users get confused while using the redirection. If a command returns a bunch of filenames as output, you cannot use those filenames as argument.

For example, if you use the find command to find all the files ending with .txt, you cannot pass it through pipe to move the found files to a new directory, not directly like this:

This is why you’ll often see find command used in conjugation with exec or xargs command. These special commands ‘convert the text with bunch of filenames into filename’ that can be passed as argument.

The input redirection

You can use stdin redirection to pass the content of a text file to a command like this:

You won’t see stdin being used a lot. It’s because most Linux commands accept filenames as argument and thus the stdin redirection is often not required.

Take this for example:

The above command could have just been head filename.txt (without the or >> redirection symbol that you used for stdout redirection.

But how do you distinguish between the stdout and stderr when they are both output data stream? By their stream ID (also called file descriptor).

Data Stream	Stream ID
stdin	0
stdout	1
stderr	2
-t,	–list
-u,	–update
-x,	–extract, –get
-j,	–bzip2
-z,	–gzip, –gunzip, –ungzip

By default when you use the output redirection symbol >, it actually means 1>. In words, you are saying that data stream with ID 1 is being outputted here.

When you have to redirect the stderr, you use its ID like 2> or 2>>. This signifies that the output redirection is for data stream stderr (ID 2).

Stderr redirection examples

Let me show it to you with some examples. Suppose you just want to save the error, you can use something like this:

That was simple. Let’s make it slightly more complicated (and useful):

In the above example, the ls command tries to display two files. For one file it gets success and for the other, it gets error. So what I did here is to redirect the stdout to ouput.txt (with >) and the stderr to the error.txt (with 2>).

You can also redirect both stdout and stderr to the same file. There are ways to do that.

In the below example, I first send the stderr (with 2>>) to combined.txt file in append mode. And then, the stdout (with >>) is sent to the same file in append mode.

Another way, and this is the preferred one, is to use something like 2>&1. Which can be roughly translated to “redirect stderr to the same address as stdout”.

Let’s take the previous example and this time use the 2>&1 to redirect both of the stdout and stderr to the same file.

Keep in mind that you cannot use 2>>&1 thinking of using it in append mode. 2>&1 goes in append mode already.

You may also use 2> first and then use 1>&2 to redirect stdout to same file as stderr. Basically, it is “>&” which redirects one out data stream to another.

Summary

• There are three data streams. One input, stdin (0) and two output data streams stdout (1) and stderr (2).
• Keyboard is the default stdin device and the screen is the default output device.
• Output redirection is used with > or >> (for append mode).
• Input redirection is used with or 2>>.
• The stderr and stdout can be combined using 2>&1.

Since you are learning about redirection, you should also know about the tee command. This command enables you to display to standard output and save to file simultaneously.

I hope you liked this detailed guide on redirection in Linux. If you still have doubts or if you have suggestions to improve this article, please let me know in the comment section.

Источник

Linux Fundamentals – I/O, Standard Streams, and Redirection.

In Linux there are three standard input streams, standard input (0 – stdin), standard output (1 – stdout), and standard error (2 – stderr). In this article we will discuss practical uses of redirection, the standard streams, and their file descriptors in the bash command line and give examples of how to use each including redirection.

Table of Contents

Standard Streams

In Linux and computer programming in general, standard streams are input and output (I/O) communication channels between a program and it’s environment. The three standard streams are standard input (stdin), standard output (stdout), and standard error (stderr). Streams are usually connected to the terminal in which they are executed, but that can be changed using redirection operators and/or pipes.

Below is a graphic to help visualize the data streams and their file descriptors.

File Descriptors

Linux often represents the three standard streams as file descriptors, here is a table to help understand the relationship.

File descriptors are not limited to the three standard streams, but these are the most commonly used predefined streams. It is possible to create custom file descriptors, however we will not cover that in this article.

Redirection Operators

Redirection operators are a subset of control operators. They allow you to direct the input or output (stream) of your command.

The Input Operator

The input operator gives input to a command or process. The left angle bracket ( The Input Operator (rw)

The input operator (rw) does the same as the input operator only it opens the file in read/write mode. The left and right angle bracket ( <> ) represents this operator. If the file does not exist, it will be created. Since most input is from the keyboard or read from a file, this is rarely used.

The Output Operator

The output operator controls the command output stream. The right angle bracket ( > ) represents this operator. If the file you are redirecting to does not exist, it will be created. If it does exist, it will be overwritten.

The Output Append Operator

The output append operator does the same as the output operator only it appends data to a file. Two right angle brackets ( >> ) represent this operator. If the file you are redirecting to does not exist, it will be created. If it does exist, this command will append data to it instead of overwriting.

The Non-Standard Operators

Non-standard operators direct both standard output (stdout) and standard error (stderr). The ampersand sign followed by a right bracket or double right brackets represent these operators. One overwrites ( &> ) and one appends ( &>> ) data to the output file.

Pipe Operator

The pipe operator is used to pass the output of a command to the input of another command. The vertical bar ( | ) represents this operator. The aptly named pipe can be thought of similar to a physical plumbing pipe which moves liquid from one vessel to another. Output from a command enters the pipe on one side and leaves the pipe as input for another command on the other side.

Examples for using the pipe operator are in the sections below.

Standard Input – STDIN

Standard input is a data stream going into a program. This is often text input from a keyboard, but can also be data piped or redirected to a program.

Standard Input From Keyboard

A simple example of a program that uses standard input is the cat command. By default cat takes standard input and prints it to the display (standard output).

To test this we can use the cat command. Open a terminal and type cat, then type a line from your favorite song and hit enter.
NOTE: The cat program will do this forever, when you are finished hit CTRL+D to exit.

Standard Input from an Input File

Standard input can also come from an input file. If you provide cat with a file as an argument, it will replace or redirect standard input from the keyboard to the data coming from that file.

To test this, enter the following command in your terminal and hit enter.

The above command takes the argument (

/.bashrc) as standard input and outputs the content of the file on the screen.

Standard Input from Redirection

Standard input can also come from any file output using the input redirection operator which is represented by a less than sign or left pointing angle bracket ( Standard Input from a Data Pipe

So far we learned how to provide standard input from a keyboard or from a file by argument or redirection. But, you can also provide standard input from the output of another program on the input stream. This is called piping, and even if you are new to Linux you probably already encountered this concept. A pipe represented by a vertical line ( | ), basically takes output from one command and puts it into the pipe, then it comes out of the pipe on the other side and into the next command.

To test this enter the following command in your terminal.

The head program outputs the first 10 lines of a file and pipes it to cat as standard input (stdin), which in turn prints it to standard out (stdout).

Standard Output – STDOUT

Standard output is a standardized stream that is sent to a display monitor by default. Often the terminal is standard output, but you can pipe or redirect output to a file, another program or device.

Standard Output to Display

You can use the output of almost any command as an example of standard output (stdout). Here we will use the stat command to get status information about a file and display it on the monitor.

As you can see the stat command took the filename as an argument, did some processing and then displayed the statistics of that file to the monitor (stdout).

Standard Output Redirected to a File

You can redirect standard output (stdout) to a file. This is useful if you want to save the output for later use, or as a log of a script.

You can redirect stdout to a file using the output redirection operator represented by a greater than sign or right pointing angle bracket ( > ). For example, you can save the output of the stat command to a file named output.txt like so:

This created a file named output.txt, and places the standard output (stdout) of the stat command into it. We can verify this by reading the file.

Using this output redirection will overwrite any data that may exist in the file already. Use the output append operator if you want to append, represented by two angle brackets ( >> ).

Standard Output Through a Data Pipe

The data pipe sends output from one command to another. Here we will pipe the output of the stat command into grep to pull only last modified time of the file.

The stat commands output enters the pipe and passed to grep for processing. When grep finishes processing the data, it sends the output to standard output (stdout).

Standard Error – STDERR

Standard error (stderr) is the destination for all error messages from a process. By default, this is the display screen but just like standard out, it can be redirected.

Standard Error to Display

Mistyping will certainly lead to an error displayed on the screen if you use the command line often. This is an example of standard error (stderr). Here we will feed the ls command an argument that doesn’t exist, and it will tell us using stderr.

An error was displayed on the screen as stderr by default.

Standard Error Redirected to a File

Just like the other streams, you can redirect stderr to a file. Often used in scripting, but handy to know for anyone using the command line. To redirect stderr, you can use the same output redirection operator as you did with stdout ( > ). But this time, we have to specify that we want to redirect stderr by using the file descriptor. As shown in the table and diagram above, the file descriptor for stderr is 2. So we would redirect it like so:

The file errors.txt now contains the errors from the output of the above command.

As we discussed in the stdout section, using the single bracket redirection will result in overwriting any data that is currently in the file. To append instead of overwrite, you have to use the append redirection ( >> ).

Standard Error Through a Data Pipe

You cannot directly pipe stderr to another command. This is by design to allow for complete error reporting. Here is an example of trying (and failing) to pipe stderr into grep.

The ls command prints errors to the screen and the grep command acts as if it received no input.

However, as with most things Linux, there are multiple ways to accomplish the same task. You can redirect stderr to stdout using their file descriptors, then pipe it through grep.

We will discuss more about file descriptors in the next section.

Using File Descriptors for Redirection

Non-Standard redirection requires file descriptors. We covered all of the standard (input / output ) redirection in the above sections. Below we will outline some additional redirects using file descriptors.

File descriptors are used to redirect streams. However, you can do a lot more with file descriptors, much more than we will cover here. The intention of this section is to give you a elementary understanding of file descriptors and making you comfortable with their basic usage.

Redirect STDERR and STDOUT to Different Files.

You can redirect both stderr and stdout to a file. Here is an example:

The above command sends stdout (File Descriptor 1) to ouput.txt and stderr (2) to errors.txt. We can verify by reading those files.

Redirect STDERR and STDOUT to Same File.

You can redirect both stderr and stdout to a single file if you wish. For this we can either redirect stderr to stdout, then stdout to a file like so:

Or we can use a shorthand (non-standard) operator like so:

Either command produces the same result, all stdout and stderr were sent to the alloutput.txt file.

Using Permanent Redirection in Scripts

If you are writing a script and want to redirect data streams without appending the redirect operator or every line, you can use the exec command like this.

You can use one, both or any number of custom file descriptors in your script.

Conclusion

Knowing the standard streams, file descriptors and redirection are an important part of being proficient on the command line. In this article we covered a lot but still only scratched the surface. I hope I touched all the basics of these topics, but if you feel I need to be corrected or should include additional information please sound off in the comments.

Источник