Piping and redirection are two tools that, although I have mentioned them before, I vastly under-utilize in the command line.
To that end, I have decided that a more thorough exploration is warranted, so that I might come to appreciate these tools to their fullest — and remember to use them!
Piping
A pipe is a hollow object through which other objects can travel. While physical pipes can channel an almost unlimited variety of entities, command line pipes are limited almost exclusively to the transfer of data.
The first question that comes to my mind is: “Why would you want/need to pipe data?” The answer is perhaps best answered by example, but I will attempt an explanation by saying that pipes are are often uses to connect two programs (commands/tools) by piping the output of one command into another command as its input.
The pipe symbol is a vertical bar | that is found at the right of the keyboard near the main Enter key. It is used between two commands, to pipe the (text) output of the command on its left, as the (text) input to the command on its right.
Now, we’ll get into some examples!
ls | grep 'Gwen' will take the results of the list (ls) command and search (grep) through them for instances of the the specified name.
ls Downloads/ | wc will pipe the results of directory listing into the word count command, to display the number of lines, words and characters in that directory.
find /home/gwen | grep 'dog' will search through my home directory and all of its sub-folders, and pipe the results into a search for the word “dog”, resulting it a list of files and folders (including the path) containing the word, which will be shown in red.
find /home/gwen | grep 'dog' | more will pipe the results of the above command into the more “file perusal filter” for easier viewing. Note that the searched-for text will no longer be in red.
To summarize, pipes can be used as long as a command’s text output can be channeled into another command’s text input, even multiple times in a row.
Redirection
Redirection is the process of channeling something in a new direction; a direction different than the original path. Command line redirection is the process of changing the flow of data.
Pipes are, at their core, redirectors, limited in their capacity to the redirection of output from one command to another. The redirectors that are actually called redirectors allow for two possibilities: sending output to files, and reading input from files.
First, let’s look at the standard input/output (I/O) streams that can be redirected. It is not essential to memorize these details, but they will help the later parts make sense.
| Descriptor | Description | Abbreviation | Example |
|---|---|---|---|
| 0 | Standard Input | STDIN | Keyboard, Etc. |
| 1 | Standard Output | STDOUT | Display, Screen, Etc. |
| 2 | Standard Error Output | STDERR | Display, Screen, Etc. |
Second, let’s look at the available redirectors:
| Redirector | Function | Description |
|---|---|---|
| > | Overwrite | Standard Output |
| < | Overwrite | Standard Input |
| 2> | Overwrite | Standard Error |
| >> | Append | Standard Output |
| << | Append | Standard Input |
| 2>> | Append | Standard Errors |
The trick is to somehow put all of these details together to form useful commands that use the appropriate redirector to channel the proper type of data.
Once again, I find the best explanation to be by example.
Note that data redirected to a file will create that file if it does not already exist, and can overwrite files if the redirector allows.
cat > cat_text.txt will redirect/save the text typed into the command line.
cat >> cat_text.txt will redirect/save the text typed into the command line, by appending it to the end of the existing file.
ls > directory-contents.txt will redirect/save the contents of the current directory in a text file.
ls | grep 'Gwen' > results.txt will redirect/save the search results into the specified text file.
touch filename.jpg 2>> custom_error_log.txt will append error messages to a specified text file.
Redirection is a great way to save command results or make them useful, but what if you want them to simply disappear? You can redirect to /dev/null to make that happen. This is useful not only in shell scripts, but also for error messages that you don’t want to see, etc. Example: command 2> /dev/null
Ok, yes, you caught me; I can’t think of a practical example to use for the < and the << input redirectors. Help?
Conclusion
I have some ideas now, of how I may be able to better utilize pipes and and redirectors while performing tasks in the command line. Now I just need an opportunity to put them to use.
What are some ways that you use — or would use — pipes and redirectors on a regular basis? Share your thoughts in the comments.