Examples of Pipe or Pipelines Command

In UNIX, Unix like operating systems including Linux, the shell of the OS have a mechanism called pipe or pipeline which looks like this |. In fact this ASCII vertical bar character “|” is named as pipe character. Unless you are from old schooling, each and every linked article are important; most IT related professional bachelors degree programs exclude the parts which are quite important in real life. Also, you should know what is Terminal, historically we are using UNIX more effectively now than we used to do 40 years back – from Android, iOS to major servers – all runs some version of UNIX or Unix like OS.  In this article we will not only discuss about the Examples of Pipe or Pipelines Command, but also go to a bit history. Specifically if you are a software engineer but it was not your major, these are very important and often neglected area. Just like, typing without looking at the keyboard without doing typographical errors need a huge time to master, application of piping needs long term usage.

 

History of Pipe or Pipelines Command

 

A pipeline consists of a chain of processing elements arranged to give the output of each element as in sequence; the name is by analogy to a physical pipeline. Usually there is some buffering the consecutive elements. The shell of the operating systems have a mechanism called pipe. This mechanism allows to chain the process so that the output of one process feeds directly. Each connection is implemented by an anonymous pipe. Programs filters are often used in this configuration. It is Douglas McIlroy who invented this concept for Unix shells. A very basic example of piping is :

The program programme1 is executed by the system that sends the results to programme2 which in turn returns the results to the standard output of the system. Working example :

This display the client IP addresses that accessed most frequently visited to a web server running Apache. Standard error (stderr) is an output stream typically used by programs to output error messages or diagnostics. Pipeline escapes this stderr. If you run man stderr on OS X 10.9.x, you will get BSD Kernel Interfaces Manual.

We usually use simple pipelines where the shell connects a series of sub-processes via pipes and executes external commands within each sub-process. But it is possible for the shell to perform processing directly, this is called Pipemill :

this is a subshell – var1, var2 etc. will not be available after the while loop terminates.

 

Examples of Pipe or Pipelines Command

 

Our Tar Extract Progress Bar tutorial uses extreme level of piping. Run this on OS X :

UNIX and BSD has UNIX Wheel Group. It gave me output :

I am not root. In the same way, to list of all sub-directories in the current directory, we can use :

I used d to grab drwx! Here is a combination of pipeline, redirection (>) and usage of && :

If I stopped before the && my output.txt, I had to work more, it would waste time; it would demand to type cat myoutput.txt – Piping is for not saving time, it is for logically making the things working. Run man pipe on OS X, you will get pipe (8) – Postfix delivery to external command manual. tee kind of sanitizes the raw usage of redirection :

You can run diff to compare over ssh :

Practically you need to understand which thing can be connected with which to get a practical result. Unless you have sufficient knowledge on UNIX commands, pipe will remain far from being useful. These are safer examples, do not run others’ pipe without knowing what are the individual commands can do.