COMP 10024 – UNIX Pipelines – Connecting the World

In our last module, we explored how to use temporary files to pass data between commands. While functional, that approach was often awkward, inefficient, and, quite frankly, ugly. Today, we learn the “magic” of Pipelines, which allow us to connect processes directly in memory.

1. The Philosophy: Everything is a File

At the heart of UNIX is the principle that “Everything is a file”. This isn’t just a clever saying; it means that the output of a program (stdout) and the input of another program (stdin) are treated as data streams that can be rerouted.

2. What is a Pipe?

A pipe is a special kind of “magic” file managed by the Kernel. It acts as a one-way conduit: what goes in one end must come out the other. In the shell, we use the vertical bar character (|) to create this connection.

When you run command1 | command2, the shell performs a few complex steps behind the scenes:

1. It forks two child processes simultaneously.
2. It connects the stdout of the first process to the stdin of the second.
3. Data flows between them through a buffer in the Kernel’s memory, never touching the hard drive.

3. Real-World Pipeline Examples

The power of pipelines comes from “chaining” multiple simple commands to produce a sophisticated result.

A. Filtering and Counting

Suppose you want to know how many times a specific word appears in a file. You don’t need a custom program; you just need a pipeline:

grep "error" logfile.txt | wc -l

Here, grep finds the lines, and wc -l counts them.

B. Sorting and Slicing

If you want to find the three largest directories on your system:

du -sk * | sort -rn | head -n 3

• du -sk *: Calculates disk usage for all files.
• sort -rn: Sorts them numerically in reverse order (largest first).
• head -n 3: Slices the list to show only the top three.

Lab: Building Data Pipelines

This lab will guide you through building increasingly complex pipelines to extract information from your system.

Task 1: Basic Word Counting

1. The file /usr/share/dict/words (on most systems) contains a massive list of English words.
2. Use a pipe to count how many words are in that file:

   cat /usr/share/dict/words | wc -l

Task 2: Finding Specific Patterns

1. How many words in that dictionary start with the letter “q”?

   grep "^q" /usr/share/dict/words | wc -l

2. Now, let’s find how many of those “q” words also contain the letter “z”:

   grep "^q" /usr/share/dict/words | grep "z" | wc -l

Task 3: Process Inspection

1. Use ps aux to see every process running on the server. It’s too much data to read!
2. Pipe it into grep to find only the processes owned by “root”:

   ps aux | grep "root"

3. Now, count them:

   ps aux | grep "root" | wc -l

Task 4: The Unique Challenge

1. The uniq command is great, but it only works on sorted data.
2. Let’s find out how many unique users are currently logged into the system:

   who | cut -d' ' -f1 | sort | uniq | wc -l

Summary Challenge

Using a single pipeline, can you list the files in /etc, find the ones that contain the string “conf”, sort them alphabetically, and save the final list to a file called conf_files.txt?

Hint: You will need to combine a Pipe (|) with an Output Redirection (>) at the very end!

Categorised in: COMP-10024, Lectures, Portfolio

This post was written by amax