Pipelines and text tools

Small tools, joined end to end

A vertical bar sends whatever the command on its left printed into the command on its right. That is a pipeline, and it is the reason the shell is worth learning: each tool does one small thing well, and you compose them until the answer is the line you wanted. The pipe is the whole connection; neither tool needs to know the other exists.

Read a pipeline left to right as a sentence. cut -d: -f1 people.txt | sort -u means "take the first field of every line, then sort those and keep one of each". Neither tool knows anything about the other; the pipe is the whole connection. Reading it left to right is the same order the data travels.

The seven you will use most

grep keeps the lines that match a pattern. sort puts lines in order, and with -nr orders numbers from largest down. uniq -c collapses repeated lines and counts them, though it only sees neighbours, so it is always paired with sort first. cut -d: -f1 takes one field from a separated line, head -n 2 keeps the first two lines, tr swaps characters, and wc -l counts the lines that went past. Each option changes one decision: which lines, in what order, or which field.

Two of these have an option worth remembering. grep -c counts matching lines instead of printing them, and wc -l reading from a file directly prints a name beside the number, which is why examples often feed it with a less-than sign so the count arrives alone. The less-than sign is a redirection, not a pipe, and it feeds the file without adding a name.

The same file, six questions

# One field from a separated line: field 1, split on colons.
cut -d: -f1 people.txt | sort -u

# Count the lines in a file, without printing its name.
wc -l < people.txt

# Count the lines that match, instead of printing them.
grep -c editor people.txt

# Sort by the third field, as a number, largest first.
sort -t: -k3 -nr people.txt | head -n 2

# How often does each role appear?
cut -d: -f2 people.txt | sort | uniq -c

# Swap characters: colons become spaces.
tr ':' ' ' < people.txt

What the pipe connects, and what it leaves out

A pipe connects the standard output of one command to the standard input of the next. Standard error is not included: an error message still goes to the terminal, so a pipeline can print a clean answer while a warning appears beside it. 2>&1 before the bar would merge the two, which is sometimes wanted and often not.

The commands in a pipeline run at the same time, not one after another. head can finish and close its end while an earlier command is still writing, and the writer then receives a broken-pipe signal. That is why a large file can be inspected with head without reading all of it. The early exit is normal, not an error, and it is why head is safe on a huge file.

Sorting, counting and choosing fields

sort compares text by default, so 10 comes before 9. Add -n for numbers, -r to reverse, and -k with -t to choose a field. uniq only collapses neighbours, which is why it is almost always preceded by sort, and uniq -c puts the count first.

cut takes a single character as its separator; awk is the tool when the separator is a word or the field logic is more involved. grep matches anywhere in the line unless you anchor it or ask for -w, so a search for art also finds start.

Building a pipeline one piece at a time

The reliable way to build a pipeline is to stop after each tool and look at the output. cut -d: -f1 people.txt on its own shows the names; adding | sort shows them in order; adding | uniq -c shows the counts. If the final answer is wrong, the mistake is in the last piece you added, not somewhere in the middle.

A pipeline is also a sentence about data, so the tool order follows the question. Filter first to make the stream smaller, then sort, then count. Sorting before filtering does the same work on more lines, and counting before filtering answers a different question. The order is not cosmetic: it changes both the cost and the meaning.

Try these next

• Sort the same three numbers with and without -n, and compare the orders.
• Use grep -c and wc -l on the same input and explain why the numbers can differ.
• Build a pipeline that prints the two longest values from a list.