Captured output, sed and awk

Keeping the answer in a variable

To keep what a command printed, wrap it in a dollar and a pair of parentheses: count=$(wc -l < notes.txt). The command runs, its output is collected, and the value lands in the name. A name holding an answer is usually better than a command printing straight to the screen, because the script can then compare it, count it or print it inside a sentence.

An older spelling of the same idea uses backticks, which are easy to confuse with an ordinary quote in a long file; the dollar form is the one to write. If the command inside prints several lines, all of them are kept, and a loop over the name will walk the lines one word at a time. The dollar form also nests, which the backtick form does not do comfortably.

Two tools for a line of text

sed is a stream editor: it reads lines and writes them out changed. The form you will use most is a substitution, sed 's/old/new/' in a file, which replaces the first match on every line. Adding a g on the end replaces every match on the line rather than just the first.

awk reads a line as a set of fields, separated by whitespace unless you choose another separator with -F. A small program in quotes then says what to do with them: { print $1 } prints the first field of each line, $3 the third, and printf shapes the output the way the printing chapter taught. awk also counts in decimals, which is the one thing the shell will not do, so it is the tool for an average. The field numbers start at one, while a shell array starts at zero.

Capturing, replacing, and reshaping

rows=$(grep -c editor /tmp/times.txt)
echo "rows for editors: $rows"

for name in $(cut -d: -f1 /tmp/times.txt); do
  echo "  - $name"
done

# Replace the first match on each line.
sed 's/editor/writer/' /tmp/times.txt

# Replace every match on each line.
sed 's/a/A/g' /tmp/times.txt

# Fields by position, with a chosen separator.
awk -F: '{ print $1 }' /tmp/times.txt
awk -F: '{ printf "%s (%s)\n", $1, $3 }' /tmp/times.txt

# Decimals, which awk can do and the shell cannot.
awk -F: '{ s += $3 } END { print s / 2 }' /tmp/times.txt

What command substitution keeps, and what it drops

$(...) collects standard output, not standard error, and removes every trailing newline. A command that prints three lines gives a value containing two newlines and no final one; printing that value with echo adds one back. That is why a captured count is a clean number and a captured file is not quite the file. The trailing newlines are gone, but the newlines between lines are kept.

Unquoted, the captured text is split on spaces and newlines before it reaches the next command, so a captured list becomes separate arguments. Quoted, it stays one value with its newlines intact. The rule is the same as everywhere else: quote unless you want the split.

sed and awk in one breath

sed is line-oriented: it reads a line, applies the commands, writes the result, and moves on. s/old/new/ replaces the first match on each line, s/old/new/g replaces every match, and -n with p prints only the lines you ask for. The file is untouched unless you redirect the output or use an in-place option.

awk treats each line as fields and runs its program once per line, with BEGIN before the first line and END after the last. $0 is the whole line, $1 the first field, NF the number of fields and NR the line number. -F sets the separator, and -v passes a value in from the shell. A rule can select lines by number or pattern, which is how a change is limited to part of a file.

Quoting the program, not the data

The program you hand to sed or awk is data for that tool, not text for the shell. Single quotes stop the shell from expanding $1, * and the rest before the tool sees them. That is why awk -F: '{ print $1 }' works while the same program in double quotes can lose its $1 to the shell.

The data itself still needs the quoting rules from chapter two. A file name with a space in it should be quoted when it is passed as an argument; the program string should be single-quoted. Two different jobs, two different quotes, and a script that keeps them apart is a script whose sed command does what it says.

Try these next

• Capture the output of printf into a variable, print it quoted, then unquoted, and compare.
• Use sed to replace the first match on each line, then add g and compare.
• Print a sum and a count from an awk program using END, then compute the average.