Questions you can answer in one line
Which clients caused the most errors last night? How many orders came from each country? Which configuration files still mention the old server name? Questions like these come up every day, and the data that answers them is usually sitting in text files: logs, CSV exports, configuration, source code.
You could write a program for each question. But Unix has carried, since the 1970s, a set of small text tools that each do one job extremely well: grep finds lines, sed edits them, awk works with fields and numbers, sort and uniq group and count, cut, tr, head, tail and others slice and reshape. Combined with pipes, they answer most such questions in one line, often faster than you could open a spreadsheet, and on files far too large for one.
This lesson teaches the tools the way people actually use them: building a pipeline one stage at a time, looking at the output after each stage. Every example uses the same small web-server log (with addresses from the ranges reserved for documentation), which each script writes into its working directory first:
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450The fields, separated by single spaces, are: date, time, client address, method, path, status code and response size in bytes.
Regular expressions
Most of these tools find text using regular expressions, a small language for describing patterns. The essential pieces:
| Pattern | Matches |
|---|---|
abc | the characters abc |
. | any one character |
[aeiou], [0-9], [^0-9] | one character from a set; or not in it |
^, $ | start and end of the line |
x* | zero or more x |
x+, x? | one or more; zero or one (ERE) |
x{2,4} | between two and four x (ERE) |
(ab)+, a|b | grouping; alternation (ERE) |
\. | a literal dot |
There are two dialects. Basic regular expressions (BRE), the default in grep and sed, require a backslash before +, ?, {, ( and | to give them their special meaning. Extended regular expressions (ERE), selected with grep -E and sed -E and used by awk and Bash's =~, do not. Use ERE: it reads like the regular expressions of every other language. (GNU grep -P offers Perl-compatible expressions with features like \d and look-ahead, but it is not available everywhere.)
Remember that regular expressions are not the same as the shell's glob patterns from lesson 2: in a glob, * alone means "anything"; in a regular expression, .* does.
grep: find lines
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG
grep -c ' 500 ' access.log # count matching lines
grep -n 'POST' access.log # show matches with line numbers
grep -v '/api/' access.log | wc -l # -v: lines that do NOT match
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -u # -o: only the matched text
grep -E ' (4|5)[0-9]{2} ' access.log | grep -i 'STOCK' | wc -l # chain filters; -i ignores case2
3:2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
3
192.0.2.77
198.51.100.23
203.0.113.5
2The options worth knowing by heart: -i (ignore case), -v (invert), -c (count), -n (line numbers), -o (print only the match), -l (list the files that match), -r (search a directory tree), -w (whole words), -q (quiet: status only, for if), -F (fixed string: no regex, fast and safe for text containing dots or brackets), and -A, -B, -C (lines of context after, before or around each match). Note that -c counts matching lines, not matches: a line with three hits counts once, so to count every occurrence use grep -o pattern file | wc -l. For searching source code, many developers use ripgrep (rg), a faster, recursive, .gitignore-aware grep, but plain grep is what you can count on being installed.
Slicing, sorting and counting
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG
echo "requests per client:"
cut -d' ' -f3 access.log | sort | uniq -c | sort -rn
echo "largest response:"
sort -k7,7n access.log | tail -n 1 | cut -d' ' -f5,7
echo "status codes seen:"
cut -d' ' -f6 access.log | sort -u | paste -sd' '
printf 'kiwi,12\napple,7\nfig,30\n' | sort -t, -k2,2n | head -n 2
echo "Hello, World" | tr -s ' ' | tr 'a-z' 'A-Z'
wc -l -w < access.logrequests per client:
3 203.0.113.5
2 198.51.100.23
2 192.0.2.77
largest response:
/index.html 5120
status codes seen:
200 201 304 404 500
apple,7
kiwi,12
HELLO, WORLD
7 49What each tool did:
cut -d' ' -f3selects fields by number, given a one-character delimiter (-cselects character positions instead). It cannot handle fields separated by runs of spaces; that is a job forawkor fortr -s ' 'first.sortsorts lines:-nnumerically,-rin reverse,-udropping duplicates,-t,to set the field separator, and-k7,7nto sort by field 7 only, numerically. Always give-kboth a start and an end field;-k7alone means "from field 7 to the end of the line".uniq -ccollapses adjacent identical lines and counts them, which is why it always followssort.sort | uniq -c | sort -rnis the "frequency table" idiom.head -n Nandtail -n Nkeep the first or last N lines (tail -ffollows a growing log).paste -sd' 'joins lines into one;paste a bputs two files side by side.trtranslates or deletes characters (-ssqueezes repeats,-ddeletes).wccounts lines, words and bytes.
Two more are handy for comparing lists: comm shows which lines of two sorted files are unique to the first, unique to the second or common to both (comm -23 a b gives lines only in a), and join merges two sorted files on a common field, like a database join.
sed: edit the stream
sed, the stream editor, applies editing commands to each line as it passes through. Its most-used command is substitution, s/pattern/replacement/flags, and most of the rest is choosing which lines a command applies to (its address: a line number, a range, or a /regex/):
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG
sed -n '2,3p' access.log # print only lines 2-3
sed -n '1p' access.log | sed 's/GET/get/' # substitute the first match
sed -E 's#^([0-9]{4})-([0-9]{2})-([0-9]{2})#\3/\2/\1#' access.log | head -n 2 # reorder the date
sed '/favicon/d' access.log | wc -l # delete matching lines
sed -n '/POST/,/favicon/p' access.log | wc -l # a range: from one match to another
cp access.log redacted.log
sed -i.bak -E 's/([0-9]+\.){3}[0-9]+/x.x.x.x/g' redacted.log # edit in place, keep a backup
head -n 1 redacted.log
ls redacted.log*2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:01:07 203.0.113.5 get /index.html 200 5120
02/11/2026 10:01:07 203.0.113.5 GET /index.html 200 5120
02/11/2026 10:01:09 198.51.100.23 GET /api/stock 500 312
6
4
2026-11-02 10:01:07 x.x.x.x GET /index.html 200 5120
redacted.log
redacted.log.bak-nwithpprints only the addressed lines; without-n,sedprints every line after editing it.s/old/new/replaces the first match on each line; thegflag replaces all of them. The delimiter can be any character, which is why the date example used#to avoid escaping slashes. In the replacement,\1,\2, … are the groups captured by( ), and&is the whole match./regex/ddeletes matching lines, and/start/,/end/addresses a range of lines.-iedits files in place. It is the one option that behaves differently between GNUsed(Linux) and BSDsed(macOS): GNU accepts-ialone, BSD requires a backup suffix argument.-i.bakwith the suffix attached works on both, and keeps a backup, which you will be glad of the first time a substitution goes wrong.
awk: fields, conditions and arithmetic
awk is a complete small programming language designed for exactly this kind of data. An awk program is a list of pattern { action } rules. For every input line, awk splits it into fields $1, $2, … (at runs of whitespace, by default), and runs the action of every rule whose pattern matches. BEGIN and END rules run before the first line and after the last. Variables need no declaration and start as 0 or empty, and arrays are associative.
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG
awk '{ print $3, $6 }' access.log | head -n 2 # fields are $1, $2, ...
awk '$6 >= 500 { n++ } END { print n, "server errors" }' access.log
awk '{ bytes[$3] += $7 } END { for (ip in bytes) printf "%-14s %5d bytes\n", ip, bytes[ip] }' access.log | sort
awk '$5 == "/api/stock" { sum += $7; n++ } END { printf "/api/stock: %d requests, mean %.1f bytes\n", n, sum / n }' access.log
printf 'warehouse,crates\nMombasa,1590\nCusco,1090\nSapporo,1900\n' > stock.csv
awk -F, 'NR > 1 && $2 > 1100 { print $1 }' stock.csv # -F sets the field separator; NR is the line number
awk 'BEGIN { printf "%.4f\n", 22 / 7 }' # awk does floating point203.0.113.5 200
198.51.100.23 500
2 server errors
192.0.2.77 0 bytes
198.51.100.23 602 bytes
203.0.113.5 6658 bytes
/api/stock: 3 requests, mean 684.0 bytes
Mombasa
Sapporo
3.1429Those few lines show why awk is the tool of choice once a question involves fields and numbers: filter by a field's value ($6 >= 500), accumulate (n++, sum += $7), group with an array keyed by a field (bytes[$3] += $7), and report at the end. Built-in variables you will use: NR (the line number), NF (the number of fields; $NF is the last one), FS and -F (the input field separator), and OFS (the output field separator). Unlike Bash, awk does floating-point arithmetic. And since the order in which for (key in array) visits keys is unspecified, pipe the result to sort when the order matters.
On Linux, awk is usually GNU awk (gawk) or mawk; on macOS it is the BSD "one true awk". Plain awk, as in these examples, works on all of them.
How professionals build a pipeline
Nobody writes grep ... | awk ... | sort | uniq -c | sort -rn | head in one go. The working method is incremental:
- Look at the data first:
head -n 5 file,wc -l file. Check the separators and whether there is a header line. - Add one stage at a time, and look at the output after each one. Pipe into
headwhile you are exploring so that a mistake does not scroll a million lines past. - Filter early.
grepfirst, so that later stages handle fewer lines; it is also the fastest tool of the set. - Set
LC_ALL=C(as every script in this lesson does) when you need predictable sorting or speed: in other locales, sorting follows language rules and character ranges like[a-z]can behave unexpectedly. Keep the user's locale when you are processing human-language text and want it handled correctly. - Save the finished one-liner in a script or your notes, with a comment saying what question it answers.
For example, here is "which paths returned server errors, and how often?" built in three steps, each checked before the next is added, followed by a use of comm to answer "which clients are not on our allow list?":
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG
echo "step 1: find the lines"
grep -E ' 5[0-9]{2} ' access.log
echo "step 2: keep only the path"
grep -E ' 5[0-9]{2} ' access.log | awk '{ print $5 }'
echo "step 3: count and rank"
grep -E ' 5[0-9]{2} ' access.log | awk '{ print $5 }' | sort | uniq -c | sort -rn
cut -d' ' -f3 access.log | sort -u > seen.txt
printf '%s\n' 192.0.2.77 203.0.113.5 > allowed.txt # comm needs sorted input
echo "clients not on the allow list:"
comm -23 seen.txt allowed.txtstep 1: find the lines
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
step 2: keep only the path
/api/stock
/api/stock
step 3: count and rank
2 /api/stock
clients not on the allow list:
198.51.100.23In a terminal you would of course not repeat the earlier stages: you press the up arrow, add | next-stage to the end of the previous command, and run it again.
Know the limits, too. These tools see lines and fields; they do not understand quoted CSV fields that contain commas, nested JSON, or XML. For JSON use jq; for CSV with real quoting use Miller or a CSV library; and once the logic needs several data structures, a short Python script is clearer and easier to test. The command-line tools are for exploration and for simple, well-structured text, which is a very large share of real work.
Fifty years of filters
These tools are among the oldest software still in daily use. grep (1973) was written by Ken Thompson from the line editor ed, whose command g/re/p meant "globally search for a regular expression and print the matching lines". sed (1974) was written by Lee McMahon, also at Bell Labs, as a non-interactive version of the same editor. awk (1977) is named after its authors, Alfred Aho, Peter Weinberger and Brian Kernighan, and their 1988 book about it is still a model of clear technical writing. The GNU Project rewrote all of them as free software, adding long options, extended regular expressions and many conveniences, which is why the GNU versions on Linux and the BSD versions on macOS still differ in details today.
Common misconceptions
Misconception
"uniq removes all duplicate lines." It removes adjacent duplicates only. Sort first, or use sort -u.
Misconception
"grep '192.168.1.1' matches only that address." The dots are regex wildcards, so it also matches 192x168y1z1, and 192.168.1.10. Use grep -F for literal text, and -w or anchors to match whole words.
Misconception
"sort sorts numbers." By default it sorts text, so 10 comes before 9. Use -n (or -h for sizes like 2K and 1G).
Misconception
"In a regular expression, * means anything, as in a file glob." It means "zero or more of the previous item". grep 'a*' matches every line (every line contains zero or more as), and the regex for "anything" is .*.
Misconception
"awk -F, reads CSV files." It splits at every comma, including commas inside quoted fields such as "Lima, Peru". For real CSV, use a tool that understands quoting: Miller, a CSV library, or the --csv option of GNU awk 5.3 and later.
Misconception
"cat file | grep x is the normal way to search a file." grep x file does the same with one process fewer, and lets grep report the file name. Reserve cat for concatenating files or showing them.
Exercises
Several exercises use the lesson's access.log. Copy it into a file to check your answers.
1. Text processing with grep, sed, awk and friends
Warm-upComputationalUsing the lesson's access.log, what does grep -c 'api' access.log print?
2. Text processing with grep, sed, awk and friends
CoreComputationalUsing the lesson's access.log, what does this print?
awk '$3 == "203.0.113.5" { s += $7 } END { print s }' access.log
3. Text processing with grep, sed, awk and friends
CoreComputationalWhat does this print?
echo "2026-11-02" | sed -E 's/([0-9]+)-([0-9]+)-([0-9]+)/\2.\3.\1/'
4. Text processing with grep, sed, awk and friends
CoreComputationalWhat does this print?
printf 'b\na\nb\na\n' | uniq | wc -l
5. Text processing with grep, sed, awk and friends
Warm-upComputationalWhat does this print?
printf '9\n10\n100\n' | sort | head -n 1
6. Text processing with grep, sed, awk and friends
StretchComputationalWhat does this print?
echo "a,b,,d" | awk -F, '{ print NF }'
7. Text processing with grep, sed, awk and friends
CoreHumanYou need the average response size for each HTTP status code in a large space-separated log. Which tool is the best fit for the core of the job?
Further reading and resources
- GNU grep manual
- GNU sed manual
- The GNU Awk User's Guide
- GNU Coreutils manualsort, uniq, cut, tr, wc, head, tail, paste, comm and join.
- jqThe standard command-line tool for JSON.
- MillerLike awk, sed and sort for CSV, TSV and JSON with named fields.