Skip to content
LumenaryAcademy
Practice 60 min

Text processing with grep, sed, awk and friends

Regular expressions, the classic filters, and how to answer real questions about data by building a pipeline one stage at a time.

Code tested · 5AI-draftedAwaiting review by a computer scientist
MotivationH

Questions you can answer in one line

Which clients caused the most errors last night? How many orders came from each country? Which configuration files still mention the old server name? Questions like these come up every day, and the data that answers them is usually sitting in text files: logs, CSV exports, configuration, source code.

You could write a program for each question. But Unix has carried, since the 1970s, a set of small text tools that each do one job extremely well: grep finds lines, sed edits them, awk works with fields and numbers, sort and uniq group and count, cut, tr, head, tail and others slice and reshape. Combined with pipes, they answer most such questions in one line, often faster than you could open a spreadsheet, and on files far too large for one.

This lesson teaches the tools the way people actually use them: building a pipeline one stage at a time, looking at the output after each stage. Every example uses the same small web-server log (with addresses from the ranges reserved for documentation), which each script writes into its working directory first:

text
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450

The fields, separated by single spaces, are: date, time, client address, method, path, status code and response size in bytes.

DefinitionsH

Regular expressions

Most of these tools find text using regular expressions, a small language for describing patterns. The essential pieces:

PatternMatches
abcthe characters abc
.any one character
[aeiou], [0-9], [^0-9]one character from a set; or not in it
^, $start and end of the line
x*zero or more x
x+, x?one or more; zero or one (ERE)
x{2,4}between two and four x (ERE)
(ab)+, a|bgrouping; alternation (ERE)
\.a literal dot

There are two dialects. Basic regular expressions (BRE), the default in grep and sed, require a backslash before +, ?, {, ( and | to give them their special meaning. Extended regular expressions (ERE), selected with grep -E and sed -E and used by awk and Bash's =~, do not. Use ERE: it reads like the regular expressions of every other language. (GNU grep -P offers Perl-compatible expressions with features like \d and look-ahead, but it is not available everywhere.)

Remember that regular expressions are not the same as the shell's glob patterns from lesson 2: in a glob, * alone means "anything"; in a regular expression, .* does.

Worked examplesH

grep: find lines

bash
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG

grep -c ' 500 ' access.log                         # count matching lines
grep -n 'POST' access.log                          # show matches with line numbers
grep -v '/api/' access.log | wc -l                 # -v: lines that do NOT match
grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' access.log | sort -u   # -o: only the matched text
grep -E ' (4|5)[0-9]{2} ' access.log | grep -i 'STOCK' | wc -l # chain filters; -i ignores case
text
2
3:2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
3
192.0.2.77
198.51.100.23
203.0.113.5
2

The options worth knowing by heart: -i (ignore case), -v (invert), -c (count), -n (line numbers), -o (print only the match), -l (list the files that match), -r (search a directory tree), -w (whole words), -q (quiet: status only, for if), -F (fixed string: no regex, fast and safe for text containing dots or brackets), and -A, -B, -C (lines of context after, before or around each match). Note that -c counts matching lines, not matches: a line with three hits counts once, so to count every occurrence use grep -o pattern file | wc -l. For searching source code, many developers use ripgrep (rg), a faster, recursive, .gitignore-aware grep, but plain grep is what you can count on being installed.

ComputationC

Slicing, sorting and counting

bash
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG

echo "requests per client:"
cut -d' ' -f3 access.log | sort | uniq -c | sort -rn

echo "largest response:"
sort -k7,7n access.log | tail -n 1 | cut -d' ' -f5,7

echo "status codes seen:"
cut -d' ' -f6 access.log | sort -u | paste -sd' '

printf 'kiwi,12\napple,7\nfig,30\n' | sort -t, -k2,2n | head -n 2
echo "Hello,   World" | tr -s ' ' | tr 'a-z' 'A-Z'
wc -l -w < access.log
text
requests per client:
      3 203.0.113.5
      2 198.51.100.23
      2 192.0.2.77
largest response:
/index.html 5120
status codes seen:
200 201 304 404 500
apple,7
kiwi,12
HELLO, WORLD
  7  49

What each tool did:

  • cut -d' ' -f3 selects fields by number, given a one-character delimiter (-c selects character positions instead). It cannot handle fields separated by runs of spaces; that is a job for awk or for tr -s ' ' first.
  • sort sorts lines: -n numerically, -r in reverse, -u dropping duplicates, -t, to set the field separator, and -k7,7n to sort by field 7 only, numerically. Always give -k both a start and an end field; -k7 alone means "from field 7 to the end of the line".
  • uniq -c collapses adjacent identical lines and counts them, which is why it always follows sort. sort | uniq -c | sort -rn is the "frequency table" idiom.
  • head -n N and tail -n N keep the first or last N lines (tail -f follows a growing log).
  • paste -sd' ' joins lines into one; paste a b puts two files side by side.
  • tr translates or deletes characters (-s squeezes repeats, -d deletes).
  • wc counts lines, words and bytes.

Two more are handy for comparing lists: comm shows which lines of two sorted files are unique to the first, unique to the second or common to both (comm -23 a b gives lines only in a), and join merges two sorted files on a common field, like a database join.

Worked examplesH

sed: edit the stream

sed, the stream editor, applies editing commands to each line as it passes through. Its most-used command is substitution, s/pattern/replacement/flags, and most of the rest is choosing which lines a command applies to (its address: a line number, a range, or a /regex/):

bash
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG

sed -n '2,3p' access.log                                   # print only lines 2-3
sed -n '1p' access.log | sed 's/GET/get/'                  # substitute the first match
sed -E 's#^([0-9]{4})-([0-9]{2})-([0-9]{2})#\3/\2/\1#' access.log | head -n 2   # reorder the date
sed '/favicon/d' access.log | wc -l                        # delete matching lines
sed -n '/POST/,/favicon/p' access.log | wc -l              # a range: from one match to another

cp access.log redacted.log
sed -i.bak -E 's/([0-9]+\.){3}[0-9]+/x.x.x.x/g' redacted.log   # edit in place, keep a backup
head -n 1 redacted.log
ls redacted.log*
text
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:01:07 203.0.113.5 get /index.html 200 5120
02/11/2026 10:01:07 203.0.113.5 GET /index.html 200 5120
02/11/2026 10:01:09 198.51.100.23 GET /api/stock 500 312
6
4
2026-11-02 10:01:07 x.x.x.x GET /index.html 200 5120
redacted.log
redacted.log.bak
  • -n with p prints only the addressed lines; without -n, sed prints every line after editing it.
  • s/old/new/ replaces the first match on each line; the g flag replaces all of them. The delimiter can be any character, which is why the date example used # to avoid escaping slashes. In the replacement, \1, \2, … are the groups captured by ( ), and & is the whole match.
  • /regex/d deletes matching lines, and /start/,/end/ addresses a range of lines.
  • -i edits files in place. It is the one option that behaves differently between GNU sed (Linux) and BSD sed (macOS): GNU accepts -i alone, BSD requires a backup suffix argument. -i.bak with the suffix attached works on both, and keeps a backup, which you will be glad of the first time a substitution goes wrong.
Worked examplesH

awk: fields, conditions and arithmetic

awk is a complete small programming language designed for exactly this kind of data. An awk program is a list of pattern { action } rules. For every input line, awk splits it into fields $1, $2, … (at runs of whitespace, by default), and runs the action of every rule whose pattern matches. BEGIN and END rules run before the first line and after the last. Variables need no declaration and start as 0 or empty, and arrays are associative.

bash
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG

awk '{ print $3, $6 }' access.log | head -n 2               # fields are $1, $2, ...
awk '$6 >= 500 { n++ } END { print n, "server errors" }' access.log
awk '{ bytes[$3] += $7 } END { for (ip in bytes) printf "%-14s %5d bytes\n", ip, bytes[ip] }' access.log | sort
awk '$5 == "/api/stock" { sum += $7; n++ } END { printf "/api/stock: %d requests, mean %.1f bytes\n", n, sum / n }' access.log

printf 'warehouse,crates\nMombasa,1590\nCusco,1090\nSapporo,1900\n' > stock.csv
awk -F, 'NR > 1 && $2 > 1100 { print $1 }' stock.csv        # -F sets the field separator; NR is the line number
awk 'BEGIN { printf "%.4f\n", 22 / 7 }'                      # awk does floating point
text
203.0.113.5 200
198.51.100.23 500
2 server errors
192.0.2.77         0 bytes
198.51.100.23    602 bytes
203.0.113.5     6658 bytes
/api/stock: 3 requests, mean 684.0 bytes
Mombasa
Sapporo
3.1429

Those few lines show why awk is the tool of choice once a question involves fields and numbers: filter by a field's value ($6 >= 500), accumulate (n++, sum += $7), group with an array keyed by a field (bytes[$3] += $7), and report at the end. Built-in variables you will use: NR (the line number), NF (the number of fields; $NF is the last one), FS and -F (the input field separator), and OFS (the output field separator). Unlike Bash, awk does floating-point arithmetic. And since the order in which for (key in array) visits keys is unspecified, pipe the result to sort when the order matters.

On Linux, awk is usually GNU awk (gawk) or mawk; on macOS it is the BSD "one true awk". Plain awk, as in these examples, works on all of them.

ExplanationH

How professionals build a pipeline

Nobody writes grep ... | awk ... | sort | uniq -c | sort -rn | head in one go. The working method is incremental:

  1. Look at the data first: head -n 5 file, wc -l file. Check the separators and whether there is a header line.
  2. Add one stage at a time, and look at the output after each one. Pipe into head while you are exploring so that a mistake does not scroll a million lines past.
  3. Filter early. grep first, so that later stages handle fewer lines; it is also the fastest tool of the set.
  4. Set LC_ALL=C (as every script in this lesson does) when you need predictable sorting or speed: in other locales, sorting follows language rules and character ranges like [a-z] can behave unexpectedly. Keep the user's locale when you are processing human-language text and want it handled correctly.
  5. Save the finished one-liner in a script or your notes, with a comment saying what question it answers.

For example, here is "which paths returned server errors, and how often?" built in three steps, each checked before the next is added, followed by a use of comm to answer "which clients are not on our allow list?":

bash
#!/usr/bin/env bash
export LC_ALL=C
cat > access.log <<'LOG'
2026-11-02 10:01:07 203.0.113.5 GET /index.html 200 5120
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:02:44 203.0.113.5 POST /api/orders 201 88
2026-11-02 10:03:01 192.0.2.77 GET /index.html 304 0
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
2026-11-02 10:06:30 192.0.2.77 GET /favicon.ico 404 0
2026-11-02 10:07:02 203.0.113.5 GET /api/stock 200 1450
LOG

echo "step 1: find the lines"
grep -E ' 5[0-9]{2} ' access.log
echo "step 2: keep only the path"
grep -E ' 5[0-9]{2} ' access.log | awk '{ print $5 }'
echo "step 3: count and rank"
grep -E ' 5[0-9]{2} ' access.log | awk '{ print $5 }' | sort | uniq -c | sort -rn

cut -d' ' -f3 access.log | sort -u > seen.txt
printf '%s\n' 192.0.2.77 203.0.113.5 > allowed.txt        # comm needs sorted input
echo "clients not on the allow list:"
comm -23 seen.txt allowed.txt
text
step 1: find the lines
2026-11-02 10:01:09 198.51.100.23 GET /api/stock 500 312
2026-11-02 10:05:12 198.51.100.23 GET /api/stock 500 290
step 2: keep only the path
/api/stock
/api/stock
step 3: count and rank
      2 /api/stock
clients not on the allow list:
198.51.100.23

In a terminal you would of course not repeat the earlier stages: you press the up arrow, add | next-stage to the end of the previous command, and run it again.

Know the limits, too. These tools see lines and fields; they do not understand quoted CSV fields that contain commas, nested JSON, or XML. For JSON use jq; for CSV with real quoting use Miller or a CSV library; and once the logic needs several data structures, a short Python script is clearer and easier to test. The command-line tools are for exploration and for simple, well-structured text, which is a very large share of real work.

History & global context

Fifty years of filters

These tools are among the oldest software still in daily use. grep (1973) was written by Ken Thompson from the line editor ed, whose command g/re/p meant "globally search for a regular expression and print the matching lines". sed (1974) was written by Lee McMahon, also at Bell Labs, as a non-interactive version of the same editor. awk (1977) is named after its authors, Alfred Aho, Peter Weinberger and Brian Kernighan, and their 1988 book about it is still a model of clear technical writing. The GNU Project rewrote all of them as free software, adding long options, extended regular expressions and many conveniences, which is why the GNU versions on Linux and the BSD versions on macOS still differ in details today.

Common misconceptionsH

Common misconceptions

Misconception

"uniq removes all duplicate lines." It removes adjacent duplicates only. Sort first, or use sort -u.

Misconception

"grep '192.168.1.1' matches only that address." The dots are regex wildcards, so it also matches 192x168y1z1, and 192.168.1.10. Use grep -F for literal text, and -w or anchors to match whole words.

Misconception

"sort sorts numbers." By default it sorts text, so 10 comes before 9. Use -n (or -h for sizes like 2K and 1G).

Misconception

"In a regular expression, * means anything, as in a file glob." It means "zero or more of the previous item". grep 'a*' matches every line (every line contains zero or more as), and the regex for "anything" is .*.

Misconception

"awk -F, reads CSV files." It splits at every comma, including commas inside quoted fields such as "Lima, Peru". For real CSV, use a tool that understands quoting: Miller, a CSV library, or the --csv option of GNU awk 5.3 and later.

Misconception

"cat file | grep x is the normal way to search a file." grep x file does the same with one process fewer, and lets grep report the file name. Reserve cat for concatenating files or showing them.

Exercises

Exercises

Several exercises use the lesson's access.log. Copy it into a file to check your answers.

1. Text processing with grep, sed, awk and friends

Warm-upComputational

Using the lesson's access.log, what does grep -c 'api' access.log print?

2. Text processing with grep, sed, awk and friends

CoreComputational

Using the lesson's access.log, what does this print?

awk '$3 == "203.0.113.5" { s += $7 } END { print s }' access.log

3. Text processing with grep, sed, awk and friends

CoreComputational

What does this print?

echo "2026-11-02" | sed -E 's/([0-9]+)-([0-9]+)-([0-9]+)/\2.\3.\1/'

4. Text processing with grep, sed, awk and friends

CoreComputational

What does this print?

printf 'b\na\nb\na\n' | uniq | wc -l

5. Text processing with grep, sed, awk and friends

Warm-upComputational

What does this print?

printf '9\n10\n100\n' | sort | head -n 1

6. Text processing with grep, sed, awk and friends

StretchComputational

What does this print?

echo "a,b,,d" | awk -F, '{ print NF }'

7. Text processing with grep, sed, awk and friends

CoreHuman

You need the average response size for each HTTP status code in a large space-separated log. Which tool is the best fit for the core of the job?

Choose one

Further reading and resources