When a one-liner becomes a program
Most shell scripts start life as a command you typed twice. Then it grows a variable, an if, a loop, an option, and one day it is two hundred lines long, run by other people, from a scheduler, with arguments you never thought of. The difference between a script that survives that growth and one that becomes everyone's nightmare is structure: functions with clear inputs and outputs, careful handling of arguments, data kept in proper arrays rather than space-separated strings, and a predictable layout with a usage message and meaningful exit statuses.
Bash's functions and arrays are less powerful than those of a general-purpose language, and their rules are a little odd: functions cannot really return values, and variables are global unless you say otherwise. This lesson shows how experienced shell programmers work within those rules. Its last section is also about knowing when not to: when a script has outgrown Bash and should be rewritten in a language like Python.
Functions
A function is defined with name() { commands; } and called like any command, with arguments. Inside it, the arguments are $1, $2, … and "$@", just like a script's. (The alternative syntax function name { ... } comes from the Korn shell; the name() form is the portable one.)
#!/usr/bin/env bash
greet() {
local name=${1:-world} # local: invisible outside the function
local mark=${2:-!}
printf 'Hello, %s%s\n' "$name" "$mark"
}
greet
greet Ama "?"
leaky() { temp="I escaped"; } # no local: the variable is global
leaky
echo "after leaky: $temp"
is_even() { (( $1 % 2 == 0 )); } # "returns" a status: its last command's
for n in 3 4; do
if is_even "$n"; then echo "$n is even"; else echo "$n is odd"; fi
done
square() { echo $(( $1 * $1 )); } # "returns" data: by printing it
result=$(square 7)
echo "7 squared is $result"
outer() { local x="outer's x"; inner; }
inner() { echo "inner sees: $x"; } # dynamic scope: sees its caller's locals
x="global x"
outer
innerHello, world!
Hello, Ama?
after leaky: I escaped
3 is odd
4 is even
7 squared is 49
inner sees: outer's x
inner sees: global xFour rules explain everything above:
- Variables are global by default.
leakysettempfor the whole script. Declare every variable a function uses withlocal, unless you mean to change a global. - A function's status is its last command's status, or the number given to
return(0 to 255). That is howis_evenworks as a condition: its body is a single(( ))test.returnis for success or failure, never for data. - To return data, print it and let the caller capture it with
$( ), assquaredoes. (This runs the function in a subshell, so it cannot change the caller's variables; when that matters, a function can assign to a global, or to a variable whose name it is given, using a nameref,local -n result=$1, available since Bash 4.3.) - Scope is dynamic. A function sees the local variables of the function that called it, which is why
innersawouter'sx. This is different from almost every modern language, and another reason to pass values as arguments rather than relying on what happens to be in scope.
Arguments and the environment
Arguments arrive as positional parameters, and the same rules apply to scripts and functions:
#!/usr/bin/env bash
show() { printf '%d args:' "$#"; printf ' <%s>' "$@"; echo; }
forward() {
show "$@" # exactly the arguments received
show $* # re-split at spaces: usually a bug
show "$*" # joined into one argument
}
forward "a b" c
consume() {
local first=$1
shift # drop $1; the rest move down
echo "first=$first, then $# more: $*"
}
consume x y z
set -- one two three # replace this shell's own positional parameters
echo "now \$# is $#, \$2 is $2"
plain="not exported"
export SHARED="exported"
bash -c 'echo "a child process sees: [${plain-}] [${SHARED-}]"'
GREETING=hej bash -c 'echo "set for one command only: $GREETING"'
echo "and afterwards: [${GREETING-}]"2 args: <a b> <c>
3 args: <a> <b> <c>
1 args: <a b c>
first=x, then 2 more: y z
now $# is 3, $2 is two
a child process sees: [] [exported]
set for one command only: hej
and afterwards: []"$@"forwards arguments exactly. Use it whenever one function or script passes its arguments on.$*unquoted re-splits them, and"$*"joins them into one string, which is right only for messages.shiftdrops the first argument and moves the rest down, the usual way to consume a leading subcommand or value.shift 2drops two.set -- ...replaces the current positional parameters, handy for supplying defaults (lesson 1'sgreet.shused it).- Only exported variables reach child processes. A child process (another program, or another
bash) inherits the environment: exported variables only.VAR=value commandsets a variable in the environment of that one command, which is the idiomatic way to configure a program for a single run, for exampleLC_ALL=C sortorTZ=UTC date.
The builtin source file (or . file) runs another file's commands in the current shell, so its functions and variables become available; that is how ~/.bashrc is loaded and how scripts share a library of functions. Running the file as a program instead would execute it in a child process, and its definitions would vanish when it exited.
Arrays
A string is a poor container for a list: you saw in lesson 2 how word splitting destroys elements that contain spaces. Bash has two kinds of array for real lists and tables. Indexed arrays are numbered from 0; associative arrays (Bash 4.0 and later, so not in the old Bash that macOS ships) map string keys to values and must be declared with declare -A.
#!/usr/bin/env bash
fruits=(mango "passion fruit" guava)
fruits+=(lychee) # append
echo "${#fruits[@]} fruits; first: ${fruits[0]}; last: ${fruits[-1]}"
for f in "${fruits[@]}"; do printf '<%s> ' "$f"; done; echo
echo "indices: ${!fruits[*]}; slice: ${fruits[*]:1:2}"
unset 'fruits[1]' # arrays can have gaps
echo "after unset: indices ${!fruits[*]}, values ${fruits[*]}"
declare -A capital=([Kenya]=Nairobi [Peru]=Lima [Japan]=Tokyo) # Bash 4.0+
capital[Ghana]=Accra
mapfile -t countries < <(printf '%s\n' "${!capital[@]}" | LC_ALL=C sort)
for c in "${countries[@]}"; do
printf '%-6s -> %s\n' "$c" "${capital[$c]}"
done
[[ -v capital[Chile] ]] || echo "no entry for Chile"
declare -A count=()
for word in the cat and the hat and the bat; do
(( ++count[$word] ))
done
for word in the and cat; do echo "$word: ${count[$word]}"; done4 fruits; first: mango; last: lychee
<mango> <passion fruit> <guava> <lychee>
indices: 0 1 2 3; slice: passion fruit guava
after unset: indices 0 2 3, values mango guava lychee
Ghana -> Accra
Japan -> Tokyo
Kenya -> Nairobi
Peru -> Lima
no entry for Chile
the: 3
and: 2
cat: 1The syntax is dense, so here it is in one place:
| Syntax | Meaning |
|---|---|
a=(x "y z"), a+=(w) | create; append |
"${a[@]}" | every element as a separate word: the form to use in loops and arguments |
${a[i]}, ${a[-1]} | one element; the last element |
${#a[@]} | number of elements |
${!a[@]} | the indices (or, for associative arrays, the keys) |
${a[@]:1:2} | a slice |
unset 'a[i]' | remove an element (the indices are not renumbered) |
[[ -v a[key] ]] | does the key exist? |
mapfile -t lines < file | read a file (or < <(command)) into an array, one line per element |
Associative arrays do not remember insertion order, so the example sorted the keys before printing them. And (( ++count[$word] )) is the shell's version of a word counter: in arithmetic, an unset element counts as 0.
Handing results back
Since a function cannot return data the way a Python or JavaScript function does, Bash programmers use three patterns, each with its place:
#!/usr/bin/env bash
# Three ways for a function to hand back results.
# 1. Print it; the caller captures it. Simple, but runs in a subshell.
basename_of() { printf '%s\n' "${1##*/}"; }
name=$(basename_of /var/log/nginx/access.log)
echo "1. $name"
# 2. Fill a variable the caller names: a nameref (Bash 4.3+). No subshell.
split_version() {
local -n parts=$1 # parts is now another name for the caller's array
IFS=. read -r -a parts <<< "$2"
}
split_version ver "5.1.16"
echo "2. major=${ver[0]} minor=${ver[1]} patch=${ver[2]}"
# 3. Return a status, and print details only on failure.
require_number() {
[[ $1 =~ ^-?[0-9]+$ ]] && return 0
echo "not a number: $1" >&2
return 1
}
for v in 42 4x2; do
if require_number "$v" 2>/dev/null; then echo "3. $v ok"; else echo "3. $v rejected (status $?)"; fi
done
# A trap: local hides the status of a command substitution
f() { local out=$(false); echo " with local on the same line: $?"; }
g() { local out; out=$(false); echo " declared first, then assigned: $?"; }
f; g1. access.log
2. major=5 minor=1 patch=16
3. 42 ok
3. 4x2 rejected (status 1)
with local on the same line: 0
declared first, then assigned: 1- Print and capture is the default: clear, composable (
$(basename_of "$f")works anywhere), and the function cannot accidentally change the caller's variables. Its costs are a subshell per call, which matters in a loop over thousands of items, and the loss of trailing newlines. - A nameref (
local -n) lets a function fill an array or several variables for the caller, without a subshell. Choose local variable names that the caller is unlikely to pass in, because a nameref pointing at a variable of the same name refers to itself. - A status plus a message on stderr suits validation and checks: the caller writes
if require_number "$v"; then, and the message is there for a human when it fails.
The last two lines show a trap worth remembering. local out=$(false) reports status 0, because the status you see is that of the local command, which succeeded, not of the command substitution. Declare first, then assign, whenever you need the status (and with set -e, which lesson 7 covers, whenever you want a failure to stop the script). ShellCheck warns about this pattern.
Anatomy of a command-line script
A script that other people use should behave like any other command: accept options in the usual way, explain itself with -h, report mistakes on stderr, and exit with 0 on success and a non-zero status on misuse (2 is the convention for bad usage). The builtin getopts parses short options, including combined ones like -un3, and stops at the first non-option argument or at --.
#!/usr/bin/env bash
usage() {
cat <<'EOT'
Usage: shout [-n COUNT] [-u] [-h] WORD...
-n COUNT repeat each word COUNT times (default 1)
-u upper-case the words
-h show this help
EOT
}
main() {
local count=1 upper=false opt OPTIND # OPTIND must be local to reuse getopts
while getopts ':n:uh' opt; do
case $opt in
n) count=$OPTARG ;;
u) upper=true ;;
h) usage; return 0 ;;
:) echo "shout: -$OPTARG needs a value" >&2; return 2 ;;
\?) echo "shout: unknown option -$OPTARG" >&2; return 2 ;;
esac
done
shift $(( OPTIND - 1 )) # what remains are the WORDs
if [[ ! $count =~ ^[1-9][0-9]*$ ]]; then
echo "shout: COUNT must be a positive whole number, not '$count'" >&2
return 2
fi
if (( $# == 0 )); then
echo "shout: give at least one WORD (try -h)" >&2
return 2
fi
local word i
for word in "$@"; do
if $upper; then word=${word^^}; fi
for (( i = 0; i < count; i++ )); do printf '%s ' "$word"; done
done
echo
}
# A real script ends with: main "$@"
# Here we call main several times, showing errors (stderr) and statuses too.
for args in "-n 2 -u hola mundo" "-h" "-un3 hi" "-- -u" "-n" "-x hi" "-n zero hi" "-u"; do
printf '$ shout %s\n' "$args"
read -r -a argv <<< "$args"
main "${argv[@]}" 2>&1
echo " (status $?)"
done$ shout -n 2 -u hola mundo
HOLA HOLA MUNDO MUNDO
(status 0)
$ shout -h
Usage: shout [-n COUNT] [-u] [-h] WORD...
-n COUNT repeat each word COUNT times (default 1)
-u upper-case the words
-h show this help
(status 0)
$ shout -un3 hi
HI HI HI
(status 0)
$ shout -- -u
-u
(status 0)
$ shout -n
shout: -n needs a value
(status 2)
$ shout -x hi
shout: unknown option -x
(status 2)
$ shout -n zero hi
shout: COUNT must be a positive whole number, not 'zero'
(status 2)
$ shout -u
shout: give at least one WORD (try -h)
(status 2)How it works:
- The option string
':n:uh'lists the options; a colon after a letter means it takes a value, delivered inOPTARG. The leading colon putsgetoptsin silent mode, so it reports problems by settingoptto:(missing value) or?(unknown option) instead of printing its own message, and the script can phrase the error itself. shift $(( OPTIND - 1 ))discards the options once they are parsed, leaving only the operands in"$@".- Validate everything that came from outside: the count must be a positive integer, and at least one word is required. Each failure prints a message naming the program, on stderr, and returns 2.
local OPTINDmatters only because the demonstration callsmainseveral times;getoptskeeps its position inOPTINDbetween calls.
getopts handles only short options. For long options (--count 3), scripts typically loop over the arguments themselves with while (( $# )); do case $1 in ...; esac; shift; done, or use the external getopt program from util-linux, which is not available everywhere. Lesson 8's project shows the loop.
How professionals lay out a script
A layout that you will recognise in well-maintained repositories, and that the Google Shell Style Guide and others broadly agree on:
#!/usr/bin/env bash
# backup-photos: copy new photos to the archive disk.
set -euo pipefail
readonly ARCHIVE_DIR=${ARCHIVE_DIR:-/mnt/archive} # settings, overridable from the environment
log() { printf '%s %s\n' "$(date +%T)" "$*" >&2; }
die() { log "error: $*"; exit 1; }
usage() { ... }
copy_new_photos() {
local source=$1 count=0
...
}
main() {
# parse options, validate, then call the functions above
...
}
main "$@"The habits it encodes: a shebang and a one-line description; strict mode (lesson 7); constants and settings at the top, in capitals, marked readonly; small functions that use local for everything; log and die helpers that write to stderr; all real work inside functions; and a single call, main "$@", on the last line. That last line also protects you from a half-downloaded script: Bash reads scripts as it runs them, and if the file is cut off, nothing runs because main is never reached. If the file is meant to be both run and sourced as a library, guard the call with [[ ${BASH_SOURCE[0]} == "$0" ]] && main "$@".
Structured this way, a script can also be tested. Because the work lives in functions and main is called only on the last line, a test file can source the script and call its functions directly with prepared inputs. The most widely used framework for this is bats-core, in which each test is a small block of shell that runs a command and asserts on its status and output. Even without a framework, a tests.sh that calls your functions and compares results with expected values catches most regressions, and it runs anywhere Bash does.
When to stop using Bash. Shell is excellent at running programs and connecting their input and output. It is poor at data structures, arithmetic beyond integers, error handling and anything that needs to be tested in depth. A common rule of thumb among engineers is to switch to Python (or another general-purpose language) when a script passes a couple of hundred lines, needs nested data structures, or does more computing than orchestrating. Keeping a small Bash wrapper that calls a well-tested program is often the best of both worlds.
Common misconceptions
Misconception
"return 42 returns 42 to the caller, like other languages." It sets the function's exit status to 42, which the caller can only inspect through $? or an if. Statuses are limited to 0 to 255 and meant for success or failure. Return data by printing it.
Misconception
"Variables inside a function are local." They are global unless declared with local (or declare inside the function). Forgetting local in a function called from a loop is a classic way to clobber the loop's own variable.
Misconception
"Inside a function, $1 is the script's first argument." Each function call has its own positional parameters. To use the script's arguments in a function, pass them in: main "$@".
Misconception
"Associative arrays work in any Bash." They need Bash 4.0 or later. The /bin/bash on macOS is version 3.2, so a script using declare -A fails there unless it is run with a newer Bash, which is one more reason to use #!/usr/bin/env bash and to state the minimum version in your README.
Misconception
"A space-separated string is a good enough list." It breaks as soon as an element contains a space or a wildcard. Use an array and "${array[@]}".
Exercises
Run the functions in a terminal to check your predictions.
1. Functions, arguments, arrays and script structure
Warm-upComputationalWhat does this print?
total=10
add() { total=$(( total + $1 )); }
add 5
add 7
echo "$total"
2. Functions, arguments, arrays and script structure
StretchComputationalWhat does this print?
f() { return 300; }
f
echo $?
3. Functions, arguments, arrays and script structure
CoreComputationalWhat does this print?
count() { echo "$#"; }
count "a b" "" c
4. Functions, arguments, arrays and script structure
CoreComputationalWhat does the last line print?
list=(x "y z" w)
unset 'list[0]'
echo "${!list[@]}"
5. Functions, arguments, arrays and script structure
CoreComputationalWhat does this print?
g() { local v="in"; }
h() { v="changed"; }
v=out
g; echo -n "$v "
h; echo "$v"
6. Functions, arguments, arrays and script structure
CoreComputationalA script calls getopts 'vo:f' opt. Which of these command lines are parsed without an error? Choose all that apply.
7. Functions, arguments, arrays and script structure
Warm-upComputationalA wrapper script should pass all of its arguments, unchanged, to rsync. Which line does that?
Further reading and resources
- Bash Reference Manual: Shell Functions
- Bash Reference Manual: Arrays
- Bash Reference Manual: Bash Builtin Commands (including getopts, mapfile, declare)
- Google Shell Style GuideOne widely used set of conventions for Bash scripts, including when not to use shell at all.