ENOSUCHBLOG

Programming, philosophy, pedaling.

Anybody can write good bash (with a little effort)

Jan 23, 2020     Tags: programming, rant    

What this post is

A gentle admonishment to use shell scripts where appropriate accept that shell scripts will appear in your codebases and to lean heavily on automated tools, modern features, safety rails, and best practices whenever possible.

Shell programming is a popular and predictable target of ire in programming communities1: virtually everybody has a horror story about a vintage, broken, or monstrous shell script underpinning a critical component of their development environment or project.

Personal favorites include:

I’ve experienced all of these, and am personally guilty of a (slight) majority of them2. Despite that (and perhaps because of it) I continue to believe that shell scripts3 have an important (and irreplaceable) niche in my development cycle, and should occupy that same niche in yours.

I’ll go through the steps I take to write (reliable, composable) bash below.

Basics

A bash script (i.e., a bash file that’s meant to be run directly) doesn’t end up in my codebases unless it:

I also put two functions in (almost) every script:

1
2
3
4
5
6
7
8
9
10
11

function installed {
  cmd=$(command -v "${1}")

  [[ -n "${cmd}" ]] && [[ -f "${cmd}" ]]
  return ${?}
}

function die {
  >&2 echo "Fatal: ${@}"
  exit 1
}

Edit: a Redditor has pointed out that this installed function is unnecessarily cautious and verbose.

These compose nicely with bash’s conditional tests and operators (and each other) to give me easy sanity checks at the top of my scripts:

1
2
3
4
5
6

[[ "${BASH_VERSINFO[0]}" -lt 4 ]] && die "Bash >=4 required"

deps=(curl nc dig)
for dep in "${deps[@]}"; do
  installed "${dep}" || die "Missing '${dep}'"
done

Some other niceties:

Automated linting and formatting

In terms of popularity and functionality, shellcheck reigns supreme. Going by its changelog, shellcheck has been around for a little under 7 years. It’s also available in just about every package manager.

As of 0.7.0, shellcheck can even auto-generate (unified-format) patches for some problems:

1

shellcheck -f diff my_script.sh | patch

And includes a (sadly optional) check for my personal bugbear: non-mandatory variable braces:

1
2
3
4
5
6
7

# Bad
foo="$bar"
stuff="$# $? $$ $_"

# Good
foo="${bar}"
stuff="${#} ${?} ${$} ${_}"

shellcheck also doesn’t complain about usage of [ (instead of [[), even when the shell is explicitly GNU bash5.

There’s also bashate and mvdan/sh, neither of which I’ve used.

Environment variables, not flags

In the past, I’ve used the shift and getopt builtins (sometimes at the same time) to do flag parsing. I’ve mostly given up on that, and have switched to the following pattern:

Compose liberally

Don’t be afraid of composing pipes and subshells:

1
2
3

# Combine the outputs of two `stage-run` invocations for
# a single pipeline into `stage-two`
(stage-one foo && stage-one bar) | stage-two

Or of using code blocks to group operations:

1
2

# Code blocks aren't subshells, so `exit` works as expected
risky-thing || { >&2 echo "risky-thing didn't work!"; exit 1; }

Subshells and blocks can be used in many of the same contexts; which one you use should depend on whether you need an independent temporary shell or not:

1
2
3
4
5
6
7

# Both of these work, but the latter preserves the variables

(read line1 && read line2 && echo "${line1} vs. ${line2}") < "${some_input}"
# line1 and line2 are undefined

{ read line1 && read line2 && echo "${line1} vs. ${line2}"; } < "${some_input}"
# line1 and line2 are defined and contain their last values

Note the slight syntactic differences: blocks require spacing and a final semicolon (when on a single line).

Use process substitution to avoid temporary file creation and management:

Bad:

1
2
3
4
5
6
7
8
9

function cleanup {
  rm -f /tmp/foo-*
}

output=$(mktemp -t foo-XXXXXX)
trap cleanup EXIT

first-stage output
second-stage --some-annoying-input-flag output

Good:

1

second-stage --some-annoying-input-flag <(first-stage)

You can also use them to cleanly process stderr:

1
2

# Drop `big-task`'s stdout and redirect its stderr to a substituted process
(big-task > /dev/null) 2> >(sed -ne '/^EMERG: /p')

Roundup

The shell is a particularly bad programming language that is particularly easy to write (unsafe, unreadable) code in.

It’s also a particularly effective language with idioms and primitives that are hard to (tersely, faithfully) reproduce in objectively better languages.

It’s also not going anywhere anytime soon: according to sloccount, kubernetes@e41bb32 has 28055 lines of shell in it6.

The moral of the story: shell is going to sneak into your projects. You should be prepared with good practices and good tooling for when it does.

If you somehow manage to keep it out of your projects7, people will use shell to deploy your projects or to integrate it into their projects. You should be prepared to justify your project’s behavior and (non-)conformity to the (again, objectively bad) status quo of UNIX-like environments for when they come knocking.

  1. It’s also a popular and predictable favorite in more niche “UNIX-way-or-the-highway” communities. The author hopes that it doesn’t need to be mentioned that this mentality is probably worse than not writing any shell at all. ↩

  2. I’m not telling which ones. ↩

  3. Really just bash scripts. I’m sorry. I know referring to “shell” generically when I mean bash is a pet peeve for a lot of people, but I’ve given up. Like Make, I’ve yet to run into an environment where GNU bash was either not the default or wasn’t a single installation step away. ↩

  4. Some people like to add set -x, but I don’t. I personally find set -x’s output format visually distracting and rarely helpful. If I need to log a command, I do it explicitly. ↩

  5. This might be an optional check that I’m missing. ↩

  6. From sloccount ${PWD} in the repo root. cloc ${PWD} says 28385 lines for sh and bash combined. ↩

  7. You won’t. ↩