You have spent five lessons writing set -euo pipefail without knowing what it does. You have twice put a return 0 at the end of log() "because otherwise the script dies". And bookings_report.sh creates the report's directory and, if something fails halfway through writing it, leaves an incomplete file that somebody will read tomorrow as if it were sound. All of that gets fixed today, and it is worth understanding why it matters so much: a script that breaks early and makes a noise is infinitely better than one that fails halfway in silence. The first wakes you at four in the morning with a clear message; the second leaves you a truncated backup you will discover the day you need it. This lesson is about achieving the first: seeing what your script does, making it stop when it should, and guaranteeing that it leaves things clean whatever happens. At the end we will finally write backup_tramontana.sh.

1. Debugging: seeing what the script really does
2. shellcheck in earnest
3. Strict mode pulled apart
4. Where set -e does not save you
5. trap and the pseudo-signals
6. Guaranteed cleanup and Ctrl+C
7. An ERR handler that is actually useful
8. Idempotency
9. Retries, safe temporary files and locking
10. Application: backup_tramontana.sh

bash -x is the main tool, and its default output (+ command) is not much use as soon as there are functions. With a decent PS4 it changes completely:

Every line now says file, line number and function, and shows the command after expanding variables: you see basename /opt/tramontana/releases/3.2.1, not basename "$(readlink -f "$BASE/app")". That difference is exactly what you are after when a variable does not hold what you thought. Tracing a whole script generates hundreds of lines, so to narrow things down surround only the suspect region with set -x … set +x. Also export the PS4 in your ~/.bashrc so you do not have to type it every time. set -v is used far less, but it is the complement to -x when the problem is in the expansion itself: -v shows what you wrote and -x what Bash understood.

You installed it back in 04-01. These are the warnings you will see most, with the reasons why:

SC2155 deserves a demonstration, because it is subtle and it turns up everywhere:

In the first, false's error disappears and, with set -e, the script carries on happily with an empty variable; in the second it is detected. The rule: declare first, assign afterwards.

Silencing a warning is legitimate when you know what you are doing, but always with a written justification, and the directive affects only the next line (put it at the top of the file and it silences the warning throughout the whole script, which is almost never what you want):

set -euo pipefail is three independent settings, and it is worth knowing exactly what each one does.

• set -e (errexit): if a command returns a non-zero code, the script ends immediately with that code. It turns silent failures into noisy stops.
• set -u (nounset): using an undefined variable is a fatal error instead of expanding to the empty string. It prevents the classic rm -rf "$DIR/" turning into rm -rf / because DIR did not exist.
• set -o pipefail: a pipeline returns the code of the last stage that failed, not that of the last stage; without it, cat nonexistent | wc -l returns 0 and your script believes everything went well.

On set -u, two practical details. With Bash 4.4 and later — including the 5.2 on your Ubuntu 24.04 — an empty "$@" and "${array[@]}" no longer trigger the error, but ${array[0]} of an empty array does. And the emergency exit for any variable that may legitimately not exist is ${VAR:-}, which you have been using since 04-02: [[ -n ${TRAMONTANA_VERBOSE:-} ]] && echo "verbose mode".

This is the material almost nobody tells you about and the reason people over-trust strict mode. set -e is disabled in five contexts, and in all of them a failure goes unnoticed:

Case 4 is the most treacherous and deserves to be seen in action:

cp failed, set -e was active, and the function carried on to the end and declared success, because using it as a condition makes Bash disable errexit throughout its body. It is fixed by checking inside (cp ... || return 1) or by not using the function as a condition: call it on its own and let set -e do its job. The practical conclusion: set -e is a safety net, not a plan. Check explicitly what matters — with || die, with if — and let set -e catch whatever slips through. And add the trap from 04-02: (( i++ )) with i at 0 returns 1 and does abort; use (( ++i )) or i=$(( i + 1 )).

trap installs a handler that runs when a signal arrives. Its syntax is trap 'commands' SIGNAL... and, besides the signals from 03-06, it accepts four pseudo-signals:

EXIT is the important one: it is the only way to guarantee that something happens whatever occurs, because it fires on a normal exit, on a set -e failure, on a Ctrl+C and on a kill; only SIGKILL skips it, and there is no defence against that. Three details that avoid surprises: the handler's quotes decide when the variables are expanded (single quotes, at the moment it fires, which is nearly always what you want); trap - EXIT uninstalls a handler; and trap -p lists the installed ones.

The pattern that makes a script safe to interrupt:

Four deliberate decisions. local code=$? goes on the first line, because any earlier command would overwrite it. The [[ -n $TMP_DIR && -d ... ]] check protects against the case where the script dies before creating the temporary directory: without it, an rm -rf "$TMP_DIR"/* with the variable empty would be catastrophic. exit "$code" preserves the original code instead of returning the rm's. And the INT handler exits with 130, which is 128 + 2 (SIGINT), the convention from 04-01; that exit 130 in turn fires the EXIT, so the cleanup runs anyway: the handlers chain, they do not override each other.

Bash publishes four variables during an ERR handler, and with them you can build a trace almost as useful as one from a language with exceptions:

In four lines you know the code, the file, the line, the exact already-expanded command and the chain of calls. Compare it with the bare cp: Permission denied you would otherwise have had and you will see why this function deserves to live in lib/common.sh.

A script is idempotent if running it twice leaves the system the same as running it once. It is the most valuable design goal in the module, because it means you can relaunch it after a failure without thinking. The canonical counter-example, which Luis wrote last month:

After four runs, grep -c '^max_connections=' /etc/tramontana/app.conf returns 4. The application reads the last one and works, so nobody finds out... until somebody edits the first one and nothing happens. The idempotent version checks before acting:

Three transferable ideas: mkdir -p instead of mkdir (it does not fail if it exists), ln -sfn instead of ln -s (it replaces the link instead of nesting it inside) and checking the desired state before applying the change. It is the model Ansible works with, and it is why its output distinguishes ok from changed; we will come back to it in 07-06.

A network operation does not fail: it fails sometimes. Retrying immediately makes things worse, so you wait longer each time:

The waits go 2, 4, 8, 16: that is exponential backoff. And never retry non-idempotent operations — creating a booking, sending an email — without an identifier that stops them being duplicated.

TMP="/tmp/backup_$$" is a real security hole: the PID is predictable, so an attacker can create a symbolic link with that name in advance pointing at /etc/tramontana/app.conf, and your script, if it runs with privileges, will overwrite the file it points to. TMP=$(mktemp -d) || die 73 "no temp directory" avoids it because it creates the file atomically (with O_EXCL, which fails if it already exists), with a random name and 600 permissions — 700 for directories. Always use it, together with the trap ... EXIT from section 6.

In 03-07 you put flock -n in front of the command in the crontab line. It is better for the lock to live inside the script, so that it also protects you when you launch it by hand:

exec 9>"$LOCKFILE" || die 73 "cannot open the lock" followed by flock -n 9 || die 75 "another backup is already running". exec 9>file opens descriptor 9 for the whole script and flock -n 9 tries to lock it without waiting. The lock is released only when the process dies, even under a kill -9, because it is held by the kernel and not by the file. Do not delete the lock file when you finish: that opens a race in which two processes could lock different files with the same name.

Promised since Module 2, with everything from this lesson:

Test the failure path as well, because that is where the script proves its worth: interrupt it halfway with Ctrl+C and check that no .backup.XXXXXXXX and no .partial is left in the destination. Cleanup you have not tested is cleanup that does not work.

• Believing set -e stops everything. It does not act in conditions, nor after &&/||, nor inside functions used as a condition, nor in local x=$(cmd): keep checking the important things by hand.
• Not capturing $? on the first line of the EXIT handler. Any earlier command overwrites it and the script returns 0 even though it failed.
• Using double quotes in the trap handler. The variables are expanded when the trap is installed, not when it fires: single quotes unless you know why. And do not delete the lock file when you finish, because it opens a race between processes: the lock is released by the kernel when the process dies.
• mktemp without trap ... EXIT. Every failed run leaves rubbish behind: they always go together. Avoid predictable names in /tmp too, because $$ is not random: it is a symbolic-link attack waiting to happen.
• Non-idempotent scripts. If relaunching duplicates a line or adds up twice, you cannot retry after a failure, which is precisely when you need to most.
• Tip: write the final file under a temporary name and rename it at the end; it is the cheap way of making sure a half-written file never exists. And when something does not work, bash -x with PS4 before rereading the code: five seconds of trace save twenty minutes of guesswork.
• Tip: test the error path, not only the success path: remove permissions, fill the disk, kill the process halfway. It is the only way to know whether your traps work.

Exercise 1. This script looks correct and fails silently. Explain why and fix it in two ways.

Exercise 2. Add to lib/common.sh a function with_cleanup: it must create a temporary directory, publish it in TRAMONTANA_TMP, install the cleanup trap and return 0. Write a test case that verifies that, after a script dies through set -e, the directory no longer exists.

Exercise 3. Make this deployment fragment of Luis's idempotent and explain what problem each change solves: mkdir /opt/tramontana/releases/3.3.0, tar -xzf /tmp/app-3.3.0.tar.gz -C /opt/tramontana/releases/3.3.0, ln -s /opt/tramontana/releases/3.3.0 /opt/tramontana/app and echo "3.3.0" >> /opt/tramontana/HISTORY.

Solution 1. The failure is case 4 from section 4: prepare_dest is used as an if condition, and that disables set -e throughout the function. The cp fails, prints its error, execution carries on, "destination prepared" is printed, the function returns 0 — the echo's code — and the if treats the result as good. The script announces that the backup is starting with the destination incomplete.

Fix A is preferable when the function can fail in ways you want to tell apart; fix B, when any failure must abort. What will not do is the original, which promises to check and does not check.

Solution 2.

The test matters because the function depends on two subtleties that are easy to break: the single quotes in the trap and the fact that EXIT fires even when what kills the script is set -e. A test pins that down; a comment does not.

Solution 3.

The four changes. mkdir -p stops the second run dying on the first line. The preceding tar -tzf detects a truncated package before anything has been touched. ln -sfn fixes the classic error: ln -s over a link that already points to a directory creates the new link inside that directory, leaving /opt/tramontana/releases/3.2.1/3.3.0, and the deployment changes nothing while you believe it has. And the grep -qxF — -x whole line, -F literal text — turns the >> into a conditional, avoiding a HISTORY with the same version repeated four times.

Your scripts no longer fail in silence, and for the first time you can interrupt them without fear.

• You debug with bash -x and a PS4 that says file, line and function, narrowing regions with set -x/set +x, and you check the syntax without running anything using bash -n. You understand SC2086, SC2046, SC2164 and SC2155, and you know how to silence a warning by justifying it on the line above.
• You know strict mode piece by piece — -e, -u, pipefail — and, above all, the five contexts where set -e does not act, with the case of a function used as a condition demonstrated and fixed in two ways. You use trap with EXIT, ERR and INT, capturing $? on the handler's first line, with single quotes, knowing that EXIT fires whatever happens except with SIGKILL.
• You write an error_trace() that takes advantage of BASH_COMMAND, BASH_LINENO and FUNCNAME to give a real trace, and you design for idempotency: mkdir -p, ln -sfn, check before appending, write to .partial and rename at the end.
• You retry with exponential backoff, you create temporary files with mktemp instead of predictable names using $$, and you lock with flock on a descriptor inside the script itself.
• And backup_tramontana.sh finally exists: with locking, temporary files that clean themselves up, atomic writing, a checksum, 640 permissions on an archive containing db_password and documented exit codes.

One last question remains, the one that separates a script that works from one that gets deployed: what happens when cron runs this at 4:20 with nobody watching? Where does the log go? How is Marta told only when there is something to do? Where does the database password come from if it cannot be in the file? And what happens if one run takes longer than the interval between runs? In Production Scripts: Best Practices we close the module with that complete checklist, the canonical template with main "$@", the configuration precedence genuinely implemented, the principle of "silence if all is well", testing before production, and the integrating project: deploy.sh, with validation, a prior backup, an atomic link switch, a curl check and automatic rollback if 8080 does not respond.