Your scripts already work. But "working" is what they do at midday, with your terminal open, your variables loaded and you watching the output. Production is something else: it is 4:20 in the morning, there is nobody there, the PATH is cron's meagre one, the disk is fuller than yesterday, the network is behaving oddly and the only evidence of what happened will be whatever your script wrote down. This lesson closes the module with the checklist that bridges that gap and with the project that proves it: deploy.sh, the Tramontana Bookings deployment script, with validation, a prior backup, an atomic switch and automatic rollback if the application does not respond. It is the module's final exam, and it uses absolutely everything you have learned.

1. The production checklist
2. Canonical structure and main "$@"
3. Configuration: the precedence, implemented
4. Secrets
5. Logging: what, how and where
6. Output for humans and for machines
7. Locking, time limits and overlap
8. Notifications: silence if all is well
9. Testing, versioning and documentation
10. When to leave Bash
11. Project: deploy.sh

None of them is optional in a script that runs on its own: the ten together are the difference between an automation and a time bomb.

The complete template, an evolution of the one from 04-01. Copy it as it is:

main at the end lets the file be read from top to bottom like a piece of text: first the constants, then the pieces, and at the end the summary of what the program does. Since Bash needs functions to be defined before they are called, main "$@" has to be the last line, and putting it there also protects against a real failure: if the file gets truncated while being copied, Bash will not run a half script, because the call to main will have disappeared with the rest. And it goes with quotes, always, for what you know from 04-03: it is the only way for the arguments to arrive intact.

The rule from 04-03 — arguments > environment > file > defaults — is implemented by applying it in that order, from the weakest to the strongest:

${VAR:-$CURRENT} takes the value from the environment if it exists and, if not, keeps whatever was already there: that little trick is the whole mechanism. And log the unknown keys instead of ignoring them, because a misspelled disk_thresold that says nothing is a wasted night.

Three rules, in order of importance:

• Never in the script. A file versioned in git with a password inside it is compromised for ever, even if you delete it: it stays in the history.
• Never on the command line. mysql -p"$PASSWORD" is visible in ps aux to any user on the machine for as long as the command lasts. Yes in a file with 600 permissions, owned by the service's user, or in an environment variable set by the service manager.

A decent minimum: no passwords in the code and active verification of the permissions. Serious management — pass, a secrets manager, TLS certificates — is the subject of 06-05.

A production script logs five things: that it has started (with its version and its parameters), every decision it takes, every change it makes to the system, that it has finished, and how long it took. With that you can reconstruct any incident.

Three details of the format. The ISO 8601 date with a time zone (%Y-%m-%dT%H:%M:%S%z) sorts alphabetically the same as chronologically and is unambiguous across a clock change. The PID in brackets lets you separate two interleaved runs. And the message goes at the same time to the file and to stderr, so that it serves equally well in cron and by hand.

The duration is measured with SECONDS, the variable Bash increments on its own: SECONDS=0 at the start and $SECONDS at the end. And a note on the boundary: in Module 5 we will replace this file with the journal, with logger -t tramontana -p daemon.info as the bridge and journalctl to query it (05-06), plus logrotate so that the file does not grow for ever.

A useful script serves two audiences: Marta wants to read, the monitoring wants to parse. The solution is not to choose, it is to offer both:

With --quiet everything informative is suppressed and only the errors are left — indispensable for cron, which emails any output — and with --json the output is a single parseable line. If your JSON starts to have nested structures, that is the signal from section 10: in Bash there is no decent way of escaping quotes inside arbitrary values.

You already have the flock on descriptor 9 from 04-06. What is missing is the other half: what happens if one run takes longer than the interval between runs. The nightly backup is launched at 04:20; if one day it takes 25 hours, tomorrow's will find the lock. Three answers, in order of preference:

• Abandon with a code of your own (75) and a warning. That is right for backups and reports: tomorrow's will do it.
• Wait with flock -w 300 9, if the overlap is brief and waiting is acceptable. And never kill the previous one from the new one: if that is needed, the problem is the interval, not the lock.

The run also needs a ceiling, because a hung process does not fail: it just sits there, holding the lock for ever.

timeout returns 124 when it cuts things short on time, which lets you tell that apart from an ordinary failure. It is the direct application of the SIGTERM → wait → SIGKILL rule from 03-06.

The principle is short and it is broken all the time: a script that reports when everything is fine teaches people to ignore its reports. If Marta gets a "backup correct" email every morning, by the third week she files it unread, and the day it says "backup FAILED" she will not read that one either. The working rule: notify only when there is something to do, and let the message say what has happened, what impact it has and what is expected of the person reading it.

So that silence is not mistaken for "the script did not run", the counterpart is a success mark: the last line of the log carrying today's date, or a LAST_SUCCESS file with the timestamp. Checking "is the mark from today?" is trivial and catches 90% of the problems, including the worst: that the task has stopped running at all.

Before a script touches production:

1. shellcheck with no warnings and bash -n, both mandatory in CI: find scripts -name '*.sh' -exec shellcheck {} +.
2. A complete --dry-run against the real system, reading the whole output: it is the last chance to see what is going to happen. And in a test environment with the same data and paths, with a snapshot of the VM before every destructive test.
3. The failure path, not only the success path: remove permissions from the destination, point the health URL at a closed port, interrupt with Ctrl+C halfway. If you have only tested it when everything goes well, you have not tested it.
4. With cron's environment, which is different from yours: env -i /home/operator/scripts/deploy.sh -n 3.3.0 /tmp/app.tgz reproduces the poverty of variables it will have at 4:20.

Versioning is not optional: ~/scripts is a git repository, with one commit per change and a message that says why. Deploying the script is then a git pull on the server — or better, copying it with rsync from a central repository, without leaving git credentials on the server. And the minimum documentation is three things: the header with usage and exit codes, a --help that always works, and a README.md in the folder saying which script does what and which one calls which. Nothing more; nothing less.

Bash is excellent at orchestrating commands. It stops being so as soon as any of these signals appears:

The most honest signal: if you are writing eval, or escaping quotes inside quotes inside quotes, you have already crossed the line. For Tramontana the natural boundary is Ansible: the moment the deployment has to be done on three servers at once, deploy.sh becomes an Ansible role, and this module will have served its purpose, which was to teach you exactly what that role does underneath.

The module's final exam. It validates, checks preconditions, backs up before touching anything, extracts, verifies, switches the link atomically, checks with curl and rolls back on its own if the application does not respond.

The service kept running. That is what this module has built: 3.3.0 did not start, the script detected it, rolled the link back in the same second, logged everything and returned a code (69) that says exactly what happened. And with -n you would have seen beforehand what it was going to attempt, without touching anything.

Go back over where each lesson lives: the shebang and the explicit exit are from 04-01; ${VAR:-}, the arrays and printf from 04-02; getopts, usage(), stdout versus stderr and the exit codes from 04-03; [[ =~ ]], case and the waiting while from 04-04; SCRIPT_DIR, the library and die() from 04-05; and set -euo pipefail, the traps, the locking, the idempotency and timeout from 04-06. Nothing here is new: it is everything that came before, together.

• Relative paths. They work from your terminal and fail from cron: always absolute, or derived from SCRIPT_DIR. And do not assume the PATH, which in cron is /usr/bin:/bin; if you use something from /usr/local/bin, give the full path or an explicit PATH= at the top.
• Putting main "$@" unquoted or in the middle of the file. The first breaks arguments with spaces; the second fails because the functions are not defined yet. And do not notify on success: it teaches people to ignore the notifications. Silence if all is well, and a checkable success mark.
• Logging only the ending. Without the intermediate decisions you cannot reconstruct anything; log what you decided not to do as well. And always set a time limit: a hung process does not fail, it sits there holding the lock until somebody kills it.
• Deploying without a prior --dry-run. Thirty seconds of reading against half an hour of restoring.
• Tip: keep ~/scripts in git from today even if it is only you, because git log answers "since when has it done this?" better than your memory. And make every script print its version on the log's first line: when two servers behave differently, that will be the first thing you compare.
• Tip: the best moment to write the folder's README is today, while you still remember why each script exists.

Exercise 1. Add to deploy.sh a --json option which, on finishing, prints on stdout one line with version, previous, status (deployed or rolled-back), code and duration. It must also work when there is a rollback, without duplicating code.

Exercise 2. Write ~/scripts/check_tasks.sh, which verifies that the three nightly tasks (cron-backup.log, cron-purge.log and the deployment) have left a mark from today and notifies only if one is missing, applying the principle of silence if all is well.

Exercise 3. Luis wants to put this line in /etc/cron.d/tramontana: 20 4 * * * operator cd ~/scripts && ./deploy.sh $VERSION /tmp/app.tar.gz. Find the five problems and rewrite it.

Solution 1. The key is not to duplicate: the single exit point already exists — the trap cleanup EXIT — and that is where the summary should be printed.

The 2>/dev/null leaves only the JSON line, because the log goes to stderr: the channel separation from 04-03 bearing fruit. And since the exit 0 on the success path also fires the EXIT, the printf '%s\n' "$version" is removed from the body of main so as not to print it twice.

Solution 2.

Without -v and with everything correct the script prints nothing and returns 0: cron sends no email and Marta gets no noise. The email is only sent on the failure path, saying what is happening and what to check, as the course's convention requires.

Solution 3. The five problems:

1. ~ is not expanded in a system crontab. The field is passed to sh -c with a minimal environment and HOME may not be defined: absolute paths.
2. $VERSION is empty. Cron does not inherit your environment, so the script would receive one argument where it expected two and the validation would reject it. Just as well.
3. The output is not redirected. Without >> log 2>&1, any message turns into an email and any error is lost if email is not configured.
4. No SHELL or PATH declared at the top of the file, and a badly chosen time: 04:20 avoids the clock-change window and the overlap with other tasks.
5. Deploying automatically from cron is a bad idea in the first place. A deployment is approved by Marta; what should run on its own is the backup.

Absolute paths, an explicit PATH, MAILTO="" because the script already writes its own log and only notifies when necessary, and the deployment kept out of cron: it is launched by hand when Marta approves it. In Module 5 we will see that a systemd timer does all of this with better logging and better overlap control.

You have closed the module, and with it the gap between "it works for me" and "it runs on its own".

• You have a ten-point production checklist and a canonical template with main "$@" as the last line, which makes the script read from top to bottom and stops it running half way if it is truncated. You implement the configuration precedence — arguments > environment > file > defaults — for real, logging even the unknown keys.
• You keep secrets out of the code and off the command line, and you verify the permissions of the file that holds them.
• You log start, decisions, changes, end and duration, with an ISO date and the PID, to a file and to stderr, knowing that in 05-06 that will move to the journal.
• You produce output for humans and for machines, with --quiet and --json. You control overlap with flock, you set a ceiling with timeout knowing that 124 means "time ran out", and you apply silence if all is well with a checkable success mark as the counterpart.
• You test with shellcheck, bash -n, --dry-run, a test environment, env -i and the failure path; you version in git and you document with a header, --help and a README.
• You recognise when to leave Bash — 300 lines, nested JSON, concurrency, eval — and where to go. And deploy.sh exists: it validates, locks, backs up before touching anything, extracts with a time limit, switches the link atomically, checks the health and rolls back on its own when 3.3.0 does not start, leaving the service standing.

In /home/operator/scripts/ there is now a real toolbox: health_check.sh, bookings_report.sh, bookings_summary.sh, purge_releases.sh, backup_tramontana.sh, check_tasks.sh, deploy.sh, the lib/common.sh library and its test_common.sh, all with the same structure, the same exit codes and the same error discipline. That is no longer "knowing Bash": it is knowing how to automate.

But look at what has been surfacing between the lines and we have kept postponing. backup_tramontana.sh writes into /var/log/tramontana/ and nobody rotates that file. deploy.sh switches the link but does not restart the application, because you do not yet know how to manage services. The backup goes to /srv/tramontana/backups without our having discussed how much disk there is or what happens when it runs out. The tramontana group still has not been formally created, svc-tramontana exists by magic, and you have been using sudo without asking yourself who decides what you can do with it. In Module 5: System Administration all of that stops being magic: you will create real users and groups, configure sudo and the special permissions, install and pin package versions, manage disks and filesystems, turn your scripts into systemd services and timers with start-up, dependencies and automatic restart, send your logs to the journal with rotation, measure the server's performance, and finish with a backup and restore strategy worthy of the name. The scripts you have written will be the pieces; Module 5 is the system that holds them up. Update your VM snapshot and we will see you there.