ShellCheck left the toolkit clean of formal errors, but it knows nothing about what it does. It cannot tell you whether veloz_percentage 0 0 returns anything sensible instead of blowing up, whether backup.sh really honors the thirty days of retention, or whether the awk we used to speed up the report in 08-02 produces the same figures as the loop it replaced. Only running the code and comparing the result against what is expected answers that. This lesson answers the question the module opened with — how do I know a change breaks nothing? — with Bats, the Bash testing framework.

1. Why operations scripts are rarely tested, and why that is a mistake
2. What is worth testing and what is not
3. Testing by hand first
4. Installing bats-core and its libraries
5. Anatomy of a .bats file
6. Running the tests
7. Isolating the test from the environment
8. Test doubles: replacing an external command
9. The structure of tests/ in the toolkit
10. Hook, CI and a coverage criterion
11. Application: tests/common.bats

Nobody argues that a web application gets tested. With system scripts the custom is different: you run them by hand once, "it works", and off to production. The usual excuses have their apparent logic: "it is only an 80-line script" — one that runs rm -rf as root every night — "I test it by running it" — once, with today's data, on your machine, with your PATH — and "there is no way to test something that touches the system", which is false and is solved in section 8.

The argument in favor is stronger than all of those: operations scripts fail at dawn with nobody watching. A user sees a downed web application within ten seconds; backup.sh can spend three weeks storing empty archives and nobody finds out until a restore is needed. And there is a second, more practical argument: tests make refactoring possible. All of Module 8 consists of changing code that works; without tests, every change is an act of faith, and with them the performance refactor from 08-02 is validated in two seconds.

The rule: test the logic, not the plumbing. If a function decides something — a threshold, a format, a validation, a sum — test it. If it only chains two system commands together, check the whole and not the pieces.

Before installing anything it is worth seeing what Bats automates. A function is tested in an interactive session:

That is already a test: run with a known input and compare against the expected output. For three checks it may be enough. What Bats brings when you go beyond three is everything missing here: readable names per case, a failure that does not stop the others, capturing output and exit code at the same time, preparing and cleaning the environment of each test and a final summary with the count of passes and failures.

bats-core is the project maintained today (the original sstephenson/bats is abandoned).

The two helper libraries are not essential, but they improve the messages a great deal: bats-support provides the error formatting and bats-assert the checks assert_success, assert_failure, assert_output, assert_line and refute_output, which on failure show what they expected and what they got instead of a terse "it did not pass".

A .bats file is a Bash script with one added piece of syntax: the @test block.

The pieces, one by one:

• @test "description" { ... }: each test with a name that reads in the output and that must say which behavior is expected, not "test 1".
• run command ...: runs the command capturing its output and its exit code without the failure aborting the test. It is Bats' central piece.
• $status: the exit code of what run executed. $output: the combined output (stdout + stderr). ${lines[@]}: the same output as an array, one line per element.
• setup() runs before each test and teardown() after it, even if it fails; setup_file()/teardown_file() run once per file, for expensive preparation. load file loads a .bash relative to the test directory, skip "reason" skips a test and BATS_TEST_DIRNAME is the directory of the .bats file, for building paths independent of where it is launched from.

Without bats-assert the same checks are written with [ "$status" -eq 0 ] and [ "$output" = "32.1" ], which is equally valid.

Useful options: --filter 'percentage' runs only the tests whose name matches the pattern, -t produces pure TAP output — the standard format CI systems consume — -x traces every command for debugging and -j 4 runs in parallel. The exit code is 0 if everything passes and 1 if something fails, which is exactly what the hook and CI need.

The most common mistake when starting out is writing tests that depend on the machine: on the real /srv/veloz/data, on the time zone, on jq being installed. Those tests pass on your laptop, fail in CI and end up disabled. The solution is for each test to build its own world in a temporary directory:

Three shipments are enough: there is a repeated city, two statuses and three couriers, which is all any aggregation needs. BATS_TMPDIR is the temporary directory Bats manages; in recent versions there is also BATS_TEST_TMPDIR, created and deleted automatically per test.

This is the technique that makes system scripts testable, and it is simpler than it looks. Bash looks for commands by walking $PATH from left to right; if you put a directory containing a script called curl at the front, that is the curl that runs.

The double achieves three things, and only the first is "not using the network". Determinism: the response is always the same, so the test does not fail because the API is slow. Cases impossible to provoke: by changing the double to return exit 7 or a truncated JSON you test the error handling, which is exactly the code that never gets exercised by hand. And call verification: the log lets you check how it was invoked, which is how the use of --netrc from 08-03 was tested above.

The same technique covers the rest: a systemctl double returning inactive tests the alerting branch of service-status.sh; an ssh one tests fleet.sh without leaving the machine; a mail one verifies that watchdog.sh warns without sending anything to anybody. And backup.sh needs no double: it is enough to point VELOZ_DATA_DIR and VELOZ_BACKUP_DIR at the temporary directory so it works over four fake files instead of over the real disk.

The tests/ directory contains the submodules (bats/, lib/) and one .bats per tested unit: common.bats for the library functions, report.bats for daily-report.sh end to end and backup.bats for retention and restore.

The edge cases are the ones that find the most bugs: a CSV with only the header makes many reports divide by zero, an amount 1,234.50 breaks the sum and a file with spaces takes apart a badly written find. And testing the exit code is not a detail: it is the only thing the systemd timer from 07-05 sees and what decides whether an alert is sent.

The pre-commit hook from 08-04 already invoked bats tests/. In CI it is added alongside the static analysis, with bats --formatter tap tests/. An important detail to make this sustainable: the tests must be fast. If they take forty seconds, the hook becomes annoying and somebody will start using --no-verify; with doubles and fake data, the toolkit's complete suite takes under two seconds, and that is what guarantees it always runs.

You do not have to test everything. The criterion, in priority order: what has already broken at least once — every resolved incident becomes a test, and this is the most valuable rule, because it guarantees the same failure does not come back and it grows the suite precisely where the system is fragile; what decides something (thresholds, validations, computations); what is destructive (retention, deletion, overwriting), because a failure there cannot be undone; and the exit codes of the scripts that govern alerts and timers. Chasing 100% coverage in Bash produces fragile tests that break with every formatting change and end up disabled, which is worse than not having them.

Five tests, under two seconds, and they cover the three kinds of case (to which it is worth adding the positive case veloz_require bash). The last one deserves attention: it checks that veloz_log treats its argument as text and does not expand it, which is exactly the injection vulnerability from 08-03. A test can watch over a security property just as it watches over a computation.

• Forgetting run. Without it, a command that fails aborts the test and you cannot check $status.
• Tests that depend on the machine. Real paths, time zone, jq installed. Isolate everything in setup with a temp dir, TZ and LC_ALL.
• Not cleaning up in teardown. Tests contaminate each other and fail depending on the execution order.
• Testing only the happy path. Bugs live in the empty CSV, the missing field and the full disk.
• Tip: turn every incident into a test. It is the cheapest way for the suite to cover exactly what matters.
• Tip: bats -x --filter 'name' runs a single test with full tracing; it is the set -x from 05-03 applied to tests.

Exercise 1. Write a Bats test verifying that daily-report.sh fails with code 2 and a message on stderr when the input CSV does not exist.

Exercise 2. Explain why this test is wrong and rewrite it.

Solution 1.

assert_failure 2 requires the exact code, not just any failure: it is what distinguishes "usage error" from "internal error" according to the conventional codes from 05-03. Since run captures stdout and stderr together, assert_output sees the message even though it is written to stderr.

Solution 2. It has four problems: it uses an absolute path in the user's home, so it only works on that machine; it runs the backup over the real data, risking overwriting the production backup from a test; it only checks the exit code, not that the backup contains anything; and its name does not say which behavior is expected.

Now the test is independent of the machine, works over fake data and verifies the effect: that the archive exists, that its checksum exists (07-03) and that the expected file is inside it.

Testing operations scripts is not a luxury: it is the only early detection available in code that runs at dawn with nobody watching, where an empty backup can go unnoticed for weeks. The best payoff comes from the pure functions in lib/common.sh, followed by each script's end-to-end behavior — exit code, output and the files it creates; plumbing with no logic does not get tested. The idea is the same as the homemade check [[ $(f x) == expected ]] || echo FAIL, and Bats adds what that one lacks: readable names, isolation between cases, a final summary and, above all, run, which captures $status, $output and ${lines[@]} without a failure aborting the test. With setup/teardown each case builds its world in a temporary directory with fake Veloz Envíos data and pins TZ and LC_ALL, so that the test does not depend on the machine — the main reason suites end up disabled. Test doubles are the technique that makes it all possible: a fake script called curl, systemctl, ssh or mail in a directory at the front of PATH lets you test veloz_api_get without a network, fleet.sh without servers and backup.sh without touching the disk, as well as provoking errors you would never see by hand and verifying how the command was invoked. The suite is organized into tests/common.bats, tests/report.bats and tests/backup.bats, each covering normal, edge and error cases with their exit codes, and it runs in the pre-commit hook and in CI (08-04, 08-05), which demands that it be fast: with doubles, under two seconds. And the coverage criterion is not a percentage but a priority: what has already broken once, what decides something and what is destructive.

We now have a readable one, a fast one, an audited one, a versioned one, an analyzed one and a tested one. One decision remains that we have taken for granted all course long: that the interpreter is Bash 5 on Ubuntu. Lesson 08-07 puts it to the test: what happens when a script has to run where /bin/sh is dash, Alpine's ash or ksh, which of the constructs we have been using are bashisms and what their POSIX equivalent is, why the differences between the GNU and BSD tools hurt more than those of the shell itself, how to check portability with checkbashisms, shellcheck -s sh and dash -n, and when pure POSIX is worth it and when it is a burden. That lesson closes the entire module.