We have nine working scripts, each with its options, its configuration and its own way of being invoked. That is not a product: it is a folder. A colleague joining tomorrow does not know which one to run, the help is scattered across nine different --help outputs, installation is "copy this and remember the permissions", and nobody knows which version is running on srv-veloz-02. This last project turns the collection into veloz-ops: a single command with subcommands, installable, configurable, versioned, tested, deployed across the fleet and documented. It is the close of the course, and the step that separates "I can program in Bash" from "I have shipped something".
Contents
- The toolkit seen whole
- The single-command-with-subcommands pattern
- Final reorganization of the repository
- Unified configuration and
veloz-ops config - Bash completion
- Idempotent installation
- Semantic versioning and
veloz-ops version - Quality before deploying
- Fleet deployment and rollback
- Documentation: README and runbook
- The toolkit seen whole
flowchart TD
U["veloz-ops (single command)"]
U --> S2[status] --> L2[libexec/system-info.sh]
U --> S3[logs] --> L3[libexec/analyze-logs.sh]
U --> S4[backup] --> L4[libexec/backup.sh]
U --> S5[network] --> L5[libexec/network-monitor.sh]
U --> S6[fleet] --> L6[libexec/fleet.sh]
L2 & L3 & L4 & L5 & L6 --> LIB["lib/common.sh<br/>veloz_log · veloz_require · veloz_api_get"]
LIB --> CFG["etc/veloz-ops.conf + etc/*.d/"]
L3 --> D1[("/var/log/veloz/*.log")]
L4 --> D2[("/srv/veloz/data + /backups")]
L5 --> D3(["veloz-api :8080 · SSH fleet"])
TMR["systemd timers"] --> U
The diagram shows the property we are after: one entry point, one shared library, one configuration. The scripts move into libexec/ because they stop being a public interface; nobody invokes them directly, veloz-ops invokes them.
- The single-command-with-subcommands pattern
It is the pattern of git, systemctl or docker, and in Bash it is implemented with case and functions (04-05):
#!/usr/bin/env bash
# veloz-ops - single entry point of the toolkit.
set -euo pipefail
export LC_ALL=C
readonly ROOT=${VELOZ_ROOT:-/opt/veloz-ops}
readonly LIBEXEC=$ROOT/libexec
# shellcheck source=lib/common.sh
. "$ROOT/lib/common.sh"
usage() {
cat <<'EOF'
Usage: veloz-ops <subcommand> [options]
status System information (09-01)
logs Log analysis (09-02)
backup Backups, verification and restore (09-03)
network Network and service monitoring (09-04)
fleet Run a command on the three servers
config Effective configuration and its origin; version; help <sub>
EOF
}
main() {
local sub=${1:-help}; shift || true
case $sub in
status) exec "$LIBEXEC/system-info.sh" "$@" ;;
logs) exec "$LIBEXEC/analyze-logs.sh" "$@" ;;
backup) exec "$LIBEXEC/backup.sh" "$@" ;;
network) exec "$LIBEXEC/network-monitor.sh" "$@" ;;
fleet) exec "$LIBEXEC/fleet.sh" "$@" ;;
config) show_config "$@" ;;
version) printf 'veloz-ops %s\n' "$VERSION" ;;
help|-h|--help) [[ $# -gt 0 ]] && usage_for "$1" || usage ;;
*) printf 'Unknown subcommand: %s\n\n' "$sub" >&2; usage >&2; exit 2 ;;
esac
}
main "$@"Three decisions. exec replaces the process instead of creating a child: it saves a process and, more importantly, the exit code and the signals reach the real script with no intermediary, so Ctrl-C on veloz-ops backup really interrupts the backup. The case is an explicit allowlist (08-03): building the path as "$LIBEXEC/$sub.sh" would be shorter and would let anyone run veloz-ops ../../bin/whatever. And an unknown subcommand exits with code 2 and writes to stderr, following the 05-03 convention.
- Final reorganization of the repository
veloz-ops/ ├── bin/veloz-ops # only executable in the PATH ├── libexec/ # internal scripts, not in the PATH │ ├── system-info.sh analyze-logs.sh backup.sh network-monitor.sh fleet.sh │ └── access.awk # separate awk programs (09-02) ├── lib/common.sh # veloz_* functions, loaded with source ├── etc/veloz-ops.conf # 600; plus backup.d/ and network-monitor.d/ ├── systemd/*.service *.timer ├── tests/*.bats ├── docs/runbook.md └── completion/ install.sh verify.sh README.md CHANGELOG.md
The criterion is worth stating out loud: bin/ is what the user invokes, libexec/ what the program invokes, lib/ what is loaded with source and never executed, etc/ what is edited without deploying. That last phrase settles the doubts: if changing it requires code review, it is code; if it is a path, a threshold or a list of targets, it is configuration.
- Unified configuration and
veloz-ops config
veloz-ops configThe 05-06 precedence —default < file < environment < options— is implemented at a single point in lib/common.sh, and it is made auditable by storing the origin of every value:
declare -A CFG ORIGIN
veloz_config_load() {
local f=$1 k v env_var
while IFS='=' read -r k v; do # file: overrides the defaults
[[ $k == \#* || -z $k ]] && continue
k=${k// /}; v=${v%\"}; v=${v#\"}; CFG[$k]=$v; ORIGIN[$k]="file:$f"
done < "$f"
for k in "${!CFG[@]}"; do # environment: overrides the file
env_var=VELOZ_$k
[[ -n ${!env_var:-} ]] && { CFG[$k]=${!env_var}; ORIGIN[$k]=environment; }
done
}${!env_var} is indirect expansion (03-06): it builds the name VELOZ_DISK_THRESHOLD and reads that variable. Storing the origin alongside the value costs one more associative array and answers the question that eats hours in production: "the threshold is at 95, but the file says 85, where is that coming from?". veloz-ops config prints it in three columns —key, value, origin— and that is why it is a first-class subcommand and not a debugging echo.
- Bash completion
# completion/veloz-ops.bash -> /etc/bash_completion.d/veloz-ops
_veloz_ops() {
local cur=${COMP_WORDS[COMP_CWORD]} options='status logs backup network fleet config version help'
(( COMP_CWORD > 1 )) && case ${COMP_WORDS[1]} in
backup) options='run verify restore report --dry-run' ;;
status) options='--format --section --help' ;;
network) options='check summary live --target' ;;
esac
mapfile -t COMPREPLY < <(compgen -W "$options" -- "$cur")
}
complete -F _veloz_ops veloz-opsCOMP_WORDS and COMP_CWORD are the variables Bash fills in when you press Tab, and compgen -W filters a list by the typed prefix. It is not cosmetic: completion is the documentation people actually read, and it prevents half the typing mistakes in a delicate subcommand like restore.
- Idempotent installation
install.sh must be runnable a hundred times with the same result (07-02):
install_toolkit() {
install -d -m 755 "$PREFIX"/{bin,libexec,lib,etc,docs}
install -m 755 bin/veloz-ops "$PREFIX/bin/veloz-ops"
install -m 755 libexec/*.sh "$PREFIX/libexec/"
install -m 644 libexec/*.awk lib/common.sh -t "$PREFIX/lib/"
[[ -e $PREFIX/etc/veloz-ops.conf ]] ||
install -m 600 etc/veloz-ops.conf.example "$PREFIX/etc/veloz-ops.conf"
install -m 644 completion/veloz-ops.bash /etc/bash_completion.d/veloz-ops
ln -sfn "$PREFIX/bin/veloz-ops" /usr/local/bin/veloz-ops
install -m 644 systemd/*.service systemd/*.timer /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now veloz-backup.timer veloz-network.timer
"$PREFIX/bin/veloz-ops" version
}install instead of cp + chmod because it creates the destination and sets the permissions in a single operation, and it is idempotent by nature; ln -sfn overwrites the link without complaining; systemctl enable --now enables and starts, and repeating it does no harm. The key line is the conditional: existing configuration is never overwritten. An installer that overwrites veloz-ops.conf wipes out the thresholds somebody tuned at three in the morning; that is why the repository versions veloz-ops.conf.example and not the real file (08-04). Uninstalling is symmetric: --uninstall stops and disables the timers, deletes the units, runs daemon-reload, removes the link and the tree, and keeps etc/ and the backups, saying where they are left.
- Semantic versioning and
veloz-ops version
veloz-ops version| Change | Increments | Example |
|---|---|---|
| A subcommand or option is removed or renamed; the format of a state file changes | MAJOR | --section becomes --area |
| New subcommand or option, backward compatible | MINOR | veloz-ops network summary is added |
| A fix with no interface change | PATCH | The median calculation is fixed |
The version comes from Git, not from a constant somebody will forget to update (08-04):
VERSION=$(git -C "$ROOT" describe --tags --dirty 2>/dev/null || cat "$ROOT/VERSION" 2>/dev/null || echo unknown)git describe --tags --dirty gives v1.4.0 on an exact tag, v1.4.0-7-gab12cd3 if there are seven commits on top —which instantly reveals that there is something unreleased on that server— and adds -dirty if there are uncommitted changes, which in production is an alert in itself. install.sh also writes a VERSION file for deployment via rsync without a repository.
- Quality before deploying
A single script gathers all the checks from module 8:
#!/usr/bin/env bash
# verify.sh - everything that must pass before deploying.
set -uo pipefail
failures=0
mapfile -t SCRIPTS < <(find bin libexec lib -type f)
step() { printf '\n== %s ==\n' "$1"; shift; "$@" || ((failures++)); }
step syntax bash -n "${SCRIPTS[@]}"
step shellcheck shellcheck -x -S style "${SCRIPTS[@]}"
step format shfmt -d -i 2 -ci "${SCRIPTS[@]}"
step tests bats tests/
step help bin/veloz-ops help
exit $(( failures > 0 ))The order goes from fast to slow: bash -n takes milliseconds and rules out a syntax error before spending thirty seconds on Bats. shellcheck -x follows the source calls so it also analyzes lib/common.sh (08-05), and -S style raises the bar above errors. shfmt -d does not reformat, it only shows the difference and fails: in CI you must fail, not fix things by surprise. The Bats battery is extended to the subcommands (08-06):
@test "unknown subcommand exits with 2 and writes to stderr" {
run bin/veloz-ops doesnotexist
[ "$status" -eq 2 ]
[[ $output == *"Unknown subcommand"* ]]
}
@test "backup --dry-run creates no directory" {
local before; before=$(find "$BATS_TMPDIR/backups" -type d | wc -l)
run bin/veloz-ops backup run --dry-run
[ "$status" -eq 0 ]
[ "$(find "$BATS_TMPDIR/backups" -type d | wc -l)" -eq "$before" ]
}The second test is the most valuable in the toolkit: it verifies that dry-run mode has no effects, which is the promise anyone leans on before launching a backup with --delete. The pre-commit hook (08-04) runs the first three stages and CI runs the full battery on every branch.
- Fleet deployment and rollback
deploy() { # $1 = tag, e.g. v1.4.0
local tag=$1 host
git tag -l "$tag" | grep -q . || veloz_die 2 "nonexistent tag: $tag"
./verify.sh || veloz_die 3 "code that does not verify is not deployed"
for host in srv-veloz-01 srv-veloz-02 srv-veloz-03; do
printf '\n--- %s ---\n' "$host"
ssh -n "$host" "cd /opt/veloz-ops && git fetch --tags -q &&
git checkout -q $tag && sudo ./install.sh" || return 1
ssh -n "$host" 'veloz-ops version && veloz-ops status --section services' || return 1
done
}Deployment is sequential and with a post-check on each host: if srv-veloz-01 fails, the second one is never touched, and in the worst case one server is left broken and two intact. ssh -n (07-06) stops the loop from eating standard input, the classic error that makes only the first host get deployed. A tag is deployed, never a branch: git checkout v1.4.0 gives exactly the code that was tested, and the rollback is the same command with the previous tag (deploy v1.3.2). That reverting is identical to deploying is what makes somebody dare to run it at four in the morning. With one caveat for the runbook: reverting the code is trivial, reverting a format change to the state files or the history is not, and that is why that kind of change bumps the MAJOR version.
- Documentation: README and runbook
They are two documents with two different readers and they are part of the deliverable. The README.md is read by whoever arrives new, cold: what it is, what it requires, how it is installed in five copy-pasteable commands, the subcommands with one real example each and where the configuration lives. If somebody cannot install it and run veloz-ops status following only the README, the README is wrong. The docs/runbook.md is read by somebody half asleep with an alert on their phone: it is organized by alert, not by script, with no prose and with copy-pasteable commands.
## ALERT: backup with failures on srv-veloz-01
1. Confirm: veloz-ops backup report
2. Check the log: journalctl -u veloz-backup.service -n 50
3. If "out of space" (code 5): df -h /backups; review the profile's KEEP_*
4. If "verification failed" (code 4): do NOT delete anything; restore from srv-veloz-03
5. Escalate to Operations if the data profile fails two nights in a row.A script with no documentation is a script only its author can run; and its author will end up on holiday exactly the night it fails.
Common Mistakes and Tips
- Building the subcommand path from the user's argument.
"$LIBEXEC/$1.sh"opens the door to../. An allowlist withcase, always. - Installers that overwrite the configuration. Version
.conf.example; the real.confis only created if it does not exist. - Deploying a branch instead of a tag.
mainmeans something different every hour; a tag always means the same thing. - A
for hostloop withsshand no-n. The firstsshconsumes the list and the rest of the fleet is left undeployed, silently. - Documenting by script instead of by alert. At four in the morning nobody searches for "analyze-logs"; they search for the text that arrived on their phone.
- Tip: run
veloz-ops configon the three servers and compare. Unexplained differences between nodes are the origin of half the "it works on this server" incidents.
Exercises
doctorsubcommand. It checks the health of the installation itself: dependencies (jq,rsync,curl,awk),etc/permissions (600), active timers, the age of the last backup and of the last monitor cycle. One symbol per check and a global 0/1/2 code.- Fleet status dashboard. A
veloz-ops fleet statusthat collects thestatusJSON from the three servers in parallel and flags the differences in version, kernel and effective configuration. - Open extension challenges. Package the toolkit as a
.debwithpostinst/prermand the configuration marked asconffiles; add aveloz-ops apithat exposes the reports over HTTP, deciding what not to expose; unify a--jsonacross every subcommand with a Bats test validating each output; and port the toolkit to Debian or Alpine, noting every bashism you have to resolve (08-07).
Solutions
1. doctor checks nothing on its own: it reuses what is already built.
doctor() {
local failures=0
chk() { local n=$1; shift
if "$@" >/dev/null 2>&1; then printf ' [OK] %s\n' "$n"
else printf ' [FAIL] %s\n' "$n"; ((failures++)); fi; }
chk "dependencies" veloz_require jq rsync curl awk
chk "etc permissions" test "$(stat -c %a "$ROOT/etc/veloz-ops.conf")" = 600
chk "backup timer" systemctl is-active --quiet veloz-backup.timer
chk "recent backup" test "$(days_since_last_backup)" -le 1
(( failures == 0 )) && return 0 || return 2
}The nested chk function, which takes a command and runs it, avoids repeating the same if ten times; and returning 0/2 lets doctor itself be watched from watchdog.sh with the 07-04 convention.
2. fleet.sh already runs in parallel, so consolidating the JSON with jq -s is enough:
veloz-ops fleet 'veloz-ops status --format json' | jq -s '
map({host, version, kernel})
| {nodes: ., distinct_kernels: (map(.kernel) | unique | length)}'Detecting the divergence —not listing the values— is what makes the dashboard a useful tool: if distinct_kernels is greater than 1, there is a server that has not been updated.
Conclusion
We started the course typing echo "Hello" in a terminal and we finish it with a product installed on three servers: a single command with subcommands and help, configuration with precedence and auditable origins, completion, an idempotent installer and a symmetric uninstaller, semantic versioning tied to Git tags, a test battery that runs on every commit, deployment by tag with rollback and documentation written for whoever will read it in the small hours. None of those pieces is advanced Bash: they are case, functions, arrays, printf, source and exit codes —the material of module 4— applied with the judgment of module 8. That is the message of the whole course: the difference between a script and a tool is not in the language, it is in the discipline.
It is worth ending with some honesty about the limits. Bash is unbeatable at gluing programs together, moving files and automating the system; it is exactly the right tool for the five projects in this module. It stops being so when nested data structures appear (more than two levels of JSON and jq is no longer enough), when you need unit tests of business logic, real concurrency, sustained floating-point arithmetic, or when the script goes past a thousand lines and every change is frightening. If you find yourself writing an expression parser or a stateful HTTP client in Bash, the signal is clear: it is time for Python, Go or whatever language your team uses. Knowing when to stop is part of mastering the tool, and rewriting a well-structured Bash script in another language is straightforward precisely because it was well structured.
And where to go next. The Bash manual (man bash, and above all its expansions section, which you read end to end once in your life) is the only definitive source. Google's shell style guide, so you can argue decisions with reasons instead of tastes. ShellCheck as a permanent teacher: every warning you do not understand, look it up by its code and you will learn something you did not know. And above all, the thing that really consolidates it: automate your own work. Take that task you do by hand every Monday, write it as a script with set -euo pipefail, give it options, add a --dry-run, test it with Bats, push it to Git and fire it with a timer. That is what the trade consists of, and you already know how to do all of it.
Bash Programming Course
Module 1: Introduction to Bash
- What Is Bash?
- Setting Up Your Environment
- Basic Command Line Navigation
- Understanding the Shell
- Finding Help: man, help and --help
Module 2: Basic Bash Commands
- File and Directory Operations
- Text Processing Commands
- File Permissions and Ownership
- Redirection and Piping
- Wildcards and Path Expansion
- History and Keyboard Shortcuts
Module 3: Scripting Fundamentals
- Creating and Running a Script
- Variables and Constants
- Basic Operators
- Conditional Statements
- Arguments and User Input
- Quoting, Expansion and Substitution
Module 4: Intermediate Scripting
- Loops in Bash
- Functions in Bash
- Arrays and Associative Arrays
- String Manipulation
- The case Statement and Interactive Menus
- Arithmetic and Numeric Calculations
Module 5: Advanced Scripting Techniques
- Advanced File Operations
- Process Management
- Error Handling and Debugging
- Regular Expressions
- Advanced I/O: Descriptors and Here-Documents
- Modular Scripts and Reusable Libraries
Module 6: Working with External Tools
Module 7: Automation and Scheduling
- Cron Jobs
- Automating Tasks
- Backup and Restore Scripts
- Monitoring and Logging
- Services and Timers with systemd
- Remote Automation with SSH
Module 8: Best Practices and Optimization
- Writing Readable Code
- Optimizing Bash Scripts
- Security Considerations
- Version Control with Git
- Static Analysis with ShellCheck and shfmt
- Automated Testing with Bats
- Portability: POSIX sh versus Bashisms
