We have nine working scripts, each with its options, its configuration and its own way of being invoked. That is not a product: it is a folder. A colleague joining tomorrow does not know which one to run, the help is scattered across nine different --help outputs, installation is "copy this and remember the permissions", and nobody knows which version is running on srv-veloz-02. This last project turns the collection into veloz-ops: a single command with subcommands, installable, configurable, versioned, tested, deployed across the fleet and documented. It is the close of the course, and the step that separates "I can program in Bash" from "I have shipped something".

Contents

  1. The toolkit seen whole
  2. The single-command-with-subcommands pattern
  3. Final reorganization of the repository
  4. Unified configuration and veloz-ops config
  5. Bash completion
  6. Idempotent installation
  7. Semantic versioning and veloz-ops version
  8. Quality before deploying
  9. Fleet deployment and rollback
  10. Documentation: README and runbook

  1. The toolkit seen whole

flowchart TD
  U["veloz-ops (single command)"]
  U --> S2[status] --> L2[libexec/system-info.sh]
  U --> S3[logs] --> L3[libexec/analyze-logs.sh]
  U --> S4[backup] --> L4[libexec/backup.sh]
  U --> S5[network] --> L5[libexec/network-monitor.sh]
  U --> S6[fleet] --> L6[libexec/fleet.sh]
  L2 & L3 & L4 & L5 & L6 --> LIB["lib/common.sh<br/>veloz_log · veloz_require · veloz_api_get"]
  LIB --> CFG["etc/veloz-ops.conf + etc/*.d/"]
  L3 --> D1[("/var/log/veloz/*.log")]
  L4 --> D2[("/srv/veloz/data + /backups")]
  L5 --> D3(["veloz-api :8080 · SSH fleet"])
  TMR["systemd timers"] --> U

The diagram shows the property we are after: one entry point, one shared library, one configuration. The scripts move into libexec/ because they stop being a public interface; nobody invokes them directly, veloz-ops invokes them.

  1. The single-command-with-subcommands pattern

It is the pattern of git, systemctl or docker, and in Bash it is implemented with case and functions (04-05):

#!/usr/bin/env bash
# veloz-ops - single entry point of the toolkit.
set -euo pipefail
export LC_ALL=C
readonly ROOT=${VELOZ_ROOT:-/opt/veloz-ops}
readonly LIBEXEC=$ROOT/libexec
# shellcheck source=lib/common.sh
. "$ROOT/lib/common.sh"

usage() {
  cat <<'EOF'
Usage: veloz-ops <subcommand> [options]
  status    System information (09-01)
  logs      Log analysis (09-02)
  backup    Backups, verification and restore (09-03)
  network   Network and service monitoring (09-04)
  fleet     Run a command on the three servers
  config    Effective configuration and its origin; version; help <sub>
EOF
}

main() {
  local sub=${1:-help}; shift || true
  case $sub in
    status)  exec "$LIBEXEC/system-info.sh"     "$@" ;;
    logs)    exec "$LIBEXEC/analyze-logs.sh"    "$@" ;;
    backup)  exec "$LIBEXEC/backup.sh"          "$@" ;;
    network) exec "$LIBEXEC/network-monitor.sh" "$@" ;;
    fleet)   exec "$LIBEXEC/fleet.sh"           "$@" ;;
    config)  show_config "$@" ;;
    version) printf 'veloz-ops %s\n' "$VERSION" ;;
    help|-h|--help) [[ $# -gt 0 ]] && usage_for "$1" || usage ;;
    *) printf 'Unknown subcommand: %s\n\n' "$sub" >&2; usage >&2; exit 2 ;;
  esac
}
main "$@"

Three decisions. exec replaces the process instead of creating a child: it saves a process and, more importantly, the exit code and the signals reach the real script with no intermediary, so Ctrl-C on veloz-ops backup really interrupts the backup. The case is an explicit allowlist (08-03): building the path as "$LIBEXEC/$sub.sh" would be shorter and would let anyone run veloz-ops ../../bin/whatever. And an unknown subcommand exits with code 2 and writes to stderr, following the 05-03 convention.

  1. Final reorganization of the repository

veloz-ops/
├── bin/veloz-ops              # only executable in the PATH
├── libexec/                   # internal scripts, not in the PATH
│   ├── system-info.sh  analyze-logs.sh  backup.sh  network-monitor.sh  fleet.sh
│   └── access.awk             # separate awk programs (09-02)
├── lib/common.sh              # veloz_* functions, loaded with source
├── etc/veloz-ops.conf         # 600; plus backup.d/ and network-monitor.d/
├── systemd/*.service *.timer   ├── tests/*.bats   ├── docs/runbook.md
└── completion/  install.sh  verify.sh  README.md  CHANGELOG.md

The criterion is worth stating out loud: bin/ is what the user invokes, libexec/ what the program invokes, lib/ what is loaded with source and never executed, etc/ what is edited without deploying. That last phrase settles the doubts: if changing it requires code review, it is code; if it is a path, a threshold or a list of targets, it is configuration.

  1. Unified configuration and veloz-ops config

The 05-06 precedence —default < file < environment < options— is implemented at a single point in lib/common.sh, and it is made auditable by storing the origin of every value:

declare -A CFG ORIGIN
veloz_config_load() {
  local f=$1 k v env_var
  while IFS='=' read -r k v; do            # file: overrides the defaults
    [[ $k == \#* || -z $k ]] && continue
    k=${k// /}; v=${v%\"}; v=${v#\"}; CFG[$k]=$v; ORIGIN[$k]="file:$f"
  done < "$f"
  for k in "${!CFG[@]}"; do                # environment: overrides the file
    env_var=VELOZ_$k
    [[ -n ${!env_var:-} ]] && { CFG[$k]=${!env_var}; ORIGIN[$k]=environment; }
  done
}

${!env_var} is indirect expansion (03-06): it builds the name VELOZ_DISK_THRESHOLD and reads that variable. Storing the origin alongside the value costs one more associative array and answers the question that eats hours in production: "the threshold is at 95, but the file says 85, where is that coming from?". veloz-ops config prints it in three columns —key, value, origin— and that is why it is a first-class subcommand and not a debugging echo.

  1. Bash completion

# completion/veloz-ops.bash  ->  /etc/bash_completion.d/veloz-ops
_veloz_ops() {
  local cur=${COMP_WORDS[COMP_CWORD]} options='status logs backup network fleet config version help'
  (( COMP_CWORD > 1 )) && case ${COMP_WORDS[1]} in
    backup)  options='run verify restore report --dry-run' ;;
    status)  options='--format --section --help' ;;
    network) options='check summary live --target' ;;
  esac
  mapfile -t COMPREPLY < <(compgen -W "$options" -- "$cur")
}
complete -F _veloz_ops veloz-ops

COMP_WORDS and COMP_CWORD are the variables Bash fills in when you press Tab, and compgen -W filters a list by the typed prefix. It is not cosmetic: completion is the documentation people actually read, and it prevents half the typing mistakes in a delicate subcommand like restore.

  1. Idempotent installation

install.sh must be runnable a hundred times with the same result (07-02):

install_toolkit() {
  install -d -m 755 "$PREFIX"/{bin,libexec,lib,etc,docs}
  install -m 755 bin/veloz-ops "$PREFIX/bin/veloz-ops"
  install -m 755 libexec/*.sh  "$PREFIX/libexec/"
  install -m 644 libexec/*.awk lib/common.sh -t "$PREFIX/lib/"
  [[ -e $PREFIX/etc/veloz-ops.conf ]] ||
    install -m 600 etc/veloz-ops.conf.example "$PREFIX/etc/veloz-ops.conf"
  install -m 644 completion/veloz-ops.bash /etc/bash_completion.d/veloz-ops
  ln -sfn "$PREFIX/bin/veloz-ops" /usr/local/bin/veloz-ops
  install -m 644 systemd/*.service systemd/*.timer /etc/systemd/system/
  systemctl daemon-reload
  systemctl enable --now veloz-backup.timer veloz-network.timer
  "$PREFIX/bin/veloz-ops" version
}

install instead of cp + chmod because it creates the destination and sets the permissions in a single operation, and it is idempotent by nature; ln -sfn overwrites the link without complaining; systemctl enable --now enables and starts, and repeating it does no harm. The key line is the conditional: existing configuration is never overwritten. An installer that overwrites veloz-ops.conf wipes out the thresholds somebody tuned at three in the morning; that is why the repository versions veloz-ops.conf.example and not the real file (08-04). Uninstalling is symmetric: --uninstall stops and disables the timers, deletes the units, runs daemon-reload, removes the link and the tree, and keeps etc/ and the backups, saying where they are left.

  1. Semantic versioning and veloz-ops version

Change Increments Example
A subcommand or option is removed or renamed; the format of a state file changes MAJOR --section becomes --area
New subcommand or option, backward compatible MINOR veloz-ops network summary is added
A fix with no interface change PATCH The median calculation is fixed

The version comes from Git, not from a constant somebody will forget to update (08-04):

VERSION=$(git -C "$ROOT" describe --tags --dirty 2>/dev/null || cat "$ROOT/VERSION" 2>/dev/null || echo unknown)

git describe --tags --dirty gives v1.4.0 on an exact tag, v1.4.0-7-gab12cd3 if there are seven commits on top —which instantly reveals that there is something unreleased on that server— and adds -dirty if there are uncommitted changes, which in production is an alert in itself. install.sh also writes a VERSION file for deployment via rsync without a repository.

  1. Quality before deploying

A single script gathers all the checks from module 8:

#!/usr/bin/env bash
# verify.sh - everything that must pass before deploying.
set -uo pipefail
failures=0
mapfile -t SCRIPTS < <(find bin libexec lib -type f)
step() { printf '\n== %s ==\n' "$1"; shift; "$@" || ((failures++)); }

step syntax     bash -n "${SCRIPTS[@]}"
step shellcheck shellcheck -x -S style "${SCRIPTS[@]}"
step format     shfmt -d -i 2 -ci "${SCRIPTS[@]}"
step tests      bats tests/
step help       bin/veloz-ops help
exit $(( failures > 0 ))

The order goes from fast to slow: bash -n takes milliseconds and rules out a syntax error before spending thirty seconds on Bats. shellcheck -x follows the source calls so it also analyzes lib/common.sh (08-05), and -S style raises the bar above errors. shfmt -d does not reformat, it only shows the difference and fails: in CI you must fail, not fix things by surprise. The Bats battery is extended to the subcommands (08-06):

@test "unknown subcommand exits with 2 and writes to stderr" {
  run bin/veloz-ops doesnotexist
  [ "$status" -eq 2 ]
  [[ $output == *"Unknown subcommand"* ]]
}

@test "backup --dry-run creates no directory" {
  local before; before=$(find "$BATS_TMPDIR/backups" -type d | wc -l)
  run bin/veloz-ops backup run --dry-run
  [ "$status" -eq 0 ]
  [ "$(find "$BATS_TMPDIR/backups" -type d | wc -l)" -eq "$before" ]
}

The second test is the most valuable in the toolkit: it verifies that dry-run mode has no effects, which is the promise anyone leans on before launching a backup with --delete. The pre-commit hook (08-04) runs the first three stages and CI runs the full battery on every branch.

  1. Fleet deployment and rollback

deploy() {  # $1 = tag, e.g. v1.4.0
  local tag=$1 host
  git tag -l "$tag" | grep -q . || veloz_die 2 "nonexistent tag: $tag"
  ./verify.sh || veloz_die 3 "code that does not verify is not deployed"
  for host in srv-veloz-01 srv-veloz-02 srv-veloz-03; do
    printf '\n--- %s ---\n' "$host"
    ssh -n "$host" "cd /opt/veloz-ops && git fetch --tags -q &&
                    git checkout -q $tag && sudo ./install.sh" || return 1
    ssh -n "$host" 'veloz-ops version && veloz-ops status --section services' || return 1
  done
}

Deployment is sequential and with a post-check on each host: if srv-veloz-01 fails, the second one is never touched, and in the worst case one server is left broken and two intact. ssh -n (07-06) stops the loop from eating standard input, the classic error that makes only the first host get deployed. A tag is deployed, never a branch: git checkout v1.4.0 gives exactly the code that was tested, and the rollback is the same command with the previous tag (deploy v1.3.2). That reverting is identical to deploying is what makes somebody dare to run it at four in the morning. With one caveat for the runbook: reverting the code is trivial, reverting a format change to the state files or the history is not, and that is why that kind of change bumps the MAJOR version.

  1. Documentation: README and runbook

They are two documents with two different readers and they are part of the deliverable. The README.md is read by whoever arrives new, cold: what it is, what it requires, how it is installed in five copy-pasteable commands, the subcommands with one real example each and where the configuration lives. If somebody cannot install it and run veloz-ops status following only the README, the README is wrong. The docs/runbook.md is read by somebody half asleep with an alert on their phone: it is organized by alert, not by script, with no prose and with copy-pasteable commands.

## ALERT: backup with failures on srv-veloz-01
1. Confirm:      veloz-ops backup report
2. Check the log: journalctl -u veloz-backup.service -n 50
3. If "out of space" (code 5): df -h /backups; review the profile's KEEP_*
4. If "verification failed" (code 4): do NOT delete anything; restore from srv-veloz-03
5. Escalate to Operations if the data profile fails two nights in a row.

A script with no documentation is a script only its author can run; and its author will end up on holiday exactly the night it fails.

Common Mistakes and Tips

  • Building the subcommand path from the user's argument. "$LIBEXEC/$1.sh" opens the door to ../. An allowlist with case, always.
  • Installers that overwrite the configuration. Version .conf.example; the real .conf is only created if it does not exist.
  • Deploying a branch instead of a tag. main means something different every hour; a tag always means the same thing.
  • A for host loop with ssh and no -n. The first ssh consumes the list and the rest of the fleet is left undeployed, silently.
  • Documenting by script instead of by alert. At four in the morning nobody searches for "analyze-logs"; they search for the text that arrived on their phone.
  • Tip: run veloz-ops config on the three servers and compare. Unexplained differences between nodes are the origin of half the "it works on this server" incidents.

Exercises

  1. doctor subcommand. It checks the health of the installation itself: dependencies (jq, rsync, curl, awk), etc/ permissions (600), active timers, the age of the last backup and of the last monitor cycle. One symbol per check and a global 0/1/2 code.
  2. Fleet status dashboard. A veloz-ops fleet status that collects the status JSON from the three servers in parallel and flags the differences in version, kernel and effective configuration.
  3. Open extension challenges. Package the toolkit as a .deb with postinst/prerm and the configuration marked as conffiles; add a veloz-ops api that exposes the reports over HTTP, deciding what not to expose; unify a --json across every subcommand with a Bats test validating each output; and port the toolkit to Debian or Alpine, noting every bashism you have to resolve (08-07).

Solutions

1. doctor checks nothing on its own: it reuses what is already built.

doctor() {
  local failures=0
  chk() { local n=$1; shift
          if "$@" >/dev/null 2>&1; then printf '  [OK]   %s\n' "$n"
          else printf '  [FAIL] %s\n' "$n"; ((failures++)); fi; }
  chk "dependencies"    veloz_require jq rsync curl awk
  chk "etc permissions" test "$(stat -c %a "$ROOT/etc/veloz-ops.conf")" = 600
  chk "backup timer"    systemctl is-active --quiet veloz-backup.timer
  chk "recent backup"   test "$(days_since_last_backup)" -le 1
  (( failures == 0 )) && return 0 || return 2
}

The nested chk function, which takes a command and runs it, avoids repeating the same if ten times; and returning 0/2 lets doctor itself be watched from watchdog.sh with the 07-04 convention.

2. fleet.sh already runs in parallel, so consolidating the JSON with jq -s is enough:

veloz-ops fleet 'veloz-ops status --format json' | jq -s '
  map({host, version, kernel})
  | {nodes: ., distinct_kernels: (map(.kernel) | unique | length)}'

Detecting the divergence —not listing the values— is what makes the dashboard a useful tool: if distinct_kernels is greater than 1, there is a server that has not been updated.

Conclusion

We started the course typing echo "Hello" in a terminal and we finish it with a product installed on three servers: a single command with subcommands and help, configuration with precedence and auditable origins, completion, an idempotent installer and a symmetric uninstaller, semantic versioning tied to Git tags, a test battery that runs on every commit, deployment by tag with rollback and documentation written for whoever will read it in the small hours. None of those pieces is advanced Bash: they are case, functions, arrays, printf, source and exit codes —the material of module 4— applied with the judgment of module 8. That is the message of the whole course: the difference between a script and a tool is not in the language, it is in the discipline.

It is worth ending with some honesty about the limits. Bash is unbeatable at gluing programs together, moving files and automating the system; it is exactly the right tool for the five projects in this module. It stops being so when nested data structures appear (more than two levels of JSON and jq is no longer enough), when you need unit tests of business logic, real concurrency, sustained floating-point arithmetic, or when the script goes past a thousand lines and every change is frightening. If you find yourself writing an expression parser or a stateful HTTP client in Bash, the signal is clear: it is time for Python, Go or whatever language your team uses. Knowing when to stop is part of mastering the tool, and rewriting a well-structured Bash script in another language is straightforward precisely because it was well structured.

And where to go next. The Bash manual (man bash, and above all its expansions section, which you read end to end once in your life) is the only definitive source. Google's shell style guide, so you can argue decisions with reasons instead of tastes. ShellCheck as a permanent teacher: every warning you do not understand, look it up by its code and you will learn something you did not know. And above all, the thing that really consolidates it: automate your own work. Take that task you do by hand every Monday, write it as a script with set -euo pipefail, give it options, add a --dry-run, test it with Bats, push it to Git and fire it with a timer. That is what the trade consists of, and you already know how to do all of it.

Bash Programming Course

Module 1: Introduction to Bash

Module 2: Basic Bash Commands

Module 3: Scripting Fundamentals

Module 4: Intermediate Scripting

Module 5: Advanced Scripting Techniques

Module 6: Working with External Tools

Module 7: Automation and Scheduling

Module 8: Best Practices and Optimization

Module 9: Real-World Projects

© Copyright 2026. All rights reserved