You already have three jobs running on their own: the report at 06:30, the status check every 10 minutes and the backup at 03:15. Each one writes to a file that grows forever, and none of them answers the question that really matters: who finds out if one of them stops working? Logging and monitoring are two sides of the same problem. Logging is the system telling you what it does, so you can reconstruct afterwards what happened; monitoring is it warning you while it happens. One is for investigating, the other for reacting, and neither works when improvised: a log with no format cannot be queried, and an alert that fires every five minutes stops being read within a week.

1. Why echo is not enough and what a log line looks like
2. The veloz_log function with levels and a threshold
3. logger, syslog and journalctl
4. Rotation with logrotate
5. copytruncate versus create
6. Checking versus watching a trend: the 0/1/2 pattern
7. Thresholds, noise, hysteresis and state
8. Typical Veloz Envíos checks
9. Notification channels, and a loop versus periodic execution
10. Application: watchdog.sh is born

An echo "processing" answers the person watching the screen right now. Inside a three-month file, that line does not say when it happened, who wrote it or whether it mattered — and those three answers are all you need when Tuesday's report fails and you investigate it on Thursday. Besides, echo writes to standard output, and that mixes two things that must stay separate (02-04): the results, which another program might consume through a pipe, and the diagnostics. The rule as always: results to stdout, log to stderr.

A log format has four fields, each with its reason to exist: the timestamp (2026-08-03T06:30:01+02:00) to correlate with other systems, the level (INFO, WARN, ERROR) to filter by severity, the component (report, backup, watchdog) to know who is speaking in a shared log, and the message with what happened. The timestamp goes first and in ISO-8601 for two practical reasons: it sorts alphabetically the same as chronologically — so sort works directly on the log — and it is unambiguous across time zones. date -Is generates it. About the message, one recommendation you will appreciate as the log grows: use key=value pairs instead of prose, because processed=1284 city=Madrid is filtered with grep and aggregated with awk (06-01), and "1284 shipments have been processed, of which…" is not. And never put personal data or credentials in the log: logs are copied, shared and kept for months (08-03).

The four classic levels are DEBUG (internal detail, invisible in production), INFO (normal milestones: start, end, results), WARN (something odd but recoverable, worth reviewing) and ERROR (a failure that prevents the work, you have to act). The complete function for lib/common.sh:

It produces lines like 2026-08-03T06:30:02+02:00 [INFO] daily-report.sh: processed=1284 issues=37, and four decisions deserve an explanation. The associative array (04-03) turns labels into numbers so they can be compared; without it, "is WARN more serious than INFO?" has no answer in Bash. The : "${VAR:=value}" assigns a default only if the variable was not already defined, so the .conf or the environment can change the threshold without touching code (05-06). It writes always to stderr and additionally to the file if one is configured, so you see the messages when running by hand and have them recorded under cron. And the final return 0 prevents a filtered log call from returning a non-zero code and aborting the script under set -e (05-03). With the default threshold the DEBUGs do not appear, so you can leave them in place and switch them on with VELOZ_LOG_LEVEL=DEBUG only when investigating.

Every Linux already has a centralized logging service — syslog on the classic ones, journald on the ones using systemd — and logger is the command that writes to it: logger -t veloz-backup -p user.err "the 2026-08-03 backup failed with code 74". -t sets the tag, so you can filter later by service, and -p the priority in facility.level format (user.info, user.warning, user.err). It also accepts standard input: command 2>&1 | logger -t veloz-report.

Your own file or syslog? The file gives you total control of the format and unlimited volume, but you have to build the rotation yourself and querying it means grep. The journal already solves rotation, offers journalctl's filters and is forwarded to a central server in a standard way, in exchange for a header and for the possibility of dropping messages under load. It is not either/or, and the practice the toolkit follows is: the detail to your own file, and the important events — a job failing, an alert raised or resolved — also to logger, so they show up where the team is already looking.

They combine: journalctl -t veloz-watchdog -p warning --since "-24h" gives the watchdog's warnings and errors over the last day. For your own file, what you learned in Module 2 is enough: tail -f, grep -F '[ERROR]' and, thanks to the level being in a fixed field, aggregations like awk '$2 == "[ERROR]" { print $3 }' watchdog.log | sort | uniq -c | sort -rn. With a free-prose log, that would not be possible.

Logs grow forever, and watchdog.sh will write every 5 minutes. A full disk brings down the whole server — including the jobs that were going to warn you — so rotation is not an extra: it is part of setting up an automated job. logrotate runs daily from cron.daily and reads the configurations in /etc/logrotate.d/:

Directive by directive: daily rotates once a day, at the pace of the jobs (there are also weekly and size 100M); rotate 14 keeps two weeks of history and deletes the rest; compress reduces a text log to about 10% of its size; delaycompress compresses from the second cycle onwards, leaving the .1 uncompressed in case it is still being written to; missingok does not fail if the file does not exist yet; notifempty avoids accumulating fourteen empty rotated files; create 0640 veloz veloz creates the new one with the right owner, because the script does not run as root; and su veloz veloz rotates with that identity, which is needed in user directories. That produces exactly the access.log.1 and app.log.2.gz you have been seeing since Module 2: now you know where they come from. And test it before trusting it: logrotate -d /etc/logrotate.d/veloz-ops simulates without touching anything — it is the --dry-run from 07-02 applied here — and logrotate -f forces a cycle now so you can check permissions and ownership without waiting for tomorrow.

This section explains a failure that puzzles a lot of people: you rotate the log and the process keeps writing to the old file. The cause lies in how the system works: a process that opened a file writes to the inode, not to the name. When logrotate renames watchdog.log to .1 and creates a new one, the process that had it open still points at the old inode, now called .1, and the new file stays empty forever. There are three strategies. create renames the old one and creates a new one, with the risk you have just seen. create plus postrotate … reload does the same but tells the process to reopen, and it is the best one if the process knows how. And copytruncate copies the content and then empties the original, with the risk of losing the lines written between copying and emptying. The toolkit's scripts are not affected: each cron run opens the log with >>, writes and finishes, and the next >> opens the new file by name. That is why create is the right choice here. It does affect a long-running process that keeps the file open, and there you need postrotate systemctl reload veloz-api; endscript or else copytruncate, which is the universal solution when the process cannot reopen, at the price of that small window of loss. Rule of thumb: create for processes that open and close; copytruncate or a reload for those that keep the file open.

Before writing the watchdog, a distinction that keeps you from building the wrong tool. A check answers "is it fine right now?" with yes/no/unclear and is there to alert; a metric answers "how is it evolving?" with a number over time and is there to diagnose and forecast. Bash is excellent for checks and for collecting metrics, but it is not the place to store time series or draw graphs — that is Prometheus, Grafana and friends. The classic monitoring systems (Nagios and its descendants) established a very useful convention that you can adopt even if you use none of them: 0 = OK, 1 = WARNING (close to the threshold, take a look), 2 = CRITICAL (broken, act now) and 3 = UNKNOWN (it could not be checked). Each check is then an independent, homogeneous function: it takes its thresholds, prints one line of summary and returns one of those four codes.

The gsub(/%/, "", $5) strips the percent sign so it can be compared as a number (06-01), and the || return 3 distinguishes "I could not measure it" from "it is bad": they are very different things, and confusing them produces false alerts every time df is slow or the path does not exist. In this shape, adding a new check means writing one more function and adding it to a list.

This is where it is decided whether your monitoring gets used or ignored. A check that runs every 5 minutes and alerts every time generates 288 messages a day for a single problem; the second time that happens, the team creates a mail filter and stops reading the alerts, including the good ones. Two mechanisms prevent it.

Hysteresis: different thresholds for raising and for clearing. If you alert at 85% disk and consider it resolved below 85%, a disk oscillating between 84.9% and 85.1% will produce an alert every five minutes; with a raise at 85% and a clear at 80%, a real improvement is needed for it to go quiet. State: remembering in a file which alerts have already been sent, and alerting only on transitions.

Notice that it also notifies on resolution: an alert that never closes leaves the team not knowing whether the problem is still alive, and it is as useless as not alerting at all. The marker also stores the start time, which lets you answer later "how long did the incident last?".

With the pattern from section 6, the complete battery fits in a few lines because all the machinery is already built in earlier modules:

One more is missing, check_log_errors, which counts the recent [ERROR]s in /var/log/veloz/app.log by comparing the timestamp with $(date -d "-15 minutes" '+%Y-%m-%d %H:%M:%S') inside awk (04-06): 1 or more is WARNING, 10 or more is CRITICAL. And check_backup is especially valuable and usually missing: it detects the job that did not run, a failure no log reveals because there is nothing to write when nothing happens. It is the practical application of the JSON status file from 07-02.

Three ways to get the alert to a person, and the good habit of using several:

The webhook applies what you learned in 06-05: the JSON body is built with jq -n --arg instead of interpolating the text into a template — a message with quotes or a newline would break the JSON — and -d @- hands it to curl on standard input. The -m 10 is mandatory: a chat service that does not answer must not be able to hang the watchdog (07-02). And notice that a failure to deliver the alert does not abort the script, because a downed recipient must not prevent the remaining checks from running. The alert destinations live in etc/veloz-ops.conf with mode 600, never in the code. One last design decision: the temptation to write while true; do check; sleep 300; done is almost always a worse idea than scheduling the run every 5 minutes. If the process dies, the loop stops watching and nobody notices, whereas the next periodic run arrives all the same; a memory leak accumulates over weeks versus a short process that starts clean; when the server reboots you have to relaunch it by hand; and when the code changes you have to restart it. The loop is only justified with sub-minute granularity or with in-memory state that is expensive to rebuild — and in that case turn it into a systemd service with Restart=on-failure, which is the subject of 07-05. For everything else: periodic execution, state in a file, short process.

The design decisions, one by one. The checks live in an array of name:command strings so you can add one without touching main, and the name is split off with the expansions from 04-04. Each one is wrapped in timeout 30s, because a single hung check must not prevent the others from running. The … && code=0 || code=$? idiom captures the code without set -e aborting the loop. Code 3 (UNKNOWN) does not worsen the overall result, so that a measurement failure does not trigger a critical alarm. And notify_transition guarantees that each incident is alerted once, and its resolution too. In the crontab it goes every 5 minutes with flock -n and redirection to the log. This is the working version; project 09-04 goes deeper into the network side, with watching several targets, latency measurement and a historical report.

• echo with no timestamp or level, or logging to stdout. The first answers no useful question a month later; the second pollutes the output another process might consume.
• Not rotating. A disk filled by logs brings down the server and, along with it, the job that was going to warn you.
• copytruncate by default. Unnecessary for scripts that open and close, and it loses lines. Use create except with long-running processes.
• Alerting on every cycle, or not alerting on resolution. 288 messages a day for a single problem make the team filter out your alerts; and an alert that never closes leaves everybody unsure whether it is still live.
• Confusing "I could not measure" with "it is bad". Return 3 (UNKNOWN) and do not count it as critical.
• A single threshold for raising and clearing, or checks without timeout. The first produces flapping (use hysteresis); the second lets a hung check block the others.
• Personal data or credentials in the log. They get copied, shared and kept for months (08-03).
• Tip: for every alert you are about to create, ask yourself "if it goes off at 3 in the morning, is there anything to do?". If the answer is no, it is not an alert: it is a metric, and its place is a dashboard, not your phone.

Exercise 1. Write the logrotate configuration for /var/log/veloz/app.log, which the veloz-api process keeps open: daily rotation, 30 copies, compressed except the most recent one, without failing if the file does not exist, and with the right strategy for a process that cannot reopen. Justify the choice.

Exercise 2. Write check_memory following the 0/1/2/3 pattern: warning (1) if available memory drops below 20% of the total and critical (2) below 10%, printing a summary line.

Solution 1.

The key directive is the last one. copytruncate is the right choice because veloz-api keeps the file open and cannot reopen it: with create, the process would keep writing to the inode renamed to .1 and the new file would stay empty forever. The price is a minimal window between copying and truncating in which lines can be lost, and it is accepted because the alternative is losing them all. If veloz-api supported a reload, the superior option would be create plus postrotate systemctl reload veloz-api; endscript. delaycompress leaves the .1 uncompressed, which avoids compressing a file that could still receive late writes.

Solution 2.

It reads MemAvailable and not MemFree (06-03): MemFree excludes cache memory, which the kernel gives up instantly if needed, so a healthy server can have almost zero "free" and be perfectly fine; MemAvailable is the real estimate of what a new process could use. Both values are extracted in a single pass of awk, and (( total > 0 )) guards against a division by zero, returning 3 (UNKNOWN) instead of blowing up.

On logging: echo is not enough because it does not say when, who or how serious; design a line with a date -Is timestamp, level, component and a message in key=value pairs, and implement it in a veloz_log with DEBUG/INFO/WARN/ERROR levels, a configurable threshold and output to stderr and optionally to a file. Add logger -t -p so that important events reach syslog or the journal, where journalctl -u, --since, -p err and -f retrieve them with filters a flat file does not offer. And rotate, with a file in /etc/logrotate.d/ (daily, rotate, compress, delaycompress, missingok, notifempty), testing it with logrotate -d: create for processes that open and close like your scripts, copytruncate or a reload for those that keep the file open, because a process writes to the inode and not to the name.

On monitoring: distinguish checking from measuring a trend; write each check as a homogeneous function that prints one line and returns 0/1/2/3 in the style of the classic plugins, with 3 reserved for "I could not measure it"; define thresholds with hysteresis and keep state in files so you alert only on transitions — and alert when it resolves too; deliver through several channels (logger, mail, webhook with jq -n and curl -m) without a downed channel aborting the cycle; and prefer periodic execution to an infinite loop. watchdog.sh already covers disk, load, process, port, API, issues, recent errors and — what almost nobody watches — that last night's backup exists.

The toolkit is complete and scheduled, but cron is starting to fall short: it cannot wait for the network to be ready, it does not recover the run it missed while the server was powered off, it does not isolate or limit a job's resources, and its logging is scattered across four files. In 07-05 we meet systemd: .service and .timer units, OnCalendar validated with systemd-analyze calendar, Persistent=true for missed runs, centralized logging in the journal and an honest table of when migrating is worth it and when cron is still the right answer.