The previous lesson ended with a diagnosis and an outstanding task: Tramontana's nightly backup runs at the wrong time, with the wrong priority and over more data than necessary. Somebody scheduled it at some point and nobody has looked at it since. It is the exact portrait of badly done automation: it works until the day it brings the service down.

cron is the Unix timer. It has been running since 1975 and it is still the most widespread way of telling a server "do this every day at four". Its syntax is short and its failures are always the same three, repeated in every company in the world. This lesson gives you the complete syntax and, above all, those three failures with their solutions, because they account for 90% of real cron incidents.

One point about scope: here the tasks will be one-liners or calls to existing commands. Writing the backup script with its checks and its error handling is Module 4.

1. What automates well and what automates badly
2. The user crontab and the system crontab
3. The five fields
4. The conflict between day of the month and day of the week
5. Classic failure 1: cron's PATH
6. Classic failure 2: the output nobody reads
7. Classic failure 3: the time zone
8. Debugging a cron task
9. Overlapping and flock
10. anacron and systemd timers
11. The Tramontana case: the nightly backup and the purge

Tramontana's nightly backup failed on the last two rows: it did not limit its I/O priority and nobody read its result. An automatic task that fails silently is worse than not having it at all, because it generates the confidence that the job is done. The first time you discover that the backups have been failing for three months is usually the day you need to restore one.

The cron daemon (the cron package on Ubuntu) reads several different places:

There is the culprit behind last night's incident.

Careful with crontab -r. It asks for no confirmation and there is no wastebasket: it deletes all your tasks at a stroke. And it is right next to -e on the keyboard. The professional habit is simple: keep your crontab in a version-controlled file and load it with crontab file. That way an unfortunate keystroke costs you a crontab ~/cron/operator.cron, not a reconstruction from memory.

The key difference between the two formats is that the lines in /etc/crontab and /etc/cron.d/ have an extra field with the user who runs the task, right before the command:

Forgetting that field is a frequent error: cron reads the user name as the command's first argument and the task fails in a baffling way.

The cron.daily directories and company are run by run-parts, which has a quirk: it ignores files whose name contains a dot. A script called backup.sh in /etc/cron.daily/ will never run. It has to be called backup, with no extension, and be executable.

That last row is genuine advice. At 00:00 exactly, everything anybody scheduled without thinking starts up; shifting by five minutes avoids competing with it.

Shortcuts that replace the five fields:

Here is a rule that contradicts intuition and makes many tasks run more often than intended.

If the "day of the month" and "day of the week" fields are both different from *, they are combined with OR, not with AND. The task runs if either of the two is satisfied.

The usual intention — "only on Friday the 13th" — requires checking inside the command, because cron cannot express it:

(Note the \%: in a crontab, the % symbol has a special meaning that we will see in section 6.)

When one of the two is *, there is no ambiguity and the other one governs. The table summarises the four cases:

It is, by a long way, number one. The symptom is always the same: the command works perfectly when you type it and does nothing from cron.

You have known the cause since 03-01: cron launches neither a login nor an interactive shell, so it does not read /etc/profile, nor ~/.profile, nor ~/.bashrc. Nothing from your environment exists there. The PATH cron provides by default is:

Two directories. /usr/local/bin is not there, nor is /usr/sbin, and of course neither is the /home/operator/scripts you added in 03-01. Your aliases do not exist either, nor your variables, nor the locale you have configured.

If rsync were installed in /usr/local/bin, a task invoking it by name would fail every time, silently and leaving no visible trace.

The three solutions, in order of preference:

1. Absolute paths always. Find them out with command -v and write them exactly.

   command -v rsync tar find gives you all of them at once: /usr/bin/rsync, /usr/bin/tar, /usr/bin/find.

2. Declare the PATH in the crontab header. Variable assignments go before the tasks and apply to all of them:

3. Make the script load the environment with . ~/.profile as its first line. It works, but it couples the task to one account's personal configuration, and that is fragile.

The rule we apply at Tramontana is number 1 reinforced with number 2: absolute paths in the commands and an explicit PATH in the header, for whatever the command may invoke internally.

Cron captures each task's stdout and stderr and sends them by local mail to the owning user. On a server with no MTA configured — like srv-tramontana — that mail goes nowhere: it is discarded or it sits in /var/mail/operator without anybody ever opening it.

The result: if you do not redirect the output, you find out nothing. Neither about the successes nor, above all, about the failures.

The 2>&1 goes at the end, for what you learned in 03-04: first stdout is redirected to the file and then that destination is copied onto stderr. The other way round, the errors — which are exactly what you care about — would end up in the phantom mail.

A dated record is achieved by prefixing a date:

MAILTO controls where the mail goes: [email protected] sends it to a real address (if there is an MTA), and MAILTO="" disables sending altogether. The professional combination is MAILTO="" plus redirection to a log file, with a monitoring system watching that file.

And the % quirk: in a crontab, % is interpreted as a newline that feeds the command's standard input. That is why date +%F inside a crontab has to be written date +\%F. It is a source of errors out of all proportion to how obscure the detail is.

cron uses the system's time zone, not the user's. Always check it:

In zones with summer time there are two problematic days a year. When the clock goes forward in March, 02:30 does not exist: a task scheduled for that time does not run that day. When it goes back in October, 02:30 happens twice, and depending on the implementation the task may run twice.

Three ways of protecting yourself:

• Schedule outside the 01:00–03:00 band. It is the simplest solution and the one that really solves the problem.
• Use UTC on the server (sudo timedatectl set-timezone UTC), a common practice on servers, at the cost of having to translate the schedules in your head.
• Declare CRON_TZ=UTC in the crontab header, which fixes the zone for those tasks only.

There is a coincidence here worth underlining: Tramontana's backup was at 03:30, inside the risky band, and it also clashed with the load peak. Moving it solves two problems at once.

Where it is recorded. The daemon notes each run in the syslog:

Three things: the backup started at 03:30:01, coinciding with the incident; the rsync ran too; and the message "No MTA installed, discarding output" is cron literally telling you it has thrown the output in the bin. There is failure 2, confessed in the log itself.

That record tells you whether the task ran, but not whether it worked: cron does not check the exit code. For that you need your own log file.

Dumping cron's environment. The definitive trick for failure 1: schedule a temporary task that saves its environment and compare it with yours.

Five variables. No LANG, and none of your extended PATH. Note as well that SHELL is /bin/sh, not bash: Bash-specific constructs may fail. If you need them, declare SHELL=/bin/bash in the header.

Testing the command with cron's environment, without waiting for it to fire:

Reproduced in two seconds. env -i starts the command with no inherited variables at all and adds only the ones you specify: it is a faithful simulation of cron. Any new task should be tested this way before being scheduled.

If one run lasts longer than the interval, cron launches the next one anyway. Two backups writing to the same file produce a corrupt archive; two simultaneous purges can cut the ground from under each other.

flock solves the problem with a lock file:

-n is almost always the right option for a periodic task: if the previous one is still running, this one is superfluous. Without -n, the runs queue up and you end up with twenty processes waiting.

The lock is released on its own when the process finishes, even if it dies abruptly, because the kernel manages it through the file descriptor. There are no orphaned lock files to clean up by hand, which is exactly the problem with implementing it with a touch and an rm.

cron assumes the machine is switched on at the scheduled time. If the server was off, the task is not recovered. anacron fills that gap: it works with a granularity of days and, on start-up, runs whatever was left pending. On Ubuntu, cron.daily, cron.weekly and cron.monthly are managed by anacron for precisely that reason.

When to choose each. cron for the simple and periodic on a server that is always on: it is universal, it fits on one line and any administrator understands it. anacron for laptops and machines that get switched off. systemd timers when you need dependencies between units, resource control, retries or the system's integrated logging; they are studied in 05-05. For Tramontana's backup, cron is sufficient and more readable.

We redo the guilty task applying everything above. The crontab is kept in a version-controlled file and loaded from there.

It is loaded with crontab ~/cron/operator.cron and checked with crontab -l. Go over each decision, because each one corrects a specific problem:

• 04:20 instead of 03:30: outside the error peak you identified in 03-05 and outside the clock-change band.
• flock -n: if a previous backup is still running, this one does not start.
• ionice -c 3: the backup only uses the disk when the service does not need it. It is the direct correction of the 03-06 incident.
• /opt/tramontana/app/ with a trailing slash instead of the whole of /opt/tramontana: app is the symbolic link to the active release, and the slash makes tar follow the link. One release is copied instead of four, and always the one that is serving.
• A dated name: data-2026-08-18.tar.gz, with an escaped \%F. A file per day, instead of overwriting the only copy you had.
• >> ... 2>&1 in both tasks: everything is recorded, with the 2>&1 at the end.
• Absolute paths in every command, including the date inside the substitution.
• The purge as a separate task and with find -mtime +30: you know from 03-03 that this means 31 days or more, ample margin for a monthly retention.

Verification before going home: the command is tested with cron's environment and the result is checked.

Exit code 0, 47 MB — one release, not four — and tar -tzvf confirms that the contents are what we expect before signing the task off. It is the course convention applied here: look inside the archive before trusting it.

The report for Marta. The nightly backup has been rescheduled to 04:20, with the lowest disk priority and limited to the release in service instead of all four. This protects against the backup leaving the application without disk again, against two backups overlapping and against losing the previous copy when the new one is generated. It does not protect against the backup failing: nobody is yet watching /var/log/tramontana/cron-backup.log, so an error would still go unnoticed, nor against the loss of the entire server, because the backups are still on the same disk. Both of those require decisions beyond a technical adjustment and are dealt with in the administration module.

• crontab -r instead of -e. It deletes everything without asking. Keep the crontab in a file and load it with crontab file.
• Forgetting the user field in /etc/cron.d/. Cron takes it as an argument to the command.
• Giving a /etc/cron.daily/ script an extension. run-parts ignores names with a dot.
• Assuming your PATH exists. Cron gives you /usr/bin:/bin and nothing else. Absolute paths.
• An unescaped %. Inside a crontab, % is a newline. Write \%.
• Not redirecting the output. Without >> log 2>&1 the failures disappear.
• Putting 2>&1 before the output redirection. It goes at the end, always.
• Scheduling between 02:00 and 03:00. That is the clock-change band.
• A * * * * * for testing and forgetting to remove it. It leaves a task running every minute forever.
• Tip: document every task with a comment saying what it does and why at that time. In a year's time you will be the one thanking yourself, not somebody else.
• Tip: make the task write a dated success marker. Checking "is the last line of the log from today?" is a trivial check and it detects 90% of the problems.

Exercise 1. Write the cron expressions for: every 20 minutes between 8 and 20 on working days; the last day of each month at 23:50; and at 06:15 on the 1st and the 15th. Justify the second case, which has a catch.

Exercise 2. Schedule a task that records every hour the number of 500 errors in errors.log in /var/log/tramontana/errors-500.log together with the date. Apply the three protections against the classic failures and prove that it would work in cron's environment before installing it.

Exercise 3. A scheduled task */5 * * * * /home/operator/scripts/synchronise >> /tmp/sync.log 2>&1 produces no lines at all in its log, even though the command works when you run it. Describe the complete diagnostic procedure, in order, saying what you would check at each step and what you would conclude from each result.

Solution 1.

The second one, "the last day of each month", cannot be expressed with the five fields: cron does not know how many days the month has, and 31 would skip February, April, June, September and November. The idiomatic solution is to schedule it every day and let the command decide:

You ask what day tomorrow will be: if it is the 1st, today is the last day of the month, whatever its length and even in a leap year. It is a good example of cron's boundary: when the condition does not fit into the five fields, run it daily and check inside. The \% is escaped, as always.

Solution 2. First the command is tested in cron's environment:

It works with only /usr/bin:/bin in the PATH, so we do not depend on anything cron does not have. Now the crontab line:

The three protections: absolute paths for date and grep, so that the minimal PATH does not matter; >> ... 2>&1 so that both the result and any error end up in the file and not in mail nobody reads; and CRON_TZ declared in the crontab header — along with choosing minute 0 of each hour, far from the clock-change band — so that the timestamps are consistent. flock is added because it is a cheap habit, even though here the risk of overlapping is minimal. And the % signs are escaped: without the backslashes, cron would cut the command off at the first % and pass the rest as standard input, which is a particularly hard failure to recognise.

A line that starts accumulating escapes like this one is the sign that the content is asking for a script of its own. That is what Module 4 will resolve.

Solution 3. The procedure, in order, from the most external to the most internal:

1. Does the task exist? crontab -l | grep synchronise. If it does not appear, another crontab was edited — root's, for example — or a crontab -r took it away.
2. Is it running? grep CRON /var/log/syslog | grep synchronise | tail -3. If there are no entries, cron is not launching it: check the syntax of the fields and that the daemon is active. If there are, cron is launching it and the problem is further in.
3. Does the log file exist and with what permissions? ls -l /tmp/sync.log. An empty but existing log confirms that the redirection works and that the command writes nothing; the file not existing points to the command not even having started.
4. Is it executable and with the right path? ls -l /home/operator/scripts/synchronise. Without the x bit, cron gets "permission denied". And here is the prime suspect: the task uses an absolute path to the script, but the script internally probably invokes commands by name, and those do depend on the PATH.
5. Reproduce with cron's environment: env -i PATH=/usr/bin:/bin HOME=/home/operator /bin/sh /home/operator/scripts/synchronise. This step almost always reproduces the failure instantly.
6. Verify writing to the destination with sudo -u operator touch in the destination directory, in case the task runs as a different user from the one you think.

The most likely conclusion is the one from step 5. And there is a clue pointing to it from the start: if the log is empty instead of containing an error message, the command produced no output at all, which fits with a shell that could not even start the binary. An empty log is information, not an absence of information.

You have turned a dangerous automatic task into a defensible one.

• You know what automates well: the idempotent, the bounded, with controlled resources and visible failure. And you distinguish the user crontab from the system crontab, with its extra user field, and you know run-parts and its fussiness about dots in names.
• You have mastered the five fields with lists, ranges, steps and shortcuts, and you know that day of the month and day of the week are combined with OR, not with AND.
• You have the three classic failures sorted: cron's minimal PATH — absolute paths always — the output nobody reads — >> log 2>&1 with the 2>&1 at the end — and the time zone with its risky band between 02:00 and 03:00.
• You debug a task methodically: grep CRON /var/log/syslog to find out whether it ran, your own log to find out whether it worked, dumping the environment with env and reproducing it exactly with env -i.
• You avoid overlapping with flock -n, knowing that the kernel releases the lock and leaves no remains.
• You know anacron for machines that get switched off and you know when a systemd timer justifies its complexity.
• And you have rescheduled Tramontana's backup with a new time, a lock, an I/O priority, a scope reduced to the active release, a dated name and logging, with the purge as a separate task and an honest report about what the measure does not cover.

One front remains untouched. Everything you have done so far happens inside srv-tramontana, and a server only exists for whoever can reach it. The module's last lesson, Networking Commands, gives you the tools to look outwards: ip to read the machine's real configuration, ping, traceroute and mtr for the path, dig and /etc/hosts for name resolution, ss to find out which ports are listening and who is holding them, and curl to check whether the service really responds. All of it organised into a layered diagnostic methodology that you will apply to the case waiting for you: Marta reports that the bookings website "will not load", and you have to work out exactly where the break is.