Module 6 ended with an uncomfortable promise: knowing what an operating system is counts for nothing at three in the morning if you don't know which command to type first. And all of those commands are typed in the same place: the shell. Before you learn to troubleshoot you need a solid grasp of the interface the troubleshooting flows through, because most of a junior administrator's mistakes are not analysis mistakes but shell mistakes: a filename with spaces that got split into two arguments, a pipe that reported success even though the first command failed, a badly set PATH that ran the wrong binary.

This lesson takes the shell apart from the inside. You will see that it is not part of the operating system: it is an ordinary user program that does exactly what you learned in 02-01 —fork, execve, wait— and whose only magic is turning text into system calls. You will understand why cd cannot be a program, why 2>&1 > file does not do what it looks like, and why set -euo pipefail should head every script you write. We will finish with a pipeline that pulls the busiest stations out of meteo-api.log and with a real script that checks Meteora's data.

Services and boot are the next lesson, and performance tooling the one after that; here we stay at the interface.

1. What a shell really is
2. Available shells and which one to use
3. Interactive, non-interactive, login: which file gets read
4. Anatomy of a command, and builtins versus external commands
5. The expansion order, step by step
6. Quoting and escaping: 80% of all failures
7. Redirection and file descriptors
8. Pipes, exit status and pipefail
9. Shell variables, environment variables and PATH
10. Job control and signals
11. The essential text tools
12. A complete pipeline over meteo-api.log
13. Scripting: from a one-off command to a program
14. Full script: verifying Meteora's data

Plenty of people believe the shell "is Linux" or that it is part of the kernel. It isn't. /bin/bash is an ordinary ELF executable, like ls or like meteo-api, running in user mode (01-06) and able to do only what any other program can do: system calls. Its main loop, stripped of everything incidental, is this: print the prompt and read a line; parse the text (split it into words, expand it, detect redirections and pipes); if the command is a builtin, run it in the shell's own process; if it is external, fork() to create a child, apply the redirections in it, execve() the program and waitpid() in the parent; store the status in $? and start over.

You can watch it happen with the tool introduced in 01-06:

What this proves. The whole mechanism, with no mystery left: the shell starts (execve of bash), duplicates itself (clone, which is what implements fork on Linux), the child turns into ls (the second execve) and the parent waits (wait4). -f follows the children and -e trace= cuts out the noise. Everything you type in a terminal ends up as this sequence.

Three practical consequences follow from that nature. A shell cannot change anything in its parent process: if a script runs export PATH=..., that variable dies with the script, which is why configuration files are loaded with source (or .), which runs them in the current shell. Every external command costs a process: a loop that launches grep a million times performs a million fork+execve pairs, and at the tens of microseconds per pair we measured in 02-01 that is minutes of pure fork; hence the text tools are designed to process whole streams in one go. And the shell is replaceable: swapping bash for zsh does not change the operating system one bit.

On Debian, /bin/sh points to dash, not to bash. Real consequence: a script that starts with #!/bin/sh but uses [[ ]] or arrays will fail on Debian and will work wherever /bin/sh is bash. It is the classic "works on my machine".

And remember from module 5 that getent passwd meteora returns meteora:x:990:990:Meteora service:/var/lib/meteora:/usr/sbin/nologin. The last field is the login shell, and nologin is a program that prints a message and exits with status 1. Since the login process does an execve of that shell, nobody can get an interactive interpreter as meteora even if they obtain its password: least privilege from 05-01 applied with a single word.

Here lives one of the most frustrating beginner mistakes. Bash has two independent axes: interactive or not (does it read commands from a terminal with a human in front of it, or does it run a script?) and login or not (is it the first shell of the session, or one started inside an existing session?). Each combination reads different files:

The classic mistake, in its exact form: you put export METEORA_HOME=/var/lib/meteora in ~/.bashrc, open a terminal and it works; then ssh meteo-01 'echo $METEORA_HOME' comes back empty and a service fails because the variable does not exist. It is not a problem with the variable: a non-interactive shell does not read ~/.bashrc.

The rule of thumb is simple. Whatever defines the environment (variables, PATH) goes in ~/.profile, which is read at login and inherited by everything you start from there. Whatever defines interactive comfort (aliases, prompt, colors, history) goes in ~/.bashrc, because it makes no sense in a script. And whatever a service needs goes in no shell file at all: it goes in its systemd unit with Environment= or EnvironmentFile= (07-02).

That is why Debian's default ~/.bashrc starts with case $- in *i*) ;; *) return;; esac: $- holds the active flags and the letter i appears only if the shell is interactive. If it isn't, return aborts the read. It prevents a real and hard-to-debug failure: if ~/.bashrc writes anything to standard output —a welcome banner, for example— it breaks scp and rsync, which expect a clean stream on that channel.

A line is split into words separated by spaces or tabs: the first one is the command, the rest are arguments, and among them there are usually options.

The -- separator is not cosmetic. If you create ./-report.txt, the command rm -report.txt answers rm: invalid option -- 'r', because rm reads the name as a string of options; rm -- -report.txt works. It is a mandatory defense in scripts that handle names of external origin: anyone who can create files in a directory you walk can inject options into your commands.

Builtin versus external. A builtin command is compiled into the shell and creates no process at all; an external one is a file on disk executed with fork+execve.

What this proves. type knows the shell's entire world —aliases, functions, builtins and external commands— whereas which only walks the PATH and therefore lies about builtins: use type. Notice echo: there are two of them, the builtin and /usr/bin/echo, and they behave differently with options such as -e. That is one of the reasons serious scripts prefer printf.

And why can't cd be an external program? Because the working directory is an attribute of the process, stored in its task_struct and visible in /proc/<pid>/cwd (04-02). If cd were an executable, the shell would fork, the child would call chdir(), change its own directory and die; the parent would stay exactly where it was. The change has to happen in the shell's own process. The same logic explains export, umask, ulimit, exec and source.

Before running anything, bash transforms the line following a fixed, non-negotiable order, which explains almost all the "weird" behavior:

1. Brace expansion: {a,b}, {1..5} — 2. Tilde: ~, ~meteora — 3. Parameters and variables: $VAR, ${VAR:-def} — 4. Command substitution: $(command) — 5. Arithmetic: $(( 2 + 2 )) — 6. Word splitting according to IFS — 7. Globbing: *, ?, [...] — 8. Quote removal.

Why the order matters, with three demonstrations that would fail under any other order:

Explanation. In (a), substituting $PATTERN gives us the text *.dat and, because globbing comes later, that text is subjected again to filename expansion; with quotes it is neither split nor globbed. In (b), when bash processes the braces it has not yet substituted $N, so it sees an invalid range: that is a structural limit of the order, not a bug. In (c), the result of $(...) contains a space and word splitting breaks it into two arguments; double quotes suppress steps 6 and 7. Measured against real incidents, (c) is the number one cause of broken scripts.

The golden rule, with no caveats: quote every variable expansion unless you have an explicit reason not to, and that reason is rare. The case of names with spaces deserves the full example, because that is where the most damage is done:

What changes. The bad version piles up three defects: it parses the output of ls (which formats for humans, not for machines), it loses the full path and, without quotes, it splits every name at its spaces, so that august report.pdf turns into two failed deletions. The good version uses the shell's globbing —which returns full paths and does not split on spaces—, checks that the pattern actually expanded (if the directory is empty, bash leaves the pattern literal) and uses --.

We pick up 04-04 again: every process starts with three descriptors open —0 stdin, 1 stdout, 2 stderr— and the shell simply manipulates them between the fork and the execve, with open() and dup2(). That detail of timing is what makes the executed program none the wiser: by the time it starts, fd 1 already points where you said.

The counterintuitive order. command > out.log 2>&1 sends everything to the file; command 2>&1 > out.log sends stderr to the screen and only stdout to the file. The reason is that 2>&1 literally means "make fd 2 a copy of wherever fd 1 points at this instant": it is a snapshot, not a permanent link. In the first case fd 1 has already moved to the file by the time 2>&1 is processed; in the second it still points at the terminal. It is the dup2() of 04-04, with no added magic.

What they do. The first one leaves separate traces and lets you alert only on the error file. The second takes advantage of find writing its warnings to stderr and its results to stdout, so discarding fd 2 leaves a clean list. In the third, look at the quotes around the delimiter: <<'EOF' does not expand variables inside the block, whereas <<EOF without quotes does; confusing the two produces configuration files with empty values.

A pipe connects one process's stdout to the next one's stdin through the pipe() of 03-03: a 64 KB buffer in the kernel, with automatic blocking when it fills up or empties. The processes are launched all at once, not one after another, so on a multi-CPU machine they really do run in parallel; and when head finishes and closes its end, the previous process receives SIGPIPE and dies, which is why head on a huge pipeline does not wait for everything to finish.

The exit status problem. By default, a pipeline's status is that of the last command, which hides failures:

Why this is serious. In a script with set -e, that pipeline does not abort execution: the script goes on believing everything is fine and processes zero lines as if they were valid data. It is the kind of silent failure that ends in "yesterday's report came out empty and nobody noticed". PIPESTATUS is a bash array with the status of every element; pipefail changes the rule so that the pipeline returns the status of the last command that failed. It is what you want in every non-trivial script.

tee duplicates a stream: aggregator 2>&1 | tee -a /var/log/meteora/aggregator.log | grep -i error stores everything in the log (with -a to append, not truncate) and at the same time passes a copy on to grep, which shows only the errors on screen. Without tee you would have to choose between seeing and storing.

What this proves. The child receives a copy of the environment block, which is the third argument to execve(); variables that were not exported never make it in there. And /proc/<pid>/environ —02-01 again— shows any process's environment, with the values separated by null bytes, hence the tr. A security note: never pass secrets through the environment, because it is readable by the process owner and shows up in core dumps; Meteora's secrets live in /etc/meteora/meteora.conf with mode 600 for exactly that reason.

PATH is the list of directories, separated by :, where external commands are looked up, left to right, stopping at the first match. Picking up 05-02 again: including . in the PATH, and above all putting it first, is a classic mistake; if an attacker drops a file called ls in /tmp and an administrator with sudo runs cd /tmp and types ls, they execute the attacker's program with root privileges. Rules: never include . or directories writable by others; in automated scripts use absolute paths or set PATH explicitly; and check with type -a what is actually going to run before you assume anything.

A job is a complete pipeline launched from the interactive shell. The shell assigns it a number and can move it between the foreground (it receives the keyboard) and the background.

What just happened. Ctrl-Z sent SIGTSTP to the foreground process group and the process moved into the T (stopped) state you saw in 02-01; bg sent it SIGCONT and left it running without the terminal. You can confirm it with ps -o pid,stat,cmd -p 12934.

SIGHUP and nohup. When you close an SSH session the terminal disappears and the kernel sends SIGHUP to the associated processes, which by default die. nohup cmd > log 2>&1 & makes the process ignore that signal; setsid cmd </dev/null >log 2>&1 goes further and creates a new session with no controlling terminal, so the signal is never even generated; and tmux is the preferable option in practice because it also lets you reconnect and see the output. For genuinely periodic tasks, none of the three: you use a systemd timer, the subject of 07-02.

Basic regular expressions: ^ and $ are start and end of line; . is any character; [0-9] a digit; *, + and ? mean zero or more, one or more and optional (the last two require -E); {3} with -E is exactly three repetitions; and | with -E is alternation.

Details that matter. -E turns on extended expressions and saves you escaping {}, | and +; -o prints only the matching part. tail -F (capital F) reopens the file if it is rotated —essential with the logrotate of 05-04—, whereas tail -f sits staring at an inode nobody writes to any more. And --line-buffered forces grep to write line by line: without it, it uses a 4 KB buffer as soon as it detects that its output is a pipe, and you would see the errors minutes late; it is exactly the buffering of 01-06.

find + xargs, safely:

Why. The null byte is the only character that cannot appear in a filename on Linux, so -print0 with xargs -0 is the only safe pairing; --no-run-if-empty stops rm from running with no arguments at all. In the third form, -exec ... + groups many files per invocation, whereas -exec ... \; launches one process per file: with 10,000 files the difference is a handful of processes versus 10,000 fork+execve pairs, that is, seconds versus minutes.

Line format in /var/log/meteora/meteo-api.log (combined log style, with the duration at the end):

Goal: the 10 stations that generate the most requests.

The awk version, which does the same in a single process:

Why it is better. -F'station=' splits each line on that string, so that $2 begins with the identifier, and NF>1 discards the lines without the parameter; split cuts at the first character that is not part of the value and p[1] is left holding EST-0142; the associative array accumulates and the END block dumps the totals. On a 5-million-line log, the six-process pipeline with its two full sorts takes about 40 seconds and writes temporary files to disk; the awk version walks the file exactly once and only sorts a few hundred final lines: between 6 and 8 seconds. The general lesson is that sort over millions of lines is usually a pipeline's bottleneck, and that aggregating before sorting changes the order of magnitude.

A script's first line, the shebang, tells the kernel which interpreter to use in the execve. The kernel reads the two bytes #!, takes the rest as an absolute path and runs interpreter path_to_script. Writing #!/bin/bash demands that bash be exactly there; #!/usr/bin/env bash looks it up in the PATH, which also works where it lives in /usr/local/bin. The trade-off is that dependency on PATH, so in scripts run with sudo or as a service it is wise to pin it.

The mandatory header, option by option:

set -e has well-known traps: it does not fire inside a function called in a condition, nor in the last command of a pipeline without pipefail, nor with local var=$(cmd) —the status that counts is local's, which is always 0—. That is why the correct way is to declare first (local output) and assign afterwards (output=$(command)).

Key details. [[ ]] is a bash construct that does not perform word splitting or globbing inside, so it is safer than [ ], which follows POSIX rules and fails with empty values. while IFS= read -r line is the canonical way to read lines: IFS= prevents trimming of leading and trailing spaces and -r stops \ from acting as an escape. A function returns the status of its last command, or whatever you give it with return N.

The positional arguments are $1, $2…, with $# for the count, "$@" for all of them each as one word (always use this form and not "$*", which joins them into a single string), $? for the last status and $$ for the PID. The exit status convention is the system's own: 0 success, 1 generic error, 2 usage error, and >2 specific errors that you define.

trap for cleanup, indispensable in any script that creates temporary files or takes a lock:

What it does. trap registers a handler for signals —the same ones from 03-03— and for the EXIT pseudo-event, which fires whenever the script ends, successfully or not, so that the temporary directory is removed even if it dies halfway. Saving $? at the top of the function and using it in exit preserves the original status, which the rm would otherwise overwrite.

This is the lesson's integration exercise: check that the files in /var/lib/meteora/readings/ are complete and up to date, and warn if the current day's file is missing.

Design decisions, explained:

• readonly and a pinned PATH. The script will run from a systemd timer, where there is no inherited PATH and no predictable working directory; pinning it removes the command hijacking of 05-02.
• log writes to stderr. That leaves standard output free in case the script ever produces data, and journald captures both streams anyway.
• mktemp -d + trap. Never use a fixed path such as /tmp/work: it is a race condition and a symlink attack vector. mktemp creates a directory with an unpredictable name and mode 700.
• find -print0 | sort -z with read -r -d ''. The complete safety trio against odd names: the separator is the null byte from end to end.
• size % 24 != 0. It is the cheapest and most informative integrity check for this format: since each Reading takes exactly 24 bytes, a size that is not a multiple of 24 means a truncated write, most likely due to a power cut during a write() with no subsequent fsync (04-05).
• A 2% tolerance. A real day rarely has exactly 720,000 readings: a station can lose coverage for a few minutes. Alerting with zero tolerance would produce daily noise and the alert would end up ignored.
• Distinct exit statuses. They let systemd or the monitoring system tell "look at it tomorrow" (1) apart from "act now" (3).

Before putting any script into production, run the static analyzer over it: shellcheck check-readings.sh catches exactly the failures in this lesson —unquoted variables, $(ls), misused [ ], cd without a check, read without -r— and it is the highest benefit-to-effort tool in the shell ecosystem.

Tips that save hours: before a mass rm or mv, replace the action with echo and review the whole output; use Ctrl-R to search the history instead of retyping long commands; debug with set -x or bash -x script.sh, which prints every command fully expanded before running it; comment the why, not the what; and try to make every automated script idempotent, because sooner or later somebody will retry it.

Without running anything, predict the exact output and say which expansion step determines it. Then check it.

Using the meteo-api.log format from this lesson, write a single pipeline (or one awk) to: (1) count the requests with a 5xx status; (2) get the 5 IPs with the most requests and their counts; (3) compute the mean latency of /v1/readings in milliseconds; (4) list the hours of the day with more than 100,000 requests.

Write archive-readings.sh that compresses with gzip the .dat files in /var/lib/meteora/readings/ older than 90 days, skips the ones already compressed, never touches the current day's file, takes a lock so it cannot overlap with itself, logs what it did, supports a -n dry-run mode and returns a correct exit status. It must be safe against names with spaces and pass shellcheck.

Solution 1

• Line 1. $A is substituted in step 3, giving the text *.dat and, because globbing is step 7, that text is expanded again against the directory. That is the trap of storing patterns in variables.
• Line 2. Double quotes suppress steps 6 and 7, so the literal value is printed.
• Line 3. Brace expansion is step 1, before variable expansion: bash sees {1..$N}, which is not a valid range, and leaves it as is. The correct form is seq 1 "$N".
• Line 4. The loop over the glob is correct: each name arrives whole, spaces included, and the order is the locale's (which is why -weird.dat comes first). If you used rm $f inside without quotes, -weird.dat would be read as options: hence rm -- "$f".

Solution 2

Comments. In (1), $9 is the status code in the combined format and n+0 forces a 0 to be printed instead of an empty string when there are no matches, a detail that matters if the output feeds an alerting system. In (2) the file is walked exactly once and only the distinct IPs are sorted, not the millions of lines. In (3), $NF is the last field (duration in seconds) and the if (n) guard avoids division by zero, which is fatal in awk; remember that the mean hides the tail, and in 07-03 you will see why p99 is the metric that matters. In (4), field 4 is [31/Aug/2026:03:12:07, so splitting it on : makes t[2] the hour.

Solution 3

Decisions, explained:

• exec 9>"$LOCK" + flock -n 9. It opens the lock file on the script's own descriptor 9 and takes a non-blocking exclusive lock: the same mechanism from 04-04 that the aggregator uses. If another instance holds it, we exit with status 0, because "it is already running" is not a system failure and must not fire an alert.
• done < <(find ...) instead of find ... | while. This is crucial: with a pipe, the while would run in a subshell and the counters would be lost when it ended, leaving them permanently at zero.
• -mtime "+$DAYS" and -print0 delegate the date filtering to find, which reads the inode directly, and guarantee that no name with spaces or newlines breaks the loop.
• Checking -e "$f.gz" avoids losing data if a previous run was left half-finished; gzip -9 is reasonable because the historical files are written once and read rarely, and on very regular 24-byte binary records it shrinks them by 60 to 70%.
• A dry-run mode before any destructive operation, and logging to stderr so journald picks it up as is.

The shell has stopped being a black box. You now know it is just another user program whose loop is read, expand, fork, execve, wait, and that this nature explains everything else: why cd has to be a builtin, why unexported variables never reach the children, why a script cannot change its parent's directory and why every external command costs a process.

You have seen the four mechanisms you simply have to master. The expansion order, with its eight steps, which explains why {1..$N} does not work and why a pattern stored in a variable gets expanded twice. Quoting, which is not decoration: it suppresses word splitting and globbing, and its absence is the leading cause of broken scripts. Redirection, which is dup2() with syntactic sugar, and from which the asymmetry of > f 2>&1 versus 2>&1 > f follows. And pipes, which are module 3's pipe() with its 64 KB buffer, its SIGPIPE and its exit-status trap that pipefail fixes.

On that foundation you have built real tools: a pipeline that pulls the busiest stations out of meteo-api.log —and its awk version, five times faster because it aggregates before sorting— and a verification script that applies every defense at once: set -euo pipefail, mktemp with trap, -print0 with read -d '', absolute paths, getopts, distinct exit statuses and integrity checks based on the real 24-bytes-per-reading format.

But a lone script is not a system. The aggregator must run every hour even if the machine was switched off, meteo-api must start only after /var/lib/meteora is mounted and restart if it falls over, and all of that has to survive a server reboot. The shell no longer solves that: the service manager does, the piece that starts as PID 1 and decides what runs, when, and under which limits.

That is the next topic: Services, Boot and systemd, where we will follow meteo-01 from the moment the power button is pressed until meteo-api accepts its first request on port 443.