In lesson 02-02 you already used | intuitively to chain cut, sort and uniq. Now you are going to understand what happens underneath and master the other half of the mechanism: redirection. This is the lesson that turns a collection of loose commands into a composition language. And it is also the one you need for veloz-ops to stop printing to the screen and start generating report files and execution logs, which is what is expected of a serious operations tool.

1. The three standard streams
2. Redirecting output: > and >>
3. The truncation risk and noclobber
4. Redirecting input: <
5. Redirecting errors: 2>, 2>&1 and &>
6. /dev/null: the system's black hole
7. tee: seeing and saving at once
8. Pipes and the concept of a filter
9. Filter chains that answer real questions
10. The exit code of a pipeline

Every process in Linux is born with three communication channels open, identified by a number called a file descriptor:

The obvious question is why there are two outputs if both end up on the screen. The answer is the key to the whole design: so you can separate them when you need to. A command writes its results to stdout and its complaints to stderr, so you can save the results to a file while the errors keep appearing in your terminal, or the other way round. See for yourself:

If you run ls /srv/veloz/data /srv/veloz/nonexistent, you will see the listing of the first directory and the No such file or directory message for the second one mixed together. They look like a single block, but they are different streams: as soon as you redirect one of them, they will separate.

The > operator sends stdout to a file instead of to the screen:

You will see nothing on screen: the output is in the file. > creates the file if it does not exist and empties it completely if it does. That emptying is total and instantaneous, and it happens before the command starts running.

The >> operator appends at the end without deleting what was there:

Practical rule: > for results that get regenerated, >> for records that accumulate. A daily report that is remade every morning uses >; the toolkit's execution log uses >>.

The danger of > is that it gives no warning. Mistakenly typing > shipments.csv instead of >> shipments.csv destroys the data file on the spot, with no confirmation and no possible recovery. It is the silent equivalent of the rm from lesson 02-01.

Bash offers a safety net, the noclobber option:

With noclobber active, > refuses to overwrite a file that already exists. When you do want to, you use the >| operator, which forces the truncation:

Notice the asymmetry of set: -o turns an option on and +o turns it off, the opposite of what intuition suggests. Turning on noclobber in your ~/.bashrc is a good idea for interactive sessions; in scripts it is better not to depend on it and to write your redirections carefully.

The < operator makes a command read from a file instead of from the keyboard:

Compare it with wc -l /srv/veloz/data/shipments.csv, which also prints the file name. The difference is conceptual: in the first case wc does not know there is a file, it only receives a stream of data on stdin; in the second, it opens the file itself. That distinction will matter when you write loops that read files line by line in Module 4.

There are commands that only accept standard input, such as tr, which does not take a file as an argument: tr 'a-z' 'A-Z' < shipments.csv is the only way to feed it from a file.

Since stderr is descriptor 2, it is redirected by putting that number in front of the operator:

Now output.txt contains only the correct listing and errors.txt only the error message. 2>> appends instead of truncating, exactly like >>.

To combine both streams into the same target you use 2>&1, which reads as "send descriptor 2 wherever descriptor 1 is pointing right now":

Order matters, and this is the classic trap. Bash processes redirections from left to right:

The second form is a real and frequent mistake: it looks like it saves everything and in reality the errors are lost in the terminal. Remember that 2>&1 copies the current target of 1, it does not create a permanent link.

Bash also offers a shorthand of its own, more readable and with no ordering risk:

&> is not POSIX: it works in Bash but not in plain sh. For portable scripts you use > file 2>&1 (portability, 08-07).

/dev/null is a special file that discards everything written to it and returns nothing when read. It serves to silence output you do not care about:

The second pattern is one of the most frequent in scripting. It is used when you only care about the command's exit code, not its output, typically in a check:

Here we do not want to see ping's statistics, only to know whether it answered. That said, silencing errors is dangerous: 2>/dev/null in a production script can hide exactly the failure you needed to see. Use it only when the error is expected and accounted for.

tee (after the plumbing "T") duplicates the stream: it writes it to a file and lets it carry on through stdout.

In a single line you have saved the complete errors to a file and seen the count on screen. With -a (append) it appends instead of truncating, and combined with sudo it solves a classic problem:

This works where sudo echo ... >> /etc/... fails, and understanding why is important: the redirection is done by your shell, not by sudo, so it is your unprivileged user who tries to open the system file. With tee, the privileged process is the one doing the writing.

A pipe | connects the stdout of the command on the left with the stdin of the one on the right. The two processes run at the same time, not one after the other: the data flows as it is produced, with no intermediate files and without loading anything entirely into memory.

A filter is a program that reads from stdin, transforms, and writes to stdout. grep, cut, sort, uniq, tr, wc, head, tail and column are all filters, and that is why they fit together in any order.

Two warnings:

• The pipe only carries stdout. The first command's errors go to the screen, not to the second. If you want to pass them along too, cmd 2>&1 | filter (Bash has the |& shortcut).
• Avoid the "useless use of cat". The previous example is better written as grep -c ERROR /var/log/veloz/app.log: one process instead of three. cat at the head of a pipeline is only justified when you are concatenating several files.

This is where the whole module converges. The 10 IPs that have made the most requests to veloz-api today:

Step by step: cut -d' ' -f1 extracts the first space-separated field, which in the combined format is the IP → sort groups the identical ones → uniq -c counts each group → sort -rn sorts by number descending → head -10 keeps the podium → tee also saves it into the toolkit.

Cities with the most issues, saving the result as a report:

And to count and record at the same time the requests that returned error 500: grep ' 500 ' /var/log/veloz/access.log | tee ~/veloz-ops/logs/errors-500.log | wc -l.

Notice the general pattern that keeps repeating: filter → extract → group → count → sort → present → save. Almost any question about tabular data or logs is answered with some variant of that sequence, and that is exactly the skeleton of the log analyzer you will build in project 09-02.

In lesson 01-04 you saw $?, the exit code of the last command. In a pipeline, $? is that of the last command in the chain, not that of the whole thing:

grep found nothing and returned 1, but wc -l ran correctly and returned 0, so $? is 0. The pipeline appears to have succeeded when in fact the filter failed. It is an inexhaustible source of silent bugs in scripts.

Bash offers two solutions: the PIPESTATUS array, which stores the code of each command in the pipeline, and the set -o pipefail option, which makes the pipeline return the code of the first command that failed. Both belong to error handling and you will study them thoroughly in 05-03; for now, just keep the warning in mind.

One final note: when you need to pass a command's output as arguments to another (not as standard input), the pipe is no use and you need xargs, which you will see in 05-01. And the advanced I/O forms —here-documents and your own descriptors— arrive in 05-05.

• Writing > file when you meant >>. It destroys the content instantly. Consider set -o noclobber in your ~/.bashrc.
• Putting 2>&1 before >. The errors stay on the screen. The correct form is cmd > file 2>&1, or simply cmd &> file.
• Believing the pipe carries the errors. It only carries stdout; use 2>&1 | if you need them.
• Using sudo cmd > /etc/file. The redirection is done by your unprivileged shell and it fails. Use sudo tee.
• Overusing 2>/dev/null. It hides the diagnosis you were going to need. Silence only expected errors.
• Trusting $? after a pipeline. It only reflects the last command (see 05-03).
• Reading and writing the same file in one command. sort file > file leaves the file empty, because > truncates it before sort reads it. Write to a temporary file and then rename it with mv.

Exercise 1 — Separating results and errors. Run an ls on /srv/veloz/data and on a nonexistent directory, so that the correct listing ends up in ~/veloz-ops/logs/listing.txt and the error message in ~/veloz-ops/logs/listing.err. Then write the variant that saves it all together into a single file, in both possible ways.

Exercise 2 — Report of couriers with issues. On shipments.csv, generate a file ~/veloz-ops/logs/issues-by-courier.txt with the number of issues for each courier, sorted from most to fewest and presented as an aligned table. You must see the result on screen at the same time as it is saved.

Exercise 3 — Debugging a command that fails silently. A colleague has this line in a script and complains that "it never detects anything":

Explain the two flaws and rewrite it correctly.

Solution to Exercise 1

The last two are equivalent in Bash. The first is preferable in scripts that might be run with sh; the second is more readable and removes the risk of inverting the order. What you must not write is ls ... 2>&1 > ~/veloz-ops/logs/all.log, which would leave the errors on screen.

Solution to Exercise 2

grep ',issue,' filters by the status field with the commas before and after, which avoids false positives if that word were to appear in another field; cut -d, -f4 extracts the courier; sort | uniq -c groups and counts; sort -rn sorts descending; column -t aligns; and tee at the end —instead of >— is what lets you see the result and save it simultaneously.

Solution to Exercise 3

The two flaws:

1. $? picks up wc's code, not grep's. wc -l always finishes successfully, so the condition is always met, whether there are timeouts or not. The script would say "timeouts found" even with an empty log.
2. The > /dev/null 2>&1 redirection contributes nothing useful here and along the way discards the number wc had computed, so the information is lost completely.

Correct version, using grep's exit code (0 if it found something, 1 if not):

grep -q (quiet) prints nothing and returns only the exit code, so it replaces both the wc and the redirection to /dev/null. If you also need the count:

The $(...) syntax is command substitution and if conditionals are formalized in Module 3; what matters here is the diagnosis: evaluate the exit code of the command that actually decides, never that of the last link in a pipeline.

You have understood the mechanism that makes Bash a composition language. You know that every process has three streams —stdin, stdout and stderr—, and why separating them is useful; you redirect output with > and >> knowing the truncation risk and the protection of noclobber; you redirect input with <; you control errors with 2>, 2>&1 and &>, knowing that order matters; you silence noise with /dev/null with due caution; you duplicate streams with tee, even to write to system files with sudo; and you chain filters with | to answer questions no single command can solve. You also know that $? after a pipeline only reflects the last link, a detail you will settle definitively in 05-03.

With this, the Module 2 commands stop being loose pieces: they are components of assembly lines that already produce real reports in ~/veloz-ops/logs.

In lesson 02-05 we close the circle of file manipulation with wildcards. You will discover that *.csv is interpreted neither by ls nor by rm, but by the shell itself before running them, and that this difference explains everything from the most famous rm * mistake in history to why a loop fails when a pattern matches nothing.