You have spent three lessons writing 2>/dev/null, | and > without anybody having fully explained to you what they are. You have copied them because they work. This lesson takes the mechanism apart, and it is probably the most important one in the module: redirection and pipes are the mechanism through which the Unix philosophy of Module 1 — small programs that do one thing well and compose together — stops being a nice phrase and becomes something you type.

This is not a subject where memorising symbols is enough. There are a handful of cases — 2>&1 >file against >file 2>&1, the exit code of a pipeline, sudo echo x > /etc/... — where intuition misleads you and everybody gets it wrong at least once. We are going after them in enough detail that it does not happen to you.

1. The three standard streams and why they exist
2. Seeing them for real in /proc
3. Output redirection
4. Input redirection
5. Error redirection and the order of evaluation
6. The special files in /dev
7. Here-documents and here-strings
8. Pipes: what they really are
9. The exit code of a pipeline
10. tee and the | sudo tee pattern
11. Buffering: when the output does not appear
12. Composing a pipeline step by step

Every process on Linux is born with three channels already open. They are not a Bash convention: the system opens them and they are identified by a number, the file descriptor.

The interesting question is why there are two output channels if both go to the screen. The answer is the key to the whole design: so that they can be separated when needed. If the results and the errors travelled down the same channel, you could not save just the useful part to a file, nor chain one program to another without the warnings contaminating the data.

> redirected only stdout to the file. The error stayed on the screen because it travels down stderr. That separation is deliberate and it is what allows a find walking /etc with permission denied warnings to still produce a clean list.

Since everything is a file, the descriptors can be looked at:

All three point to /dev/pts/0, your pseudo-terminal. Number 255 is for Bash's internal use. Now look at what happens inside a redirection:

Descriptors 1 and 2 no longer point to the terminal but to the file. Redirection is not a function of the command: it is something Bash does to the child process before starting it. That is why it works with any program, without the program having to know anything about it.

> truncates the file before running the command, and creates it if it does not exist. That means a > over a file with data destroys it even if the command fails afterwards.

noclobber turns > into an operation that fails if the destination exists, and leaves >| as the explicit way of saying "yes, overwrite". It is a good setting for a working session in production; check it with set -o | grep noclobber and disable it with set +o noclobber. The course convention — a .bak-$(date +%F) copy before touching anything — and noclobber reinforce each other.

< file connects stdin to the file. Many commands accept the file name as an argument, so the difference looks cosmetic. It is not:

With <, the program does not know the file's name: it only receives a stream of bytes, and that is why it does not print it. It is what you want when the number is going to feed another calculation. (26 lines: the header plus the 25 records.)

Here is the point where everybody goes wrong. Bash processes redirections from left to right, and 2>&1 does not mean "join the two streams forever": it means "copy 1's current destination onto 2". It is a photograph of the state at that instant.

Step by step: > /tmp/ok.txt points 1 at the file; then 2>&1 copies that destination onto 2. Both end up in the file. Correct.

Step by step: 2>&1 copies 1's current destination, which is still the terminal, onto 2; then > /tmp/wrong.txt moves 1 to the file, but 2 was left pointing at the terminal. Result: the error comes out on screen and only the normal output goes to the file. It is exactly the opposite of what most people think they are writing.

The practical rule: 2>&1 always goes at the end. And if you do not need compatibility with other shells, &> file is unambiguous and does not allow the mistake.

There is a legitimate use for the "wrong" order: command 2>&1 >/dev/null | grep something sends only the errors into the pipe and discards the normal output. It is unusual, but when you need it, it is the only way.

2>/dev/null is the noise silencer you were already using in 03-03. Use it with judgement: it discards all the errors, including the ones you did want to see. If you only want to ignore the permission denied ones, filtering is better than silencing blindly.

/dev/stdout as a path is useful when a command demands a file name and you want its output in the pipeline, as in tar -cf /dev/stdout. And /dev/zero is handy for generating a file of a known size to test a backup with:

A here-document feeds stdin with text written on the command line itself, up to a delimiter.

With the delimiter in single quotes nothing is expanded:

The rule: quote the delimiter unless you deliberately want expansion. It is the same quoting rule from 03-01, applied to blocks.

A here-string <<< is the single-line version, and it avoids the echo ... | people write out of habit:

A pipe is not a temporary file. It is a buffer in kernel memory — typically 64 KiB — with two ends: the standard output of the process on the left and the standard input of the one on the right.

And here is the point to understand: the two processes run at the same time, not one after the other. The one on the right starts consuming as soon as there is data. If the buffer fills up because the consumer is slower, the kernel puts the producer to sleep until there is room; if the buffer empties, it puts the consumer to sleep. That automatic synchronisation is why you can do cat 50-GB-file | grep something without exhausting memory.

Two simultaneous processes. grep writes into the buffer, wc reads from it, and neither of them knows the other exists: one thinks it is writing to the screen and the other thinks it is reading from the keyboard. That is exactly what makes it possible to compose programs that were never designed to work together.

A pipe only carries stdout. The errors from the command on the left still go to the screen. To put them into the pipe as well, |& (equivalent to 2>&1 |):

By default, $? returns the code of the last command in the pipeline. The previous ones are lost, and that hides failures:

cat failed, but wc finished cleanly and the pipeline reports success. A procedure that checks $? would accept as good an operation that read nothing at all.

The PIPESTATUS array holds the code of each element:

And set -o pipefail makes the pipeline return the code of the last command that failed:

pipefail is indispensable in serious scripts and you will see it built into set -euo pipefail in Module 4. Here just take away the idea: a pipeline can lie to you about its success, and you know two ways of stopping it.

tee reads from stdin and writes at the same time to stdout and to one or more files. It forks the stream.

You see the count and you also keep the lines in a file. With -a it appends instead of truncating, and with - as an extra destination it can duplicate to the screen in the middle of a pipeline, which makes it an excellent debugging tool.

It looks contradictory: we wrote sudo. The explanation lies in the division of labour. The redirection is carried out by Bash, not by sudo. Bash opens the destination file before launching anything, and it does so with operator's privileges, and he cannot write to a file owned by root. The sudo affects echo, which is precisely the part that did not need privileges.

The solution is to make the privileged process the one that writes:

Now tee runs under sudo and it is the one that opens the file. The trailing > /dev/null discards the copy tee sends to the screen, which adds nothing here. And to append, | sudo tee -a, never >>, for the same reason. Applied with the course convention:

The diff reveals something important: we have added a second log_level key instead of correcting the existing one. The file now has two, and which one wins depends on how the application reads it. This is why the convention demands the diff -u: not to admire the change, but to discover that it was not the one you wanted. Correcting in place is sed's job, in the next lesson.

A program writes into a buffer and only flushes when it fills up or when it finishes. The C library applies a rule that catches people out: if stdout is a terminal, the buffer is line-based; if it is a pipe or a file, it is block-based in 4 KiB chunks.

The practical consequence: tail -F access.log | grep 'ERROR 500' may show nothing for minutes, not because there are no errors, but because grep is accumulating 4 KiB before releasing them. It looks as though the pipeline is broken and it is working perfectly.

stdbuf -oL forces line buffering on the following command. Alternatives: grep --line-buffered, awk with fflush(), sed -u. Remember that the course convention is tail -F and never -f in production, because -F survives the file being rotated.

The goal: the paths with the most 500 errors in access.log. It is built incrementally, verifying each link, just as you did with the regexes.

Of 412 lines, 14 remain. Now we extract just the path and group:

Nine out of fourteen on /bookings, and the ms=30012 values in the sample are just under 30 seconds, which is exactly the query_timeout=30 in app.conf. The pipeline has not produced a fact: it has produced a hypothesis — the database queries are hitting the time limit — which fits with the active_connections=200 we saw in 03-03 and with the configured max_connections=200. That is what you take to Marta.

Go over the composition again: grep filters, grep -oE extracts, sort groups equal things together, uniq -c counts, sort -rn orders by frequency. Five programs that know nothing about each other, chained by a kernel buffer. None of them knows how to produce the report; together, they do.

• Writing 2>&1 before the output redirection. It is the classic mistake. It goes at the end, or use &>.
• command > file with the same file as input. sort data.txt > data.txt empties it: Bash truncates the destination before sort reads. Use sort -o data.txt data.txt or a temporary file.
• sudo command > /protected/file. The redirection is done by your shell. Use | sudo tee.
• Trusting $? after a pipeline. It is the last command's. Use PIPESTATUS or pipefail.
• Silencing with 2>/dev/null as a reflex. You are also discarding the errors you needed. Look at them first.
• Believing that a pipeline is sequential. The processes run at the same time; that is why it works with enormous files and why buffering can delay the output.
• Forgetting to quote a here-doc's delimiter. If the text contains $ or quotes, it will be expanded.
• Tip: insert | tee /tmp/step1.txt | in the middle of a long pipeline to inspect what is travelling through that point without breaking the chain.
• Tip: command | cat disables colour and column output in many programs, because they detect that they are not writing to a terminal. If you are missing the colour, --color=always forces it.

Exercise 1. Run a find over /etc as a normal user, so that the list of results ends up in /tmp/findings.txt and the permission errors in /tmp/failures.txt, with nothing appearing on screen. Then count how many errors there were and explain why you could not have separated them with &>.

Exercise 2. Add the line # reviewed 2026-08-18 to the end of /etc/tramontana/app.conf keeping the convention of a prior copy and a subsequent verification. Explain why sudo echo ... >> does not work.

Exercise 3. Build step by step a pipeline that shows the five IP addresses with the most requests in access.log, while saving the complete listing to /tmp/ips.txt at the same time. Verify that the total number of requests counted matches the file's 412 lines and explain what you would do if it did not match.

Solution 1.

Two independent redirections, each stream to its own file, and the screen stays clean. &> does not work because it mixes both streams into a single destination, which is precisely the opposite of what the exercise asked for. The six errors are directories such as /etc/ssl/private, legitimately closed to a normal user: reviewing them in their own file is better practice than discarding them with 2>/dev/null, because an unexpected error in there would be a signal you would not want to lose.

Solution 2.

sudo echo '...' >> /etc/tramontana/app.conf fails because the redirection is carried out by your shell before sudo is invoked, and your shell runs as operator, who has no write permission on a 640 file owned by root. The sudo would apply to echo, which needs no privilege at all to print text. With | sudo tee -a, the process that opens the file is tee, and that one does run as root. The cp -p preserves permissions and ownership in the copy, and ls -l confirms that the original is still 640 root:tramontana: writing with tee under sudo has not changed the file's ownership, because tee writes into the existing file instead of creating it anew.

Solution 3.

The first command is the indispensable verification: 412 matches for 412 lines, so every line has exactly one ip= field. If it had come out lower, there would be lines in another format silently being left out of the report; if higher, some line with two IPs and an inflated count. In either case you would have to examine the discordant lines with grep -vc 'ip=' before going on, because a report built on an unverified assumption is a report that can be contradicted.

tee forks: the file receives the complete listing and head -5 keeps the podium. The final sum with awk — which you will study in the next lesson — closes the circle: 412 again, so no request has been lost or duplicated along the way. And there is a finding: 10.0.2.77 accounts for 138 requests, a third of the total and almost 50% more than the next one. It deserves investigation.

You have gone from copying symbols to understanding where every byte travels.

• You know the three standard streams — stdin 0, stdout 1, stderr 2 — you know why there are two output channels, and you have seen them for real in /proc/<pid>/fd.
• You handle >, >>, noclobber with >|, and <, knowing that with < the program does not know the file's name.
• You have the classic mistake sorted: 2>&1 copies 1's current destination, it is evaluated from left to right and that is why it goes at the end; &> is the unambiguous shortcut.
• You use /dev/null, /dev/zero and /dev/stdout with judgement, knowing that silencing errors blindly hides the ones that did matter.
• You write here-documents and here-strings, and you quote the delimiter when you do not want expansion.
• You know that a pipe is a kernel buffer between two processes running at the same time, that it only carries stdout, and that |& includes the errors.
• You do not trust a pipeline's exit code: you know PIPESTATUS and set -o pipefail.
• You apply | sudo tee to write to protected files, and you can explain why sudo echo x > /etc/... fails: the redirection is done by your shell, not by sudo.
• You recognise block buffering when a log does not appear in real time and you fix it with stdbuf -oL.
• And you have composed a pipeline incrementally until 412 lines of log became a defensible diagnostic hypothesis.

sort, uniq -c and even an awk have turned up in those pipelines without being explained. That is the debt the next lesson settles. Text Processing: cut, sort, uniq, sed and awk closes the circle opened by the Unix philosophy of Module 1: plain text as the universal interface between programs. You will learn to cut out fields, sort by whichever column you like, count occurrences, transform with sed without destroying the original, and group and total with awk until half a pipeline is replaced by a single command. By the end of it you will be able to hand Marta the monthly billing report from bookings.csv and the top error paths from access.log without writing a line of code.