There is one skill that separates the administrator who resolves incidents from the one who escalates them: not needing the internet to know what a command does. Linux comes with its own documentation installed, written by the people who developed each tool, matching the exact version you have in front of you and available even when the server is isolated on a network with no way out.

That last part is not a theoretical assumption. A properly configured production server is usually behind a firewall that does not allow browsing. A banking or healthcare environment may be on a completely separate network. And when there is a serious incident, the first thing to be cut off is external access. If your working method is "I'll look it up", at that moment you are left without a method.

This lesson teaches you the alternative method. By the end of it you will know how to read a whole manual page — including that notation with brackets and bars that looks like hieroglyphics — how to find a command when you cannot remember its name, and which of the five different documentation sources holds the answer you need. It is probably the most profitable lesson in the module.

1. The five documentation sources on a Linux system
2. man: the system manual
3. How to read the SYNOPSIS notation
4. The manual sections and why they exist
5. Searching the manual: whatis, apropos and man -k
6. Navigating with less inside man
7. --help: the quick summary
8. info: the extensive GNU documentation
9. help: the Bash builtins and why man cd does not work
10. Local documentation in /usr/share/doc
11. tldr and cheat: the complement, not the substitute
12. Exit codes and $?: knowing whether something failed
13. A search strategy for when you do not even know the name

Before going into detail, the map. On srv-tramontana five different sources live side by side, and choosing the wrong one wastes your time:

And a sixth, optional one, which needs placing properly: tldr, which gives practical examples but is not official documentation and is not exhaustive.

The operating rule: --help to recall, man to understand, info to go deep, /usr/share/doc to configure.

man (from manual) shows the official documentation page for a command, a configuration file, a system call or a file format.

It opens full screen in a pager. You leave with q.

Manual pages have followed a standardised structure since the 1970s. Knowing it lets you jump straight to what you are after instead of reading from top to bottom:

Two pieces of advice that change how man gets used:

• Start with EXAMPLES if it exists. Not every page has one (GNU pages usually do not), but when it is there it settles the question in thirty seconds.
• Finish with SEE ALSO when you have picked the wrong command. It is the list of related tools written by the people who know the domain best.

A real example of selective reading. You need to know which files the SSH service uses:

Inside, you press /FILES and Enter: you jump straight to the list of files, without reading the three hundred lines before it. We will come back to this technique in section 6.

The SYNOPSIS is the part most people skip and the one that contains the most information. It is written in a formal notation with fixed rules:

Let us see it applied. The SYNOPSIS of cp:

That page is telling you three things that are not written out in prose anywhere:

1. There are three valid ways of invoking cp, not one. Each line is a complete form.
2. In the first one, SOURCE and DEST appear without brackets: they are mandatory. cp with no arguments fails.
3. In the second one, SOURCE... carries the ellipsis and the last argument is called DIRECTORY: you can copy several files at once, but then the destination has to be a directory. That explains an error you will see in lesson 02-04.

Another example, man 5 crontab against man 1 crontab:

Here [-l | -r | -e] tells you that listing, deleting and editing are mutually exclusive: you cannot ask for two at once. The whole logic of the command is in a single line.

A mental exercise you should always do: before reading OPTIONS, read the SYNOPSIS and ask yourself how many invocation forms there are and what is mandatory. Very often the answer is already there.

The manual is divided into eight numbered sections. The reason is that the same name can refer to different things: passwd is a command and a configuration file; printf is a command and a C function.

The three you will use constantly as an administrator are 1 (commands), 5 (configuration files) and 8 (root tools).

When you run man passwd with no number, man gives you the first section where it finds that page, normally section 1:

But if what you want is to understand the format of the /etc/passwd file, that is a completely different page:

The header always tells you which section you are in: PASSWD(1) against PASSWD(5). Get into the habit of looking at that number: it is how you know whether you are reading the right page.

To find out which sections a page exists in:

man -a shows you all of them in sequence: when you leave one with q, it asks whether you want to see the next.

Two section 7 pages worth knowing about already:

These two tools answer the two most frequent questions.

"What is this command I have just seen in a script?" → whatis (equivalent to man -f), which shows only the NAME line:

"I do not know what the command I need is called" → apropos (equivalent to man -k), which searches for the word in the descriptions of all the installed pages:

In three seconds, with no search engine, you have the three tools that exist for the problem. This is the answer to the "the CLI is not discoverable" criticism from section 1 of the previous lesson: it is discoverable, but you discover it with apropos, not with menus.

When the search returns too much, you narrow it down:

-s 8 restricts it to the administration section. You can also limit the search to the name instead of the description:

-e forces an exact match, which is useful because apropos tar would return dozens of pages containing "tar" inside other words.

If apropos answers nothing appropriate, the index database may not have been generated. You regenerate it with:

man draws nothing by itself: it sends the text to a pager, which on Ubuntu is less. The shortcuts you learn here also work for viewing files and logs (lesson 02-05), so they pay off twice over.

The real workflow of somebody with experience is not to read the page: it is to go in and search.

Suppose you want to know what the -h option of du does. Instead of walking through the page:

Inside, type /-h and Enter. less jumps to the first occurrence; with n you move forward to the one in the OPTIONS section:

A very useful trick: to look for one specific option and avoid the dozens of stray appearances of that letter, search for the pattern with spaces:

The ^ means "at the beginning of the line" and manual pages indent their options, so this jumps straight to the definition. (That ^ is a regular expression; the full topic belongs to lesson 03-02, but you can adopt this one isolated use right away.)

And a practical warning: if you are inside man and do not know how to get out, it is q. It is the most frequently asked question of the whole course.

Almost every command accepts --help (or -h) and answers with a summary in the terminal itself, with no pager:

Differences from man:

That last point has a practical consequence: in a minimal Docker container (Module 7) there are usually no manual pages installed, because they are removed to reduce the image size. There, --help is all you have.

Watch out for one detail: in some commands, -h does not mean "help". In ls, du and df it means human-readable. If in doubt, use --help, which is unambiguous.

And another: when the output of --help does not fit on the screen, do not chase it upwards. Send it to the pager:

That | is a pipe; it is explained thoroughly in lesson 03-04, but you can adopt this use from today.

The GNU project decided at the time that manual pages were too limited for documenting its tools and created its own system: Texinfo, which you consult with info. It is organised into linked nodes, like a book with navigable chapters.

A practical consequence worth knowing: in the GNU tools (ls, cp, mv, tar, grep, sed...), the manual page is often a summary and the real documentation is in info. Many pages end with an explicit note:

Basic info navigation:

When info is worth the trouble: when man gives you the option but not the why. For instance, to understand the exact behaviour of cp with symbolic links, or the formats of date, info explains the edge cases that man merely lists.

If you find info awkward (many people do, because of its node-based navigation), there is a way out:

That way you read the same content in less, with the shortcuts you already know.

Try this:

The explanation is in the previous lesson: cd is not a program, it is a Bash internal command. No package installs it, so there is no manual page documenting it. Its documentation is provided by Bash itself through the help builtin:

And the complete list of builtins, which is worth looking at once in your life:

The complete decision tree:

There is an intermediate case worth knowing about: man builtins shows all the documented builtins on a single page, and man bash contains the shell's complete documentation. It is one of the longest manual pages on the system (more than five thousand lines), and for that very reason /word is indispensable in there.

Every installed package leaves documentation in /usr/share/doc/<package>/. It is the most ignored source and the one that most often contains exactly what you need: example configuration files.

What to expect in each type of file:

Many are compressed as .gz. You read them without decompressing with zless or zcat:

README.Debian is especially valuable because it documents the differences between the original tool and the way Ubuntu packages it: changed paths, different defaults, systemd integration. That does not appear in the project's official documentation and it is the cause of half the confusion when you follow a tutorial written for another distribution.

When you get to configuring services in Module 5 and Module 8, this folder will be your first stop, ahead of any tutorial.

tldr (from too long; didn't read) is a community collection of practical examples. Where man tar gives you a thousand lines, tldr tar gives you the six invocations used 95 % of the time.

It is enormously useful, and you have to be aware of its three limits:

1. It is not exhaustive. It shows the common case, not what you need in a rare one. And rare cases are precisely what incidents are made of.
2. It is not official. The community writes it; it may be out of date with respect to your version.
3. It may not be installed, and on a production server it probably is not (and you probably should not install it just for that).

Professional criterion: tldr to get going quickly with a tool you already understand; man to decide something that is going to production. If you are about to run a destructive command or modify the configuration of a running service, the source is the manual.

Remember from section 9 of the previous lesson: every command returns a number when it finishes. Now we are going to look at it directly, because it is the information that tells you whether something worked when the command said nothing.

$? holds the code of the last command run. The universal convention:

Each command can define its own, and that is where the EXIT STATUS section of the manual comes in:

Searching for /EXIT STATUS:

That means that in grep, 1 is not a failure: it means "no matches". Confusing "found nothing" with "broke" is a classic source of bugs in monitoring scripts. And the only way to know is by reading that section of the manual.

Watch out for one trap: $? is overwritten by every command, including the echo that displays it.

That second 0 is the code of the previous echo, which worked perfectly. Check $? immediately, or store it in a variable (Module 4).

This is the complete procedure, in order, for when you face a problem without knowing which tool solves it:

1. Describe the problem in two or three words in English. "space", "compress", "rename", "monitor processes".
2. apropos with those words, narrowing down with -s 1 or -s 8.
3. From the list, whatis on the candidates, to discard them quickly.
4. man on the chosen one: read NAME, SYNOPSIS and jump to EXAMPLES.
5. If it was not the right one, the SEE ALSO of that page usually contains the one that is.
6. If the command exists but you do not know how to use it, tldr for a quick start, then back to the manual for the details.
7. If it is a configuration file, man 5 <name> and /usr/share/doc/<package>/examples/.
8. If it is a command with no manual, type will tell you whether it is a builtin and then help.

A complete run-through. Marta Vidal asks how much space the application's directory takes up:

Inside the manual, /summarize:

And /human:

The result, without leaving the server and without a search engine:

Three commands and two searches inside one page. That is the method.

Not knowing how to get out of man. It is q. If you are in info, also q. If you have ended up in vim by accident, it is :q! (lesson 02-05).

Reading the whole page from top to bottom. Manual pages are reference material, not tutorials. Go in and search with /.

Ignoring the section number. If man passwd talks to you about changing passwords and you were after the file format, it is not that the manual is wrong: you are missing the 5.

Searching the internet before apropos. The search engine will give you an answer written for another distribution, another version and another year. man documents exactly the binary you have installed.

Copying a command from a forum without reading its options. Before running something you do not understand, put its options through man. It literally takes a minute, and it is the difference between resolving one incident and causing another.

Tip: man -k is your local search engine. Get into the habit of trying it before opening the browser. In the first week it will feel slower; from the third week on you will be faster.

Tip: save what you learn. Create /home/operator/scripts/notes/ and write down there the commands you have had to look up. Your own documentation, with your own cases, is quicker to consult than any manual.

Tip: read man 7 hier once. It is the FHS from Module 1 told by the system itself. Half an hour well spent.

Tip: on a server with no manual pages, check whether they are disabled before giving them up for lost. Some images exclude them in the package installation configuration; how to get them back is the subject of lesson 05-03.

Without running the commands, only by reading their manual pages, answer:

1. Does mkdir accept creating several directories in a single command? How do you know from the SYNOPSIS?
2. How many different invocation forms does ln have?
3. In man 1 tar, are -c, -x and -t compatible with each other?

Luis writes to you: "I need to keep only the last 50 lines of /var/log/tramontana/access.log and I cannot remember the command. I also want to watch it updating live while I test the application. Can you Google it for me?"

Constraint: srv-tramontana is on a network with no internet access. Solve it using only system documentation, and document the path you followed (which search commands you used and in what order).

Marta Vidal wants to know exactly what the line 0 3 * * * /home/operator/scripts/backup.sh does — it appears in the scheduled task configuration of the previous server — and whether the format is reliable.

Using only the system manual, find out:

1. Which manual section documents the format of that file (not the command).
2. What each of the five fields means.
3. What exit code the crontab command returns if the file has a syntax error.

Write the answer as a short paragraph addressed to Marta, who is not technical.

Solution 1

Yes, it accepts several. The proof is in the ellipsis after DIRECTORY: it means "one or more". And since DIRECTORY appears without brackets, at least one is mandatory. All that information is in a single line, without reading a word of prose.

Four forms. Look at the second one: ln TARGET with no link name. It creates the link in the current directory with the same name as the target. It is a detail that takes people by surprise and that you only learn by reading the SYNOPSIS. You will use it in lesson 02-06.

They are grouped under the heading "Operation mode", which indicates that you choose one and only one. The page confirms it further up:

They are not compatible. Trying tar -cx gives an error along the lines of "You may not specify more than one -Acdtrux option". The clue was in the organisation of the page itself: when a manual groups options under a "mode" heading, they are usually mutually exclusive.

Solution 2

The path followed.

First, search the manual by concept:

Confirm it is what it looks like:

Go to the manual and search inside it for how to set the number of lines. Inside man tail, /lines:

And for the second part, following the file live, /follow:

The two commands that answer Luis's request:

You leave follow mode with Ctrl+C.

The nuance the documentation contributes and that a quick search-engine answer would not have given: the difference between -f and -F. -f follows the file descriptor; if logrotate rotates the log at midnight and creates a new one, tail -f carries on staring at the old, now renamed file, and stops showing anything without warning. -F follows the name, so it reattaches to the new file.

To follow a production log that rotates, the correct option is tail -F. That is literally in the manual page, and it is exactly the kind of detail a three-line tutorial leaves out. We will come back to it in lesson 02-05.

What you reply to Luis is not the command: it is the method. apropos with two English words, whatis to confirm and /option inside the manual. Thirty seconds, with no internet.

Solution 3

1. Manual section. The crontab command is in section 1, but the file format is in section 5:

2. The five fields. Inside the page, searching for /field:

Applied to 0 3 * * *: minute 0, hour 3, any day of the month, any month, any day of the week. That is, every day at 3:00.

3. Exit code on a syntax error. In man 1 crontab, searching for /EXIT:

And in the DIAGNOSTICS section of the same page it is explained that a file with syntax errors is rejected outright: crontab does not install a partial version. That nuance is the important one for Marta, and it only turns up by reading the page.

Report for Marta:

Marta, the line 0 3 * * * /home/operator/scripts/backup.sh is a scheduled system task. The first five values are a clock: minute 0, hour 3, and the three asterisks mean "any day, any month, any day of the week". Taken together: the backup script runs automatically every day at 3:00 in the morning.

On the reliability of the format: it is the Unix standard, it has been in use for more than forty years and it is documented on the system itself. When a task is installed, the program validates the syntax and rejects the whole file if there is an error, rather than accepting it halfway. That means a badly written task does not get installed, instead of getting installed and running at the wrong time.

The one caveat is that the format says when the script is launched, it does not guarantee that the script works. Verifying that the backup is actually being made requires reviewing its execution log, something I suggest we set up once we have the monitoring system running.

All the information in this report came from two manual pages installed on the server itself.

You no longer depend on a search engine to work on Linux.

• A Linux system brings five documentation sources: man to understand, --help to recall, info to go deep, help for the builtins and /usr/share/doc to configure.
• You know how to read a manual page by jumping to what you need: EXAMPLES first when it exists, FILES to locate configuration, EXIT STATUS before scripting and SEE ALSO when you have picked the wrong tool.
• You understand the SYNOPSIS notation: brackets for the optional, ellipses for the repeatable, bars for the mutually exclusive, and several lines for several invocation forms.
• You know the eight sections, and in particular the difference between man 1 passwd and man 5 passwd, which is the trap everybody falls into.
• You can search without knowing the name with apropos and confirm with whatis.
• You move around inside less with /, n, N, g, G and q, shortcuts you will reuse for reading files and logs.
• You know why man cd does not exist and what to do instead.
• And you know how to read the exit code with $?, and that a 1 does not always mean an error, as grep demonstrates.

You have the interpreter and you have the manual. What is missing is the terrain. In the next lesson, Navigating the File System, you will go down into the tree you met in Module 1, but this time to walk it: pwd and the working directory, cd in all its forms including the ones almost nobody uses, ls broken down column by column, tree, stat with its three timestamps, file to know what a file really is, and du/df to see how much space /var/log/tramontana takes up. At the end you will do a complete, documented tour of the Tramontana Bookings tree. And when something does not add up, you already know where to look before asking.