You already know who you are and which machine you are on. What remains is the question that completes your orientation: where you are and what is around you.
When you open the file explorer on Windows you see C:, perhaps D:, and a Program Files folder where software is installed. On Linux none of that exists. There is a single tree growing from / and absolutely everything hangs from it: programs, configuration, your documents, the disks, and even information that is not on any disk at all, such as the processor's temperature.
This lesson gives you the map. By the end, when you see a path like /var/log/tramontana/access.log you will read it as a sentence with meaning, and when you have to decide where to put a new piece of software you will not do it on instinct but with the standard in hand. That is the difference between a server anyone can administer and a server where things are "wherever so-and-so put them".
Contents
- A single tree versus Windows drives
- What mounting means
- The FHS standard: a tour of the directories
- Static and variable, shareable and unshareable data
- Everything is a file: the seven types
- The pseudo-file systems /proc and /sys
- Hidden files and the dot convention
- Absolute and relative paths
- Where each piece of Tramontana goes and why
- A single tree versus Windows drives
The structural difference from Windows runs deep; it is not a matter of naming.
| Windows | Linux | |
|---|---|---|
| Model | Several trees, one per drive | A single tree from / |
| Root | C:\, D:\, E:\... |
/ and only / |
| Separator | Backslash \ |
Slash / |
| A new disk | Appears as a new letter | Is mounted on a directory of the tree |
| Software | C:\Program Files\ |
Distributed according to its nature |
| Case | Insensitive | Always sensitive |
On Windows, the structure of the storage is reflected in the paths: if a file is on the second disk, its path starts with D:. If that disk changes letter tomorrow, every path changes.
On Linux, the path describes what the file is, not where it physically sits. /var/log/tramontana/access.log means "a log file belonging to the Tramontana application". Whether it is on the first disk, the second, an LVM volume or a network share is irrelevant to whoever uses it and can change without any program noticing.
That independence between logical structure and physical layout is one of the best ideas in the Unix design, and the mechanism that makes it possible is called mounting.
- What mounting means
Mounting is connecting the contents of a storage device to a directory of the tree. That directory is called the mount point, and from the moment it is mounted, entering it means entering the device.
Look at it on your server:
Filesystem Size Used Avail Use% Mounted on /dev/sda2 23G 6.4G 15G 30% / /dev/sda1 1.1G 6.4M 1.1G 1% /boot/efi tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 389M 1.3M 388M 1% /run
How to read it: the partition /dev/sda2 is mounted on /, and /dev/sda1 on /boot/efi. When you access a file inside /boot/efi, you are reading from the second device, even though from your point of view you have merely entered a subdirectory.
flowchart TD
R["/ (root)<br/>partition /dev/sda2"]
R --> ETC["/etc"]
R --> VAR["/var"]
R --> HOME["/home"]
R --> BOOT["/boot"]
BOOT --> EFI["/boot/efi<br/>partition /dev/sda1 (FAT32)"]
R --> SRV["/srv"]
SRV --> BK["/srv/tramontana/backups<br/>could be a separate disk"]
R --> MNT["/mnt"]
MNT --> NAS["/mnt/nas<br/>could be a network share"]
style EFI fill:#e8f0fe,stroke:#4285f4
style BK fill:#e8f0fe,stroke:#4285f4
style NAS fill:#e8f0fe,stroke:#4285f4
The practical consequences of this model are enormous:
- You can add a new 2 TB disk and mount it on
/srv/tramontana/backups. The backup scripts carry on writing to the same path as always and never notice a thing. - You can mount a network share on
/mnt/nasand work with it as if it were local. - You can isolate risks: if
/varis on its own partition, a runaway log does not fill the root file system, as we saw in lesson 01-04.
Mounting in depth, the /etc/fstab file and automatic mounting are lesson 05-04.
- The FHS standard: a tour of the directories
The FHS (Filesystem Hierarchy Standard) is the document that defines what goes in each directory. It is maintained by the Linux Foundation and followed by every significant distribution.
Its value is enormous and very concrete: thanks to it, an administrator arriving at a Rocky Linux server knows the configuration is in /etc without having to look for it. The FHS is what makes Linux predictable.
Look at the root of your server:
bin dev home lib64 media opt root sbin srv tmp var boot etc lib lost+found mnt proc run sbin.usr-is-merged sys usr
Full reference table
| Directory | Name | Contents | Do you touch anything here? |
|---|---|---|---|
/ |
root | The origin of the tree | Not directly |
/bin |
binaries | Essential commands for all users | No |
/sbin |
system binaries | Administration commands (they require root) | No |
/lib, /lib64 |
libraries | Shared libraries and kernel modules | No |
/usr |
Unix System Resources | Most of the installed software | Not by hand |
/usr/bin |
Non-essential user commands | No | |
/usr/sbin |
Non-essential administration commands | No | |
/usr/local |
Software compiled and installed manually | Yes | |
/usr/share |
Architecture-independent data: documentation, icons | Rarely | |
/etc |
et cetera | All the system configuration, in plain text | Yes, a lot |
/var |
variable | Data that changes: logs, queues, caches, databases | Yes |
/var/log |
The system and application logs | Yes, a lot | |
/home |
Users' home directories | Yes | |
/root |
The home directory of root (not to be confused with /) |
Rarely | |
/opt |
optional | Self-contained third-party software | Yes |
/srv |
service | Data served by this server | Yes |
/tmp |
temporary | Temporary files; wiped on reboot | Yes, in passing |
/boot |
Kernel, initramfs and the GRUB boot loader | With care | |
/dev |
devices | Device files: disks, terminals, ports | For reference |
/proc |
processes | Pseudo-file system: processes and kernel information | Read only |
/sys |
system | Pseudo-file system: devices and kernel parameters | With great care |
/run |
Runtime data: PIDs, sockets. In RAM | No | |
/mnt |
mount | A temporary, manual mount point | Yes, occasionally |
/media |
Automatic mounting of removable media: USB, CD | Automatic |
The ones you will use most
Five directories account for 90 % of your work. They are worth detailing.
/etc — the configuration. Everything that configures the system and its services is here, and almost always in plain text. It is the most valuable directory on a server: if you back up /etc, you can rebuild the machine's configuration.
Every name is a piece you will study: passwd and group (users, lesson 05-01), fstab (mounts, 05-04), crontab (scheduled tasks, 03-07), ssh (remote access, 06-02), netplan (networking, 06-01).
And one implication of it being plain text that is worth appreciating now: /etc can be versioned with Git. Many administrators do exactly that, and so they have the complete history of configuration changes on their servers, with who changed what and when.
/var — what grows. Variable: data that changes during normal operation.
| Subdirectory | Contents |
|---|---|
/var/log |
System and application logs |
/var/lib |
Persistent state of services (databases, for example) |
/var/cache |
Caches that can be deleted without losing data |
/var/spool |
Queues: printing, mail, cron jobs |
/var/tmp |
Temporary files that survive a reboot, unlike /tmp |
/var/www |
Web content served by Apache or nginx |
It is the directory an administrator watches most closely, because it is the one that fills up.
/home — the users. Every user has their own: /home/operator, /home/student. It is the only place where a normal user can write freely, and where /home/operator/scripts will live.
/opt and /usr/local — software that does not come from the package manager. There is a fine distinction here that confuses many people:
/opt |
/usr/local |
|
|---|---|---|
| Philosophy | Each application in its own subdirectory, self-contained | The system structure is replicated (bin, lib, share) |
| Structure | /opt/tramontana/ with everything inside |
/usr/local/bin/program, /usr/local/lib/... |
| Typical of | Commercial software, deployed applications | Software compiled from source |
| Uninstalling | Delete one directory | Track down the scattered files |
/srv — served data. It is the least used and the worst understood. The FHS defines it as the data this server serves to third parties: websites, FTP, shares, backups. The difference from /var/www is that /srv is meant to be organised by service or by client, and many distributions leave it empty so that the administrator can structure it.
The visual tree
flowchart TD
ROOT["/"]
ROOT --> BIN["bin, sbin, lib<br/>→ links to /usr"]
ROOT --> USR["usr<br/>system software"]
ROOT --> ETC["etc<br/>configuration"]
ROOT --> VAR["var<br/>data that changes"]
ROOT --> HOME["home<br/>users"]
ROOT --> OPT["opt<br/>third-party software"]
ROOT --> SRV["srv<br/>served data"]
ROOT --> VIRT["proc, sys, dev, run<br/>virtual / in RAM"]
ROOT --> BOOT["boot<br/>booting"]
USR --> UB["bin · sbin · lib"]
USR --> UL["local<br/>compiled by hand"]
USR --> USH["share<br/>docs, icons"]
VAR --> VL["log<br/>LOGS"]
VAR --> VLIB["lib<br/>service state"]
VAR --> VC["cache · spool"]
HOME --> HO["operator"]
OPT --> OT["tramontana/app"]
SRV --> ST["tramontana/backups"]
VL --> VLT["tramontana/<br/>access.log · errors.log"]
style OT fill:#fff4e5,stroke:#f59e0b
style ST fill:#fff4e5,stroke:#f59e0b
style VLT fill:#fff4e5,stroke:#f59e0b
A note on /bin, /sbin and /lib
If you list the root in detail you will see something curious:
lrwxrwxrwx 1 root root 7 Apr 22 13:08 bin -> usr/bin lrwxrwxrwx 1 root root 7 Apr 22 13:08 lib -> usr/lib lrwxrwxrwx 1 root root 9 Apr 22 13:08 lib64 -> usr/lib64 lrwxrwxrwx 1 root root 8 Apr 22 13:08 sbin -> usr/sbin
That leading l and the arrow indicate that they are not real directories but symbolic links to their equivalents inside /usr. This is the so-called usr merge, completed in Ubuntu and in most modern distributions.
Historically, /bin contained what was essential for booting and repairing the system (because /usr could be on another partition or even on the network and unavailable early in the boot), and /usr/bin held the rest. Today initramfs handles early boot and that separation no longer adds anything, so it was unified. The links are kept so that the old paths in thousands of scripts keep working.
It is a good example of how the FHS evolves while preserving compatibility.
- Static and variable, shareable and unshareable data
The FHS is not an arbitrary list of names. Behind it there are two axes of classification, and understanding them lets you deduce where each thing goes instead of memorising it.
| Shareable (several machines can use the same ones) |
Unshareable (specific to each machine) |
|
|---|---|---|
| Static (does not change without intervention) |
/usr, /opt |
/etc, /boot |
| Variable (changes by itself, in operation) |
/var/mail, /home |
/var/log, /var/run, /proc |
The two axes:
- Static versus variable. Static is what only changes when an administrator installs or modifies something: the programs in
/usr, the configuration in/etc. Variable is what changes by itself during operation: the logs in/var/log, the queues, the caches. - Shareable versus unshareable. Shareable is what it would make sense to serve over the network to several machines: the binaries in
/usrare identical on every server running the same distribution. Unshareable is what is specific to one machine: its network configuration, its hostname, its logs.
Why this is genuinely useful
Four practical consequences that come up constantly in administration:
-
It determines what you back up and how often.
/etc(static, unshareable) is small and extremely valuable: copy it daily./usr(static, shareable) does not need backing up, it is reinstalled with the package manager./var/log(variable) is backed up or rotated according to the retention policy. -
It determines what can be mounted read-only. On hardened servers,
/usris mounted read-only because it must not change during operation. Any attempt to write there is a sign of compromise./varand/etc, by contrast, must be writable. -
It determines what needs its own partition. Variable data grows unpredictably: that is why
/vardeserves its own partition, as we saw in lesson 01-04. Static data has a bounded size. -
It determines where to put your own things. And here is the direct application: the Tramontana application is static (it only changes when Luis deploys a new version), its configuration is static and unshareable, and its logs are variable and unshareable. Three different natures demanding three different locations. We wrap that up in section 9.
- Everything is a file: the seven types
You already saw the principle in lesson 01-02. Now it is time to see it in action.
On Linux, almost everything is presented with the same file interface, so that the same syscalls (open, read, write, close) and the same tools work for radically different things.
There are seven types, and ls -l identifies them with the first letter of each line:
| Letter | Type | What it is | Example |
|---|---|---|---|
- |
Regular file | Data: text, binaries, images | /etc/hostname |
d |
Directory | A container for other files | /home |
l |
Symbolic link | A pointer to another path | /bin → usr/bin |
c |
Character device | Read and written byte by byte, unbuffered | /dev/null, /dev/tty1 |
b |
Block device | Accessed in blocks, buffered | /dev/sda |
s |
Socket | Communication between processes | /run/systemd/private |
p |
Named pipe (FIFO) | A data channel between processes | Uncommon |
Check it:
lrwxrwxrwx 1 root root 7 Apr 22 13:08 /bin -> usr/bin -rw-r--r-- 1 root root 15 Aug 18 09:02 /etc/hostname drwxr-xr-x 4 root root 4096 Aug 18 08:41 /home crw-rw-rw- 1 root root 1, 3 Aug 18 09:02 /dev/null brw-rw---- 1 root disk 8, 0 Aug 18 09:02 /dev/sda
Notice two revealing details:
- The first letter of each line is the type:
l,-,d,c,b. - On the device lines, where ordinary files show the size, two numbers appear (
1, 3and8, 0). They are the major and minor numbers: the major identifies which kernel driver manages the device, and the minor identifies which of them it is. A device has no size because it contains no data: it is a door to a driver.
The special devices
Three files in /dev that you will use constantly.
/dev/null — the black hole. Anything you write there disappears; reading it returns end of file immediately.
There is no output. Neither operation produces anything.
Its usefulness is a daily matter: discarding output you do not care about. You already used it without knowing in lesson 01-04, when you verified the ISO with sha256sum -c SHA256SUMS 2>/dev/null to discard the error messages for the images you had not downloaded. Redirection is lesson 03-04.
/dev/zero — the infinite source of zeros. Reading it returns zero bytes without limit. It is used to create fixed-size files or to wipe data.
head -c 20 takes 20 bytes and od -c displays them readably: twenty zeros.
/dev/sda — the whole disk. It is not a metaphor: it is the disk. You can read it byte by byte and, if you are root, write to it. It is what let dd write an ISO to a USB stick in lesson 01-04, and also what makes that command so dangerous.
/dev/random and /dev/urandom generate cryptographically secure random data, and they are the basis of the key generation you will see in Module 6.
Why this principle matters
This list may look like a curiosity, but it has a consequence that runs through the whole course: the same tools work for everything.
cat can display a text file, the processor's information (/proc/cpuinfo) or the bytes of a disk. grep can search in a log, in another program's output or in a kernel pseudo-file. Redirecting a program's output to a file, to a device or to another program is exactly the same operation.
When you chain tools with pipes in Module 3, that uniformity is what makes it possible. On a system where disks, processes and the network each had their own API, you would need different programs for each thing.
- The pseudo-file systems /proc and /sys
/proc and /sys are the most spectacular application of the previous principle. They take up not a single byte on disk: they are virtual file systems the kernel generates in memory at the moment you read them.
Every time you open /proc/meminfo, the kernel builds that "file" on the fly with the current state of memory. It is a query interface disguised as a file system.
/proc |
/sys |
|
|---|---|---|
| Origin | The 1990s, inherited from Unix | 2002, kernel 2.6 |
| Contents | Processes + general kernel information | Devices, drivers and kernel parameters |
| Organisation | Historical, somewhat untidy | Hierarchical and systematic |
| Typical use | Diagnostics and information | Configuring hardware and parameters |
/proc in practice
Processor information:
grep -m1 shows only the first match, because the file repeats the information for every core.
Memory information:
Here is the interesting fact: this is exactly where free -h gets its numbers from. free is nothing more than a program that reads /proc/meminfo and presents it readably. The same goes for uptime and /proc/uptime, or for ps and the /proc/<pid>/ directories.
Verifying this changes the way you see the system: diagnostic tools have no special powers, they simply read files that you can read too.
One specific process. Every running process has a directory in /proc named after its PID:
/proc/1/ is process 1, that is, systemd. What is inside:
| File | Contents |
|---|---|
cmdline |
The command line it was launched with |
cwd |
A link to its current working directory |
environ |
Its environment variables |
exe |
A link to the binary it is running |
fd/ |
The files it has open right now |
status |
State, user, memory used, threads |
(The tr '\0' ' ' replaces the null characters separating the arguments with spaces, to make it readable.)
Other useful files in /proc:
| Path | Contents |
|---|---|
/proc/version |
The full kernel version |
/proc/uptime |
Seconds of uptime |
/proc/loadavg |
Average load (the source for uptime) |
/proc/mounts |
Mounted file systems |
/proc/partitions |
Detected partitions |
/proc/sys/ |
Kernel parameters modifiable on the fly |
/sys and modifying on the fly
/sys exposes devices and kernel parameters, and many of its files are writable. Changing one modifies the kernel's behaviour instantly, without rebooting.
That 0 means IP packet forwarding is disabled: the server does not act as a router. Enabling it would be as simple as writing a 1 into that file.
This mechanism is what makes kernel tuning possible, and it is the subject of lesson 07-03. It is mentioned here only so that you understand the nature of what you are looking at.
A clear warning: /proc and /sys are safe to read — do it fearlessly, it is the best way to understand what your system is doing. Writing to them without knowing what you are doing can degrade performance or destabilise the machine. During this module, stick to looking.
- Hidden files and the dot convention
On Linux, a file whose name begins with a dot is hidden. There is no special attribute as on Windows: it is just a naming convention that the tools respect.
. .bash_history .bashrc .profile .ssh scripts .. .bash_logout .cache .local .sudo_as_admin_successful
-a stands for all. Many more items appear.
The first two are special and will always be present in any directory:
.is the current directory...is the parent directory.
They are not decoration: they are real directory entries, and they are the basis of the relative paths in the next section.
The usual hidden files in your home directory:
| File | Contents |
|---|---|
.bashrc |
Bash configuration for interactive sessions: aliases, prompt, functions |
.profile |
Runs at login: environment variables |
.bash_history |
Command history (the one you scroll through with the arrows) |
.ssh/ |
SSH keys and configuration |
.cache/, .local/ |
Caches and data of the user's applications |
The purpose of the convention is purely practical: personal configuration is stored in the user's directory, but it must not get in the way when they list their documents. Programs put their configuration behind a leading dot and it disappears from view.
An important detail that already follows from the table: a hidden directory can contain perfectly visible files. .ssh is hidden, but ls -a ~/.ssh shows its contents normally. Hiding affects the item, not what is inside it.
And an operational warning: when backing up a home directory, you must make sure to include the hidden files. It is a classic source of incomplete backups in which all the configuration and the SSH keys are lost.
- Absolute and relative paths
There are two ways of naming a file, and the distinction is as simple as it is fundamental.
Absolute path: it starts at the root / and describes the complete route. It is the same wherever the person writing it happens to be.
Relative path: it starts at the current directory. Its meaning depends on where you are.
scripts/backup.sh ← from /home/operator ../log/tramontana/ ← from /var/lib ./backup.sh ← the file backup.sh in the current directory
The symbols used in paths:
| Symbol | Meaning |
|---|---|
/ (at the start) |
The root: the path is absolute |
/ (in the middle) |
A separator between directories |
. |
The current directory |
.. |
The parent directory |
~ |
The current user's home directory |
~operator |
The home directory of operator |
- |
The previous directory (with cd) |
When to use each one:
| Situation | Recommendation |
|---|---|
| Scripts and cron jobs | Always absolute. A script does not control where it is run from |
| Configuration files | Absolute |
| Interactive work | Relative, they are more convenient |
| References inside a project | Relative, so the project is portable |
The first row is the one that prevents the most grief. A script using the relative path logs/access.log works when you test it from your own directory, and fails mysteriously when cron runs it from somewhere else. It is one of the most frequent mistakes when starting out with automation, and we will come back to it in lessons 03-07 and 04-07.
The practical use of paths — moving around with cd, knowing where you are with pwd, listing with ls — is lesson 02-03. Here you only needed the definition.
- Where each piece of Tramontana goes and why
The moment has come to apply the whole map. Luis Ferrer is ready to deploy Tramontana Bookings on srv-tramontana and asks you where to put each thing. The answer is not an opinion: it follows from the FHS and from the static/variable classification.
| Piece | Location | Nature | FHS justification |
|---|---|---|---|
| Application | /opt/tramontana/app |
Static, shareable | /opt is for self-contained third-party software packages, outside the package manager |
| Configuration | /etc/tramontana/ |
Static, unshareable | /etc is the configuration specific to this machine |
| Logs | /var/log/tramontana/ |
Variable, unshareable | /var/log is the place for logs; it grows during operation |
| Backups | /srv/tramontana/backups |
Variable, shareable | /srv is for data the server serves or holds in custody |
| Scripts | /home/operator/scripts |
Variable, unshareable | The administrator's personal tools |
Let us go one by one, because the reasoning matters more than the conclusion.
/opt/tramontana/app — the application
Tramontana Bookings does not come from the Ubuntu repositories: it is in-house software that Luis deploys manually. The FHS reserves /opt for exactly that, with the convention /opt/<vendor>/<product>.
It is static because it only changes when a new version is deployed, not during operation. And it is shareable: if there were a second application server tomorrow, it would have exactly the same content at that path.
An additional advantage of the self-contained nature of /opt: uninstalling means deleting one directory, and backing up the application means copying one directory.
A rejected alternative: /usr/local. It is correct for compiled software that integrates into the system, spreading binaries into /usr/local/bin and libraries into /usr/local/lib. For a self-contained web application, /opt is cleaner.
/etc/tramontana/ — the configuration
This is where the database connection string, the application's parameters and its credentials go. /etc is, by the FHS definition, static configuration specific to this machine.
That last part is the key, and it is what justifies separating the configuration from the application instead of leaving it inside /opt/tramontana/app/config:
- The test server and the production server run the same code with different configuration. Separating them allows the same artefact to be deployed to both.
- A backup of
/etcautomatically picks up the configuration of every service, ours included. - When the application is updated (by deleting and recreating
/opt/tramontana/app), the configuration is not at risk at all. - The configuration contains secrets and needs restrictive permissions, different from those of the code. Permissions are lesson 02-07 and secrets are 06-05.
/var/log/tramontana/ — the logs
access.log and errors.log are the perfect example of variable data: they grow continuously without anyone intervening.
Putting them in /var/log is not a formality. It has very concrete consequences:
- It is where the log rotation tools (
logrotate) expect to find them, so they can compress and delete the old ones automatically. Without that, a log grows until it fills the disk. - It is where any administrator will look first when investigating a problem.
- As we saw in lesson 01-04,
/varcan be on its own partition: if the logs run away, the root file system is unaffected.
A rejected alternative: leaving them inside /opt/tramontana/app/logs, which is what many applications do by default. It is a classic mistake: it mixes static data with variable data, breaks automatic rotation, complicates backups and makes an overflow hit the wrong partition.
/srv/tramontana/backups — the backups
/srv is the directory for data this server serves or holds in custody, organised by service.
And there is a weighty operational reason to keep them separate: being in their own top-level directory, /srv/tramontana/backups can be mounted on a different physical disk with a single line in /etc/fstab, without touching a single path in the scripts. It is exactly the scenario from section 2 on mounting.
And that matters a great deal, because a backup on the same disk as the original data protects against accidental deletion, but not against disk failure. The complete backup strategy is lesson 05-08.
/home/operator/scripts — your tools
The automation scripts you will write in Module 4 are yours: your home directory is their natural place while you are developing them.
A professional nuance worth anticipating: when a script stops being an experiment and becomes part of operations — cron runs it, a backup depends on it — it should no longer live in a personal directory. Its place becomes /usr/local/bin or /opt/tramontana/bin, because if operator ceases to exist tomorrow or their directory is cleaned out, operations cannot depend on that. We will see it in lesson 04-07.
The result
This is how srv-tramontana looks once the deployment is complete:
/
├── etc/
│ └── tramontana/ ← configuration (static, for this machine)
│ └── app.conf
├── opt/
│ └── tramontana/
│ └── app/ ← the application (static, shareable)
├── var/
│ └── log/
│ └── tramontana/ ← logs (variable, they grow)
│ ├── access.log
│ └── errors.log
├── srv/
│ └── tramontana/
│ └── backups/ ← backups (could be another disk)
│ └── bookings.csv
└── home/
└── operator/
└── scripts/ ← your toolsFive directories, five different natures, five justifications. Any Linux administrator in the world would understand this structure without anyone explaining it. That is what following a standard gives you.
Common Mistakes and Tips
- Confusing
/rootwith/./is the root of the tree;/rootis the home directory of the root user. They sound alike and they are different things. - Looking for a Linux "Program Files". There is no single place: software is distributed according to its nature (
/usr/binfor package binaries,/optfor third-party applications,/usr/localfor hand-compiled things,/etcfor their configuration). - Installing your own software in
/usr/bin. That directory belongs to the package manager. Anything you put there by hand can be overwritten or removed in an update. Use/usr/local/binor/opt. - Leaving the logs inside the application's directory. It breaks automatic rotation, mixes static with variable data and puts the wrong partition at risk.
- Storing important data in
/tmp. It is cleaned on reboot and, on many systems, periodically as well. If you need temporary files that survive, use/var/tmp. - Writing to
/procor/syswithout understanding what it does. Reading them is safe and educational; writing to them can destabilise the system instantly. - Forgetting hidden files when copying a home directory. You take the documents and leave behind the configuration and the SSH keys.
ls -abefore copying. - Using relative paths in scripts. They work when you test them and fail when cron runs them from another directory. In scripts, absolute paths.
- Tip. When you are unsure where to put something, ask yourself two questions: does this change by itself during operation? (if so, it goes under
/var) and is it specific to this machine? (if so,/etcfor configuration). Those two questions settle most cases. - Tip. Spend a while exploring
/etc,/var/logand/procwithlsandcat. There is no risk in reading, and familiarity with the terrain is gained by walking it. It is the best investment of time before starting Module 2. - Tip. The full FHS is published and is surprisingly readable. When you have a real doubt about where something goes, consulting it gives you an authoritative answer, not an opinion.
Exercises
Exercise 1
For each item, state which directory it should go in according to the FHS, and justify your answer using the static/variable and shareable/unshareable classification:
- The TLS certificate for the Tramontana Bookings domain.
- A PostgreSQL database holding the bookings.
- A script Marta runs every morning to generate a report.
- The file
houses.txtwith the catalogue of rural cottages that the application serves to the agencies. - A 3 GB temporary file generated during a migration, which must survive a reboot.
- A hand-compiled version of nginx with special modules.
Exercise 2
You run ls -l in a directory and get this output:
drwxr-xr-x 3 operator tramontana 4096 Aug 18 09:14 reports -rw-r--r-- 1 operator tramontana 15243 Aug 18 09:12 bookings.csv lrwxrwxrwx 1 operator tramontana 24 Aug 18 09:15 current -> /var/log/tramontana brw-rw---- 1 root disk 8, 16 Aug 18 08:41 data-disk crw-rw-rw- 1 root root 1, 3 Aug 18 08:41 empty srw-rw-rw- 1 root root 0 Aug 18 08:41 app.sock
- Identify the type of each item and explain how you determined it.
- Why do two of them show two numbers where the others show a size? What do those numbers mean?
- What would happen if you deleted the item
current? And if you deleted/var/log/tramontana?
Exercise 3
Luis Ferrer proposes this structure for deploying Tramontana Bookings:
/home/luis/tramontana/ ├── app/ ├── config/ │ └── app.conf ├── logs/ │ ├── access.log │ └── errors.log └── backups/
Argue, point by point, four concrete problems with this proposal, and present the alternative structure with its FHS justification. For each problem, describe a real scenario in which it would cause an incident.
Solutions
Solution to Exercise 1
1. TLS certificate → /etc/ssl/certs/ (or /etc/tramontana/ssl/).
Static and unshareable. It is configuration: it only changes when it is renewed, and it is specific to this machine and its domain. It goes in /etc by definition. The associated private key must go in /etc/ssl/private/ with very restrictive permissions, because whoever obtains it can impersonate the server. It is covered in lesson 06-05.
2. PostgreSQL database → /var/lib/postgresql/.
Variable and unshareable. It changes continuously with every booking; it is the persistent state of a service, which is exactly the definition of /var/lib. It is the location PostgreSQL uses by default on Ubuntu, and respecting it ensures that backup tools and SELinux or AppArmor policies expect to find it there.
3. Daily report script → /usr/local/bin/ (or /opt/tramontana/bin/).
This one has a nuance. While it is being developed, /home/operator/scripts is fine. But the exercise says that Marta runs it every morning: it is an operational tool somebody else depends on. It should move to /usr/local/bin, which is the FHS place for local executables added by the administrator, with two advantages: it is in every user's PATH (Marta invokes it by name, with no path) and it does not depend on the operator account continuing to exist.
4. houses.txt served to the agencies → /srv/tramontana/.
Variable and shareable. The key word in the statement is serves: it is data this server delivers to third parties, which is the literal definition of /srv in the FHS. If it were served exclusively over the web through nginx, /var/www/ would also be defensible; /srv is preferable when the data is served through several channels or conceptually belongs to the service.
5. A 3 GB temporary file that survives a reboot → /var/tmp/.
Both requirements point to the same place. /tmp is ruled out because it is emptied on reboot (and on many systems it is a tmpfs in RAM, where 3 GB would be a serious memory problem). /var/tmp is designed precisely for temporary files that persist across reboots, and it is on disk.
6. Hand-compiled nginx → /usr/local/.
Static and shareable. /usr/local is the parallel tree reserved for software the administrator compiles and installs manually: the binary in /usr/local/sbin/nginx, the configuration in /usr/local/etc/nginx/. Its whole reason for existing is not to collide with /usr, which belongs to the package manager: that way an Ubuntu update will never overwrite your build, and your build will never break the official package.
Solution to Exercise 2
1. Types, determined by the first letter of each line:
| Item | Letter | Type |
|---|---|---|
reports |
d |
Directory |
bookings.csv |
- |
Regular file |
current |
l |
Symbolic link (the -> arrow confirms it and shows the target) |
data-disk |
b |
Block device |
empty |
c |
Character device |
app.sock |
s |
Socket |
Details that confirm each reading: the link shows -> /var/log/tramontana and has rwxrwxrwx permissions (symbolic links always do; the ones that count are the target's). The block device belongs to the disk group, which is usual for disks. And empty with numbers 1, 3 is in fact /dev/null: those are its characteristic major and minor numbers.
2. The two numbers.
They are shown by data-disk (8, 16) and empty (1, 3), the two devices. On a normal file, that column shows the size in bytes; a device has no size because it contains no data: it is an access door to a kernel driver.
In its place appear:
- The major number identifies the kernel driver managing the device.
8corresponds to SCSI/SATA disks;1to memory devices, among which is/dev/null. - The minor number identifies which of the devices managed by that driver it is.
16corresponds to/dev/sdb(0would be/dev/sda);3to/dev/nullspecifically.
In other words, data-disk is the system's second disk and empty is the black hole.
3. Consequences of deleting.
If you delete current: nothing of consequence happens. A symbolic link is only a pointer, a tiny file containing a path as text. Deleting it removes the shortcut; the directory /var/log/tramontana and all its contents remain intact. You can recreate the link in a second.
If you delete /var/log/tramontana: you really do lose the logs. And on top of that the current link is left dangling: it still exists and still points to a path that no longer does. Any program trying to follow it will get a "No such file or directory" error, which is confusing because the link does appear when you list the directory.
Dangling links are easy to spot because ls shows them in flashing red on most colour terminals.
This asymmetry — the link depends on the target, the target does not depend on the link — is the essential characteristic of symbolic links, and it explains how they differ from hard links. That is the subject of lesson 02-06.
Solution to Exercise 3
Problem 1: everything hangs from a personal directory.
The entire production infrastructure depends on the luis account existing. /home is reserved for users' personal data, not for services.
Incident scenario: Luis leaves the company. The normal offboarding procedure is followed: remove the account with userdel -r, which deletes the home directory. The production application, its configuration, its logs and its backups are all deleted at once. Nobody connected offboarding an employee with destroying the service, because in a well-organised system there is no reason for a connection.
An equivalent and quieter incident: the permissions on /home/luis restrict access, and the web service user (www-data) cannot read the application. It ends up being solved with excessively open permissions on a personal directory, which opens a security hole.
Problem 2: the logs live with the application.
logs/ is inside the same tree as app/, mixing variable data with static data.
Incident scenario: a bug in the code causes a loop that writes to errors.log non-stop. The log grows until it fills the partition holding /home, which in many installations is the same as /. With the root disk full, the system cannot write temporary files: services start failing in cascade, journald cannot record anything and it may not even be possible to log in to diagnose it. With the logs in /var/log and /var on its own partition, the damage would have stayed contained.
A second effect: logrotate does not manage those logs, because they are not where it expects them. Nobody compresses or deletes them, so they grow indefinitely.
Problem 3: the configuration is inside the application.
config/app.conf hangs from the same tree as the code.
Incident scenario: Luis deploys version 2.0 by deleting and recreating the tramontana/ directory. app.conf is lost, along with the database connection string and the credentials. The service does not start and the configuration has to be rebuilt from memory, in the middle of an outage, under the pressure of having the service down.
An associated structural problem: not being separate, the same artefact cannot be deployed to test and to production with different configurations. You end up maintaining two divergent copies of the code, which is how the bugs that "only happen in production" appear.
Problem 4: the backups are in the same tree as the data.
backups/ is inside /home/luis/tramontana/, that is, on the same disk and under the same directory it protects.
Incident scenario: the disk fails. The original data and all the backups are lost together. A misdirected rm -rf on /home/luis/tramontana/ is equally enough: it takes out both the object of the backup and the backup itself.
A backup that shares a destination with the original protects only against the selective deletion of a file. It does not protect against hardware failure, or against a wide deletion, or against ransomware encryption, which are precisely the scenarios that justify having backups in the first place.
Alternative structure:
| Piece | Correct path | FHS justification |
|---|---|---|
| Application | /opt/tramontana/app |
Self-contained third-party software; static and shareable |
| Configuration | /etc/tramontana/app.conf |
Static configuration specific to this machine; backed up with /etc; survives deployments |
| Logs | /var/log/tramontana/ |
Variable data; managed by logrotate; can be isolated on its own partition |
| Backups | /srv/tramontana/backups |
Data held in custody by the server; mountable on a different disk with one line in /etc/fstab |
How to present it to Luis (because being right is not enough): the argument that convinces is not "the FHS says so", but the four incident scenarios. The correct structure is not more bureaucratic: it is the one that means offboarding an employee does not take the service down, that a runaway log does not fill the root disk, that a deployment does not delete the credentials and that the backups are worth something when the disk fails. Besides, setting it up properly from the start costs exactly the same as setting it up badly.
Conclusion
With this lesson you close Module 1 and you have the complete map of the territory:
- Linux organises storage in a single tree from
/, not in separate drives, and mounting is what connects devices to points on that tree, separating the logical structure from the physical layout. - The FHS defines what goes in each directory and is what makes any Linux system predictable:
/etcfor configuration,/varfor what grows,/usrfor system software,/optfor third-party applications,/srvfor what is served,/homefor users. - The static/variable and shareable/unshareable classification is not theory: it determines what you back up, what you mount read-only, what needs its own partition and where you put your own things.
- Everything is a file, and
ls -ltells you which type by its first letter:-,d,l,c,b,s,p. Devices such as/dev/null,/dev/zeroand/dev/sdaare doors to kernel drivers. /procand/systake up no disk: they are windows onto the kernel generated on the fly, and they are the source that tools such asfree,uptimeandpsdrink from.- Hidden files are only a naming convention, and
.and..are real entries in every directory. - Absolute paths start at
/and are the ones you must use in scripts; relative paths start wherever you are and are convenient when working by hand. - And you can justify, standard in hand, why Tramontana Bookings goes in
/opt/tramontana/app, its configuration in/etc/tramontana/, its logs in/var/log/tramontana/, its backups in/srv/tramontana/backupsand your scripts in/home/operator/scripts.
Take stock of where you were when the module started and where you are now. You know what Linux is and what exactly the kernel is; you know the Unix philosophy that explains the design of everything you will see; you understand the distribution landscape and why Ubuntu Server 24.04 LTS is the right choice for srv-tramontana; you have the server installed, verified, updated and with snapshots; you can read the prompt, run basic commands and carry out reconnaissance on a machine; and you have the complete map of its file system. That is a working lab and a shared vocabulary with any systems administrator.
So far you have been looking. In Module 2: Basic Commands you will start working for real: you will learn to handle the command line fluently, to consult the system documentation so you do not depend on search engines, to navigate the directory tree you have just got to know, to create, copy, move and delete files, to view and edit their contents, to handle hard and symbolic links, and to master permissions and ownership, which is the piece separating a server that works from a server that is also secure. Prepare your snapshot, open a session on srv-tramontana and I will see you in the first lesson.
Linux Course: From Beginner to System Administrator
Module 1: Introduction to Linux
- What Is Linux?
- History of Linux
- Linux Distributions
- Installing Linux
- First Contact with the System
- The Linux File System Structure
Module 2: Basic Linux Commands
- Introduction to the Command Line
- Getting Help and System Documentation
- Navigating the File System
- File and Directory Operations
- Viewing and Editing Files
- Hard and Symbolic Links
- File Permissions and Ownership
Module 3: Advanced Command-Line Skills
- The Shell Environment: Variables, Aliases and History
- Using Wildcards and Regular Expressions
- Searching Files and Content: find, locate and grep
- Pipes and Redirection
- Text Processing: cut, sort, uniq, sed and awk
- Process Management
- Scheduling Tasks with Cron
- Networking Commands
Module 4: Shell Scripting
- Introduction to Shell Scripting
- Variables and Data Types
- Script Input, Output and Arguments
- Control Structures
- Functions and Libraries
- Debugging and Error Handling
- Production Scripts: Best Practices
Module 5: System Administration
- User and Group Management
- sudo and Special Permissions
- Package Management
- Disk Management
- systemd and Service Management
- System Logs: journald and syslog
- System Monitoring and Performance Tuning
- Backup and Restore
Module 6: Networking and Security
- Network Configuration
- SSH and Remote Access
- Firewalls and Perimeter Security
- Intrusion Detection Systems
- Secrets Management and TLS Certificates
- Securing Linux Systems
Module 7: Advanced Topics
- The Boot Process and System Recovery
- Advanced Diagnostics: strace, perf and eBPF
- Linux Kernel Tuning
- Virtualization with Linux
- Linux Containers and Docker
- Automation with Ansible
- High Availability and Load Balancing
