Throughout this module you have worked with an implicit assumption: that each name corresponds to one file and each file has one name. That assumption is false, and understanding why is what separates somebody who uses Linux from somebody who understands it.

In a Unix file system, the name and the content are separate things. The content lives in a structure called an inode; the name is simply an entry in a directory pointing at that inode. Nothing stops two entries pointing at the same place, nor stops a file existing whose content is the path of another. That is where the two kinds of link in this lesson come from.

This is not file system theory for curiosity's sake. Links are everywhere on a Linux system: /bin is a link, your python3 is a link, and the system with which Debian and Ubuntu decide which version of a tool gets used is built on links. And at the end of the lesson you will design the pattern with which Tramontana Bookings will turn a deployment — and a rollback — into an instantaneous change of link, with no service outage window.

1. What an inode is
2. Seeing the inode and the link count
3. Hard links with ln
4. The two limitations of hard links
5. Symbolic links with ln -s
6. Absolute versus relative
7. Broken links
8. Which commands follow the link and which do not
9. Hard versus symbolic: the table
10. Links in a real Linux system
11. update-alternatives
12. Tramontana's deployment pattern

When you save a file, the file system creates three distinct things:

1. The data blocks: the content itself, spread across the disk.
2. The inode: a structure with all the file's metadata — permissions, owner, group, size, timestamps, link count and the addresses of the data blocks. Each inode has a unique number within its file system.
3. The directory entry: a pair (name, inode number) stored in the directory.

Notice what the inode does not contain: the file's name. The name is not a property of the file; it is a property of the directory containing it.

A directory, seen this way, does not contain files: it contains a list of names with inode numbers. It is an index.

This architecture explains at a stroke several things that until now were loose facts:

That second-to-last point is the definitive explanation of the mystery from lesson 02-03: you delete a 2 GB log, du stops seeing it, but df still says the space is occupied. The name has gone; the inode is still alive because the process has it open, and with it its blocks. Until the process closes the file — or you restart it — nothing is freed.

Two tools you already know, now with links in mind.

The first column, thanks to -i, is the inode number. The column immediately after the permissions — the 1 — is the link count: how many names point to this inode.

stat spells it out:

Inode: 262149 and Links: 1. That counter is what governs when a file's space is released, as you will see in the next section.

And now the behaviour of directories that was left pending in lesson 02-03 makes sense:

A count of 4, not 1. The four names pointing at that inode are:

1. The app entry inside /opt/tramontana.
2. The . entry inside /opt/tramontana/app.
3. The .. entry inside /opt/tramontana/app/templates.
4. The .. entry inside another subdirectory.

Hence the formula: a directory's links = 2 + number of subdirectories. A directory with no subdirectories has 2; one with seven subdirectories has 9.

An inode number only makes sense within its own file system. Two files on different disks can have the same number with no relationship whatsoever. That is why, to compare, you also have to look at the device:

The device:inode pair does identify a file uniquely across the whole machine.

A hard link is, quite simply, another directory entry pointing at the same inode. It is not a copy nor a shortcut: it is another, equally legitimate name for the same file.

Three things to confirm in that output:

1. The same inode (262149) on accommodation.txt and houses.txt.
2. The count went up to 2 on both: there are two names pointing there.
3. The size is shown twice (418 and 418), but the file takes up 418 bytes in total, not 836. ls shows the inode's size, and the inode is the same one.

That last point has a practical consequence: du counts the space only once if it finds both names in the same walk, but it can count it twice if they are in different walks. It is a classic source of discrepancies when working out the size of backups that use hard links.

There is no original and no copy. The two names are exactly equivalent; the system does not record which was created first. Modifying one modifies the other, because they are the same file:

And the same goes for the metadata: chmod on one changes the permissions of the other, because the permissions live in the inode.

Here is the behaviour you have to understand properly:

The content is still intact. rm does not delete files: it decrements the link count and removes the name. The inode and its blocks are only released when two things happen at once:

• The link count reaches 0.
• No process has the file open.

That is why the low-level command is called unlink(), not delete(). rm literally means "unlink".

The three possible outcomes of an rm:

The third row is the one behind the mystery of the space that is not released. It is also a mechanism used on purpose: a program that needs a temporary file can create it, open it and delete it immediately. The file carries on working while the program has it open, and disappears automatically when the program finishes, even if it terminates abnormally. There is nothing to clean up.

The reason: a hard link is a (name, inode number) entry. The inode number only has meaning within its own file system: inode 262149 on /dev/sda2 has nothing to do with inode 262149 on /dev/sda1. An entry in a directory on /boot saying "262149" would point at inode 262149 of /boot, which is a completely different file.

It is not an arbitrary restriction: it is that the reference means nothing outside its own scope.

How to find out whether two paths are on the same file system, before you try:

/home and /opt share /dev/sda2: you can link between them. /boot is on /dev/sda1: you cannot.

Not even root can on Linux. The reason is more interesting:

The Unix directory tree is, by design, an acyclic graph: you can go down and up, but not round in circles. If you could create a hard link to a directory, you could build a loop:

The immediate consequences:

• find, du, tar, rsync and any recursive walk would enter an infinite loop, because there would be no way of detecting that they had already been there without keeping a record of every inode visited.
• The link count would stop being useful for knowing when to release the space: a cycle of directories referencing each other would have counts greater than zero even though nobody could reach them from the root. It would be unreachable garbage, impossible to collect.
• .. would stop having a single meaning: a directory with two parents cannot have a single ...

The . and .. entries are the exception: they are hard links to directories, created by the kernel, and that is why directory counts are greater than 1. But they are controlled and their topology is known, so they break nothing.

Both limitations disappear with the other kind of link.

A symbolic link (or symlink, or soft link) is a file in its own right, with its own inode, whose content is a text path. When something opens it, the kernel reads that path and redirects the operation.

Everything you need to read there:

That size confirms what a symlink is: a file whose content is a text string.

When you use it, everything works as if it were the target:

And the two limitations of the hard link do not exist:

It crosses file systems and points at directories without any trouble, because it does not reference an inode: it references a path, and a path is text that gets resolved at the moment of use.

A link to a directory behaves like the directory:

There you have the explanation of pwd -P that was left pending in lesson 02-03: pwd shows the logical path, the one you used to get there; pwd -P shows the physical one, resolving the links.

Options of ln worth knowing:

The combination -sfn is the deployment one and you will see it in action in section 12. Without -n, replacing a link that points at a directory creates the new link inside that directory instead of replacing it, which is a very easy mistake to make.

The content of a symbolic link is a path, and that path can be of either kind.

Both work exactly the same right now. They differ in what happens when something moves.

A relative link is resolved from the directory where the link lives, not from where you are:

It works from your home directory because releases/3.2.1 is resolved relative to /opt/tramontana/, which is where the link lives.

Now, the test that tells them apart: copying the whole tree somewhere else.

• app-abs still points at /opt/tramontana/..., that is, at the original tree. The copy is not self-contained: it depends on the original. If you delete /opt/tramontana, the copy is left broken.
• app-rel points at releases/3.2.1 inside the copy. The copied tree is self-sufficient and consistent.

The rule of use:

• Relative within a single application tree (/opt/tramontana/app → releases/...), because the whole set is copied, backed up and restored as one unit.
• Absolute when the target is somewhere else in the system and has no structural relationship with the link (for example, a link in your home directory to /var/log/tramontana).

ln -r works out the relative path for you:

You give it absolute paths, which are easy to type with Tab without making a mistake, and it generates the correct relative link. It is the best of both worlds.

Because a symlink stores a path and not a reference to an inode, nothing guarantees that the target exists. You can create a link to something that does not exist yet, and the target can disappear afterwards.

ls -l shows the link without complaining, because the link exists perfectly well: it is the target that does not. The error only appears when you try to use it, and the message mentions the name of the link, which is confusing: it looks as if ~/future does not exist when ~/future is right there.

With colour enabled, ls shows broken links in flashing red, which is hard to miss. To detect them reliably:

find -xtype l means "items that, once the link is resolved, are of type link", which only happens if the link cannot be resolved. find is studied in depth in lesson 03-03; you can adopt this invocation now as a recipe.

And readlink is the specific tool for inspecting links:

readlink -e is the checker: code 0 if the link leads to something real, non-zero if it is broken. It is exactly what you will use in the deployment script to check that the app link points at an existing release before starting the service.

A broken link is not always a mistake. It is a normal situation when the target will be created later, or when it points at a removable device that is not mounted right now. What there must not be are broken links that nobody knows why they are there.

This is the part that causes the most mistakes in practice: each command decides for itself whether it acts on the link or on its target.

The three cases you must be clear about:

rm on a link deletes the link. Always. It is safe:

The target is still intact. This is the behaviour people fear and that in reality never lets you down.

cp without -a undoes links:

A normal cp produced a real file with the content; cp -P produced a link. When copying a tree containing links, this difference decides whether you get a faithful copy or a bloated one in which every link has turned into a complete copy of the file.

The trailing-slash trap with links to directories. This one bites everybody:

The trailing slash means "treat it as a directory", and that forces the kernel to resolve the link. And now the dangerous part:

That last line, with -rf and a trailing slash, goes through the link and deletes the contents of /opt/tramontana/app. The production application, wiped out by one slash too many.

The golden rule: never put a trailing slash after the name of a link in a destructive command. And in general, before an rm -rf on something that might be a link, check:

When to choose each, in practice:

Use symbolic links by default. They are visible in ls -l, they tell you where they point, they cross file systems, they work with directories and they can be replaced atomically. 95% of the links you create and that you find on a Linux system are symbolic.

Use hard links in two specific cases:

1. Incremental backups with deduplication. rsync --link-dest creates what looks like a full copy every day, but the files that have not changed are hard links to the previous day's: they take up zero extra space. It is the basis of Time Machine-style backups and you will see it in lesson 05-08.
2. When you need the file to survive the deletion of the original name, without leaving a broken link behind.

Links are not a curiosity: they are the structure of the system.

The root itself:

This is the reorganisation known as the /usr merge: the binaries were consolidated under /usr and links were left at the root so that the thousands of existing references to /bin/bash would carry on working. This is what you saw with the arrow in tree -L 1 / in lesson 02-03.

Notice that they are relative (usr/bin, not /usr/bin). Deliberately so: that way they stay valid if the tree is mounted somewhere else, for example when repairing the system from a rescue disk mounted at /mnt.

The Python interpreter:

python3 is a link to the specific version. Upgrading Python to 3.13 is, in essence, changing that link: every script beginning with #!/usr/bin/python3 starts using the new version without touching a single one.

Chains of links:

Two hops. readlink -f follows the whole chain to the end. The kernel allows about 40 levels of nesting before giving Too many levels of symbolic links.

systemd services:

Enabling a service in systemd is literally creating a symbolic link, and disabling it is deleting it. When you run systemctl enable in lesson 05-05, what happens underneath is an ln -s.

Rotated logs:

There are no links here, but it is worth knowing that some services keep a current.log as a link to the day's file.

Debian and Ubuntu use links to solve a specific problem: several packages providing the same function. Which editor does sudo editor open? Which java runs if three are installed? What is vi if there are vim, nvi and elvis?

The mechanism is a double layer of symbolic links:

The first layer is fixed and is on the PATH. The second one, in /etc/alternatives/, is the one that gets changed. That way the administrator never touches /usr/bin, which belongs to the packages.

Changing it:

The second-layer link has changed. The main commands:

Why this matters for your session's EDITOR: when you run sudo visudo or crontab -e, the editor that opens is decided by the EDITOR environment variable and, if it is not defined, by the editor alternative. If you have never configured anything and nano appears, you now know why and how to change it. The EDITOR variable belongs to lesson 03-01.

This is where everything above becomes an architectural decision. It is the pattern Tramontana Bookings is going to adopt and which will reappear in lesson 04-07 and in Module 8.

Today, deploying a new version would mean: stop the service, delete or move /opt/tramontana/app, copy the new version, start it. The problems:

• A window of unavailability proportional to the copy time. Copying 48 MB takes seconds; in production, copying a large release can take minutes.
• An inconsistent intermediate state: during the copy, the directory contains half the old version and half the new one. If the service starts at that moment, it behaves unpredictably.
• Rolling back is another complete deployment, with its own window and its own risks, carried out under pressure because something has failed.
• The previous version is no longer there, so rolling back means recovering it from a backup.

Separate the content (the releases, each in its own immutable directory) from the pointer (a symbolic link saying which one is active).

The complete deployment:

The rollback:

One command. That is the whole rollback. Nothing copied, nothing recovered from a backup, no maintenance window. And it can be run at three in the morning by somebody who is nervous, with no risk of getting it wrong.

The three letters matter:

• -s symbolic, obviously.
• -f forces the replacement. Without it, ln fails because app already exists.
• -n is the critical one. Without -n, ln sees that app is a link to a directory, goes inside it and creates the link there:

It has created releases/3.2.1/3.3.0 instead of changing app. The app link still points at the old version, the service is not updated, and on top of that you have dirtied the previous release's directory, which was supposed to be immutable. Always -n.

Replacing a symbolic link is done, underneath, with the rename() system call, which the kernel guarantees to be atomic: there is no instant at which app does not exist or points half-way. Any process opening /opt/tramontana/app at any moment will see either the complete old version or the complete new one, never an intermediate state.

Compare that with the naive alternative of rm app && ln -s releases/3.3.0 app: between the two commands there is a window, brief but real, in which app does not exist. A process trying to open a file just then would get an error. That is why the pattern is ln -sfn, in a single command.

The trade-off is in the last row: you have to remove the old releases or /opt will grow without limit. The usual policy is to keep the last three or five, and you will write the script that does it in Module 4.

Another detail of the scheme: the shared/ directory. Data that must survive deployments — files uploaded by users, persistent caches — cannot live inside the release, because the next release would not have it. It goes in shared/ and each release links to it:

This pattern, which you are assembling here by hand, is exactly the one implemented by professional deployment tools (Capistrano popularised it, and the deployment structures of many PaaS platforms replicate it). You will automate it with a script in lesson 04-07 and put it into production in Module 8.

Believing a hard link is a copy. You modify one and you change both, because they are the same file.

Believing that deleting a hard link deletes the data. Only if it was the last name.

Running ln -sf without -n on a link to a directory. It creates the link inside instead of replacing it, silently.

Putting a trailing slash after a link in an rm -rf. It goes through the link and deletes the real target. The most expensive mistake in this lesson.

Copying trees containing links with cp -r. It turns the links into real copies and bloats the result. Use cp -a or rsync -a.

Using absolute links inside a tree that is going to be copied or moved. The copy ends up depending on the original. ln -sr.

Trusting ls -l to tell you whether a link works. It shows the broken ones exactly like the good ones. readlink -e or find -xtype l.

Tip: readlink -f before any destructive operation on a path that might contain links. It tells you where it really points.

Tip: use relative links within a single tree and absolute ones between different trees. It is the rule that avoids 90% of broken links.

Tip: run find /path -xtype l from time to time. Broken links are a symptom: something moved or was deleted and nobody updated what pointed at it.

Tip: document your structural links. A README in /opt/tramontana explaining the release pattern stops the next administrator deleting releases/ thinking it is surplus.

In /tmp, without looking back at the theory:

1. Create original.txt with three lines. Note its inode and its link count.
2. Create a hard link hard.txt and a symbolic link soft.txt. Show all three with ls -li and explain each column.
3. Add a line through hard.txt. What do original.txt and soft.txt see?
4. Delete original.txt. What happens to each of the other two? Explain why in terms of inodes.
5. Create a new original.txt with different content. What does soft.txt point at now? And hard.txt?

Set up the complete release structure on srv-tramontana:

1. Create /opt/tramontana/releases/3.2.1 with the current contents of /opt/tramontana/app.
2. Replace /opt/tramontana/app with a relative link to that release, without the content ceasing to be accessible at any verifiable moment.
3. Simulate the deployment of a version 3.3.0 and activate it.
4. Simulate it failing and go back to 3.2.1.
5. Write a one-line check that verifies that the link points at a release that really exists.

Luis has set up a shared working directory on his laptop and sends you this to replicate on the server:

Marta asks you whether she can approve it. Analyse the proposal, list the problems and propose the correct alternative with your reasoning.

Solution 1

Inode 393221, count 1: a single name points there.

Reading it column by column:

• hard.txt and original.txt share inode 393221 and both show a count of 2. They are two names for the same file, with no hierarchy between them.
• soft.txt has its own inode (393225), type l, permissions lrwxrwxrwx (which mean nothing), size 12 — the characters of original.txt — and the arrow to the target.

All three see the same thing. hard.txt and original.txt because they are the same inode; soft.txt because it resolves to original.txt, which still exists.

hard.txt works perfectly and its count has dropped from 2 to 1. rm removed a directory entry and decremented the count; since it did not reach 0, the inode and its data are still there.

soft.txt is broken. It contained the text original.txt, and that name no longer exists in the directory. The link held no reference to the inode, only a string.

This last step is the one that fixes the concept:

• soft.txt has "repaired" itself and now points at the new file, which has a different inode (393230). A symbolic link is resolved by name, at the moment of use, so it follows whatever file occupies that name. That is its virtue and its danger: it can end up pointing at something with no relation at all to the original.
• hard.txt still has the old content in inode 393221. It never depended on the name.

A one-sentence summary: the hard link is tied to the content; the symbolic one is tied to the name.

Solution 2

rsync -a with a trailing slash on the source copies the contents, not the folder. And diff -rq with no output confirms the copy is identical before you touch anything.

The important detail is mv instead of rm in the intermediate step. Renaming is instantaneous and reversible; deleting is not. If the link had gone wrong, an mv back would have restored everything.

One command and you are on the previous version. And releases/3.3.0 is still on disk, so retrying the deployment after fixing the problem is another single command.

The breakdown: readlink -e resolves the link and fails with a non-zero code if the target does not exist. With the && and || of lesson 02-01, the line reports in both cases. It is exactly the check that will go at the start of the deployment script in Module 4, so that the service is never started with the link pointing at nothing.

Solution 3

The problems with Luis's proposal, from most to least serious:

1. Production would depend on a developer's personal directory. The link points at /home/luis/projects/tramontana/build. That means the production application runs whatever code Luis has in his working folder at that instant. Every time Luis builds, production changes. Without review, without Marta's approval, without warning. It is exactly the scenario the deployment process exists to prevent.

2. That directory does not exist on srv-tramontana. /home/luis is a path on Luis's laptop. On the server, the link would be born broken:

ls -l shows it quite happily, but the service would not start. And the error message would be No such file /opt/tramontana/app, when app does exist: it is the target that does not. Diagnosing that without knowing about links takes a good while.

3. sudo rm app destroys the original before you have anything. If the new link goes wrong — and it is going to go wrong — the application is no longer there. There is no way back. It should be mv, and you only delete once the new thing has been verified.

4. It breaks the FHS. Module 1 justified /opt/tramontana/app precisely because /home/user/project is the worst possible place: offboarding an employee brings the service down. Here the link reintroduces that dependency through the back door, with the aggravating factor that in ls -l /opt/tramontana everything looks fine.

5. A permissions and security problem. The production code would be under the control of an unprivileged user, who can modify it at any time without sudo. Anybody who compromises Luis's account controls the production application. Lesson 02-07 will give you the exact vocabulary to explain why this is serious.

6. There is no versioning and no way back. There is no way to know which version is deployed nor how to recover the previous one.

The correct alternative, which is the pattern from section 12:

The answer for Marta:

The proposal should not be approved as it stands. The underlying idea — being able to switch versions quickly — is a good one, and in fact it is the one we are going to implement; the problem is this particular execution.

As it is framed, the production application would start running Luis's laptop working directory directly. That means three things: that any build of his would change production instantly and without warning, that we would never know which version is running, and that if Luis leaves the company or changes laptop, the service stops working. On top of that, on the server that path does not even exist, so the change would take the application down immediately.

The alternative I propose achieves the same goal with guarantees: each version is installed in its own numbered directory on the server, and a pointer indicates which one is active. Deploying means changing the pointer, and going back to the previous version means the same, in both cases with a single command and without copying anything. The previous version stays untouched on the server, so the rollback is immediate and safe. And at any moment we can look up which version is in production and who approved it.

I will talk to Luis: what we need from him is the version package, not direct access from his machine.

You have dismantled the assumption you started the module with: a file is not a name.

• An inode holds all the metadata and the block addresses, except the name. Directories are indexes of (name, inode) pairs.
• That architecture explains why mv is instantaneous, why . shares an inode with its directory, why you can delete an open file and why df and du sometimes disagree.
• A hard link is another name for the same inode. rm unlinks: the content lives on as long as the count has not reached zero and nobody has it open.
• Hard links do not cross file systems because the inode number means nothing outside its own, and do not point at directories because they would allow cycles that would break recursive walks and space reclamation.
• A symbolic link is a file whose content is a path. It crosses file systems, points at directories, can point at something that does not exist, and is resolved by name at the moment of use.
• Relative within a single tree, absolute between different trees, and ln -sr works them out for you.
• Broken links are not detected by ls: they are detected by readlink -e and find -xtype l.
• Every command decides whether it follows the link or not, and the serious traps are cp without -a, which undoes links, and the trailing slash in an rm -rf, which goes through the link and deletes the real target.
• Links are the structure of the system: /bin, /usr/bin/python3, /etc/alternatives and enabling a systemd service means creating a link.
• And Tramontana now has its deployment pattern: immutable releases/ and an app -> releases/N that turns publishing and reverting into one atomic ln -sfn, with -n because without it the deployment fails silently.

One last piece of the module remains, and it is the one with the most consequences. You have seen that /etc/tramontana/app.conf is -rw-r----- and belongs to root:tramontana, that the executable is -rwxr-xr-x, and that the permissions of a symbolic link mean nothing. You have used sudo without ever quite asking yourself why it was needed. In the next lesson, File Permissions and Ownership, all of that stops being noise: you will read the ls -l string character by character, understand why x means different things on a file and on a directory, translate between octal and symbolic notation in both directions, work out what the umask does, recognise SUID, SGID and the sticky bit when you see them, and design the correct permissions for every Tramontana path. It is the lesson that separates a server that works from a server that is also secure.