In Module 1 you drew the map: you know what /etc is, what distinguishes /opt from /srv and why /var holds what grows. But knowing a map and walking the terrain are different things. This lesson is the walk.

Moving around the file system looks trivial — cd to go, ls to look — and yet it is where the most time gets lost over a career. The difference between someone who navigates fluently and someone who does not lies in concrete details: knowing how to get back to the previous directory with two keys, reading the output of ls -l at a glance, understanding exactly what each timestamp on a file is telling you, and not confusing "this takes up 4 KB" with "this declares 18 bytes".

A warning before we start: until the end of this lesson you are not going to modify anything. Every command you will see is read-only. That is deliberate: first you learn to look, and in the next lesson you learn to touch.

1. pwd and the working directory
2. cd in all its forms
3. Absolute and relative paths in practice
4. . and ..: real directory entries
5. ls in depth
6. Reading ls -l column by column
7. tree: seeing the structure at a glance
8. stat and the three timestamps
9. file: what a file really is
10. du and df: how much it takes and how much is left
11. Never getting lost: habits of orientation
12. Guided practice: a tour of the Tramontana tree

Every process on the system has a current working directory (cwd): the point in the tree from which it interprets relative paths. Your shell is a process and it has one too.

pwd (print working directory) always answers with an absolute path, starting from /.

When you log in, your working directory is your personal directory, which on srv-tramontana is /home/operator. That is why the prompt shows ~: it is the abbreviation for the home directory.

pwd has two options that only make sense once symbolic links are involved (lesson 02-06):

That difference will matter when Tramontana adopts the deployment pattern based on links. For now, take away that pwd -P tells you where you really are.

cd (change directory) changes the working directory. It is a builtin, and in lesson 02-01 you saw why it has to be one.

A complete walk:

Notice how the prompt reflects the current directory. It is your permanent orientation instrument and it costs nothing to look at.

The two forms that save the most time, and that many people do not know about:

cd - toggles between the last two directories, like a "back" button. When you use it, Bash prints where it has gone:

It is perfect when you are comparing the configuration against the logs: two keys to jump from one side to the other.

~user is the path to another user's home directory, resolved by the shell:

Watch out for ~root: it is /root, not /home/root. The administration account has its home outside /home precisely so that it remains accessible even if /home is on a partition that has not been mounted, something that matters in system recovery (lesson 07-01).

One behaviour to bear in mind: if cd fails, you do not move.

The prompt did not change and pwd confirms it. This is the detail that made the ; of the previous lesson dangerous: cd fails, but the next command runs anyway, in the wrong place.

You saw the definition in Module 1. Here you are going to use it.

• Absolute: starts with /. It means the same thing from anywhere.
• Relative: does not start with /. It is interpreted from the current directory.

The Tramontana tree you will use as a reference:

Translation exercises. You are in /var/log/tramontana. How do you get to each destination in both ways?

A practical check of the third row:

Three .. because you have to go up from tramontana to log, from log to var, and from var to /. Count the levels in the prompt and they come out on their own.

When to use each one:

A mnemonic: relative for your fingers, absolute for machines.

. is the current directory and .. is the parent. But they are not a shell trick: they are real entries inside every directory, written to disk.

The first two lines are genuine entries, with their permissions, their owner and their date. The definitive check:

The same number, 262148. That number is the inode, and the fact that it matches proves that . and the directory are literally the same object. Inodes are the topic of lesson 02-06; hold on to this fact.

Two practical consequences:

./ is mandatory for running a program from the current directory. As you saw in 02-01, the current directory is not in the PATH:

./ protects against names starting with a hyphen, as you saw with the -- separator. rm ./-report.txt always works.

A detail about the root directory: in /, the .. entry points to itself. cd /.. leaves you in /. The tree ends there.

ls is the command you will run more times than any other in your life. It is worth knowing well.

Without options, ls is terse because the old shell was:

Combinations that really do get used:

ls -ltr deserves a comment: by sorting from oldest to newest, the last thing modified ends up just above the prompt, where your eyes already are. In a directory with two hundred files, that is the difference between seeing the answer and having to look for it.

-d is the least intuitive option and the most necessary. Without it, ls -l on a directory lists its contents; with it, it describes the directory itself:

The first tells you what is inside; the second tells you who can get in. When you reach permissions in lesson 02-07, ls -ld will be your main tool.

-R walks the whole tree. Use it with care: ls -R / produces hundreds of thousands of lines.

You need to be able to read this output at a glance, because it contains almost all the relevant information about a file:

Taken apart:

A detail about the sixth column: ls shows the time if the file was modified in the last six months, and the year if it is older. That asymmetry is disconcerting the first time:

If you want complete, unambiguous dates:

The second column, the link count, has a behaviour that takes people by surprise: on directories it is never 1.

It is 4 because the following count: the app entry in /opt/tramontana, the . entry inside app, and the .. entries of its two subdirectories. The link count of a directory is 2 + the number of subdirectories it contains. It is a trick for counting subdirectories without going in, and it will become completely clear in lesson 02-06.

And the size of a directory (4096) is not what it contains: it is what the list of names itself takes up. To find out what it contains there is du, in section 10.

tree shows the hierarchy drawn out. It does not come installed by default on Ubuntu Server:

(Package management is covered in lesson 05-03; here we install it and move on.)

Essential options:

-L is practically mandatory on large directories:

There you see something interesting: bin, lib and sbin appear with an arrow. They are symbolic links to directories inside /usr, the result of the /usr-merge reorganisation carried out by modern distributions. Lesson 02-06 explains exactly what that arrow means.

A very useful combination for documenting the structure of a project:

ls -l gives a summary. stat gives everything:

What it adds over ls -l:

• The inode number (262151) and the device it lives on.
• The permissions in octal and symbolic form at once (0644/-rw-r--r--), very useful for lesson 02-07.
• The numeric UID and GID as well as the names.
• The blocks actually occupied.
• And the timestamps with nanosecond precision.

The three classic timestamps get confused constantly. This table settles the confusion:

The most widespread mistake is believing that ctime means creation time. It does not: it is change time, the moment the inode changed. Changing a file's permissions updates its ctime but not its mtime.

A demonstration:

The mtime (%y) did not move: the contents did not change. The ctime (%z) did: the inode was modified when the permissions changed. This distinction is the basis of intrusion detection (lesson 06-04): an attacker can forge the mtime, but not the ctime.

The fourth line, Birth (birth time), is relatively modern. It exists on ext4 and other current file systems, but not every tool exposes it and it cannot be queried portably.

A detail about atime worth knowing: updating it every time a file is read would mean one disk write per read, which would be a performance disaster. That is why Linux mounts file systems with the relatime option, which only updates atime if it is older than mtime or if more than 24 hours have passed. The consequence: the atime you see is approximate. Do not use it for precise auditing.

stat accepts custom formats with -c:

That form is the one you will use in the Module 4 scripts to generate reports.

On Windows, the extension determines the type. On Linux the extension is just part of the name: it means nothing to the system. What determines the type is the content, and the one who interprets it is the program that opens it.

file examines the first bytes of a file and compares them against a database of known signatures (the magic numbers):

Definitive proof that the extension is not in charge:

It is called .jpg and it is still text. file is not fooled by the name.

Cases where file saves you from a problem:

That last example is a classic: with CRLF line terminators means the file has Windows line endings. Bash tries to run /bin/bash\r and fails with an incomprehensible message. file gives you the diagnosis in a second.

Useful options:

A professional habit: run file over any unknown file before opening it. In lesson 02-05 you will see why running cat on a binary wrecks your terminal; file is the check that prevents it.

Two commands that get confused and that answer different questions:

• df (disk free): how much space is left on the mounted file systems. It asks the file system.
• du (disk usage): how much space one specific directory takes up. It walks and adds up.

There you see the 23 GB disk at 30 % that you know from the installation. The tmpfs lines are file systems in RAM, not on disk: that is why /run and /dev/shm disappear on reboot.

An essential option that almost nobody uses:

-i shows the inodes, not the bytes. A disk can have free space and still say "No space left on device" if the inodes have run out, typically because of millions of tiny files. It is an incident that baffles anyone who does not know it exists, and df -hi is the diagnosis in a second.

du answers the opposite question:

Without -s, du breaks down every subdirectory, which on / produces an avalanche. With -d 1 you get a one-level breakdown, which is what you want for hunting culprits:

The last line is the total for /var/log, including the loose files that are not in any subdirectory.

The difference between the declared size and the space occupied explains a discrepancy that causes confusion:

The file declares 5 bytes, but it occupies 4 KB because the file system allocates space in blocks and the minimum ext4 block is 4 KB. A directory with a hundred thousand 5-byte files takes up 400 MB for real.

That is why du (which measures what is occupied) and df (which measures what is free on the device) can fail to agree. Another classic cause of a mismatch: a deleted file that is still open by a process. It disappears from the tree, so du does not see it, but the space is not released until the process closes it, so df does count it. It is the reason why deleting a huge log without restarting the service does not free space, and you will come back to it in lesson 05-07.

Three habits that prevent 90 % of the frights:

1. Read the prompt before every destructive command. The Ubuntu prompt already shows the current directory. It costs nothing to look at it, and in the instant before an rm it is what separates deleting the right thing from deleting the other thing.

2. pwd as a reflex. After any chain of cds, after coming back from a break, after connecting by SSH to another machine. It is free.

3. Before acting, look. The pattern you already saw in 02-01 and that will recur throughout the course:

A fourth habit, for later on: when you work on several servers at once, customise the prompt so that it shows the machine name in a different colour. It is the way not to run in production what you meant to run in test. It is done with the PS1 variable (lesson 03-01).

We are going to walk through the whole Tramontana Bookings deployment, describing each piece. This tour is the one you would do on receiving a server you do not know.

Starting point and overview:

A single file system for the whole tree, at 30 %. Plenty of space, and it also means that anything growing out of control affects the entire system.

The application:

Diagnosis: third-party software in /opt as per the FHS, owned by root (users cannot modify the code: correct), 48 MB, one compiled binary and its templates. Static data: it does not change except in a deployment.

The configuration:

Two relevant things. The first: permissions 640, that is, nobody who is not root or a member of the tramontana group can read it, which is correct because it contains credentials. The second: the tramontana group appears here, and we have not formally created it yet; that will be done in lesson 05-01. The exact meaning of the 640 is lesson 02-07.

Notice the mtime too: it was modified this morning at 8:47. On a server that should not be changing, a recently modified configuration file is the first thing you investigate when there is an incident.

The logs:

Variable data: they grow with use. 26 KB today is nothing, but an access log in production grows linearly with the traffic, and that is exactly the scenario that justified putting it in /var. That it is ASCII text matters: it means all the text-processing tools of Module 3 will work on it.

The backups and the working data:

The working data is in your home directory, separate from the application. bookings.csv uses the semicolon as its separator, a detail that will matter when processing it in lesson 03-05.

A summary of the tour, in the report format Marta expects:

With five commands — df, tree, ls -l, stat, du — you have characterised a complete server without touching anything. That is the initial reconnaissance you carry out when you inherit a system.

Confusing .. with .. cd . does nothing, cd .. goes up. One character, a great deal of difference.

Using relative paths in scripts. It works while you run it by hand from the right directory, and it fails the day cron launches it from /. Absolute always in scripts.

Believing that the size of a directory in ls -l is what it contains. It is not: it is 4096 bytes of index. Use du -sh.

Confusing du with df. If you are asking "how much is left on the disk", it is df. If you are asking "what is taking up so much", it is du.

Believing that ctime is the creation date. It is change time. The real creation date, when available, is shown by stat as "Birth".

Trusting the extension. A .txt can be a binary and a file with no extension can be an executable. file is the answer.

Tip: cd - and pushd/popd. cd - toggles between two. If you need a stack of directories, pushd and popd implement one; they are builtins and their documentation is in help pushd.

Tip: ls -ltr in directories with many files. The most recent ends up stuck to the prompt.

Tip: ls -ld is the most forgotten option. Every time you want to know something about the directory and not about its contents, it is -d.

Tip: df -hi when "there is no space left" but df -h says there is. The inodes have run out.

Without running anything, starting from the Tramontana tree in section 3 and being in /opt/tramontana/app/templates:

1. The relative path to /opt/tramontana/app/version.txt.
2. The relative path to /etc/tramontana/app.conf.
3. Which directory is ../../../../var/log?
4. What does cd ../.. do from there, and what is the resulting pwd?

Then check your answers on the server.

Marta asks you to review the /var/log directory on srv-tramontana because "the disk seems to be filling up". Without deleting anything, find out and present:

1. How much free space is left on the root partition and what percentage is in use.
2. How much /var/log takes up in total.
3. Which three subdirectories of /var/log take up the most.
4. Which is the most recently modified file inside /var/log/tramontana and at what time.
5. Whether that file is text or binary.

In your home directory, with a test file:

1. Create /tmp/times.txt with some content and note down its three timestamps.
2. Read it with cat. Which one changed?
3. Change its permissions with chmod 600. Which one changed this time?
4. Append content to it. Which ones changed?
5. Explain why an attacker who wants to hide the fact that they have modified a system file can forge the mtime but finds it far harder to forge the ctime.

Solution 1

1. ../version.txt — you go up from templates to app and the file is there.

2. ../../../etc/tramontana/app.conf. Count the levels: from templates to app (one), to tramontana (two), to opt (three)... and from opt you have to go up one more to reach /. That makes four: ../../../../etc/tramontana/app.conf.

Let us check it, because this is exactly the kind of mistake that gets made:

The current absolute path has four components (opt, tramontana, app, templates), so four .. are needed to reach the root. This is exactly why absolute paths are better as soon as the distance goes beyond a couple of levels: /etc/tramontana/app.conf does not allow this mistake.

3. ../../../../var/log is /var/log, by the same calculation.

4. cd ../.. goes up two levels: from templates to app, and from app to tramontana.

Solution 2

15 GB free, 30 % used. No problem for now.

1.3 MB out of 23 GB: /var/log is not the cause of the disk filling up. This is the figure that settles Marta's question.

(The | sort -h | tail is a pipe; it is explained thoroughly in Module 3. You can get the same thing by reading the output of du -h -d 1 /var/log by eye.)

The biggest consumer is journal, the systemd log, with 912 KB. That is normal and it has a configurable size limit (lesson 05-06).

access.log, modified at 09:14. -t sorts by date descending, so the first one is the most recent.

Report for Marta: the root disk is at 30 % with 15 GB free. /var/log takes up 1.3 MB in total, so it is not what is filling the disk; it is worth looking elsewhere. Within the logs, the biggest consumer is the system journal (912 KB) and the application logs add up to 26 KB. The access log was last updated at 09:14 today, which confirms that the application is running and writing normally.

Solution 3

When you create it, all three match.

Only atime changed: you have read the contents, you have not modified them. (If you do not see the change, it is because of relatime: if atime was already more recent than mtime and 24 hours have not passed, the kernel does not update it. That is the normal, expected behaviour.)

Only ctime changed: you have modified the inode (the permissions live there), not the contents, and you have not read it either. This is the demonstration that ctime is not creation time.

mtime and ctime changed. The contents changed (mtime) and, in changing the contents, the size recorded in the inode also changed (ctime). atime did not change because writing is not reading.

Point 5: why ctime is hard to forge. The touch command lets you set atime and mtime to whatever value you like (touch -t, lesson 02-04). An attacker who modifies /etc/tramontana/app.conf can leave the mtime exactly as it was and the file will look untouched in ls -l.

But the ctime cannot be set. There is no system call that allows an arbitrary value to be written into it: the kernel updates it automatically every time the inode changes, including the change caused by touch itself when forging the mtime. Forging the ctime requires direct access to the block device underneath the file system, or changing the system clock, and both of those leave other traces.

Hence the forensic practice: when an intrusion is suspected, mtime and ctime are compared. A file whose ctime is much later than its mtime has been touched in a way that deserves an explanation. It is one of the indicators used by the tools in lesson 06-04.

You no longer navigate the Module 1 tree by feel: you walk it with instruments.

• pwd tells you where you are, and pwd -P where you really are when links are involved.
• cd has more forms than it seemed: cd - to toggle, plain cd to go home, ~user for somebody else's home directory. And if it fails, you do not move, which is precisely what made the ; dangerous.
• You can translate between absolute and relative paths by counting levels, and you have the criterion: relative for your fingers, absolute for scripts and cron.
• . and .. are real entries with their own inode, and that is why ./program is the way to run something from the current directory.
• You read ls column by column, and you know the combinations used daily: -lah, -ltr, -lhS and the indispensable -ld.
• tree -L gives you the structure at a glance without drowning you in depth.
• stat gives you the complete metadata, and you understand atime, mtime and ctime precisely, including why ctime is relevant to security.
• file tells you what a file really is, because on Linux the extension is not in charge.
• du measures what something takes up, df measures what is left, and you know why they sometimes disagree and what to do when the disk "is full" without being full.
• And you have characterised the complete Tramontana Bookings deployment with five commands, without modifying anything.

Looking is over. In the next lesson, File and Directory Operations, you start writing to the disk: touch and mkdir to create, cp and mv with all their traps — including the trailing slash on the destination, which bites everybody — rm and the serious conversation about rm -rf that has to be had once in a lifetime, rsync to copy large directories properly, and tar with gzip, bzip2 and xz to package things up. You will finish by preparing a package of /opt/tramontana/app to hand over to Luis. Make sure your VM snapshot is up to date before you begin: from here on, the commands no longer just look.