Everything you have handled so far was static: files, permissions, text. This lesson changes its object and deals with what is alive. A process is a program in execution, with its own memory, its open file descriptors, its identity and its place in a family tree that starts at systemd. Understanding them is what lets you answer the question Marta is going to ask you sooner or later: "why is the server slow?".

In the previous lesson you discovered that 65% of the errors in access.log are concentrated in the 03:00 slot, and that everything points to something competing for the database connections during the night-time window. Here are the tools to find out what that something is, and to act on it without bringing the service down.

1. What a process is
2. The process tree
3. Life cycle: fork, exec, wait, exit
4. Zombies and orphans
5. Process states
6. ps properly
7. top and htop: reading the header
8. pgrep and pkill
9. Signals
10. Priorities: nice, renice and ionice
11. Job control in the interactive session
12. lsof and fuser
13. The Tramontana case: the app hanging in the early hours

Every process has a record in the kernel containing, among other things:

The distinction between VSZ and RSS is the one that causes the most misunderstandings: VSZ includes memory reserved but never touched and shared libraries, so adding up the VSZ of every process gives a number far above the installed RAM without that meaning anything at all. The number that matters is RSS.

Every process descends from another. The root is systemd, with PID 1, the first process the kernel starts.

There is a lot to read there. executable(1284) is the Tramontana application with three threads (in braces, they are not independent processes). The sshd chain shows your own connection: the main daemon, the connection's process, the one for the already-authenticated session, your bash and the pstree you have just launched as its child. There is the environment inheritance from 03-01, turned into a diagram.

pstree -ps 1284 shows the ancestors of a specific PID and returns systemd(1)───executable(1284): the app hangs directly off systemd, not off a user session. That means it survives you closing the SSH connection, and it is the difference between a service and a program launched by hand.

When you type ls in Bash this is what happens: Bash calls fork(), which creates a child identical to itself; the child calls exec(), which replaces its contents with the ls binary while keeping the PID and the open descriptors — that is where the redirections Bash prepared beforehand fit in; ls finishes by calling exit(0); and Bash, which was in wait(), collects that exit code, which is what you later read in $?.

That separation between fork and exec is what makes redirection possible in the way you studied it: there is a moment, between the two calls, when the child already exists and is not yet the new program, and that is where the shell reconfigures the descriptors.

They are two opposite situations and it is worth not confusing them.

A zombie (state Z) is a process that has already finished but whose parent has not called wait(). It consumes no CPU or memory: it merely occupies an entry in the process table with its exit code, waiting for somebody to collect it.

A zombie cannot be killed. It is already dead; kill -9 on it does nothing, because signals are received by living processes. The only way to get rid of it is for its parent to collect it or for the parent to die. A handful of zombies is irrelevant; thousands indicate a badly written parent, and the correct action is to restart the parent.

To look for them: ps -eo stat,ppid,pid,comm | awk '$1 ~ /^Z/'. On srv-tramontana it returns nothing.

An orphan is the opposite: a living process whose parent has died. It is not a problem. The kernel reassigns it to systemd (the PPID becomes 1), which does call wait() correctly. It is the mechanism nohup and daemons rely on.

Modifiers you will see attached: s session leader, l multi-threaded, + in the foreground, < high priority, N low priority.

Why a process in D cannot be killed. It is blocked inside the kernel waiting for a disk or network operation to complete, and the kernel does not deliver signals to it until that operation finishes. kill -9 is queued: the signal will be delivered when the process returns to state S or R. If a process has spent minutes in D, the problem is not the process: it is the storage — a failing disk, a dead NFS mount — and that is where you have to look.

ps has two historical syntaxes that coexist: BSD (without a hyphen) and UNIX (with one). The two combinations that get used:

Careful with %CPU: it is an average since the process started. A process that devoured the CPU eight hours ago and is now asleep may still show a high percentage. To find out what is consuming CPU right now, use top.

ps -ef gives the UNIX view, with an explicit PPID, which is what you want for following parent-child relationships. And ps -o builds custom output:

--sort=-rss sorts by resident memory in descending order (the - reverses it). etime gives the time elapsed since it started, far more useful than the absolute time when you are diagnosing.

Filters: -u operator by user, -C executable by command name, -p 1284 by PID, --forest to see the hierarchy in ps's own output.

The three numbers are the average of processes in state R or D over 1, 5 and 15 minutes. Two consequences that almost nobody is clear about:

1. It is not a percentage. It has to be compared with the number of cores: on the srv-tramontana VM, with 2 vCPUs, a load of 2.00 is full occupancy and 3.42 means there is more work than the machine can handle.
2. It includes processes in D, that is, those waiting on disk. A load of 3.42 with the CPU almost idle does not indicate a lack of CPU: it indicates that there are processes stuck waiting on I/O.

And comparing the three numbers gives you the trend: 3.42, 1.87, 0.94 is a rising curve. The problem is getting worse right now, not easing off.

That 64.8 wa is the figure in the header above. The CPU is not saturated: it is waiting for the disk or for the database. Chasing a process that consumes CPU would be looking in the wrong place. And a high st on a virtual machine means the problem is not in your VM but on the host, something worth knowing before optimising code.

On memory: buff/cache is not lost memory, it is disk cache the kernel releases as soon as anybody needs it. The figure that matters is avail Mem. Seeing little free memory on Linux is normal and desirable.

Useful keys in top: M sorts by memory, P by CPU, 1 breaks it down per core, k sends a signal, u filters by user, c shows the full command line. htop does the same with colours, mouse support and a tree with F5; it is installed separately but it is worth it.

-a shows the command line, -c counts, -u filters by user, -f matches against the full line and not just the name, -x requires an exact match, -n/-o the newest / the oldest.

Why they are better than ps | grep | awk | kill: that pipeline has two flaws. The first is that grep itself appears in the list and you can end up killing a PID that no longer exists or, worse, a recycled one. The second is that a partial match takes out processes you did not mean to touch. pgrep queries /proc directly and does not include itself.

Before pkill, always run the equivalent pgrep. It is the same convention as ls before rm:

A signal is an asynchronous notification the kernel delivers to a process.

kill -l lists the system's 64 signals. kill 1284 sends SIGTERM. For any other: kill -TERM 1284, kill -15 1284 or kill -s SIGTERM 1284, all equivalent. killall name acts by name instead of by PID — with the obvious risk of reaching more than you intended.

SIGTERM is a request: the process receives it, runs its shutdown routine and terminates when it has finished. SIGKILL is a summary execution: the kernel destroys the process without telling it. The process never finds out and therefore can do nothing.

What you lose with -9:

• The data in buffers that had not yet been written to disk.
• The open database transactions, which are left uncommitted and locking rows until they expire.
• The HTTP requests in flight, which are cut off dead: the client sees an error.
• The temporary files and the lock files, which are left orphaned and can prevent the next start-up.
• The chance for the process to write in the log why it shut down.

The correct procedure:

Ten seconds is a reasonable margin for a web application; if the process is gone, we have finished cleanly. Only if it is still alive after that period do you resort to kill -9, and then it is documented as an incident, because it means the application's shutdown routine does not work and that is a defect that has to be fixed.

SIGHUP deserves a mention: by convention, many daemons interpret it as "reread your configuration without restarting". When it works, it is the way to apply a change in app.conf without dropping a single request.

The nice value runs from -20 (highest priority) to 19 (lowest). The name comes from "being nice": the higher it is, the more the process gives way to the others. A normal user can only raise it — lower their own priority; reducing it requires root.

nice launches a command with a modified priority; renice changes that of a process already running, also by user (-u) or by group. And for I/O, which in our case is the scarce resource, there is ionice. Its class 3 is idle: the backup only reads from the disk when nobody else needs it. With a wa of 64.8%, ionice solves more than nice: lowering the CPU priority of a process that does not use CPU achieves nothing. Diagnosing before acting also means choosing the right lever.

When the session closes, the shell sends SIGHUP to its jobs. nohup makes them immune and redirects the output to nohup.out; disown -h achieves the same for an already-launched job.

And now the important warning: none of this is any use for a real service. A nohup ./executable & does not restart if the process dies, does not start when the server boots, has no resource control, does not manage its logs and cannot be stopped cleanly by anybody but you. Job control is for long tasks in your session — a backup, a compilation — not for production. Services are managed with systemd, and that is lesson 05-05.

lsof ("list open files") answers the question "who has this open?". Since on Linux almost everything is a file, it also works for sockets and ports.

The "I cannot unmount" case.

Two culprits: a bash whose working directory (cwd) is inside it — a cd out of there is enough — and a tar writing (3w, descriptor 3 in write mode). With that you already know what to expect and who to warn, instead of forcing the unmount.

The "port 8080 is in use" case.

PID 1284 has the port listening. It is the direct answer to "I cannot start the new application because the port is in use": the old one is still alive.

fuser is terser and very practical for taking action: fuser -v /path lists the processes, fuser -k /path kills them (carefully) and fuser -k -TERM 8080/tcp sends SIGTERM to whoever is occupying that port.

/proc/<pid>/ is the source of truth that all these tools draw on: cmdline the exact command, environ the environment it started with, cwd and exe as symbolic links, fd/ the descriptors, status a readable summary, limits the resource limits. When a tool gives you a doubtful figure, go to the file.

03:05. The application is not responding. Marta messages you. Here is the procedure, with the conclusion from 03-05 as the starting hypothesis.

Step 1: is it alive and in what state?

State D: it is not hung in a loop, it is blocked waiting for I/O. With only 2.1% CPU, it is not a computation problem. A kill -9 here would not even take effect immediately.

Step 2: what does the load say? The load average: 3.42, 1.87, 0.94 with 64.8 wa from the header in section 7 confirms the same thing from another angle: a growing queue and I/O waiting, not a lack of CPU.

Step 3: who is competing?

There it is. A nightly backup has spent five minutes reading the whole of /opt/tramontana — including the four releases with their 48 MB executable files — and is saturating the VM's disk. The application, which needs the disk to serve queries, is left waiting.

Step 4: relieve it without killing anything. The backup is legitimate; what is wrong is its priority.

Both changes are applied to the running process, without interrupting it. Two minutes later the wa drops and the application responds again.

Step 5: if restarting the app had been necessary, the procedure would have been kill -TERM $(pgrep -x executable), waiting for the requests in flight to finish, verifying with pgrep that it is gone and only then starting it again. Never kill -9 as the first move: it would cut off any bookings that were being confirmed at that moment, and a half-finished booking is a problem with a customer standing in front of you.

The report for Marta. The outage was not an application failure but competition for the disk: the nightly backup was running at the same priority as the service and was leaving it without access to storage. The measure applied — giving the backup the lowest I/O priority — protects against it blocking the service again, and does not protect against a general increase in load: if the number of bookings grows, more resources will be needed or the database will have to be separated out. In addition, the backup includes all four releases when the active one would be enough, which makes it unnecessarily heavy. Outstanding: adjust the scope of the backup and its schedule, which is exactly what we will look at in the next lesson.

• Using kill -9 as the first option. It is the last. SIGTERM, wait, verify.
• Trying to kill a zombie. It is already dead. Act on the parent.
• Persisting with a process in D. It will not receive the signal until the I/O finishes. Investigate the storage.
• Reading ps's %CPU as an instantaneous value. It is an average since start-up. Use top.
• Being alarmed by low free memory. buff/cache is reusable cache. Look at avail Mem.
• Comparing the load average with 100. Compare it with the number of cores, and remember it includes I/O waiting.
• ps | grep | kill on a short name. grep itself shows up in the list and a partial match kills too much. Use pgrep -a and then pkill.
• Leaving a service running with nohup &. It does not survive a reboot and does not recover by itself.
• Tip: faced with "the server is slow", always look at top's wa first. If it is high, the bottleneck is the disk and half the hypotheses are ruled out at a stroke.
• Tip: ps -eo accepts any combination of columns; keep an alias with your own in ~/.bashrc, as you learned in 03-01.

Exercise 1. Find out with what exact command line, with what working directory and with what environment variables the Tramontana process started, without using ps. Explain what each piece of information tells you.

Exercise 2. Launch a long task in the background, suspend it, resume it in the background, lower its CPU and I/O priority, and detach it from your session so that it survives an SSH disconnection. Show the verification of each step.

Exercise 3. Simulate the diagnosis of "port 8080 will not accept the new version": identify which process is occupying it, who it belongs to, since when, and stop that process cleanly, verifying that it has finished before treating the step as done.

Solution 1.

cmdline stores the arguments separated by null bytes, hence the tr. It confirms that the app reads /etc/tramontana/app.conf, so a change there affects it. The cwd is /opt/tramontana/app, the symbolic link: it means its relative paths are resolved through it.

The revealing piece of information is exe, which points to /opt/tramontana/releases/3.2.1/executable, the already-resolved path. It is the proof that the atomic deployment works as we expected: the kernel fixed the real binary at start-up, so changing the app link to another release does not affect the running process. You know which version is genuinely being served, not which one the link says right now. And environ shows a minimal PATH and LANG=C.UTF-8: the service does not inherit your interactive environment, something you already knew from 03-01 and which will come up again in the next lesson.

Solution 2.

Each check contributes something: STAT at T confirms that Ctrl+Z really did stop it (SIGSTOP), and after bg and the adjustments, SN indicates that it is sleeping at low priority (N) and NI is 15. The renice to 15 did not need sudo because raising your own nice value is permitted to any user; lowering it would require privileges. disown -h does not remove it from the job list, but it marks it as not to receive SIGHUP when the session closes. If we had known in advance that the task was a long one, the clean approach would have been nohup ... & from the start.

Solution 3.

We now have the three answers: it is occupied by executable, it belongs to svc-tramontana and it has been running for four and a half hours. -sTCP:LISTEN filters the listening sockets and discards the established connections, which would be noise.

Two different verifications and both necessary. pgrep confirms that the process no longer exists; lsof with no output confirms that the port is free, which is not exactly the same thing: a socket can stay in TIME_WAIT for a few seconds after the process closes, and trying to start the new version in that gap would fail for a different and baffling reason. Checking the resource you actually need, and not just the process, is what avoids that mistaken diagnosis. You will see the TIME_WAIT state and the rest of the TCP states in 03-08.

You have stopped looking at files and started looking at what is running, and with that you have resolved a real incident from start to finish.

• You know what the kernel keeps about each process — PID, PPID, effective UID, state, priority, VSZ and RSS — and that the memory figure that matters is RSS.
• You walk the process tree with pstree from systemd (PID 1) and you understand the fork / exec / wait / exit cycle, which is where the redirection from 03-04 fits in.
• You distinguish a zombie from an orphan: a zombie cannot be killed and you act on the parent; an orphan is adopted by systemd and is not a problem.
• You know the states R, S, D, T, Z and why a process in D responds to no signal, not even to -9.
• You read ps aux and ps -ef column by column, you build views with ps -eo ... --sort=, and you know that ps's %CPU is an average since start-up.
• You interpret top's header: the load average against the number of cores and its trend, the us/sy/ni/id/wa/hi/si/st breakdown — with wa as the sign that the bottleneck is the disk — and the fact that buff/cache is not lost memory.
• You use pgrep/pkill instead of ps | grep | kill, always checking before killing, and you apply the professional rule for signals: SIGTERM, wait, verify and only then SIGKILL, knowing what you lose with -9.
• You adjust priorities with nice, renice and ionice, choosing the lever according to where the bottleneck is.
• You control jobs with &, jobs, fg, bg, Ctrl+Z, disown and nohup, and you are clear that none of it is any use for a production service.
• You answer "who has this open?" and "who is occupying this port?" with lsof and fuser, and you go to /proc/<pid>/ when you want the truth with no intermediaries.

The diagnosis has left a very specific outstanding task: the nightly backup runs at the wrong time, with the wrong priority and over more data than necessary. Fixing it is the subject of the next lesson. Scheduling Tasks with Cron will teach you the syntax of the five fields down to its most convoluted cases, the three classic failures that cause 90% of the incidents — the minimal PATH, the output email nobody reads and the time zone — how to debug a task that "works by hand and not in cron", and how to stop two runs overlapping with flock, which is precisely what may have happened in the early hours this morning.