Throughout module 1 we used the word process as if it were self-evident: ingestor, aggregator and meteo-api showed up in the output of ps, crossed the boundary into kernel mode and consumed resources. But we never opened that box. In this lesson we open it all the way. You are going to see what exactly a process is in memory, what data structure represents it inside the Linux kernel, why it moves through some states and not others, how it is born — with the strangest and most elegant mechanism in UNIX, fork() — and what it really costs for the CPU to stop running one and start running another.

By the end you will be able to read /proc/<pid>/ the way a doctor reads a chart, you will understand why a zombie process cannot be killed with kill -9, and you will be able to write a supervisor in C that launches and watches over the aggregator. It is the most fundamental lesson in the course: almost everything that follows — scheduling, memory, concurrency, containers — rests on what you see here.

1. Program and process: the difference that changes everything
2. The memory image of a process
3. The process control block: task_struct field by field
4. Process states and their transitions
5. The real state codes reported by ps
6. Process creation: fork() and copy-on-write
7. execve(): replacing the program without changing process
8. wait(), exit codes, zombies and orphans
9. A complete example: a supervisor for the aggregator
10. The process hierarchy and init/systemd
11. The context switch: what gets saved and what it costs
12. Hands-on inspection: /proc, ps -eo, pstree

A program is a file on disk. It is passive, it does nothing, it is a sequence of bytes in a specific format (on Linux, ELF) describing what code to load and where.

A process is a program in execution: it is active, it has state, it has resources assigned to it and it has a lifetime.

On meteo-01 you can see this literally:

A single 87 KB file, two processes. Each has its own PID, its own working memory, its own open files and its own point of execution. They share the code (the kernel maps it into RAM just once and references it from both), but nothing else.

The program→process relationship is 1 to N. And not only that: a single process can execute several programs one after another over its lifetime, as we will see with execve().

When the kernel starts a program running, it builds a memory image in the process's address space with several clearly distinct regions. From low addresses to high ones:

Each region has a different purpose and different permissions, and this is not decoration: it is real protection enforced by the MMU.

Two subtleties that are almost always overlooked:

• .bss takes up no space in the executable file. If you declare static struct Reading cache[100000]; (2.4 MB), the ELF does not grow by 2.4 MB: it merely records "reserve 2,400,000 zeroed bytes". The kernel materializes them when the process starts. That is why .bss historically stands for block started by symbol, and why there are 90 KB executables that occupy 50 MB of RAM.
• .text is read-only and shared. The two aggregator processes above have their code pages pointing at the same physical RAM frames. There is only one copy of the code in memory no matter how many processes run it. This saves memory and is the reason why running 200 instances of a service does not cost 200 × the size of the binary.

That whole memory image is the process seen from outside. Inside the kernel, each process is represented by a data structure: the process control block (PCB). On Linux it is called task_struct and it lives in include/linux/sched.h. It is one of the largest structures in the kernel: around 7 KB on a typical x86-64, with more than 200 fields.

You do not need to know them all, but you do need the groups and the reason for each one:

An important detail that connects to what you already know: the mm field is a pointer. That means two different task_structs can point to the same address space. When that happens, what you have is not two processes but two threads of the same process. We will look at it in depth in Threads and Processes, but you can already sense Linux's central idea: there are not two structures, one for processes and one for threads; there is a single one, task_struct, and what changes is how much they share.

You can see many of these fields from user space:

Field by field:

• State: S: sleeping, waiting for something (we will cover this in the next section).
• Tgid: 1877 equal to Pid: 1877: it is the main thread of the group. If it were a secondary thread, Pid would differ from Tgid.
• PPid: 1: its parent is PID 1, systemd. It is a system service; no shell launched it.
• Uid: 998: it runs as the meteora user, not as root. Exactly what we want.
• VmSize of 408 MB against VmRSS of 31 MB: it reserves a lot of address space but only 31 MB are actually resident in RAM. That difference is the essence of virtual memory and we will develop it in Virtual Memory and Paging.
• voluntary_ctxt_switches: 18422: it has stepped off the CPU voluntarily 18,422 times (because it blocked waiting for I/O). Against only 291 involuntary ones (the CPU was taken away from it). This process is clearly I/O-bound, a fact we will reuse in the next lesson.

A process is not always running. On a four-core machine, at most four processes are on the CPU at any given instant; the other 300 are in some other state. The classic model has five:

What matters is understanding why each transition exists:

• New → Ready: the kernel has finished building the task_struct and setting up the address space. It is now a valid candidate for the CPU.
• Ready → Running: decided by the scheduler, and that is the entire subject of the next lesson.
• Running → Ready (preemption): the process asked for nothing; its turn simply ran out or a higher-priority one arrived. This arrow is what distinguishes a preemptive multitasking system from a cooperative one, as we saw in 01-03.
• Running → Blocked: the process asks for something that is not available right now — a read() on a socket with no data, for example. It would be absurd to leave it occupying the CPU while it waits. The kernel sets it aside and picks another.
• Blocked → Ready: the data arrives, typically via a device interrupt (lesson 02-07). Note carefully: it does not go straight to Running. It goes to Ready and competes again.
• Running → Terminated: it calls exit() or receives a fatal signal.
• Terminated → gone: only when its parent collects the exit code. This is where zombies appear.

The transition that confuses beginners most is Blocked → Ready rather than Blocked → Running. The reason is simple: when the data ingestor was waiting for arrives, there may be ten other ready processes and a single free CPU. A process ceasing to be blocked does not entitle it to run immediately; it only gives it back the right to compete.

Linux refines the theoretical model. These are the codes you will actually see:

Two things deserve special attention:

R means "runnable", not "running". Linux does not distinguish between Ready and Running in its state representation; both are TASK_RUNNING. If ps shows you 30 processes in R on a four-core machine, there is no contradiction: 4 are on the CPU and 26 are in the run queues waiting their turn. When this happens persistently, you have a saturated CPU, and that is how you detect it.

D is the state that produces the worst production incidents. A process in D is inside the kernel, in the middle of a disk operation, at a point where the code cannot be aborted without corrupting structures. That is why it accepts no signals: kill -9 stays pending until the process leaves D. If the aggregator gets stuck in D because the disk holding /var/lib/meteora has errors or an NFS mount is not responding, you will not be able to kill it, and you will see the load average shoot up even though the CPU is at 0% (on Linux the load average counts processes in D too, not just those in R).

Modifiers you will see next to the state:

• s: it is a session leader.
• l: it is multithreaded (it has several task_structs with the same mm).
• +: it is in the foreground on a terminal.
• < / N: high / low priority (you will see this in 02-02).

Here comes the part that strikes everybody as odd the first time. In UNIX, the only way to create a process is to duplicate an existing one with fork().

What happens, and why it is disconcerting:

• fork() is called once and returns twice. It returns in the parent (with the child's PID) and it returns in the child (with 0). There is no magic: the kernel creates a second task_struct that is a copy of the first, with the return register set to 0. When the scheduler runs that new process, it will continue right after the fork, exactly where the parent is.
• The child inherits almost everything: the address space (a logical copy), the open file descriptors, the working directory, the umask, the credentials, the resource limits.
• It does not inherit: the PID (it is new), the PPID (it now points at the parent), the accumulated CPU times (they start at zero), pending alarms and pending signals.
• The order of the lines is not guaranteed. In the output above the parent printed first, but it could have been the other way round. Who runs first is up to the scheduler. Any code that depends on that order is wrong.

The obvious question: if meteo-api has 400 MB of address space, does fork() copy 400 MB? That would be a disaster, especially because in 95% of cases the next thing the child does is call execve() and throw that whole copy away.

Modern UNIX systems use copy-on-write (COW):

1. fork() copies only the task_struct and the page tables, not the data pages.
2. All the data pages are marked read-only in both processes, and noted as shared.
3. As long as both only read, they physically share the same RAM. Cost: zero copies.
4. When one of the two writes to a page, the MMU raises a protection fault. The kernel intercepts it, copies that single 4 KB page, gives it exclusively to the writer and marks it read/write.

The result: fork() on a 400 MB process copies a few hundred KB of tables and costs on the order of 0.5 ms instead of hundreds of milliseconds. And if the child calls execve() immediately, hardly any page ever gets copied at all.

This mechanism rests directly on the MMU and the page fault, which we will develop in Virtual Memory and Paging. For now, keep the idea: share until somebody writes is one of the most profitable patterns in all of operating system design.

fork() duplicates. execve() replaces: it keeps the task_struct (same PID, same parent, same open descriptors) but throws away the entire memory image and loads a new program in its place.

Key points:

• execve() does not return if it succeeds. There is no "afterwards". The code that called it no longer exists in memory: it has been replaced. That is why the perror below only runs on error, and why it must always be there.
• The PID does not change. It is the same process running a different program. This has enormous practical consequences: systemd can launch a process, apply limits to it and then let it turn into the real service without losing track of it.
• File descriptors survive by default. That is exactly what makes shell redirections possible: the shell does a fork, in the child it redirects descriptor 1 to a file and then does an exec. The new program finds its output already redirected without knowing anything about it.
• envp replaces the entire environment. In the example, the aggregator will only see METEORA_CONF. If you want to inherit the current environment, you use the execv variant with the global environ variable.

The fork + exec pattern is the foundation of everything in UNIX. When you type ls in the shell, this is what happens:

When a process terminates, it calls _exit(code) (directly or through return in main). The kernel then:

1. Frees its address space, its file descriptors and almost all of its resources.
2. Keeps the task_struct with the exit code and the usage statistics.
3. Sends the SIGCHLD signal to the parent.
4. Marks the process as EXIT_ZOMBIE.

The process is dead but its record is still there. That is a zombie: it consumes no CPU and no memory (beyond a few KB of structure), but it occupies an entry in the process table and a PID.

Why do zombies exist? Because the exit code is information that belongs to the parent. If the kernel deleted the record immediately, a parent that came late to ask "how did my child do?" would have nobody to ask. The zombie is the note the child leaves stuck on the fridge. The parent collects it with wait() or waitpid(), and then — and only then — it disappears.

The macros are necessary because status is an integer with the fields packed into it: it is not directly the exit code.

By universal convention, 0 means success and any other value between 1 and 255 means a specific error. The shell exposes the last one in $?, and that is what makes command_a && command_b work.

Detecting them:

That Z with PPID 1877 says exactly what needs fixing: the aggregator (1877) is creating children and not reaping them. And notice the key nuance: killing the zombie achieves nothing, you have to fix the parent. If you restart process 1877, all its zombies become orphans, systemd adopts them, systemd does call wait(), and they vanish instantly.

An orphan process, by contrast, is harmless: the kernel reassigns PID 1 as its parent (or the nearest subreaper, in the case of services under systemd), and that adoptive parent has a permanent loop calling wait(). This is, in fact, the primordial function of init since 1970.

Let us put it all together into something that could really run on meteo-01: a supervisor that launches the aggregator, waits for it to finish and restarts it if it crashes, with a retry limit.

Output from a run in which the aggregator ran out of memory and was killed:

An analysis of the design decisions, which are what separate this code from a toy example:

• _exit(127) rather than exit(127) after a failed execve. exit() runs the handlers registered with atexit and flushes the stdio buffers, which the child inherited from the parent through COW. That would duplicate output the parent has already printed. _exit() terminates without further ceremony, which is the right thing in a child that never managed to become another program.
• Code 127 is not arbitrary: it is the shell convention for "command not found". A real supervisor would distinguish this case (do not retry: the binary does not exist, retrying is pointless) from an execution failure (where retrying makes sense).
• waitpid(pid, ...) rather than wait(NULL). wait() reaps any child; waitpid() waits for exactly the one we care about. In a supervisor with several children, wait() would lead to subtle bugs.
• The five-second wait avoids the frantic restart loop: if the binary fails at startup, without that pause you would do thousands of forks per second and saturate the machine. systemd has the same mechanism (RestartSec), and for the same reason.
• The retry limit avoids restarting something broken forever. systemd has that too (StartLimitBurst), and you will see it in Services, Boot and systemd.

This program is, in miniature, what systemd does for you with every service. Writing it once saves you years of treating service managers as black boxes.

Since every process is born from another, all the processes on a system form a tree whose root is PID 1. On Linux, PID 1 is created by the kernel during boot and runs /sbin/init, which on modern distributions is systemd.

How to read it:

• The names in braces, {aggregator}(1878), are threads, not processes: they share 1877's mm. That is how pstree distinguishes them.
• meteo-api has three children that are not in braces: they are real processes, a classic preforking model (one master process and N workers).
• The chain sshd → sshd → bash → pstree is the complete trace of your session: the SSH daemon, the process for your connection, your shell and the command you just launched.

PID 1 is special in three ways:

1. It is the universal adoptive parent: it inherits every orphan and reaps them with wait().
2. It cannot die. If PID 1 terminates, the kernel panics: there is nobody left to manage the system.
3. Default signal actions do not affect it. The kernel ignores signals without an explicit handler that are aimed at PID 1, precisely so that an accidental kill -9 1 does not bring the machine down.

When the kernel decides that the CPU should stop running ingestor and start running the aggregator, a context switch happens. It is the operation that makes multitasking possible, and also one of the most expensive.

What has to be saved and restored:

Added up, the direct cost is around 1-3 microseconds. But that is not the real cost. The bulk of it is indirect:

• TLB flush. When cr3 changes, the cached address translations become useless. The new process starts out faulting on every access until it repopulates the TLB. (Process-context identifiers, PCID, mitigate this on modern CPUs.)
• Data cache pollution. The L1 and L2 cache lines are full of the previous process's data. The new one starts cold, and a miss all the way to main memory costs ~100 ns, as we saw in the memory hierarchy in 01-01.

All told, the effective cost of a context switch is between 3 and 10 microseconds. Let us do the number that matters:

From this comes a rule that will reappear in the next lesson: the quantum must be much larger than the cost of a context switch, or the system spends more time switching than working. A factor of 100 to 1000 is the reasonable range.

One distinction worth nailing down: a context switch is not the same as a system call. In 01-06 we saw that a system call changes mode (user→kernel) but it is still the same process: cr3 is not touched and the task_struct does not change. It costs 50-500 ns. A context switch changes process, and costs an order of magnitude more.

You can measure the real context switches on your system:

The cs column is context switches per second. Between 8,000 and 14,000 on a machine with an I/O workload is perfectly normal. If you saw 300,000, you would have a serious contention problem to investigate. The in column is interrupts per second, and we will come back to it in 02-07.

/proc is a virtual file system: it takes up no disk space, and its files are generated on the fly by reading kernel structures. Every process has its directory at /proc/<pid>/.

The most useful ones day to day:

A complete practical case: ingestor is not saving readings and you want to know what it is doing.

There is the problem, and it jumps right out: descriptor 4 points at 2026-08-30.dat, yesterday's file. The process has been running for more than a day and does not rotate the data file when the day changes. Today's readings are being written into yesterday's file. We did not even need to read the source code.

The cmdline needs tr '\0' ' ' because the arguments are separated by null bytes, not spaces; without that conversion you would see everything run together.

And ps -eo lets you build exactly the view you need:

Column by column: NI is the niceness value (nice) and PRI the resulting priority — both are the subject of the next lesson; RSS is the physical memory actually occupied, in KB; ELAPSED is the time since startup (sshd has been up for 6 days); NLWP is the number of threads. An operational decision is already visible here: ingestor has NI -5 (raised priority, because losing readings from the network is irreversible) and the aggregator has NI 5 (lowered, because it can wait).

Believing that fork() copies all the memory. It does not copy it: it shares with copy-on-write and only duplicates the pages that get written. Practical consequence: fork() on an 8 GB process is fast, but if the child writes all over the place you will end up paying for the copy anyway. And there is a treacherous case: if the system does not allow overcommit, fork() can fail with ENOMEM even though it is not going to copy anything, because the kernel reserves just in case.

Forgetting wait() in a process that creates children. It is the number one cause of zombies. The minimal solution in a daemon is to install a SIGCHLD handler that calls waitpid(-1, NULL, WNOHANG) in a loop until it returns 0, or simply signal(SIGCHLD, SIG_IGN) if you do not care about the exit codes (this tells the kernel not to generate zombies).

Trying to kill a zombie with kill -9. It does not work and never will: it is already dead. What you have to fix is the parent. And if you are in a hurry, restarting the parent turns its zombies into orphans, and systemd cleans them up instantly.

Confusing state R with "consuming CPU". R includes those waiting their turn. To find out who is really consuming CPU, look at the %CPU column in top or ps, not the state.

Panicking at a huge VSZ. A process with a VSZ of 4 GB and an RSS of 50 MB is perfectly normal: it reserves a lot of address space (which is free) and uses little physical memory. The metric that matters for RAM is RSS, and even that comes with caveats we will see in 02-04.

Using exit() instead of _exit() in a child after fork. The child inherited the parent's stdio buffers; exit() flushes them and you duplicate output that was already written. In the child, always _exit().

Diagnostic tip: when a process "has hung", the order that works is: ps -o stat to see the state, cat /proc/<pid>/stack if it is in D (it tells you what kernel function it is stuck in), strace -p <pid> if it is in S (it tells you what system call it is waiting on), and perf top -p <pid> if it is in R at 100% (it tells you what code is burning the CPU). Each state is investigated with a different tool.

Given this program, how many processes are created in total (counting the original) and how many times is the letter printed? Explain why.

Then answer: if you replace printf("M\n") with printf("M") (no newline) and redirect the output to a file, does the number of letters printed change? Why?

On meteo-01 you observe this:

1. What exactly is going on?
2. How often does the problem occur, and what does that tell you about the design of the aggregator?
3. What would happen if you left the system like this for a week? Work it out.
4. Give two solutions: an immediate one as an operator and a correct one as a developer.

Write a command that shows, for all processes belonging to the meteora user, the PID, the state, the number of threads, and the voluntary and involuntary context switches, sorted by involuntary context switches from highest to lowest. Then interpret what it would mean if the aggregator had 200,000 involuntary switches and only 300 voluntary ones.

Solution 1

Four processes are created in total and four letters are printed.

The reasoning step by step:

• After the first fork() there are 2 processes: P and A.
• Both execute the second fork(), because the child continues right after the fork that created it. P creates B, A creates C.
• All 4 reach the printf and each prints once. Total: 4 lines.

The general formula is 2^n processes for n consecutive fork()s with no conditionals.

On the second part: yes, it changes, and more than 4 can end up being printed. This is the subtle detail:

• With \n and output to a terminal, stdio uses line buffering: every printf flushes immediately. 4 letters.
• Without \n and output to a file, stdio uses full buffering of 4096 bytes. The "M" stays in the user-space buffer and is only written when the program ends.
• The problem: the stdio buffer lives in the process's address space, so it is inherited across the fork. If the order were printf("M"); fork();, the child would inherit a buffer that already contains "M" and would write it too: you would see more letters than expected.

In the code as written (the forks come before the printf) it is still 4 letters, but the experiment reveals the underlying reason why a child uses _exit() and not exit(), and why it is a good idea to call fflush(NULL) before a fork if there is pending output. It is a real and baffling bug when it shows up in production.

Solution 2

1. What is going on. The aggregator (PID 1877) launches child processes called rotator — presumably to rotate the daily file in /var/lib/meteora/readings/ — and never calls wait(). The children finish their work correctly, but their task_structs stay in state Z because nobody collects their exit code. The children are not hung: they are dead and unburied.

2. How often. Looking at the ELAPSED column: 03:12, 02:11, 01:10, 00:09. The differences are approximately 61 seconds. That is, one per minute. This indicates that the aggregator has a loop or a timer that launches the rotator every minute, which is suspicious in itself: rotating a daily file does not require checking it every 60 seconds, and it suggests a design with a fork inside the main loop.

3. A week like this. The calculation:

The default PID limit on Linux (/proc/sys/kernel/pid_max) is usually 32,768 in conservative configurations. With that value:

That is, in a week you would have 60,480 zombies... except that the limit of 32,768 is reached first, around day 22 if nothing else were consuming PIDs. But the disaster does not wait for that moment: PIDs are assigned in a circular fashion, so long before they run out you will start seeing fork: Cannot allocate memory on any new command, including the ssh you would use to log in and fix it. That is the genuinely ugly scenario: the system has not gone down, but you cannot run anything on it.

4. The two solutions.

Immediate, as an operator: restart the parent process.

When 1877 dies, its zombies become orphans, systemd adopts them and its wait() loop cleans them up instantly. Killing the zombies directly with kill -9 4021 would do nothing.

Correct, as a developer: have the aggregator reap its children. The most robust approach is a SIGCHLD handler:

Three details make this work properly: the loop is essential because signals are not queued (if three children die at almost the same time, a single SIGCHLD may arrive); WNOHANG stops the handler from blocking when there is nothing left to reap; and preserving errno avoids corrupting the value the interrupted code was working with.

If the exit code is of no interest at all, the one-line alternative is signal(SIGCHLD, SIG_IGN), which tells the kernel to discard the children without creating zombies.

Solution 3

The command:

Typical output:

A shorter alternative if your ps supports it, using the extended format:

although for the context switch counters there is no way around going to /proc/<pid>/status, because ps does not expose them.

Interpreting the case described (200,000 involuntary, 300 voluntary).

The two types mean opposite things:

An aggregator with 200,000 involuntary and only 300 voluntary switches says three things clearly:

1. It hardly ever waits for I/O. With 300 voluntary blocks over hours of execution, it barely touches the disk or the network: it has loaded the data and it computes.
2. It is a CPU-bound process that wants to run continuously.
3. There is real competition for the CPU. 200,000 preemptions mean the scheduler keeps taking it away because there are other runnable processes. If the aggregator were the only active process, its involuntary count would be low.

The operational conclusion: the aggregator is competing with ingestor and meteo-api for the CPU, and since its tasks are deferred (hourly averages) while the other two are latency-sensitive, the right decision is to lower its priority, not raise it:

That does not take CPU away from it when the machine is idle — it will still use all of it — but it guarantees that it yields when ingestor has readings to attend to. Exactly why it works that way, and what the scheduler does with that number, is precisely the subject of the next lesson.

A program is a passive file; a process is that program alive, with a memory image in four regions — shared read-only code, data, a heap that grows upward and a stack that grows downward — and a record in the kernel. That record is the PCB, which on Linux is task_struct: about 7 KB holding the identity, the state, the scheduling data, the CPU context, the pointer to the address space, the descriptor table and the credentials. The fact that mm is a pointer is what will make threads possible.

The states are not a theoretical whim: they reflect that the CPU is a scarce resource and that blocking on I/O must release it. The real ps codes refine the model, and two are especially revealing: R means runnable, not running, and D is an uninterruptible sleep on disk that not even kill -9 gets you out of.

Creation through fork() + execve() looks strange until you see what it enables: between the duplication and the replacement there is a window in which the child can change user, redirect descriptors or apply limits before turning into another program. And copy-on-write makes duplicating a multi-gigabyte process cost half a millisecond. Zombies exist because the exit code belongs to the parent, and they are fixed by fixing the parent; orphans are adopted by PID 1, which is the original purpose of init. The context switch costs between 3 and 10 µs once you count the TLB and cold caches, which settles once and for all the order of magnitude of the quantum.

You now know what a process is, how it is born, what states it lives in and what it costs to switch from one to another. What is missing is the question we have been deferring in every section: when there are ten processes in state R and four cores, which one gets the CPU, for how long, and on what criterion? That decision, taken thousands of times per second, is what we will see in CPU Scheduling, where we will finally understand what that renice -n 10 we closed the last exercise with really does.