We closed module 1 with web-store running in rutas-norte-dev and an uncomfortable lesson learned: when we deleted that bare pod, it never came back. Before fixing that with controllers, you need to properly understand the piece everything else is built on. The Pod is the smallest deployable unit in Kubernetes: you do not deploy containers, you deploy pods that contain containers. All the machinery in this course —ReplicaSets, Deployments, Services, probes, autoscaling, network policies— exists to create, maintain, reach, watch or protect pods. In this lesson we open the pod right up: why it, and not the container, is the smallest unit, what exactly the containers inside one share, how to read a manifest field by field, what every phase and every state you will see in kubectl get pods means, how a pod is born and how it dies, and how to use ephemeral pods as a diagnostic tool. By the end, you will be able to interpret an ImagePullBackOff or a CrashLoopBackOff without searching the internet.

1. Why the smallest unit is the pod and not the container
2. What containers in the same pod share (and what they do not)
3. The pause container: the one that holds the pod together
4. Anatomy of a pod manifest: bookings-api field by field
5. Pod phases and container states
6. Typical reasons: ContainerCreating, ImagePullBackOff, CrashLoopBackOff
7. restartPolicy: who restarts what
8. The full lifecycle: from creation to SIGTERM
9. Ephemeral pods for debugging
10. Why bare pods are never managed in production

This is the first question everybody coming from Docker asks. If Kubernetes orchestrates containers, why introduce an intermediate wrapper instead of scheduling containers directly?

The answer is that some processes need to be glued together. Not "nearby" or "on the same machine", but literally sharing network and disk, starting and dying together, scheduled as a single indivisible unit. Examples you will find on any serious platform:

• An application server and a log collector that reads the files the server writes.
• A service and a proxy that terminates TLS for it and adds metrics (the service mesh pattern).
• A main container and another one that downloads configuration or certificates before it starts.

If the smallest unit were the container, the scheduler could place the log collector on one node and the application on another, and the design would fall apart. Kubernetes solves this with a clear contract:

A pod is a group of one or more containers scheduled together on the same node, sharing network and storage, and living and dying as a unit.

Two practical consequences worth committing to memory right away:

• The pod is the unit of scheduling. The scheduler assigns pods to nodes, not containers. The resources added up to decide whether it "fits" on a node are those of all its containers.
• The pod is the unit of scaling. When you scale bookings-api to 6 replicas, you create 6 complete pods. You cannot scale a single container inside a pod.

That second consequence is the most important design rule: if two processes scale at different rates, they do not belong in the same pod. At Rutas Norte, bookings-api and redis-cache might look like good pod mates (the API queries the cache constantly), but it would be a serious mistake: every API replica would have its own isolated cache, and scaling the API would multiply the caches. They go in separate pods and talk to each other through a Service.

The overwhelmingly common case, and the one we will use almost always in this course, is one container per pod. The multi-container patterns (init containers and sidecars) have their own lesson: Init Containers, Sidecars and Multi-Container Patterns.

A pod is not a metaphor: it is a very concrete construction of Linux namespaces shared between processes. This table is the exact reference.

The most useful and the most dangerous consequence of the shared network namespace:

• Useful: inside a pod, two containers talk over http://localhost:8080. No DNS, no Service, no network latency.
• Dangerous: two containers cannot listen on the same port inside a pod. If you put two nginx containers in the pod, the second will fail with "address already in use".

How does Kubernetes get several containers to share the network if each of them starts and dies on its own? With an elegant trick: for every pod, the kubelet starts an invisible container called the pause container (pause, also known as the infrastructure container or sandbox).

How it behaves:

1. It is created first, before any container of yours.
2. It creates and owns the pod's network, IPC and UTS namespaces. It is the one that receives the IP.
3. The pod's other containers are started by joining the pause container's namespaces.
4. Its only job is to sleep. Its source code is a few dozen lines long: it blocks waiting for signals and reaps orphaned processes.

Its value is that it survives your containers restarting. If the bookings-api process crashes and the kubelet restarts it, the pause container is still alive, so the pod IP does not change. Only when the whole pod is destroyed does the sandbox disappear and the IP get released. This explains exactly what you observed in the final exercise of module 1: killing the process pushed RESTARTS up to 1 but the IP stayed the same.

You can see it on the node, even though it never shows up in kubectl get pods:

You never have to manage it. But knowing about it saves you confusion when reading documentation or debugging at a low level.

Let's write the pod for bookings-api, the second component of Rutas Norte. Since registry.rutasnorte.example is fictional, we simulate the API with a public Node.js image that starts a minimal server. Save it in the project repository.

A field-by-field walkthrough:

• apiVersion: v1 / kind: Pod: the Pod belongs to the API's core group, which is why it carries no group prefix. We saw this in Objects, YAML Manifests and the Declarative Model.
• metadata.name: unique within the namespace. It will also be the pod's hostname.
• metadata.labels: the project's three mandatory labels. Nobody uses them yet, but in the next lesson they will be the key by which a ReplicaSet recognises "its" pods. Their full syntax is in Labels, Selectors and Annotations.
• spec.restartPolicy: Always: this is the default value; we write it out to make it explicit. Section 7.
• spec.terminationGracePeriodSeconds: 30: the seconds granted to the container to shut down cleanly before it is killed by force. This is also the default value. Section 8.
• containers[].name: the container's identifier inside the pod. It is what you pass to kubectl logs -c or kubectl exec -c when there is more than one.
• image: an explicit tag, never latest, following the project convention.
• command / args: they override the image's ENTRYPOINT and CMD respectively. Here we use them to make a generic Node image behave like our API. Watch out for the mapping, a classic source of mistakes:

• ports: informational, it opens nothing. It serves to document and, above all, to give the port a name (http) that a Service can later reference by name instead of by number.
• env: environment variables. The first is a literal value; the second uses fieldRef to inject a piece of data from the pod itself (the downward API). The serious sources of configuration —ConfigMaps and Secrets— arrive in Environment Variables.
• resources: requests is what the scheduler reserves; limits is the ceiling the kubelet enforces. Full detail in Resource Quotas and Limits.
• volumeMounts / volumes: the emptyDir volume is an empty directory that lives and dies with the pod. Useful for temporary files and for sharing data between containers in the same pod. Persistent storage is module 5.

Apply it and check it:

Notice that we called localhost:3000 from inside the pod itself: that is the shared network namespace in action.

Here we have two concepts that get constantly confused and that are worth separating sharply.

It is a high-level summary of the whole pod. It has only five possible values:

One important detail: Running does not mean "working properly". It means "there are live processes". A pod can be Running and return 500s to everybody. Telling "alive" apart from "healthy" is exactly the job of probes, in Health Checks and Probes.

Each container has its own state, with only three values, each carrying a reason that is what really tells you what is going on:

The STATUS column of kubectl get pods is an invention of the CLI that mixes the two things to be useful: it shows the phase, unless there is a more informative container reason, in which case it shows that instead. That is why you see ImagePullBackOff in that column even though it is not a phase.

To see the real data, without the mixing:

These three reasons cover the vast majority of the problems you will run into. They deserve individual treatment because each one is diagnosed differently.

A transient and normal state: the kubelet is pulling the image, mounting volumes and setting up the pod's network. It is only a problem if it never moves on. If it sits there for minutes, kubectl describe will tell you why: almost always a volume that cannot be mounted or a Secret/ConfigMap that does not exist.

The kubelet cannot pull the image. First you will see ErrImagePull (immediate failure) and then ImagePullBackOff (retries with increasing waits). Let's trigger it on purpose, using the project's fictional registry:

The diagnosis is always in the events:

The four possible causes, in order of frequency:

1. A misspelled name or tag (by far the most common).
2. A private registry with no credentials: an imagePullSecrets is missing. Covered in module 8.
3. An unreachable registry from the node, as in this example.
4. The tag does not exist in the registry: typical when deploying a version that has not been published yet.

Clean up the test:

The most misunderstood one. It is not an error in itself: it means the container starts, finishes, the kubelet restarts it, and it finishes again. To avoid burning through the node, the kubelet waits longer and longer between retries: 10s, 20s, 40s, 80s… up to a maximum of 5 minutes. CrashLoopBackOff is the name of that wait.

The key to the diagnosis: the reason is never in describe, it is in the logs of the previous attempt.

There is the real cause. Remember the --previous flag (or -p): without it, kubectl logs tries to read the current container, which may still be in the wait and have nothing to say.

The most frequent causes of CrashLoopBackOff:

spec.restartPolicy applies to every container in the pod and accepts only three values. It is an immutable field: it cannot be changed on an already created pod.

Three clarifications that avoid very common confusion:

1. The policy is applied by the node's kubelet, not by a control plane controller. That is why it works even on bare pods, with no Deployment behind them.
2. Restarting a container is not recreating the pod. The pod is still the same object, with the same name, the same node and the same IP. Only the RESTARTS counter goes up. This is the exact difference you verified in exercise 3 of module 1.
3. Deployments require Always. As we will see in Deployments, their pod template accepts no other value. OnFailure and Never are the territory of Jobs and CronJobs.

This is the journey you already saw in Kubernetes Architecture, now focused on the pod:

This is the part almost nobody studies and the one that causes 502 errors during deployments. When you delete a pod, this is exactly what happens:

1. The API marks the pod with a deletionTimestamp and gives it a grace period (terminationGracePeriodSeconds, 30 s by default). The phase becomes Terminating.
2. In parallel, the pod is removed from the Endpoints of every Service that selected it, so it stops receiving new traffic (02-05).
3. The kubelet sends SIGTERM to the PID 1 process of each container. This is where your application must stop accepting new connections, finish the ones in flight and close cleanly.
4. It waits until the grace period runs out.
5. If the process is still alive when the period ends, the kubelet sends SIGKILL. No negotiation possible.
6. The sandbox is destroyed and the object disappears from the API.

To watch it in slow motion, delete it with a long period from another terminal and observe:

Practical consequences for Rutas Norte:

• notifications-worker must catch SIGTERM. If it ignores it, every deployment kills it after 30 seconds with SIGKILL halfway through sending an email, and a customer never gets their confirmation.
• --grace-period=0 --force is dangerous. It tells the API to treat the pod as dead without waiting for confirmation from the kubelet. On a database like bookings-postgres it can cause corruption or two instances writing at once. Use it only for pods stuck on a lost node.
• Too short a grace period cuts off requests in flight. If bookings-api has requests that take 20 seconds, a period of 10 will cut them dead.

A pod is not only a workload: it is also the best Swiss army knife for diagnosis inside the cluster. kubectl run with --rm -it creates a pod, gives you a terminal inside it and deletes it when you exit.

Inside you have curl, dig, nslookup, ping, tcpdump, netstat... everything your application image does not carry. A breakdown of the flags:

A real example we will use a lot from the Services lesson onwards: checking whether bookings-api responds from inside the cluster.

If what you want is to debug a pod that already exists and whose image has no tooling (nginx:alpine does not even ship curl), the modern solution is ephemeral containers, which are injected into a live pod without restarting it:

It shares the nginx container's network namespace, so you can run curl http://localhost:80 as if you were it. It is the tool we will go deeper into in Application Debugging and Cluster Events.

You already experienced this at the end of module 1, but now that you know the lifecycle you can understand the deeper reason: a pod has no recovery mechanism of its own. The kubelet restarts its containers, yes, but the pod as an object only exists as long as somebody declares it.

And there are situations, very frequent ones, in which the pod disappears without anybody deleting it:

For Rutas Norte, any of those four rows amounts to the four-and-a-half-hour night-time outage the company suffered in March. That is why the project rule is blunt:

At Rutas Norte, the only bare pod allowed is an ephemeral debugging pod with --rm. Everything else is governed by a controller.

What we need is something that watches permanently and says "I want 3 pods like this one, always". That is a ReplicaSet, and it is the next lesson.

Before moving on, leave the cluster with both pods running:

• Putting several unrelated processes in the same pod. The control question is: "would they scale at the same rate and die together?". If the answer is no, they are separate pods. A bookings-api with redis-cache inside it would be a serious design mistake.
• Confusing "restarting the container" with "recreating the pod". The first is done by the kubelet, keeps the name and IP and bumps RESTARTS. The second is done by a controller and produces a new object with a new IP.
• Looking for the cause of a CrashLoopBackOff in kubectl describe. It is in kubectl logs --previous. describe will tell you that it restarts, not why.
• Reading Running as "it works". It only means there are live processes. Real health is determined by probes.
• Forgetting --previous with restarting containers. Without it you lose precisely the log you care about.
• Ignoring SIGTERM in the application. It is the number one cause of requests cut off during deployments. Catch the signal, stop accepting new connections, drain the open ones and exit with code 0.
• Overusing --force --grace-period=0. It skips the graceful shutdown. On stateful workloads it can corrupt data.
• Putting two containers listening on the same port inside a pod. They share the port space: the second will fail to start.
• Tip: generate the skeleton of any manifest with kubectl run bookings-api --image=node:20-alpine --dry-run=client -o yaml > pod.yaml and edit it. It is far faster and safer than writing YAML from scratch.
• Tip: kubectl explain pod.spec.containers.lifecycle --recursive is your offline documentation and always matches your cluster's version.

Deliberately create these three pods in rutas-norte-dev and, for each one, identify the pod phase, the container state and reason and the exact command that reveals the root cause:

1. pod-a: image nginx:9.9.9-invented.
2. pod-b: image redis:7.2-alpine with command: ["redis-server", "--nonexistent-file"].
3. pod-c: image busybox:1.36 with command: ["sh", "-c", "echo report generated"] and restartPolicy: Never.

Explain why the third one is not an error, even though its STATUS is not Running.

Write k8s/base/shared-pod.yaml with a pod called shared-demo in rutas-norte-dev that has two containers and an emptyDir volume called exchange:

• writer: image busybox:1.36, writes a line with the date to /data/occupancy.log every 5 seconds.
• reader: image busybox:1.36, runs tail -f on that same file mounted at /input/occupancy.log.

Then demonstrate with commands: (a) that the reader sees what the writer writes, (b) that both containers share the same IP and (c) that they do not share the root filesystem.

Create a notifications-worker pod with busybox:1.36 that catches SIGTERM, writes closing: finishing pending sends to the log, waits 5 seconds and exits. Set terminationGracePeriodSeconds: 30.

1. Delete it and check in the logs that the message was written before it died.
2. Repeat the experiment with terminationGracePeriodSeconds: 2 and explain what changes and why it is dangerous for Rutas Norte.

Solution 1

pod-c is not an error: pod-a's 9.9.9-invented tag does not exist in the registry and pod-b's redis-server rejects the parameter. pod-c, by contrast, did exactly what it was asked to do, exited with code 0 and, thanks to restartPolicy: Never, did not restart: phase Succeeded. This is the normal behaviour of a finite task such as occupancy-reports. Had we left it with restartPolicy: Always, it would fall into CrashLoopBackOff despite never failing, because Always also restarts successful exits.

Solution 2

The file only exists under /data for the writer and under /input for the reader: the volume is shared, the root filesystem is not. Each container sees the volume at the path where it mounted it, and nothing else of the other one.

Solution 3

The process received SIGTERM, ran its shutdown routine and exited under its own steam within the 30-second period.

With terminationGracePeriodSeconds: 2, the kubelet sends SIGTERM, waits only 2 seconds and sends SIGKILL while the worker is still in its sleep 5. The process dies halfway through shutting down. For Rutas Norte that means a booking confirmation email left half sent on every deployment: the customer has paid for their ticket and receives no receipt. The rule is to size the grace period above the real time the component takes to drain its pending work.

The pod has stopped being a black box. You know why Kubernetes chose the pod and not the container as the smallest unit: some processes need to share network, volumes and lifecycle, and the scheduler has to treat them as a single indivisible thing. You know exactly what their containers share —a network namespace with a single IP, localhost, IPC, volumes, node and fate— and what they do not —root filesystem, PIDs and resources—, and you know that what holds those namespaces together is the discreet pause container, the reason the IP survives restarts.

You have written the bookings-api manifest field by field, telling command/args apart from ENTRYPOINT/CMD, and you have learned to read a pod's real state by separating the five phases (Pending, Running, Succeeded, Failed, Unknown) from the three container states (Waiting, Running, Terminated) with their reasons: ContainerCreating is normal, ImagePullBackOff is diagnosed in the events and CrashLoopBackOff in kubectl logs --previous. You know what restartPolicy does and who applies it, and you have followed the full lifecycle through to its uncomfortable end: SIGTERM, grace period and SIGKILL, the difference between a clean deployment and a customer without their booking confirmation. And you now have ephemeral --rm -it pods and kubectl debug on your tool belt as diagnostic instruments.

What brought us here is still outstanding: a pod, on its own, does not recover. If the node goes down, if it is evicted for lack of memory or if somebody deletes it, bookings-api vanishes from the cluster and does not come back. In the next lesson, ReplicaSets, we will meet the first controller of the course: the one that tirelessly runs the reconciliation loop we studied in the architecture, counts how many pods there are against how many should exist and creates the missing ones. We will delete a web-store pod on purpose and watch it be reborn in seconds.