Three questions were left open at the end of the previous lesson. How do we make the schema migration run automatically before bookings-api starts, instead of as a separate Job? How do we stop bookings-api from going into CrashLoopBackOff when it starts before bookings-postgres? How do we export PostgreSQL metrics without touching the official image?

All three have the same answer: by putting more than one container in the same pod.

Since lesson 02-01 we have known that a pod is a group of containers sharing a network namespace — and therefore localhost —, volumes and a life cycle. Until now we had not made use of that capability: every Rutas Norte pod has a single container. In this lesson we will use it with judgement, which is the hard part: most badly designed multi-container pods come from lumping two applications together because "they are related", and that always ends badly.

1. When a pod needs several containers and when that is a mistake
2. Init containers: sequential preparation before start-up
3. Init container use cases in Rutas Norte
4. Diagnosing failed init containers
5. Native sidecars: init containers with restartPolicy: Always
6. The three classic patterns: sidecar, ambassador and adapter
7. Sidecar pattern: a metrics exporter alongside bookings-postgres
8. Ambassador pattern: the connection to the payment gateway
9. Adapter pattern: normalising the notifications-worker log
10. Start-up and termination order, and the real cost

The golden rule, stated precisely:

Two processes go in the same pod when they are so tightly coupled that it makes no sense to scale, update or run them separately, and one of them exists solely to serve the other.

The immediate consequence of sharing a pod is that they share a fate: they are scheduled on the same node, they scale together, they restart together and they are updated together. If bookings-api needs 5 replicas and notifications-worker needs 2, putting them in the same pod forces you to have 5 of each. That is a design mistake, not an optimisation.

Three sanity-check questions before adding a container to an existing pod:

1. Do they always need to scale together? If the answer is no, they are two different workloads.
2. Does the second one serve only the first? If other pods need it too, it is a separate Service.
3. Do they need to share localhost or a file system? If not, there is no technical reason to put them together.

Kubernetes offers two categories of auxiliary container:

• initContainers: they run first, in order, one after another, and must finish successfully.
• Auxiliary containers that accompany the main one throughout the pod's life: the sidecars.

An init container is a container that runs to completion before any main container starts. If there are several, they run in the order in which they appear in the manifest, each waiting for the previous one to finish successfully.

Properties that set them apart from normal containers:

That last row is more useful than it looks: the init container can use an image with psql, curl or git that the production image does not have, without fattening the final image or widening its attack surface.

Basic syntax:

Init containers also consume resources and take part in the calculation of what the pod asks the scheduler for. The effective formula is:

That is, an init container asking for 2 GiB forces the scheduler to find a node with 2 GiB free, even if the application only needs 256 MiB. It pays to keep them light.

When an environment is brought up from scratch, bookings-api may start before the database is ready, fail to connect, exit and go into CrashLoopBackOff. It eventually recovers, but the start-up takes minutes and the events fill up with noise that masks real problems.

pg_isready is a utility from the PostgreSQL image itself that returns 0 if the server accepts connections. The loop deliberately has no attempt limit: the pod will sit in Init:0/1 indefinitely, which in kubectl get pods is a crystal-clear signal of "the database is not available", far better than a CrashLoopBackOff whose cause you have to go and dig out of the logs.

An honest design note: this does not replace the application handling reconnection. If bookings-postgres goes down two hours later, the init container is no longer around to help. It is a convenience for start-up, not a guarantee of resilience.

Here we pick up the question left open by lesson 06-03. The migration can live in an init container instead of a separate Job:

The obvious advantage: the migration is coupled to the deployment. It is impossible to deploy the 2.5.0 code without having migrated, because the application container does not start unless the init container finishes successfully.

But there is an important trap you need to know about:

With 4 replicas of bookings-api, the migration init container runs 4 times, potentially in parallel during an update.

Consequences and mitigations:

• The migration must be idempotent and protected by a lock. IF NOT EXISTS is not enough: two simultaneous ALTER TABLEs can block each other. A serious migration tool (Flyway, Liquibase, golang-migrate) takes an application lock in the database and the other instances wait.
• If the migration takes minutes, every replica pays that wait, and the deployment drags on.

Recommendation for Rutas Norte: a separate Job for large or destructive migrations (indexes over millions of rows, type changes), an initContainer for the small, idempotent day-to-day migrations.

web-store serves static HTML from nginx, and the footer template changes with the environment. Instead of building three images, an init container generates the file on a shared volume:

The mechanism is exactly the emptyDir of 05-01: an empty volume created with the pod and shared by all its containers. The init container writes, nginx reads it read-only. When the pod dies, the emptyDir disappears, and it does not matter: it is regenerated on the next start-up.

Init containers have their own states in the STATUS column, and knowing how to read them saves a lot of time.

A real case: bookings-api stuck because the database does not answer.

Init:0/2 sustained for four minutes: the first init container (wait-for-postgres) is still in its loop.

The key is -c <name>. Without that flag, kubectl logs tries to read the main container, which does not exist yet, and answers with a confusing error.

Another case: the migration fails.

Diagnosis: the migration is not idempotent. The IF NOT EXISTS was missing.

Useful commands for inspecting init containers:

In the describe output, the init containers appear in an Init Containers: block before the Containers: block, each with its state, its exit code and its termination reason.

For years, a sidecar was simply "one more container in the containers list". It worked, but it had two serious flaws that Kubernetes 1.29 fixed and that have been stable since 1.33.

Problem 1: there is no start-up order. All the containers in containers start in parallel. If the sidecar is a proxy through which the application has to reach the network, and the application starts before the proxy, the first requests fail.

Problem 2: Jobs never finished. A Job whose pod has a sidecar was a problem with no clean solution. The main container finishes successfully, but the sidecar stays alive, so the pod never reaches Succeeded and the Job hangs indefinitely. The only way out was ugly workarounds: sentinel files in an emptyDir, or having the main container kill the sidecar through the API.

That single field completely changes the container's semantics:

The four practical consequences, in order of importance:

1. It starts before the main containers, guaranteeing that the proxy or the exporter is ready when the application starts working.
2. It stays alive for the whole life of the pod and is restarted if it dies, which a normal init container does not do.
3. It is terminated after the main containers, so a log sidecar captures the last messages of the shutdown.
4. It does not stop a Job from finishing. When the containers in containers are done, the kubelet stops the sidecars and the pod reaches Succeeded. This unblocks all batch work with sidecars: a CronJob with a proxy, with a metrics exporter or with a log adapter simply works.

A clarification about ordering: the pod does not wait for the sidecar to finish (it never will), but for it to be started — and to pass its startupProbe if it has one. Unlike normal init containers, sidecars do support probes, the subject of lesson 07-01.

A simple decision rule: if the auxiliary container has to be ready before the application, or if the pod belongs to a Job, use a native sidecar. In every other case, a normal container in containers is still perfectly valid.

The three names come from the foundational article by Brendan Burns and Dave Oppenheimer on design patterns for distributed systems. They differ in which way the flow goes and what they transform.

Another way of remembering it:

• The sidecar adds something.
• The ambassador simplifies what the application sees of the outside world.
• The adapter simplifies what the outside world sees of the application.

All three rely on the same two mechanisms, which you already know from 02-01:

• Shared localhost: all the containers in the pod share the network namespace, so they talk to each other over 127.0.0.1 without going through the cluster network, without DNS and with no appreciable latency. Corollary: two containers in the same pod cannot use the same port.
• Shared volumes: an emptyDir mounted in both allows files to be passed along. It is the channel used by the adapter pattern.

The postgres:16.4 image does not expose metrics in Prometheus format. Modifying it would be a mistake: we would lose the official updates and would have to maintain an image of our own. The solution is a sidecar that connects to PostgreSQL over localhost, runs status queries and publishes the result at /metrics.

We add the sidecar to the StatefulSet we built in 06-01:

Points that explain the pattern:

• DATA_SOURCE_URI: localhost:5432: there is no service name and no DNS. The sidecar talks to PostgreSQL through the pod's loopback. That is the fundamental advantage: minimal latency, no need to expose port 5432 to the cluster network for this, and no need for an extra NetworkPolicy, because the traffic never leaves the pod.
• Native sidecar: it starts before PostgreSQL. Strictly speaking it is not essential here, but it guarantees that we do not lose the metrics of the first seconds of life, which are precisely those of the start-up and the WAL recovery.
• Modest, separate resources: 20m of CPU and 48 MiB. They are added to PostgreSQL's in the scheduler's calculation.
• Its own hardening: the sidecar runs as nobody and with a read-only file system. It needs nothing more, and this way a failure in the exporter does not compromise the database container.

An important note on QoS: in 03-05 we established that bookings-postgres is of class Guaranteed. For the pod to stay that way, all of its containers — sidecar included — must have requests equal to limits. With the values above (20m/100m, 48Mi/96Mi) the pod would drop to Burstable. If we want to keep Guaranteed, we have to make them equal:

It is an unintuitive consequence of adding sidecars that is worth keeping in mind.

The Service that exposes the metrics:

Checking it:

pg_up 1 confirms that the exporter can connect; numbackends gives the active connections. These metrics are the ones Prometheus will collect in lesson 07-03; here we have only built the source.

bookings-api takes payments through pagos.proveedorexterno.example, an external service with the usual complications: mutual TLS with a client certificate, retries with backoff, a circuit breaker for when the provider is slow, a request-per-second limit and a different test endpoint in each environment.

Putting all that logic into bookings-api means writing it in Node.js, maintaining it, testing it and doing it all again if a second provider appears tomorrow. The ambassador takes it out of the code: a local proxy that listens on localhost and takes care of everything.

And the proxy configuration, boiled down to the essentials:

What Rutas Norte has gained:

And above all: bookings-api makes a plain HTTP POST to http://127.0.0.1:8081/charges and that is that. In local development testing, that endpoint can be a simulator; the application cannot tell the difference.

The NetworkPolicy from 04-06 that authorises egress to the gateway still applies to the pod, not to the container, so it does not change: the whole pod needs permission to reach external port 443.

A necessary clarification: if this idea is applied to all the traffic of all the pods, with a control plane distributing the configuration, it is no longer called an ambassador but a service mesh (Istio, Linkerd). That is the territory of lesson 08-04; here we solve one specific case without adopting a whole platform.

notifications-worker is a legacy component that writes to a file, with a format of its own and multi-line traces when there is an exception:

The DaemonSet log collector from 06-02 reads the containers' standard output, not arbitrary files, and even if it read them, that format is not queryable: there are no fields, and an exception is split into three unrelated entries.

The adapter solves both problems: it reads the file from a shared volume, joins the continuation lines and emits structured JSON on its own standard output, where the collector does pick it up.

The result on the adapter's standard output, which is what the node collector takes away:

The exception travels whole in a single record, with its fields separated and tagged with the component and the environment. In 07-05 this will allow queries such as "all the SEND_ERRs for booking 4472".

Here the native sidecar contributes something concrete and important: because it is terminated after the main container, it captures the last messages notifications-worker writes during its orderly shutdown on SIGTERM (02-01). With a normal container in containers, both receive SIGTERM at the same time and those last lines — often the ones explaining why it stopped — are lost.

The sizeLimit: 256Mi on the emptyDir is not optional: an emptyDir with no limit writes to the node's disk until it fills it, and then the node goes into disk-pressure and evicts other people's pods.

With init containers and native sidecars, a pod's life cycle looks like this:

Points to remember:

1. Normal init containers run sequentially and to completion.
2. Native sidecars start in the declared order, and only need to be started.
3. The main containers all start at once, with no ordering guarantee among them.
4. On termination, the main ones first; then the sidecars, in reverse order.
5. terminationGracePeriodSeconds belongs to the pod, not to each container: it is the total budget for the whole shutdown.

Each sidecar is one more container for every pod, and that multiplication is easy to underestimate. Rutas Norte's production figures:

Half a core and 600 MiB in auxiliary containers alone. During a bank-holiday peak, with bookings-api autoscaled to 20 replicas (09-01), the ambassador alone is 1 CPU and 1.25 GiB.

And there are less visible costs:

• Every sidecar is an image that has to be maintained, scanned and updated (08-05). Three sidecars are three more supply chains.
• Every sidecar can fail and, with restartPolicy: Always, go into a restart loop dragging the whole pod with it.
• Every sidecar adds time to the start-up, and the native ones add it sequentially before the application.
• Debugging gets harder: every kubectl logs and kubectl exec now needs the -c flag.

Sanity-check questions before adding a sidecar:

1. Could a DaemonSet do it, one per node instead of one per pod? For node logs and metrics, almost always yes.
2. Could the application itself do it with a library? Sometimes five lines of code replace a 60 MiB container.
3. Does the sidecar justify its cost multiplied by the maximum number of replicas? Work out the worst case, not the usual one.

Forgetting -c <container> in kubectl logs and kubectl exec. With several containers, kubectl demands to know which one. The error a container name must be specified is unambiguous, but when there are init containers the message can be misleading because the main container does not exist yet.

Putting an init container into an infinite loop with no visibility. An until ... done without an echo leaves the pod in Init:0/1 with no log explaining the wait. Always print something on each iteration.

Heavy init containers. The scheduler reserves the maximum among the init containers, so one asking for 2 GiB forces it to find a node with 2 GiB free even if the application needs 256 MiB. Keep them small.

Migrations in an initContainer without a lock. With N replicas, the migration runs N times, potentially in parallel. Without idempotency and without an application lock, the result is a half-migrated database. Use a migration tool that takes a lock, or a separate Job (06-03).

Two containers in the same pod listening on the same port. They share the network namespace, so the second one fails with address already in use. Keep a record of which port each auxiliary container uses (3000 the API, 8081 the ambassador, 9187 the exporter...).

A classic sidecar in a Job's pod. This is the historic trap: the Job never finishes. The solution in 1.29+ is a native sidecar with restartPolicy: Always as an init container.

An emptyDir without sizeLimit for logs. A log growing out of control fills the node's disk and causes disk-pressure, evicting pods that had nothing to do with it. Always set sizeLimit.

Breaking the QoS class by adding a sidecar. A pod is Guaranteed only if all of its containers have requests == limits. Adding a sidecar with different values downgrades the pod to Burstable and changes its eviction priority (03-05).

Tip: name your containers after their function, not their technology. payments-ambassador is far more useful in an alert at three in the morning than envoy.

Tip: kubectl describe pod is the best view. It shows Init Containers: and Containers: in separate blocks, with the state, the exit code and the reason for each. It is faster than chaining jsonpath expressions.

Tip: kubectl logs --all-containers=true dumps the whole pod at once, useful for reconstructing the sequence of a troublesome start-up.

In rutas-norte-dev, create a Deployment api-demo with one replica of nginx:1.27.2-alpine and an init container that waits for a Service called dependency-demo to exist and answer. Apply the Deployment before creating the dependency and watch the pod's state. Then create the dependency (a Deployment and a Service with nginx) and check that api-demo starts.

In rutas-norte-dev, create a Deployment worker-demo with:

• A main container worker that every 5 seconds writes a line in a proprietary format to /logs/output.log (for example 2026-08-05 10:00:00 | SEND_OK | booking=4471).
• A native sidecar adapter (an init container with restartPolicy: Always) that follows that file and emits each line on its standard output with the prefix [adapted].
• A shared emptyDir with sizeLimit: 64Mi.

Verify that the sidecar starts before the main one and that it emits the lines.

In rutas-norte-dev, create a Job report-with-sidecar whose pod has a main container that takes 15 seconds and finishes, and a native sidecar that writes something every 3 seconds indefinitely. Check that the Job does reach Complete. Then reason about what would have happened if the sidecar had been declared in containers instead of as an init container with restartPolicy: Always.

Solution 1

Init:0/1: the init container is running and none has completed.

Now the dependency:

The pod went from Init:0/1 to PodInitializing and then to Running with no restarts at all. That RESTARTS: 0 is the improvement over letting the application go into CrashLoopBackOff while it waits.

Solution 2

The first line confirms the ordering: the sidecar printed its start-up message before the worker wrote anything, because native sidecars start before the containers in containers. The main container, on the other hand, prints nothing on its standard output:

All of its logging goes to the file, and it only reaches the node collector thanks to the adapter. That is precisely the purpose of the pattern.

Solution 3

The Job reaches Complete in 19 seconds even though the sidecar was still printing heartbeats indefinitely.

The pod is Completed with both of its containers stopped.

What would have happened with the sidecar in containers: the generator container would have finished successfully after 15 seconds, but batch-metrics would have stayed alive. A pod only reaches the Succeeded phase when all of its containers have finished, so the pod would have sat indefinitely in Running with 1/2 containers ready, and the Job at 0/1 completions forever. Only activeDeadlineSeconds would have cut it short, and with a Failed state.

That was exactly the historic problem native sidecars solved.

A pod with several containers is a powerful tool and an easy one to misuse. The rule that governs it is that processes should share a pod only when they are so tightly coupled that it makes no sense to scale or deploy them separately, and when one exists to serve the other.

Init containers run in order, to completion, before any main container starts, and in Rutas Norte they let us wait for bookings-postgres, apply small idempotent migrations, and prepare content in a shared emptyDir. Their states — Init:0/2, Init:Error, Init:CrashLoopBackOff — are diagnoses in themselves, and kubectl logs -c <name> is the command to internalise.

The native sidecars of Kubernetes 1.29+ are init containers with restartPolicy: Always: they start before the main containers, live for the whole pod, restart if they die, stop after the main ones and — the thing that settled a years-old problem — do not stop a Job from finishing.

The three classic patterns are distinguished by the direction of the flow: the sidecar adds a capability (the bookings-postgres metrics exporter that Prometheus will consume in 07-03); the ambassador mediates the outbound connection (the proxy that handles mutual TLS, retries and circuit breaking against pagos.proveedorexterno.example); the adapter normalises what the application produces (the JSON converter for the proprietary notifications-worker log). All three rely on localhost and shared volumes, and all three cost resources multiplied by the number of pods, something you have to work out for the worst case of scaling.

So far we have decided what runs and how each pod is composed. The question we have not touched is where: until now the kube-scheduler has placed our pods wherever it liked, and that has been enough. But bookings-postgres ought to be on a node with an SSD, the bookings-api replicas ought not to share a node — if that node goes down, the whole API goes with it —, and analytics workloads ought not to compete with ticket sales. All of that is controlled with affinity, taints and tolerations, and it is the subject of the next lesson: Scheduling.