We closed module 6 with an uncomfortable diagnosis: the Rutas Norte platform is already powerful — it has state, it is extensible and it is automated — but it is completely opaque. We do not know whether bookings-api is genuinely healthy, nor how much memory bookings-postgres really consumes, nor what happened last night at three in the morning. Module 7 is devoted to opening that black box, and the first step is not to measure: it is teaching Kubernetes to tell a live container apart from a working application.

This lesson is about probes: the three mechanisms Kubernetes uses to ask each container periodically whether it is alive, whether it is ready to receive traffic and whether it has finished starting up. They are the piece that has been missing since module 2: there we said explicitly that a zero-downtime deployment is not possible without probes, and here we settle that debt. They are also, by a wide margin, the Kubernetes feature that causes the most production incidents when misconfigured: an overly aggressive livenessProbe can bring down an entire platform in a matter of minutes. Let us understand them thoroughly.

1. The problem: "the process is alive" does not mean "the application works"
2. The three probes and what Kubernetes does with each result
3. The four handlers: httpGet, tcpSocket, exec and grpc
4. The timing parameters and the exact detection-time calculation
5. The most expensive mistake in the module: cascading liveness
6. startupProbe: the answer to the slow start-up of bookings-postgres
7. Probes and zero-downtime deployments: settling the debt from 02-04
8. Clean shutdown: terminationGracePeriodSeconds, preStop and the 502s
9. Designing the probes for every Rutas Norte component
10. Common mistakes and tips
11. Exercises

Until now, the only health signal Kubernetes had about our containers was brutally simple: is the main process (PID 1) still running? If the process ends, the kubelet applies the restartPolicy we studied in 02-01 and restarts the container. If the process does not end, Kubernetes assumes everything is fine.

That assumption is false most of the time. Let us look at a real case involving bookings-api.

bookings-api is a Node.js API that keeps a connection pool against bookings-postgres, with a maximum of 20 connections. One Tuesday afternoon, a badly indexed query on the "travel history" screen starts taking 40 seconds. Users hit reload. Within two minutes, all 20 connections in the pool are taken up by slow queries. From that point on:

• The Node.js process is still alive: PID 1 exists, no uncaught exception has been thrown.
• The HTTP server is still accepting connections: the socket is open.
• But every request that needs the database waits for a free slot in the pool and eventually times out.

From Kubernetes' point of view, the pod is perfect. From the point of view of the customer trying to buy a Bilbao–Santander ticket, the platform is down. And worse still: the bookings-api Service keeps sending traffic to that pod, and to the other four that are in exactly the same state.

Probes exist precisely to close this gap. They give the application the chance to answer for itself two different questions:

• Am I so broken that the best thing you can do is restart me? → livenessProbe.
• Can I serve requests right now? → readinessProbe.

They are different questions, with different consequences, and confusing them is the root of almost every disaster in this lesson.

Kubernetes defines three probes per container. All of them are optional and all of them are declared inside the container specification, not the pod's.

It checks whether the container is in a state it can no longer recover from on its own. If it fails the configured number of times, the kubelet kills the container and applies the restartPolicy (usually Always, so it restarts it). The pod is not recreated: it is the same pod, with the same name and the same IP, with its RESTARTS counter incremented.

Legitimate cases: a deadlock the process cannot escape, an infinite loop that stops responding, a known memory leak with no short-term fix. If your application has no state that can only be escaped by restarting, it is perfectly valid not to define a livenessProbe.

It checks whether the container can serve requests right now. If it fails, Kubernetes restarts nothing: it simply marks the pod as not ready (Ready=False) and the endpoints controller removes its IP address from the Service's EndpointSlice. Traffic stops arriving, but the pod stays alive. When the probe passes again, the IP is added back automatically.

This is the correct mechanism for the exhausted-pool case: the pod steps aside for a moment, stops receiving requests it cannot serve, drains its slow queries and comes back.

It is a start-up probe. While it is running and has not yet passed, the other two probes are disabled. When it passes for the first time, it stops running for good and liveness and readiness come into play. If it never passes within its time budget, the container is killed and restarted.

It is meant for applications with slow or highly variable start-ups, where setting a huge initialDelaySeconds on liveness would make real failure detection painfully slow for the rest of the container's life.

One rule worth memorising: liveness protects the container from itself; readiness protects the container's clients.

Each probe uses exactly one of these four mechanisms to perform the check.

The most common one and the most advisable for HTTP services. The kubelet issues a GET request to the pod's IP, on the given port and path.

Important points for beginners:

• Any response code between 200 and 399 counts as a success, both inclusive. A 204 is a success. A 301 is too, and this comes as a surprise: if your application redirects /health to /login, the probe will pass even though the application is broken. A 400, 404, 500 or 503 is a failure.
• The request is made by the node's kubelet, not by another pod. That is why the NetworkPolicies in rutas-norte-pro do not block it: it does not go through the normal pod network.
• The port field accepts the name of a port declared in ports, which is far more readable and resilient to change.
• The httpHeaders field serves two very practical purposes: identifying in the application logs which requests come from the kubelet (so you can exclude them from traffic metrics) and passing headers the application requires, such as a specific Host.
• The kubelet does not send cookies and does not follow authentication: the health endpoint must be reachable without credentials from inside the pod.

The kubelet tries to open a TCP connection to the given port. If the handshake completes, it is a success; if the connection is refused or times out, it is a failure.

This is the option for services that do not speak HTTP, such as redis-cache or PostgreSQL itself at a low level. Its big limitation: it only checks that something is listening. PostgreSQL can have the port open and be refusing connections because it has hit max_connections, or be in recovery mode. The TCP probe would pass all the same.

The kubelet runs a command inside the container. Success if the exit code is 0, failure in any other case.

It is the most flexible and the most expensive. Every execution means creating a new process inside the container: it consumes the pod's CPU and memory (counted against its limits, which can trigger an OOMKilled if you are cutting it fine) and adds load to the node's runtime. With periodSeconds: 5 across 60 pods you are launching 12 processes per second in the cluster just to ask how things are going.

Tips:

• Use exec only when there is no HTTP or TCP alternative.
• Raise periodSeconds (10–30 s) compared with what you would use with httpGet.
• The binary must exist in the image. It is a common mistake to write a probe using curl in a distroless image that does not have it: the probe always fails and the container enters CrashLoopBackOff.

Since Kubernetes 1.24 it has been a stable feature. The kubelet acts as a client of the standard gRPC health checking protocol.

The application must implement the grpc.health.v1.Health service. It is a success if it answers SERVING. This avoids having to install grpc_health_probe as a binary in the image, which was the previous solution using exec.

The five timing variables are identical for all three probes. Really understanding them means being able to answer the question "how long does Kubernetes take to notice that this container is dead?" with a number, not with a hunch.

Two subtleties that are always forgotten:

• successThreshold must be 1 in livenessProbe and startupProbe. Only readinessProbe accepts larger values. It makes sense: you cannot "half restart" something.
• timeoutSeconds: 1 (the default) is dangerously low for a real application under load. A health endpoint that normally takes 20 ms can take 1.5 s when the node is saturated, and then the probe fails on timeout even though the application is fine.

The maximum time from a container ceasing to respond until the kubelet kills it is:

Plus a margin of up to periodSeconds, because the failure may happen right after a successful check. In practice the pessimistic formula is used:

An example with the configuration we will use on bookings-api:

In other words: if bookings-api hangs, up to 33 seconds will pass before the kubelet restarts it, and after that you have to add the container's start-up time. If that is unacceptable for the business, you lower periodSeconds to 5 and failureThreshold to 3 → 18 seconds. But the more aggressive the probe, the greater the risk of false positives, and that risk is the subject of the next section.

And for readiness, the same calculation determines how long traffic will keep arriving at a pod that can no longer serve it:

Rule of thumb: readiness should be faster and more sensitive than liveness. Taking a pod out of rotation is cheap and reversible; restarting it is not.

This is the section to read twice. It is the failure that has brought down whole production platforms more often than any other, and always for the same reason.

A Rutas Norte engineer, with the best of intentions, configures the liveness of bookings-api like this:

The /health/full endpoint runs a SELECT 1 against bookings-postgres, a PING to redis-cache and a call to pagos.proveedorexterno.example. It looks thorough. It is a bomb.

The May bank-holiday weekend arrives. Traffic multiplies sixfold. bookings-postgres starts responding in 1.2 s instead of 5 ms. Then:

1. The health probe times out (1 s) on all ten bookings-api pods at once, because they all depend on the same database.
2. After 10 seconds (2 * 5), the kubelet kills all ten containers simultaneously.
3. All ten start again. On start-up, each one opens its pool of 20 connections against PostgreSQL: 200 new connections at once against a database that was already drowning.
4. PostgreSQL becomes even more saturated. The probes fail again. Back to step 2.
5. Kubernetes applies exponential backoff to the restarts, so the pods enter CrashLoopBackOff. The platform goes from slow to completely down.

Note the essential point: the database was slow, not down. Without probes, the platform would have been slow for twenty minutes and would have recovered on its own. With that liveness, it went down completely. The probe amplified the failure instead of mitigating it.

Rule 1: livenessProbe must NEVER check external dependencies. Not databases, not caches, not third-party APIs, not other microservices. It must only answer "is this process, in isolation, still capable of handling a request?".

Rule 2: readinessProbe SHOULD check the dependencies it needs in order to serve. If bookings-api cannot talk to PostgreSQL, it cannot serve bookings, and the correct behaviour is for it to leave the Endpoints until it can.

And what happens if readiness fails on all ten pods at once? The Service is left with no Endpoints and the Ingress returns 503. That is bad, but it is honest and reversible: as soon as PostgreSQL recovers, the ten pods are ready again within five seconds, with no cold starts, no connection avalanche, no CrashLoopBackOff. That is the difference between a service degradation and a disaster.

This is why bookings-api will have two different endpoints:

A note about the payment gateway: pagos.proveedorexterno.example is a third party outside our control. If we put it in readiness, an outage at the provider leaves the whole platform without Endpoints, including timetable lookups, which do not need payments. The correct decision is not to include it in any probe and to handle it with a circuit breaker inside the application that returns a specific error only in the payment flow.

bookings-postgres, after last year's growth, takes between 20 and 180 seconds to accept connections when it starts: if the previous shutdown was dirty, it has to replay the WAL, and that time depends on the pending volume.

Without a startupProbe, we have two bad options:

• Option A: initialDelaySeconds: 200 on liveness. It covers the worst case, but during the first 200 seconds of every restart nobody is watching the container. And worse: you cannot lower the detection period afterwards.
• Option B: failureThreshold: 40 with periodSeconds: 5. It covers start-up, but it also means that, in steady state, a hang will take 40 * 5 = 200 seconds to detect.

The startupProbe separates the two time budgets: a generous one for starting up, a strict one for steady-state life.

How this behaves over time:

If after 300 seconds the startupProbe is still failing, the container is killed and restarted. That is the correct behaviour: something really is wrong.

An important detail about the exec: pg_isready and psql do exist in the postgres:16.4 image, and we connect to 127.0.0.1 (inside the pod itself), not to the Service. Checking the Service from a pod's probe would be checking an external dependency, with all the problems described in section 5.

In 02-04 we saw RollingUpdate with maxSurge and maxUnavailable, and we wrote that this was not yet a zero-downtime deployment. Now it is clear why.

The Deployment controller considers a new pod "fine" when the pod is Ready. Without a readinessProbe, a pod is considered Ready as soon as its containers start. For bookings-api, that happens about 400 ms after launching node, long before Express is listening on 8080 and the connection pool is open.

The sequence of the silent disaster:

1. kubectl apply with the new image.
2. The new pod is created. After 400 ms it is Running and, without readiness, Ready.
3. The endpoints controller adds its IP to the EndpointSlice: it starts receiving real traffic.
4. The Deployment, seeing a new pod ready, terminates an old pod.
5. For the next 3–8 seconds, the new pod returns ECONNREFUSED to all the traffic arriving at it.
6. This repeats pod by pod. The result: a few seconds of errors for each replica replaced, invisible in kubectl get pods, very visible in the error rate.

With a readinessProbe, step 2 changes: the pod is Running but Ready=False, it does not join the Endpoints and the Deployment does not move on until the probe passes. That is, literally, the mechanism behind zero-downtime deployment.

There is a residual case: applications that pass readiness and then fall over three seconds later (for example, on receiving the first real request that hits faulty new code). minReadySeconds forces the Deployment to wait N seconds with the pod continuously ready before accepting it and carrying on replacing pods.

With maxUnavailable: 0 + readinessProbe + minReadySeconds: 15, a bookings-api deployment is genuinely zero-downtime. The price: it takes longer. A deployment of 6 replicas goes from 40 seconds to about 3 minutes. That is a reasonable price.

A word of warning about progressDeadlineSeconds: if readiness never passes (for example, because the new version has a configuration error), the Deployment gets stuck but the old pods keep serving. After 300 seconds the Deployment is marked as ProgressDeadlineExceeded, which allows it to be detected automatically. It does not roll back on its own: you decide that with kubectl rollout undo.

We have solved the arrival of new pods. What remains is the departure of the old ones, which is where the 502s Rutas Norte sees on every deployment come from.

When Kubernetes decides to terminate a pod, two things happen in parallel, with no coordination between them:

Path B (killing the process) is almost instantaneous. Path A (propagating the endpoint removal to every node and to the Ingress controllers) takes from a few hundred milliseconds to a couple of seconds in a loaded cluster. In that window, the load balancer keeps sending requests to a process that is already shutting down. Each one of those requests is a 502 for a Rutas Norte customer.

The preStop hook runs before the SIGTERM, and the kubelet waits for it to finish before sending it. Inserting a pause there gives path A time to complete.

It looks like a dirty trick, and in a way it is, but it is the solution recommended by the Kubernetes documentation itself and the one practically everybody uses in production. During those 10 seconds:

• The container carries on serving requests as normal (nobody has told it to stop).
• The endpoints controller has already removed it, so no new traffic arrives.
• In-flight requests finish quietly.

If your application has an endpoint that deliberately makes readiness fail, a more elegant variant is to call it and then wait. But the sleep works and requires no changes to the application.

The real timeline:

A critical point that is always forgotten: the preStop time is deducted from the grace period, not added to it. With terminationGracePeriodSeconds: 45 and a 10 s preStop, the application only has 35 seconds to shut down after the SIGTERM. Set the number with that in mind.

And as we saw in 02-01, if your process starts via a shell (CMD npm start), the SIGTERM may reach the shell and not Node. Use the exec form (CMD ["node", "server.js"]) or shareProcessNamespace/tini to make sure the signal reaches the right process.

For notifications-worker, which processes confirmation emails, the grace period must cover the longest send in progress: we set terminationGracePeriodSeconds: 90 and the application stops taking new messages from the queue when it receives the SIGTERM.

We now apply all of the above component by component. The design table first, the manifests afterwards.

Two decisions that deserve an explanation:

• notifications-worker without readiness: readiness only makes sense if something queries the Endpoints. This worker sits behind no Service; it consumes from a queue. Adding readiness would do nothing. It does have liveness, with a small internal HTTP server that answers 200 while the consumption loop is active.
• CronJob without probes: a pod that lives for 4 minutes and whose success is measured by its exit code gains nothing from probes. Instead, in 07-04 we will alert on kube_job_status_failed.

Pseudocode for /health (liveness). Cheap, local, no outbound network:

Pseudocode for /ready (readiness). It checks dependencies, with short timeouts:

Typical output for a pod whose readiness is failing:

Pay attention: STATUS is Running and RESTARTS is 0. Everything "looks" fine. The only signal is the 0/1 in the READY column. And in the events:

That is the trace to look for. In 07-06 we will systematise this way of reading events.

1. Using the same probe for liveness and readiness, checking dependencies. This is the mistake from section 5 in disguise. If you copy the same httpGet /health/full into both, you have created the cascading restart loop. Different endpoints, always.

2. timeoutSeconds: 1 (the default). Under load, a healthy endpoint easily takes more than a second. Raise it to 2–3 s on readiness and 3–5 s on liveness. This default has caused more false positives than any other.

3. Forgetting the startupProbe on slow applications. The classic symptom: CrashLoopBackOff on a container that "starts fine locally". What is happening is that liveness kills it after 30 seconds and it never finishes starting. If you see cyclic restarts with no application error logs, suspect this first.

4. An exec probe with a binary that does not exist. curl, wget or nc are not present in distroless images nor in many minimal alpine ones. Check with kubectl exec -it <pod> -- which curl before writing the probe. The telltale event is Liveness probe errored: exec: "curl": executable file not found in $PATH.

5. A health endpoint that requires authentication. The kubelet sends no credentials. A /health protected by JWT returns 401 and the probe always fails. Leave the health endpoint open, with no sensitive data in the response, and protect it at the network level if you need to.

6. Trusting 3xx codes. Remember: 200–399 is success. A /health that redirects to / will keep "passing" even if the application is completely broken. Return an explicit 200.

7. readinessProbe without preStop. You have zero-downtime deployments on the way in but you are still generating 502s on the way out. The two mechanisms are complementary and you need both.

8. Probes that are too expensive. A /ready that runs SELECT count(*) FROM bookings is issuing a heavy query every 5 seconds for every replica. With 6 replicas that is 72 queries a minute just to ask how we are. The checks must be trivial: SELECT 1 and little more.

9. Using probes for what a PodDisruptionBudget is for. Probes do not protect you from a node drain or a cluster upgrade. That is 09-05.

10. A preStop longer than the grace period. If you set preStop: sleep 60 with terminationGracePeriodSeconds: 30, the SIGKILL arrives after 30 seconds and the application never receives the SIGTERM: it closes nothing cleanly. The preStop must always be comfortably shorter than the grace period.

A colleague has configured the liveness of notifications-worker like this:

Answer:

1. In the worst case, how long does it take to detect that the worker has hung?
2. The business requires a hung worker to be restarted in under 60 seconds. Propose a configuration that meets this without making it prone to false positives.
3. The worker takes 8 seconds to start and connect to the queue. Is initialDelaySeconds: 20 correct? What better alternative exists?

This is the real web-store manifest in rutas-norte-pre. It contains three problems related to probes and shutdown. Identify them and write the corrected manifest.

Rutas Norte adds a new component: routes-search, a Java service (Spring Boot) that keeps an in-memory index of every route and timetable. The facts:

• On start-up it loads the full index from bookings-postgres: between 45 and 150 seconds depending on the volume.
• Once the index is loaded, it never touches PostgreSQL again: it serves everything from memory.
• It exposes /actuator/health/liveness and /actuator/health/readiness (Spring Boot Actuator).
• It sits behind a routes-search Service that web-store queries.
• It occasionally suffers long garbage collector pauses (up to 4 seconds).

Write the complete probe block and justify every value.

Solution 1

1. Detection time.

More than two and a half minutes with the worker hung and not sending a single confirmation email. Far above the 60-second requirement.

2. Proposed configuration.

The period has been lowered from 30 to 10 s (faster detection) and timeoutSeconds raised from 1 to 3 s (fewer false positives). The failureThreshold: 4 leaves room for a one-off spike not to trigger a restart: four consecutive failures are needed, that is, 40 seconds of sustained trouble.

3. The initialDelaySeconds.

It is a crude solution. With a fixed 20 s: if one day start-up takes 25 s because the node is slow, liveness starts failing during start-up and the pod may enter CrashLoopBackOff. And if it starts in 8 s, we have lost 12 s of watching.

The correct alternative is a startupProbe that separates the budgets:

An added benefit: the startupProbe with periodSeconds: 3 detects the finished start-up as soon as it happens, so a pod that starts in 8 s is ready in ~9 s, not in 20.

Solution 2

The three problems:

1. Liveness checks external dependencies. /api/full-status is a proxy towards bookings-api and from there to PostgreSQL. If the API or the database are slow, all three web-store pods restart at once, leaving the public website completely down even though nginx was perfectly fine. It is exactly the scenario from section 5.
2. There is no readinessProbe. Combined with maxUnavailable: 1, every deployment sends traffic to nginx pods that have not finished starting → errors during every update.
3. There is no preStop and no adjusted grace period. The pods being removed die before the endpoint removal propagates → 502 on every deployment. On top of that: timeoutSeconds: 1 is far too tight.

Corrected manifest:

With the corresponding nginx configuration:

The access_log off; stops the probes (one every 5 s per pod, 3 pods → more than 50,000 lines a day) from flooding the logs we will centralise in 07-05.

Solution 3

Justification of each decision:

• startupProbe with a 240 s budget. Start-up ranges from 45 to 150 s. Allowing 60 % headroom over the known worst case avoids spurious restarts on the day PostgreSQL is slow during the load. Without this probe, liveness would kill the pod after ~50 s and it would enter an endless CrashLoopBackOff: it would never manage to load the index.
• timeoutSeconds: 6 on liveness and readiness. Garbage collector pauses reach 4 seconds. If the timeout were 3 s, every long pause would count as a failure. With 6 s, a GC pause triggers nothing. This is a JVM-specific adjustment that has to be made deliberately.
• periodSeconds: 15 on liveness. The component is stable; we do not need sub-second detection. A long period reduces the probe pressure on the application.
• Liveness against /actuator/health/liveness and not against the general endpoint. Spring Boot separates the liveness (JVM state) and readiness (dependencies ready) groups by default. Using plain /actuator/health would include the PostgreSQL indicators in liveness: the mistake from section 5.
• Readiness that does check the index. Even though routes-search does not touch PostgreSQL in steady state, it does need the index loaded. Readiness failing while the index is not ready is correct: the pod should not receive searches it cannot resolve.
• successThreshold: 1. Return to service quickly as soon as it recovers.

A practical detail: in Spring Boot you have to enable the probe groups with management.endpoint.health.probes.enabled=true (it is enabled automatically if it detects that it is running in Kubernetes).

Probes are the first step towards an observable platform, and also the first towards a reliable one. In this lesson we have seen that:

• A live process is not a healthy application, and the exhausted connection pool case in bookings-api proves it beyond any doubt.
• livenessProbe restarts, readinessProbe removes from the Service, startupProbe delays the other two. Three different questions with three different consequences.
• The four handlers (httpGet, tcpSocket, exec, grpc) have very different costs and precision; exec is the most flexible and the most expensive.
• Detection time is calculated, not guessed: failureThreshold * periodSeconds + timeoutSeconds.
• The rule that prevents the most expensive outage in the module: liveness never checks external dependencies; readiness should.
• Probes settle the debt from 02-04: together with maxUnavailable: 0, minReadySeconds and a preStop that wins the race against the SIGTERM, we finally have genuinely zero-downtime deployments without the 502s we used to see on every update.

Rutas Norte now knows whether each component is healthy. What it still does not know is how much it consumes. In 03-04 we set the requests and limits of each component practically by eye, and we wrote that we would calibrate them by observing real consumption. That moment has arrived: in the next lesson we will install metrics-server, the first source of consumption data in the cluster, and with kubectl top we will compare what we ask for with what we actually spend.