The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Application Debugging and Cluster Events

Next ►
►

As the previous lesson closed we had the three signals of observability: the probes say whether a component is healthy, the metrics say how much and how well it works, and the logs say exactly what happened. What we still do not have is a method for using them together.

Because when the phone rings at 03:14 and the web store returns 502, knowing how to use Grafana and Kibana is not enough. Under pressure, without sleep, with the director asking every five minutes, the difference between resolving it in ten minutes or in two hours is not knowing more commands: it is having a procedure that goes from symptom to cause without wandering.

This lesson builds that methodology. We will look at the decision tree that structures the investigation, Kubernetes events as the most important and least exploited source — with the critical fact that they expire after an hour —, a master table of symptom → probable cause → the command that confirms it, the inspection tools including kubectl debug's ephemeral containers, and finally a real Rutas Norte incident resolved step by step using everything learned in the module. It is module 7's last lesson.

Contents

  1. A methodology, not a list of commands
  2. The decision tree: from symptom to cause
  3. Events: the main and least exploited source
  4. The master table: symptom → probable causes → command
  5. Detailed diagnosis of each symptom
  6. The inspection tools
  7. kubectl debug: ephemeral containers, copies and nodes
  8. A real case: the store returns 502 during deployments
  9. Gather the evidence before restarting
  10. Common mistakes and tips
  11. Exercises

  1. A methodology, not a list of commands

The commonest mistake when debugging in Kubernetes is not being unaware of a command: it is starting to look in the wrong place.

Faced with "the website is broken", the instinctive reaction is usually kubectl logs on the first pod that comes up. But if the pod is Pending, there are no logs to read, because it never started. If the problem is that the Service has no Endpoints, the application logs will be perfect and will say nothing. Half an hour wasted looking where the answer is not.

A methodology organises the work with two principles:

Principle 1: follow the pod's life cycle, in order. A pod passes through successive phases, and a failure in one phase makes everything after it irrelevant. There is no point asking whether a pod receives traffic if it has not yet been scheduled on a node.

Principle 2: confirm every hypothesis with a specific command. "I think it is the memory" is not a diagnosis. kubectl get pod X -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' returning OOMKilled is.

The five questions in order, which are the backbone of everything that follows:

# Question If the answer is NO, look at...
1 Does the pod exist? The controller (Deployment, StatefulSet, CronJob)
2 Is it scheduled on a node? The scheduler: resources, taints, affinity, PVC
3 Has the container started? Image, volumes, ConfigMaps, Secrets
4 Is it ready? The probes from 07-01 and their dependencies
5 Is it receiving traffic? Service, EndpointSlice, selector, Ingress, NetworkPolicy

Each question has a command that answers it in a second. The discipline consists of skipping none of them.

  1. The decision tree: from symptom to cause

flowchart TD
    S["Reported symptom"] --> Q1{"Does the pod exist?<br/>kubectl get pods"}
    Q1 -->|No| C1["Check the controller:<br/>kubectl describe deploy/sts<br/>ReplicaSet created? Quota exhausted?"]
    Q1 -->|Yes| Q2{"What is its STATUS?"}

    Q2 -->|Pending| C2["The scheduler cannot place it:<br/>resources, taints, affinity,<br/>unbound PVC"]
    Q2 -->|ContainerCreating| C3["The kubelet cannot start it:<br/>a volume, Secret or ConfigMap<br/>that does not exist"]
    Q2 -->|ImagePullBackOff| C4["It cannot pull the image:<br/>name, tag,<br/>registry credentials"]
    Q2 -->|CrashLoopBackOff| C5["It starts and dies:<br/>logs --previous,<br/>OOMKilled, liveness"]
    Q2 -->|Terminating| C6["It will not finish:<br/>finalizers,<br/>grace period"]
    Q2 -->|Running| Q3{"Is READY n/n?"}

    Q3 -->|No| C7["The probes are failing:<br/>describe → Unhealthy events<br/>07-01"]
    Q3 -->|Yes| Q4{"Does the Service have<br/>Endpoints?"}

    Q4 -->|No| C8["Wrong selector, or no<br/>pod is Ready"]
    Q4 -->|Yes| Q5{"Does it respond through<br/>a direct port-forward?"}

    Q5 -->|No| C9["A problem in the application:<br/>logs, exec, metrics"]
    Q5 -->|Yes| C10["A problem in the network layer:<br/>Ingress, NetworkPolicy,<br/>DNS, TLS"]

The commands that answer each decision node:

NS=rutas-norte-pro

# Q1 — Does the pod exist?
kubectl -n $NS get pods -l app=bookings-api

# Q2 — What is its status? With -o wide you also see which node it is on
kubectl -n $NS get pods -o wide

# Q3 — Is it ready? The READY column is the one that matters
kubectl -n $NS get pods -l app=bookings-api

# Q4 — Does the Service have Endpoints? THE MOST UNDERRATED CHECK
kubectl -n $NS get endpointslices -l kubernetes.io/service-name=bookings-api

# Q5 — Does it respond bypassing the Service and the Ingress?
kubectl -n $NS port-forward pod/bookings-api-7d9f8c4b5-x2klm 8080:8080
curl -s localhost:8080/health

Question 4 deserves a special comment. An empty EndpointSlice is the cause of 80 % of "the Service does not work", and it is checked in two seconds:

kubectl -n rutas-norte-pro get endpointslices -l kubernetes.io/service-name=bookings-api \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\t"}{.conditions.ready}{"\n"}{end}'
10.244.2.17	true
10.244.1.23	true
10.244.3.41	false

That shows directly which pods are receiving traffic and which are not.

  1. Events: the main and least exploited source

Events are Kubernetes API objects that record what the cluster's components do and observe: the scheduler when it cannot place a pod, the kubelet when it fails to pull an image, the endpoints controller when it updates a Service.

They are, by a wide margin, the most useful diagnostic source and the least consulted.

How to query them

NS=rutas-norte-pro

# THE MOST USEFUL COMMAND IN THIS LESSON: events in chronological order.
# Without --sort-by they come out in arbitrary order and are nearly useless.
kubectl -n $NS get events --sort-by=.lastTimestamp

# Only the problems
kubectl -n $NS get events --field-selector type=Warning --sort-by=.lastTimestamp

# Across the whole cluster (essential when the problem is infrastructure)
kubectl get events -A --sort-by=.lastTimestamp | tail -40

# Those of a specific object
kubectl -n $NS get events --field-selector involvedObject.name=bookings-api-7d9f8c4b5-x2klm

# Combining field selectors
kubectl -n $NS get events \
  --field-selector type=Warning,involvedObject.kind=Pod \
  --sort-by=.lastTimestamp

# The modern command (Kubernetes 1.26+), more readable
kubectl -n $NS events --for pod/bookings-api-7d9f8c4b5-x2klm
kubectl -n $NS events --types=Warning
kubectl -n $NS events --watch                # in real time

# And the most frequent route in practice: the Events section of describe
kubectl -n $NS describe pod bookings-api-7d9f8c4b5-x2klm | tail -20

Typical output:

LAST SEEN   TYPE      REASON              OBJECT                              MESSAGE
5m12s       Normal    Scheduled           pod/bookings-api-7d9f8c4b5-x2klm    Successfully assigned rutas-norte-pro/bookings-api-7d9f8c4b5-x2klm to rutas-norte-worker-2
5m10s       Normal    Pulled              pod/bookings-api-7d9f8c4b5-x2klm    Container image "registry.rutasnorte.example/bookings-api:2.8.1" already present on machine
5m10s       Normal    Created             pod/bookings-api-7d9f8c4b5-x2klm    Created container api
5m09s       Normal    Started             pod/bookings-api-7d9f8c4b5-x2klm    Started container api
4m22s       Warning   Unhealthy           pod/bookings-api-7d9f8c4b5-x2klm    Readiness probe failed: HTTP probe failed with statuscode: 503
3m18s       Warning   BackOff             pod/bookings-api-7d9f8c4b5-x2klm    Back-off restarting failed container api

That sequence tells a complete story in six lines: it was scheduled fine, the image was there, it started, readiness began failing with a 503 and it ended up in backoff. The diagnosis is practically done.

Normal versus Warning

Type Meaning Examples
Normal An expected life cycle operation Scheduled, Pulled, Created, Started, Killing
Warning Something did not go as it should FailedScheduling, Failed, BackOff, Unhealthy, FailedMount, Evicted

Rule of thumb: always start with the Warnings. But do not ignore the Normal ones: the absence of an expected Scheduled event, or a Killing you were not expecting, is valuable information. And in the case in section 8, a Normal event will be the key piece of the diagnosis.

The most frequent events and what they mean

REASON Emitter What it means
FailedScheduling scheduler There is no node where the pod fits
Scheduled scheduler Assigned to a node
Pulling / Pulled kubelet Downloading / image ready
Failed (ErrImagePull) kubelet It could not pull the image
Created / Started kubelet Container created / started
BackOff kubelet Exponential wait before retrying
Unhealthy kubelet A probe has failed (07-01)
Killing kubelet Terminating the container and why
FailedMount kubelet Volume, Secret or ConfigMap not available
FailedAttachVolume volume controller The volume is still attached to another node
Evicted kubelet Evicted because of node resource pressure
NodeNotReady node controller The node stopped reporting
Preempting scheduler Evicting lower-priority pods (06-05)

⚠️ The critical fact: events expire after an hour

This is the point to burn into your memory, because it completely changes the way you work.

Kubernetes events are stored in etcd with a default TTL of 1 hour. After that time, the API Server deletes them automatically. It is not a rotation by space: it is a deletion by time.

# See the TTL configured in the cluster (an API Server parameter)
kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep event-ttl
"--event-ttl=1h0m0s"

The consequences are very concrete:

  • If the incident happened at 03:14 and you investigate it at 09:00, the events no longer exist. kubectl describe will show nothing useful, even if the problem is still present.
  • A pod that has been in CrashLoopBackOff for three days only has events from the last hour, not from the moment it started failing.
  • The most valuable evidence for reconstructing a timeline disappears on its own, without warning.

How to solve it: export the events.

Option A — An immediate manual capture. The first thing you do when you start investigating:

kubectl get events -A --sort-by=.lastTimestamp \
  -o json > /tmp/incident-events-$(date +%Y%m%d-%H%M).json

Option B — kube-state-metrics. As we saw in 07-03, it exposes metrics about the state of the objects. They are not the events themselves, but they allow a lot to be reconstructed:

# Restarts over time: it survives the events' TTL
increase(kube_pod_container_status_restarts_total{namespace="rutas-norte-pro"}[1h])

# The reason for the container's last termination
kube_pod_container_status_last_terminated_reason{namespace="rutas-norte-pro"}

Option C (the recommended one) — Export the events to the logging system. A component such as kubernetes-event-exporter watches the events and sends them to Elasticsearch or Loki, where they fall under the 30-day retention from 07-05, not the one-hour one.

# k8s/base/logging/event-exporter-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: event-exporter-config
  namespace: logging
data:
  config.yaml: |
    logLevel: info
    logFormat: json
    route:
      routes:
        - match:
            - receiver: centralized-logging
    receivers:
      - name: centralized-logging
        # Output to stdout: Fluent Bit collects it like any other log,
        # so it inherits the whole stack and the retention from 07-05.
        stdout:
          deDot: true
          layout:
            timestamp: "{{ .LastTimestamp }}"
            level: "{{ if eq .Type \"Warning\" }}warn{{ else }}info{{ end }}"
            component: "kubernetes-events"
            message: "{{ .Message }}"
            reason: "{{ .Reason }}"
            object_kind: "{{ .InvolvedObject.Kind }}"
            object_name: "{{ .InvolvedObject.Name }}"
            namespace: "{{ .InvolvedObject.Namespace }}"
            node: "{{ .Source.Host }}"
            count: "{{ .Count }}"

With this, in Kibana you can search:

component: "kubernetes-events" and reason: "OOMKilled" and namespace: "rutas-norte-pro"

and see every OOMKilled from the last four weeks. It is one of the best value-for-effort improvements in the whole of observability.

Rutas Norte operating rule: the first command of any investigation is to capture the events to a file. The second is to look at the time of the incident and check whether the TTL has already taken them.

  1. The master table: symptom → probable causes → command

This table is the mental map to internalise. Each row's diagnosis is developed in the next section.

Symptom Probable causes (in order of frequency) Command that confirms it
Pending Insufficient resources; unbound PVC; untolerated taints; impossible affinity; exhausted quota kubectl describe pod → FailedScheduling event
ImagePullBackOff / ErrImagePull Wrong name or tag; registry credentials; deleted image; the node's network kubectl describe pod → Failed event with the registry's message
CrashLoopBackOff Start-up failure; missing configuration; OOMKilled; liveness too aggressive; wrong command kubectl logs --previous
OOMKilled (exit 137) limits.memory too low; memory leak; legitimate spike kubectl get pod -o jsonpath='{...lastState.terminated}'
Unexpected Error / Completed A non-zero exit code; a process that ends when it should not kubectl logs --previous + exitCode
Stuck ContainerCreating Non-existent Secret or ConfigMap; unbound PVC; volume stuck on another node kubectl describe pod → FailedMount event
Endless Terminating Pending finalizers; a process that ignores SIGTERM; a downed node kubectl get pod -o jsonpath='{.metadata.finalizers}'
0/3 Ready Readiness probe failing; a dependency down kubectl describe pod → Unhealthy event
Ingress 503 Service with no Endpoints; wrong selector; no pod ready kubectl get endpointslices
Ingress 502 The backend closes the connection; the SIGTERM race; a timeout Ingress controller logs + preStop (07-01)
Evicted Memory or disk pressure on the node; BestEffort QoS kubectl describe node → pressure conditions

  1. Detailed diagnosis of each symptom

Pending: the scheduler cannot place it

kubectl -n rutas-norte-pro describe pod bookings-api-7d9f8c4b5-x2klm | tail -15
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m14s  default-scheduler  0/4 nodes are available:
           1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
           2 Insufficient memory,
           1 node(s) had volume node affinity conflict.
           preemption: 0/4 nodes are available: 4 No preemption victims found.

The FailedScheduling message is a complete diagnosis, node by node. A translation of that line:

Fragment Meaning Fix
1 node(s) had untolerated taint The control plane has a taint (06-05) Correct: we do not want workloads there
2 Insufficient memory Two nodes with no free memory for the requests Lower the requests (07-02) or add nodes
1 node(s) had volume node affinity conflict The PV exists in another zone (05-02) The pod must go where its volume is

Complementary checks:

# How much is really reserved on each node?
kubectl describe node rutas-norte-worker-2 | grep -A 8 "Allocated resources"

# Has the namespace's ResourceQuota been exhausted? (03-04)
kubectl -n rutas-norte-pro describe resourcequota

# Is the PVC bound?
kubectl -n rutas-norte-pro get pvc

# What taints do the nodes have?
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

One case that throws people off badly: if the ResourceQuota is exhausted, the pod is not even created. You will not see a pod in Pending: you will see no pod at all. The error is on the ReplicaSet:

kubectl -n rutas-norte-pro describe replicaset bookings-api-7d9f8c4b5 | tail -8
Warning  FailedCreate  1m  replicaset-controller  Error creating: pods "bookings-api-7d9f8c4b5-" is
forbidden: exceeded quota: quota-pro, requested: requests.memory=512Mi, used: requests.memory=15872Mi,
limited: requests.memory=16Gi

It is the answer to question 1 of the decision tree: when the pod does not exist, you have to look at the controller.

ImagePullBackOff and ErrImagePull

Events:
  Warning  Failed   45s (x4 over 2m)  kubelet  Failed to pull image
  "registry.rutasnorte.example/bookings-api:2.8.2": rpc error: code = NotFound
  desc = failed to pull and unpack image: not found
  Warning  Failed   45s (x4 over 2m)  kubelet  Error: ErrImagePull
  Normal   BackOff  20s (x6 over 2m)  kubelet  Back-off pulling image

The difference between the two states: ErrImagePull is the first failure; ImagePullBackOff is the state after several attempts, with a growing exponential wait.

The four causes and their distinguishing messages:

Registry message Cause Fix
not found / manifest unknown Wrong tag or name Verify the exact tag
unauthorized / authentication required The imagePullSecrets is missing Create a docker-registry type Secret
no such host The node cannot resolve the registry The node's DNS, not the cluster's
context deadline exceeded Slow network or a huge image Raise the timeout, pre-pull the image 
# Verify the exact name of the declared image
kubectl -n rutas-norte-pro get deploy bookings-api \
  -o jsonpath='{.spec.template.spec.containers[*].image}'

# Is the registry secret declared?
kubectl -n rutas-norte-pro get deploy bookings-api \
  -o jsonpath='{.spec.template.spec.imagePullSecrets}'

# Create the secret if it is missing
kubectl -n rutas-norte-pro create secret docker-registry rutasnorte-registry \
  --docker-server=registry.rutasnorte.example \
  --docker-username=deployments \
  --docker-password="$REGISTRY_PASSWORD"

An especially treacherous case: mutable tags. If somebody uses :latest (something the Rutas Norte convention expressly forbids: immutable image tags), the pod may work on a node where the image is cached and fail on another where it is not. An intermittent and inexplicable problem.

CrashLoopBackOff: it starts and dies

The most common one and the one that needs the most steps.

Step 1: the logs of the previous run. The current container has just started and will say nothing; the cause is in the one that died.

kubectl -n rutas-norte-pro logs bookings-api-7d9f8c4b5-x2klm --previous
{"level":"fatal","component":"bookings-api","message":"Environment variable DB_PASSWORD not defined"}

An immediate diagnosis. But quite often the logs are empty, and then:

Step 2: the reason and the exit code of the termination.

kubectl -n rutas-norte-pro get pod bookings-api-7d9f8c4b5-x2klm \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq
{
  "containerID": "containerd://3f8a2c...",
  "exitCode": 137,
  "finishedAt": "2026-08-06T03:14:22Z",
  "reason": "OOMKilled",
  "startedAt": "2026-08-06T03:12:58Z"
}

A table of exit codes:

Code Meaning Usual cause
0 A clean termination The process ended when it should not have
1 A generic application error An uncaught exception
2 Incorrect use of a shell command Bad arguments
126 The command is not executable Missing execute permission
127 Command not found A binary missing from the image
137 SIGKILL (128+9) OOMKilled or the grace period ran out
139 SIGSEGV (128+11) A segmentation fault
143 SIGTERM (128+15) A normal termination by Kubernetes

Step 3: rule out the cause from 07-01. An overly aggressive livenessProbe kills perfectly healthy containers that are simply slow to start. The clue:

Warning  Unhealthy  2m (x9 over 5m)  kubelet  Liveness probe failed: Get "http://10.244.2.17:8080/health":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal   Killing    2m (x3 over 5m)  kubelet  Container api failed liveness probe, will be restarted

If the Killing event is preceded by Unhealthy and the application logs show no error at all, it is almost certain that the probe is the problem, not the application. The fix is the startupProbe from 07-01.

Step 4: the trick for debugging a container that dies instantly. If the container dies in under a second, there is no time to exec. You launch a copy with the command replaced by something that does not end:

kubectl -n rutas-norte-pro debug pod/bookings-api-7d9f8c4b5-x2klm \
  --copy-to=api-debug \
  --container=api \
  -- sleep 3600

# Now the pod stays alive and you can go in and investigate
kubectl -n rutas-norte-pro exec -it api-debug -c api -- sh

Inside you can check everything the application needed:

env | grep DB_          # are the variables there?
ls -la /etc/config/     # is the ConfigMap mounted?
cat /etc/secrets/db     # does the Secret contain what it expects?
node server.js          # run it by hand and see the complete error

OOMKilled and exit code 137

# Find every OOMKilled in the namespace
kubectl -n rutas-norte-pro get pods -o json | jq -r '
  .items[] |
  select(.status.containerStatuses[]?.lastState.terminated.reason == "OOMKilled") |
  "\(.metadata.name)\t\(.status.containerStatuses[0].lastState.terminated.finishedAt)"'

Distinguishing the two situations, which need different solutions:

Situation How to tell it apart Fix
The limit is too low Consumption is stable and close to the limit Raise limits.memory (07-02)
A memory leak Consumption grows linearly until it dies Fix the code; the limit only delays it

The distinction is made with the metric from 07-03:

container_memory_working_set_bytes{namespace="rutas-norte-pro", pod=~"bookings-api-.*", container="api"}

A rising sawtooth that reaches the limit and drops to zero repeatedly is the unmistakable signature of a memory leak. A stable value that one day grazes the limit is a limit set too low.

Unexpected Error and Completed

A pod in Completed with restartPolicy: Always ends up in CrashLoopBackOff, because Kubernetes restarts it and it ends again. The usual cause: the main process is not a long-running service.

kubectl -n rutas-norte-pro get pod X -o jsonpath='{.spec.containers[0].command} {.spec.containers[0].args}'

Typical mistakes: a command that runs a script which ends, a server launched in the background while PID 1 exits, or a CMD that starts in daemon mode.

For the occupancy-reports CronJob (06-03), on the other hand, Completed with code 0 is the correct outcome. The diagnosis is the opposite:

kubectl -n rutas-norte-pro get jobs -l app=occupancy-reports
kubectl -n rutas-norte-pro logs job/occupancy-reports-28934520
kubectl -n rutas-norte-pro describe job occupancy-reports-28934520 | grep -A 5 "Pod Statuses"

Stuck ContainerCreating

The pod is scheduled but the kubelet cannot create the container.

Events:
  Warning  FailedMount  1m (x8 over 5m)  kubelet  MountVolume.SetUp failed for volume
  "config-api" : configmap "bookings-api-config" not found

The four causes:

Cause Message Check
Non-existent ConfigMap configmap "X" not found kubectl get cm X
Non-existent Secret secret "X" not found kubectl get secret X
Unbound PVC PersistentVolumeClaim is not bound kubectl get pvc
Volume on another node Multi-Attach error for volume kubectl get volumeattachment

The last one is the most tiresome. A ReadWriteOnce volume (05-03) can only be mounted on one node at a time. If the pod is recreated on another node before the volume is released from the previous one, it stays blocked:

Warning  FailedAttachVolume  2m  attachdetach-controller  Multi-Attach error for volume
"pvc-4f8a2c91-..." Volume is already exclusively attached to one node and can't be attached to another

It resolves itself within a few minutes when the CSI releases the volume. If the original node is dead, it can take six minutes or more. It is a weighty reason for bookings-postgres to be a StatefulSet (06-01) with stable scheduling.

An endlessly Terminating pod

kubectl -n rutas-norte-pro get pod X
NAME   READY   STATUS        RESTARTS   AGE
X      1/1     Terminating   0          47m

Three causes, in order of frequency:

Cause 1: finalizers. A finalizer is a marker that prevents deletion until a controller has done its clean-up work. If that controller is down, the object stays blocked forever.

kubectl -n rutas-norte-pro get pod X -o jsonpath='{.metadata.finalizers}'
["kubernetes.io/pvc-protection","rutasnorte.example/cache-cleanup"]

The first is normal (it protects the PVC). The second is from an in-house controller: if that controller is down, it needs fixing.

Forcing the deletion by removing the finalizer is the last resort, because it skips the clean-up that finalizer guaranteed:

# ⚠️ LAST RESORT: it can leave orphaned resources
kubectl -n rutas-norte-pro patch pod X -p '{"metadata":{"finalizers":null}}' --type=merge

Cause 2: the process ignores the SIGTERM. As we saw in 07-01, if PID 1 is a shell that does not propagate signals, the real process never receives the SIGTERM and you have to wait for the SIGKILL at the end of the grace period. If terminationGracePeriodSeconds is 3600, the pod takes an hour to die.

kubectl -n rutas-norte-pro get pod X -o jsonpath='{.spec.terminationGracePeriodSeconds}'

Cause 3: the node is down. If the kubelet does not respond, nobody confirms that the pod has died. Kubernetes waits (5 minutes by default) before giving it up for lost.

kubectl get nodes
kubectl describe node rutas-norte-worker-2 | grep -A 6 Conditions

0/3 Ready: the probes are failing

NAME                            READY   STATUS    RESTARTS   AGE
bookings-api-7d9f8c4b5-x2klm    0/1     Running   0          8m

Running with 0/1 is the exact signature of a failing readiness probe. The container is alive, but it does not enter the Endpoints.

kubectl -n rutas-norte-pro describe pod bookings-api-7d9f8c4b5-x2klm | grep -A 5 Events
Warning  Unhealthy  30s (x48 over 8m)  kubelet  Readiness probe failed:
HTTP probe failed with statuscode: 503

The next step is to run the probe by hand from inside the pod, which is what gives the real answer:

kubectl -n rutas-norte-pro exec -it bookings-api-7d9f8c4b5-x2klm -c api -- \
  curl -s -w "\nHTTP %{http_code}\n" localhost:8080/ready
{
  "status": "not-ready",
  "reason": "connection pool exhausted",
  "checks": { "poolFree": false }
}

There is the cause, and it is exactly the scenario we opened 07-01 with. Readiness is doing its job: taking the pod aside until it can serve. The problem is not the probe, it is PostgreSQL.

Ingress 503 and 502

They are different errors with different causes, and confusing them costs time.

Code Meaning Cause in Kubernetes
503 Service Unavailable The Ingress has nobody to send to Service with no Endpoints; no pod Ready
502 Bad Gateway The backend exists but failed The pod closed the connection; a timeout; the SIGTERM race

Diagnosing a 503:

# Are there Endpoints? If it is empty, there is your answer
kubectl -n rutas-norte-pro get endpointslices -l kubernetes.io/service-name=web-store

# Does the Service's selector match the pods' labels?
kubectl -n rutas-norte-pro get svc web-store -o jsonpath='{.spec.selector}'
kubectl -n rutas-norte-pro get pods -l app=web-store --show-labels

A classic source of error with our convention: the Service selects on app and environment, and somebody deploys the pods with environment: production instead of environment: pro. The pods are perfect, the Service is perfect, and they never find each other.

Diagnosing a 502: that is the case in section 8.

  1. The inspection tools

kubectl describe: the complete view

kubectl -n rutas-norte-pro describe pod bookings-api-7d9f8c4b5-x2klm

The sections to read, in order of usefulness:

Section What to look for
Events (at the end) Always start here
Status / Conditions Ready, ContainersReady, PodScheduled and their reasons
Containers → State / Last State The current state and the reason for the previous termination
Containers → Restart Count How many times it has died
Node Where it is: it allows correlation with node problems
Mounts / Volumes What it mounts and from where
QoS Class Guaranteed, Burstable or BestEffort (03-05)

kubectl logs, exec and port-forward

We already know them from 01-05 and 07-05. The specific uses for debugging:

# Logs of the previous run: THE command in the face of a CrashLoopBackOff
kubectl -n rutas-norte-pro logs POD --previous -c api

# Get inside the container
kubectl -n rutas-norte-pro exec -it POD -c api -- sh

# Usual checks once inside
env | sort
cat /etc/config/application.yaml
nslookup bookings-postgres.rutas-norte-pro.svc.cluster.local
wget -qO- http://localhost:8080/ready

# port-forward: BYPASS the Ingress and the Service to isolate the problem
kubectl -n rutas-norte-pro port-forward pod/bookings-api-7d9f8c4b5-x2klm 8080:8080

A port-forward straight to the pod is the most valuable isolation test there is. If it works, you have ruled out in one go: the Ingress, the TLS certificate, the Service, kube-proxy and the NetworkPolicies. The problem is in the network layer, not in the application. And if it does not work, the problem is in the application and there is no need to look at the network.

A netshoot pod for network problems

Production images are minimal and carry no network tools. A disposable pod with the whole toolkit:

kubectl -n rutas-norte-pro run netshoot --rm -it \
  --image=nicolaka/netshoot:latest \
  --labels="app=debug,environment=pro" \
  --restart=Never -- bash
# Does cluster DNS resolve? (04-03)
nslookup bookings-postgres.rutas-norte-pro.svc.cluster.local
dig +short bookings-api.rutas-norte-pro.svc.cluster.local

# Can you reach the port? It confirms or rules out a NetworkPolicy (04-06)
nc -zv bookings-postgres 5432
curl -v http://bookings-api/health

# What is the route and where does it get lost?
traceroute 10.244.2.17
mtr --report --report-cycles 10 bookings-api

# Capture traffic
tcpdump -i any -n port 8080

An important warning: the pod's labels matter. In rutas-norte-pro there is a deny-all NetworkPolicy, so a netshoot pod without the right labels will not be able to talk to anything, and you will mistake a policy problem for an application problem. That is why we launch it with --labels.

  1. kubectl debug: ephemeral containers, copies and nodes

kubectl debug is the most powerful and least known tool, stable since Kubernetes 1.25. It has three very different modes.

Mode 1: ephemeral containers

An ephemeral container is added to a pod that is already running, sharing its network namespace and optionally its process namespace. It is the solution to the problem of distroless images, which do not even have a shell:

kubectl -n rutas-norte-pro debug -it bookings-api-7d9f8c4b5-x2klm \
  --image=nicolaka/netshoot:latest \
  --target=api \
  -- bash
Flag Function
--image An image with the tools you need
--target=api Shares the process namespace with that container
-it An interactive session

With --target, from the ephemeral container you see the original container's processes:

ps aux
PID   USER     COMMAND
1     node     node /app/server.js          ← the bookings-api process
28    root     bash                          ← our debugging session

And from there:

# See the real process's open file descriptors
ls -l /proc/1/fd | head -20

# See its environment variables
cat /proc/1/environ | tr '\0' '\n'

# Dump the JVM's stack or run a profile
kill -QUIT 1

# Check the network FROM inside the pod, with its own IP
curl -v localhost:8080/ready
netstat -tulpn

Important points:

  • The ephemeral container cannot be removed once added: it stays until the pod dies.
  • It has no probes and no resources, and it cannot modify the pod.
  • It restarts nothing: the pod keeps serving traffic while you investigate. That is its great advantage.

Mode 2: --copy-to, debugging a copy without touching production

It creates a new pod, a copy of the original, with the modifications you ask for. The original pod is left untouched.

# A copy with a debugging container added
kubectl -n rutas-norte-pro debug bookings-api-7d9f8c4b5-x2klm \
  --copy-to=api-debug \
  --image=nicolaka/netshoot:latest \
  --share-processes \
  -it -- bash

# A copy with the command replaced: for debugging a CrashLoopBackOff
kubectl -n rutas-norte-pro debug bookings-api-7d9f8c4b5-x2klm \
  --copy-to=api-debug \
  --container=api \
  -- sleep 7200

# A copy with a different image: to test whether a previous version works
kubectl -n rutas-norte-pro debug bookings-api-7d9f8c4b5-x2klm \
  --copy-to=api-previous-version \
  --set-image=api=registry.rutasnorte.example/bookings-api:2.8.0

Why it is so valuable: the copied pod does not carry the Service selector's labels, so it receives no production traffic. You can break it, restart it, modify it and experiment without any customer noticing. When you are done:

kubectl -n rutas-norte-pro delete pod api-debug

This is the mode that solves the classic dilemma of "I need to debug in production but I cannot touch production".

Mode 3: kubectl debug node/, getting into the node

When the problem is the node's and not the pod's:

kubectl debug node/rutas-norte-worker-2 -it --image=busybox

It creates a pod on that node with the host's filesystem mounted at /host:

chroot /host

# Disk space: the commonest cause of evictions
df -h

# What is going on in the node?
top
journalctl -u kubelet --since "1 hour ago" | tail -50
crictl ps -a | head
crictl logs <container-id>

# The container logs, exactly as we saw them in 07-05
ls -la /var/log/containers/ | grep bookings-api

Security warning: kubectl debug node/ creates a privileged pod with full access to the node. Whoever can run it effectively controls that machine and can read the secrets of every pod running on it. It must be restricted by RBAC to a very small group. Module 8 develops these controls.

A comparison of the three modes

Mode Touches the original pod Receives traffic What for
Ephemeral container Yes (it adds a container) Yes, it keeps serving Inspecting a live pod with no shell
--copy-to No No Risk-free experimenting; debugging CrashLoopBackOff
node/ No (a new pod) No Node problems: disk, kubelet, runtime

  1. A real case: the store returns 502 during deployments

Now we bring it all together. This is the incident that has been open since 02-04, and we are going to close it using the module's three signals.

The symptom

Every time a new version of bookings-api is deployed in rutas-norte-pro, for about 90 seconds some customers get a 502 error when trying to buy. Not all of them, and not always the same ones. Outside deployments, everything is perfect.

The team has been ignoring it for months: "it only happens when deploying, and it goes away on its own".

Step 1 — The metric confirms the problem exists and bounds it

On the 07-04 dashboard, the bookings-api errors panel, with the range set to yesterday's deployment:

sum(rate(rutasnorte_requests_total{environment="pro", code=~"5.."}[1m]))
  / sum(rate(rutasnorte_requests_total{environment="pro"}[1m]))

The graph shows three error spikes of 4–6 %, about 25 seconds each, separated by about 40 seconds. The error rate is zero before and after.

A first important clue: three spikes, not one. The Deployment has 6 replicas with maxSurge: 2, so the rollout advances in three batches. One spike per batch. That rules out a one-off start-up problem with the new version: it is a pattern that repeats with each pod replacement.

And a second query rules out the external provider:

sum(rate(rutasnorte_payment_gateway_calls_total{result=~"error|timeout"}[1m]))
  / sum(rate(rutasnorte_payment_gateway_calls_total[1m]))

Flat at zero throughout the deployment. The problem is ours.

Step 2 — The logs say who fails and who does not

In Kibana, narrowing to the deployment's window:

kubernetes.namespace_name: "rutas-norte-pro" and http_code: 502

The result: 4,812 documents, all from web-store (which acts as a proxy towards the API), none from bookings-api.

This is very revealing and worth stopping to think about. If bookings-api were generating errors, its own logs would show 500 codes. There is not a single one. bookings-api never saw those requests. The 502 is generated by whoever is trying to talk to it.

The next query, over the Ingress controller's logs:

kubernetes.labels.app: "ingress-nginx" and message: *502*
upstream prematurely closed connection while reading response header from upstream,
upstream: "http://10.244.2.17:8080/api/bookings"

"Upstream prematurely closed connection": the backend closed the connection while the Ingress was waiting for the response. It did not refuse it (that would give connection refused): it accepted it and closed it halfway.

Step 3 — The events give the exact timeline

This is where the event exporter from section 3 proves its worth: the deployment was yesterday, so the original events expired 20 hours ago. But they are in Elasticsearch:

component: "kubernetes-events" and namespace: "rutas-norte-pro"
  and object_name: bookings-api-*

Sorted chronologically:

14:32:01  Normal  Killing     bookings-api-7d9f8c4b5-x2klm  Stopping container api
14:32:01  Normal  Scheduled   bookings-api-9f2c1a8b6-kp3nm  Successfully assigned to worker-1
14:32:04  Normal  Started     bookings-api-9f2c1a8b6-kp3nm  Started container api
14:32:19  Normal  Killing     bookings-api-7d9f8c4b5-mn8pq  Stopping container api

And now the decisive observation, which requires looking at the IP:

Pod 10.244.2.17 (bookings-api-7d9f8c4b5-x2klm) received "Killing" at 14:32:01.
The 502 errors towards 10.244.2.17 in the Ingress logs are timestamped
between 14:32:01.340 and 14:32:03.180.

The Ingress kept sending traffic to that pod for 1.8 seconds after the kubelet started terminating it.

That is exactly the phenomenon we studied in 07-01: the race between the endpoint removal and the SIGTERM. When Kubernetes decides to terminate a pod, two paths advance in parallel with no coordination: the kubelet sends SIGTERM almost instantly, while the endpoint removal has to propagate through the endpoints controller, kube-proxy and the Ingress controller. The process dies before traffic stops arriving.

Step 4 — Confirm the hypothesis against the manifest

kubectl -n rutas-norte-pro get deploy bookings-api -o yaml | \
  grep -A 6 -E "terminationGracePeriodSeconds|lifecycle|preStop|readinessProbe"
      terminationGracePeriodSeconds: 30
      containers:
      - name: api
        readinessProbe:
          httpGet:
            path: /ready
            port: http

Confirmed: there is no lifecycle.preStop. The manifest that reached production went in without that part of what we designed in 07-01.

And an additional check that refines the diagnosis:

kubectl -n rutas-norte-pro get deploy bookings-api \
  -o jsonpath='{.spec.strategy.rollingUpdate}'
{"maxSurge":2,"maxUnavailable":1}

A second problem: maxUnavailable: 1 allows only 5 pods to be serving instead of 6 during the deployment, which makes each spike's impact worse.

Step 5 — Reproduce it in pre-production

You do not fix production blind. You reproduce it in rutas-norte-pre with synthetic load:

# Terminal 1: generate constant load
kubectl -n rutas-norte-pre run load --rm -it --image=williamyeh/hey --restart=Never -- \
  -z 300s -c 20 http://bookings-api/api/routes

# Terminal 2: watch the Endpoints in real time
kubectl -n rutas-norte-pre get endpointslices -l kubernetes.io/service-name=bookings-api -w

# Terminal 3: trigger the deployment
kubectl -n rutas-norte-pre rollout restart deployment/bookings-api

The result: 1.4 % errors during the deployment. Reproduced.

Step 6 — The fix

# k8s/environments/pro/bookings-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
spec:
  replicas: 6
  minReadySeconds: 15
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0        # FIX 2: never fewer than 6 serving
  template:
    spec:
      # FIX 3: 45 s in total. With the 10 s preStop, 35 s are left
      # for the application to shut down cleanly after the SIGTERM.
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          # FIX 1 (the main one): win the race against the SIGTERM.
          # During these 10 seconds the container KEEPS SERVING as
          # normal, while the endpoint removal propagates through
          # the endpoints controller, kube-proxy and the Ingress.
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

And a complementary fix on the Ingress controller, so that it reacts faster to endpoint changes:

# Annotations on the bookings-api Ingress
metadata:
  annotations:
    # Retry on another backend if this one closes the connection
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout http_502"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"

Step 7 — Verify

kubectl -n rutas-norte-pre apply -f k8s/environments/pre/bookings-api-deployment.yaml
# Repeat the load test from step 5

The result: 0 errors during the deployment. The same scenario previously produced 1.4 %.

It is taken to production and verified on the 07-04 dashboard:

sum(increase(rutasnorte_requests_total{environment="pro", code=~"5.."}[10m]))

Flat at zero throughout the deployment.

Step 8 — Prevent a recurrence

The incident is not closed until it cannot happen again:

  1. A specific alert (07-04): detect errors during deployments, instead of waiting for a customer to complain.
- alert: ErrorsDuringDeployment
  expr: |
    (
      sum(rate(rutasnorte_requests_total{environment="pro", code=~"5.."}[2m]))
      / sum(rate(rutasnorte_requests_total{environment="pro"}[2m]))
    ) > 0.01
    and on()
    (changes(kube_deployment_status_observed_generation{
      namespace="rutas-norte-pro"}[10m]) > 0)
  for: 2m
  labels:
    severity: warning
    team: platform
  annotations:
    summary: "There are 5xx errors coinciding with a deployment"
    description: >
      An error rate of {{ $value | humanizePercentage }} while a
      Deployment is updating. Review the affected component's preStop,
      readinessProbe and maxUnavailable.
    runbook_url: "https://runbooks.rutasnorte.example/deployment-errors"

  1. An automatic check in continuous integration: reject any manifest for a component with a Service that does not declare preStop and readinessProbe.

  2. A load test during deployment as a mandatory step in rutas-norte-pre before promoting to production.

The lesson of the case

No single signal solved the problem on its own:

Signal What it contributed What it could not contribute
Metrics (07-03/04) That it existed, when, and the three-spike pattern Why
Logs (07-05) That the 502 is generated by the Ingress, not the API; the exact message When each pod was terminated
Events (07-06) The exact timeline to the millisecond The impact on customers
Probes (07-01) The conceptual framework: the SIGTERM race The evidence

And one detail that is far from minor: the original events had expired 20 hours earlier. Without the event export to Elasticsearch from section 3, step 3 — the one that gave the answer — would have been impossible, and the team would still be trading theories.

  1. Gather the evidence before restarting

This short section is the most important advice in the lesson.

When something fails, the universal impulse is to restart it. kubectl delete pod, kubectl rollout restart, reboot the node. And it often works: the symptom disappears.

But restarting destroys the evidence. And if you do not know why it failed, it will fail again, probably at a worse moment.

What exactly is lost when you delete a pod:

Evidence Is it lost?
The current container's logs Yes, if they are not centralized (07-05)
The previous run's logs (--previous) Yes, irrecoverably
The state of the memory and the processes Yes
Open file descriptors and connections Yes
The contents of emptyDir volumes Yes
lastState.terminated with the exit code Yes
The pod's events Yes, as soon as the 1-hour TTL passes
Metrics No (Prometheus keeps them)
The contents of PVCs No

The capture checklist

Run it before touching anything. It takes 30 seconds and it saves the investigation:

#!/bin/bash
# capture-evidence.sh <namespace> <pod>
# ALWAYS run this before deleting or restarting a problematic pod.

NS=$1
POD=$2
DIR="/tmp/incident-$(date +%Y%m%d-%H%M%S)-${POD}"
mkdir -p "$DIR"

echo "Capturing evidence in $DIR ..."

# 1. The pod's complete definition and state
kubectl -n "$NS" get pod "$POD" -o yaml > "$DIR/pod.yaml"

# 2. describe: it includes the events, which EXPIRE IN 1 HOUR
kubectl -n "$NS" describe pod "$POD" > "$DIR/describe.txt"

# 3. Current and previous logs, from ALL containers
for C in $(kubectl -n "$NS" get pod "$POD" \
           -o jsonpath='{.spec.containers[*].name} {.spec.initContainers[*].name}'); do
  kubectl -n "$NS" logs "$POD" -c "$C" > "$DIR/logs-$C.txt" 2>&1
  kubectl -n "$NS" logs "$POD" -c "$C" --previous > "$DIR/logs-$C-previous.txt" 2>&1
done

# 4. The whole namespace's events: the incident's timeline
kubectl -n "$NS" get events --sort-by=.lastTimestamp -o yaml > "$DIR/events.yaml"
kubectl get events -A --sort-by=.lastTimestamp > "$DIR/events-cluster.txt"

# 5. The state of the node the pod lives on
NODE=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.spec.nodeName}')
kubectl describe node "$NODE" > "$DIR/node-$NODE.txt"

# 6. The state of the Service and its Endpoints
APP=$(kubectl -n "$NS" get pod "$POD" -o jsonpath='{.metadata.labels.app}')
kubectl -n "$NS" get svc,endpointslices -l app="$APP" -o yaml > "$DIR/network.yaml"

# 7. Current consumption (07-02)
kubectl -n "$NS" top pod "$POD" --containers > "$DIR/usage.txt" 2>&1

# 8. The namespace's general state, for context
kubectl -n "$NS" get all -o wide > "$DIR/namespace.txt"

tar czf "$DIR.tar.gz" -C /tmp "$(basename "$DIR")"
echo "Evidence packaged in $DIR.tar.gz"
echo "NOW you can restart."

When you do have to restart immediately

There is one legitimate exception, and we have to be honest about it: when the service is down and restoring it is more urgent than understanding it.

In that case, the correct order is:

  1. Run the capture script (30 seconds, no more).
  2. Restore the service.
  3. Investigate with the captured evidence.

Never "restart and we'll investigate later", because by nine in the morning there will be nothing left to investigate.

The alternative: isolate instead of restarting

If the problem affects a single pod out of several replicas, there is a much better option than deleting it: take it out of the Service without killing it.

# Change one of the selector's labels: the pod leaves the Endpoints
# but stays alive, with all its memory and state intact.
kubectl -n rutas-norte-pro label pod bookings-api-7d9f8c4b5-x2klm app=bookings-api-isolated --overwrite

The effects, all of them desirable:

  • The pod stops receiving traffic immediately: customers stop suffering it.
  • The ReplicaSet sees that it is a replica short and creates a new, healthy pod: the service is restored.
  • The problematic pod stays alive with all its state: you can exec into it, dump the memory, look at the open descriptors and investigate calmly.

It is the best tool for debugging a problem that only shows up in production and only on one replica. When you are done:

kubectl -n rutas-norte-pro delete pod bookings-api-7d9f8c4b5-x2klm

Common Mistakes and Tips

1. Starting with kubectl logs without looking at the status. If the pod is Pending, there are no logs. Follow the decision tree in order.

2. Forgetting --sort-by=.lastTimestamp on the events. Without it they come out in arbitrary order and the timeline, which is the valuable part, is lost.

3. Not knowing that events expire after an hour. It is the most important fact in this lesson. Export the events to the logging stack.

4. Forgetting --previous in the face of a CrashLoopBackOff. The current container's logs say nothing: it has just started. The cause is in the one that died.

5. Restarting before capturing. It destroys --previous, the memory state and the events. Thirty seconds of capture save hours.

6. Not using port-forward to isolate the layer. It is the test that rules out the Ingress, the Service, kube-proxy and the NetworkPolicies in one go.

7. Confusing 502 with 503. A 503 is "there is no backend" (empty Endpoints); a 502 is "the backend failed" (the SIGTERM race). Different causes and different fixes.

8. Launching a debugging pod without the right labels. With deny-all in rutas-norte-pro, a netshoot pod with no labels talks to nothing, and you will mistake a NetworkPolicy for an application problem.

9. Debugging in production by modifying the real pod. Use kubectl debug --copy-to: the copy does not carry the selector's labels, it receives no traffic and you can do whatever you like with it.

10. Ignoring the Normal events. In the case in section 8, the decisive event was a Killing of type Normal. The absence of an expected event also tells you something.

11. Not correlating the three signals. A metric says when, a log says what and an event says in what order. None is enough on its own.

12. Closing the incident when the service is restored. Without an identified root cause and without a preventive measure, it will happen again. Step 8 of section 8 is not optional.

Exercises

Exercise 1 — Diagnose three pods with the master table

After a deployment in rutas-norte-pre, this is the state:

NAME                                     READY   STATUS              RESTARTS      AGE
bookings-api-9f2c1a8b6-kp3nm             0/1     CrashLoopBackOff    7 (30s ago)   12m
notifications-worker-5f7c8b9d4-tz2mv     0/1     ContainerCreating   0             12m
web-store-6c8b9d7f4-hj3ks                1/1     Running             0             3d
redis-cache-0                            0/1     Pending             0             12m

And these events:

LAST SEEN   TYPE      REASON              OBJECT                                MESSAGE
30s         Warning   BackOff             pod/bookings-api-9f2c1a8b6-kp3nm      Back-off restarting failed container api
2m          Warning   FailedMount         pod/notifications-worker-5f7c8b9d4-tz2mv  MountVolume.SetUp failed for volume "smtp-credentials": secret "smtp-credentials-v2" not found
11m         Warning   FailedScheduling    pod/redis-cache-0                     0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims

In addition, web-store returns 503 to customers despite being 1/1 Running.

For each of the four problems:

  1. Identify the probable cause.
  2. Write the sequence of commands that confirms it.
  3. Propose the fix.
  4. Explain why web-store gives a 503 while being Running and ready.

Exercise 2 — Debug a container with no shell

bookings-api has been migrated to a distroless image (no shell, no curl, no ps). In production, one particular pod out of six responds with 8-second latencies while the other five run at 40 ms. The metric confirms it, the logs show no errors, and the pod is 1/1 Running.

  1. Why can you not use kubectl exec -it POD -- sh?
  2. Write the commands to investigate the pod without restarting it and without taking it out of production.
  3. Write the alternative that takes it out of production without killing it, explaining what you gain.
  4. Once inside, what three specific checks would you make to explain the latency?

Exercise 3 — Reconstruct a past incident

On Sunday at 04:12, occupancy-reports (the CronJob from 06-03) failed. On Monday at 10:00 you are asked to explain what happened. You check:

$ kubectl -n rutas-norte-pro get jobs -l app=occupancy-reports
NAME                          STATUS     COMPLETIONS   DURATION   AGE
occupancy-reports-28936960    Failed     0/1           14m        30h

$ kubectl -n rutas-norte-pro get events --field-selector involvedObject.name=occupancy-reports-28936960
No resources found in rutas-norte-pro namespace.

$ kubectl -n rutas-norte-pro logs job/occupancy-reports-28936960
Error from server (BadRequest): pod for job "occupancy-reports-28936960" not found
  1. Why are there no events and no logs?
  2. List every source of information that is still available, with the specific command or query for each one.
  3. Reconstruct what may have happened if Prometheus shows that node rutas-norte-worker-2 was under memory pressure between 04:05 and 04:30, and that the data-bookings-postgres-0 volume was at 96 %.
  4. Propose three measures so that the next night-time incident is investigable.

Solutions

Solution 1

Problem 1: bookings-api in CrashLoopBackOff.

Probable cause: the container starts and dies. The candidates, in order: a configuration failure, OOMKilled, or an overly aggressive liveness probe (07-01).

Confirmation:

NS=rutas-norte-pre
POD=bookings-api-9f2c1a8b6-kp3nm

# 1. The cause is usually here
kubectl -n $NS logs $POD --previous

# 2. The reason and the exit code of the termination
kubectl -n $NS get pod $POD \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' | jq

# 3. Are there Unhealthy events before the Killing ones? (liveness)
kubectl -n $NS describe pod $POD | grep -B 3 -A 12 Events

# 4. If the logs are empty: launch a copy with the command replaced
kubectl -n $NS debug pod/$POD --copy-to=api-debug --container=api -- sleep 3600
kubectl -n $NS exec -it api-debug -c api -- sh

The fix, depending on what step 2 returns:

reason / exitCode Cause Fix
OOMKilled / 137 Memory limit too low or a leak Raise limits.memory (07-02) or fix the leak
Error / 1 with a config log A variable or Secret is missing Fix the ConfigMap or the Secret
Error / 127 Binary not found Fix the command or the image
No log, preceded by Unhealthy Liveness too aggressive Add a startupProbe (07-01)

Problem 2: notifications-worker in ContainerCreating.

Cause: the event gives it directly: secret "smtp-credentials-v2" not found. The manifest references a Secret that does not exist in this namespace. A typical pattern: the Secret was renamed to -v2 in pro but was not created in pre.

Confirmation:

# Does the Secret it expects exist?
kubectl -n rutas-norte-pre get secret smtp-credentials-v2

# Which one actually exists?
kubectl -n rutas-norte-pre get secrets | grep smtp

# What name does the Deployment reference?
kubectl -n rutas-norte-pre get deploy notifications-worker \
  -o jsonpath='{.spec.template.spec.volumes}' | jq

The fix: create the missing Secret, or fix the manifest's reference:

kubectl -n rutas-norte-pre create secret generic smtp-credentials-v2 \
  [email protected] \
  --from-literal=password="$SMTP_PASSWORD"

The pod moves to Running automatically as soon as the Secret exists: the kubelet retries the mount periodically. There is no need to delete it.

Prevention: this error is a symptom of managing per-environment manifests by hand. It is exactly the problem that Kustomize (10-04) and GitOps (10-05) solve.

Problem 3: redis-cache-0 in Pending.

Cause: the event says it: 3 pod has unbound immediate PersistentVolumeClaims. The StatefulSet's PVC is not bound.

Confirmation:

# The PVC's state
kubectl -n rutas-norte-pre get pvc -l app=redis-cache
NAME                     STATUS    VOLUME   CAPACITY   STORAGECLASS          AGE
data-redis-cache-0       Pending                       rutasnorte-fast       12m
# Why is it not being provisioned?
kubectl -n rutas-norte-pre describe pvc data-redis-cache-0 | tail -10

# Does the StorageClass exist? (05-04)
kubectl get storageclass

The three typical causes:

Cause Check
The StorageClass does not exist in this cluster kubectl get sc rutasnorte-fast
The CSI provisioner is down kubectl -n kube-system get pods | grep csi
There is no capacity left in the backend The PVC's events

The fix: if it is the first (the most likely in pre, where the fast class may not exist), create the StorageClass or change the volumeClaimTemplates to rutasnorte-standard.

Beware of a detail from 06-01: volumeClaimTemplates is immutable. You cannot change an existing StatefulSet's StorageClass; you have to delete it with --cascade=orphan, recreate the manifest and create it again.

Problem 4: web-store gives a 503 while being 1/1 Running.

This is the most instructive part of the exercise, because the pod looks perfect.

Cause: an Ingress 503 means "there is no backend to send to". If the pod is Running and Ready, the cause lies between the Service and the pod:

  1. The Service's selector does not match the pod's labels.
  2. The Service's port points to an incorrect targetPort.
  3. The Ingress points to a Service with the wrong name or port.

Confirmation, in this exact order:

NS=rutas-norte-pre

# 1. THE KEY CHECK: are there Endpoints?
kubectl -n $NS get endpointslices -l kubernetes.io/service-name=web-store
NAME              ADDRESSTYPE   PORTS   ENDPOINTS   AGE
web-store-x4k2p   IPv4          <unset> <unset>     3d

Empty Endpoints with a Ready pod: the selector does not match.

# 2. Compare the selector and the labels
kubectl -n $NS get svc web-store -o jsonpath='{.spec.selector}'
{"app":"web-store","environment":"preproduction"}
kubectl -n $NS get pods -l app=web-store --show-labels
NAME                        READY   STATUS    LABELS
web-store-6c8b9d7f4-hj3ks   1/1     Running   app=web-store,environment=pre,...

There it is. The Service is looking for environment: preproduction and the pod has environment: pre. Both objects are valid, both are healthy, and they never find each other.

# 3. Confirm the pod works, bypassing the Service
kubectl -n $NS port-forward pod/web-store-6c8b9d7f4-hj3ks 8080:80
curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/
# → 200: the pod is perfect, the problem is the selector

The fix: align it with the Rutas Norte convention, which uses dev/pre/pro:

kubectl -n rutas-norte-pre patch svc web-store \
  -p '{"spec":{"selector":{"app":"web-store","environment":"pre"}}}'

# Verify that the Endpoints appear
kubectl -n rutas-norte-pre get endpointslices -l kubernetes.io/service-name=web-store

The lesson: a pod's status says nothing about whether it receives traffic. A pod can be 1/1 Running for days without a single request reaching it. Question 4 of the decision tree (does the Service have Endpoints?) is the one to ask, and it is answered in two seconds.

Solution 2

1. Why kubectl exec -it POD -- sh fails.

A distroless image contains only the language runtime and the application: there is no /bin/sh, no /bin/bash, no utility binary at all. It is an excellent security decision (a minimal attack surface, which we will see in 08-05), but it prevents the classic exec:

$ kubectl -n rutas-norte-pro exec -it bookings-api-9f2c1a8b6-kp3nm -- sh
OCI runtime exec failed: exec failed: unable to start container process:
exec: "sh": executable file not found in $PATH: unknown

2. Investigating without restarting and without taking it out of production: an ephemeral container.

NS=rutas-norte-pro
POD=bookings-api-9f2c1a8b6-kp3nm

kubectl -n $NS debug -it $POD \
  --image=nicolaka/netshoot:latest \
  --target=api \
  --profile=general \
  -- bash

The key points of this command:

  • --target=api shares the process namespace with the api container, which lets you see and inspect its process.
  • The ephemeral container also shares the network namespace: the same IP, the same ports, the same view of the network.
  • The pod keeps serving traffic throughout the session. Nothing is restarted.

A warning: the ephemeral container cannot be removed. It stays in the pod until the pod dies. With six replicas, that is acceptable.

3. The alternative that takes it out of production without killing it.

# Change the selector's label
kubectl -n rutas-norte-pro label pod bookings-api-9f2c1a8b6-kp3nm \
  app=bookings-api-quarantine --overwrite

What exactly you gain:

Effect Benefit
It leaves the Endpoints instantly Customers stop suffering the 8-second latency
The ReplicaSet creates a new pod The service goes back to 6 healthy replicas
The problematic pod stays alive You keep the memory, connections, descriptors and state
No incoming traffic You can take dumps and profiles without affecting anybody

It is strictly better than kubectl delete pod: you restore the service just as fast and you keep all the evidence.

A complementary alternative with --copy-to, if you want to experiment with changes:

kubectl -n rutas-norte-pro debug $POD \
  --copy-to=api-analysis \
  --image=nicolaka/netshoot:latest \
  --share-processes -it -- bash

4. The three checks that explain the latency.

Check A — Is the process saturated or blocked?

# How much CPU is it really using?
top -b -n 1 -p 1

# What is it blocked on? State D = uninterruptible I/O wait
cat /proc/1/status | grep -E "State|Threads|VmRSS"

# What system calls is it making? (if tracing is permitted)
strace -p 1 -c -f -T 2>&1 | head -25

A process at 100 % CPU points to a loop or a runaway GC. A process in state D with low CPU points to waiting on the network or on disk: the most likely answer in this case.

Check B — Where are the connections going and how many are there?

# Established connections, grouped by destination
ss -tanp state established | awk '{print $5}' | sort | uniq -c | sort -rn
     20 10.244.3.88:5432      ← bookings-postgres: 20 connections (the pool is FULL)
      1 10.244.1.45:6379      ← redis-cache

Twenty connections to PostgreSQL with a pool maximum of 20: the pool is exhausted. It is exactly the scenario from 07-01. Every new request waits for a connection to be released, and hence the 8 seconds.

# Are connections piling up in the listen queue?
ss -tln | grep 8080
# The Recv-Q column shows requests waiting to be accepted

Check C — Is the latency on our side or on PostgreSQL's?

# Measure the real latency towards the database from THIS pod
for i in $(seq 1 10); do
  time nc -zv bookings-postgres.rutas-norte-pro.svc.cluster.local 5432
done

# Does DNS resolve normally? Slow DNS explains odd latencies
time nslookup bookings-postgres.rutas-norte-pro.svc.cluster.local

# The readiness endpoint from 07-01 responds with the diagnosis
curl -s -w "\ntime: %{time_total}s\n" localhost:8080/ready
{"status":"not-ready","reason":"connection pool exhausted","checks":{"poolFree":false}}

Confirmed. And the question remains why only this pod out of six. Two hypotheses to test against the metrics from 07-03:

# Does this pod receive more traffic than the others?
sum by (pod) (rate(rutasnorte_requests_total{environment="pro"}[5m]))

# Is the pod on a node with problems?
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{
  namespace="rutas-norte-pro", pod=~"bookings-api-.*"}[5m]))
  / sum by (pod) (rate(container_cpu_cfs_periods_total{
  namespace="rutas-norte-pro", pod=~"bookings-api-.*"}[5m]))

If CPU throttling is high on this pod alone, the explanation is that it shares a node with occupancy-reports or with another heavy process, and it releases connections more slowly than the others. The fix: an anti-affinity rule (06-05) or a review of the limits (07-02).

Solution 3

1. Why there are no events and no logs.

There are no events because they expire after an hour. The failure was at 04:12 on Sunday; it is now 10:00 on Monday: 30 hours have passed. The API Server deleted them at 05:12 on Sunday, 29 hours earlier.

There are no logs because the pod no longer exists. A CronJob with failedJobsHistoryLimit (1 by default) keeps the Job object, but the pods are collected when the limit is exceeded or when garbage collection kicks in. kubectl logs job/... needs the pod to exist: it only reads from the node's file.

And even if the pod existed, the kubelet's rotation (07-05) would probably already have taken the file.

2. The sources that are still available.

Source What it contributes Command or query
The Job object State, timings, number of attempts kubectl -n rutas-norte-pro get job occupancy-reports-28936960 -o yaml
Centralized logs (07-05) The pod's complete log component: "occupancy-reports" and @timestamp >= "2026-08-02T04:00:00Z"
Exported events (07-06) The timeline, surviving the TTL component: "kubernetes-events" and object_name: occupancy-reports-*
Prometheus (07-03) The state of the job, the node and the resources kube_job_status_failed, container_memory_working_set_bytes
Grafana (07-04) The dashboards with the range set to the incident The occupancy-reports panel
Alertmanager Which alerts fired and when The alert history or the Slack channel
kube-state-metrics The reason for the last termination kube_pod_container_status_last_terminated_reason
Git Whether there was a configuration change git log --since="2026-08-01" -- k8s/

Specific commands and queries:

# The Job object retains a lot of information
kubectl -n rutas-norte-pro get job occupancy-reports-28936960 -o yaml | \
  yq '.status, .spec.backoffLimit, .spec.activeDeadlineSeconds'
status:
  conditions:
    - type: Failed
      status: "True"
      reason: BackoffLimitExceeded
      message: Job has reached the specified backoff limit
      lastTransitionTime: "2026-08-02T04:26:11Z"
  failed: 3
  startTime: "2026-08-02T04:12:00Z"

Now we know: it started at 04:12, failed three times and was given up at 04:26.

In Kibana:

component: "occupancy-reports"
  and @timestamp >= "2026-08-02T04:00:00.000Z"
  and @timestamp <= "2026-08-02T04:30:00.000Z"

And in Prometheus:

# The node's memory during the incident
node_memory_MemAvailable_bytes{instance=~"rutas-norte-worker-2.*"}

# Space on the PostgreSQL volume
kubelet_volume_stats_available_bytes{persistentvolumeclaim="data-bookings-postgres-0"}
  / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="data-bookings-postgres-0"}

# The reason for the last termination of the job's pods
kube_pod_container_status_last_terminated_reason{
  namespace="rutas-norte-pro", pod=~"occupancy-reports-.*"}

# Evictions on that node
increase(kube_pod_status_reason{reason="Evicted", node="rutas-norte-worker-2"}[1h])

3. Reconstructing the incident.

With the two facts provided (memory pressure on worker-2 between 04:05 and 04:30, and the PostgreSQL volume at 96 %), the most likely reconstruction is:

~04:00  The scheduled pg_dump backup (05-06) starts and writes the dump
        onto the bookings-postgres volume, which was already very full.
        The volume reaches 96%.

04:05   The heavy writing saturates the page cache on node worker-2.
        The node's available memory starts to fall.

04:12   The occupancy-reports CronJob starts its first pod, scheduled on
        worker-2 (the node with the most free CPU at that moment). The report
        needs ~800 Mi of memory and runs heavy queries against
        bookings-postgres.

04:14   worker-2's kubelet detects memory pressure and begins to
        evict pods, starting with the lowest QoS ones. The job's pod,
        with Burstable QoS, is a preferred candidate over the platform's
        Guaranteed pods.
        → The pod is evicted (Evicted) or dies OOMKilled.

04:15   The Job controller retries (attempt 2 of 3).
        The node is still under pressure. It fails again.

04:20   Third attempt. On top of that, the queries against PostgreSQL may
        now be failing or running extremely slowly too, because the volume
        at 96% degrades write performance for the temporary files
        the aggregation query needs.

04:26   BackoffLimitExceeded: the Job is marked as Failed.
        There is no occupancy report for Sunday.

Confirming the hypothesis (the commands that validate or rule it out):

# Were there evictions on worker-2? It would confirm the eviction theory
increase(kube_pod_status_reason{reason="Evicted", node="rutas-norte-worker-2"}[2h])

# Or was it OOMKilled? That would be an alternative hypothesis
kube_pod_container_status_last_terminated_reason{
  pod=~"occupancy-reports-.*", reason="OOMKilled"}

And in the logs:

component: "kubernetes-events"
  and object_name: occupancy-reports-*
  and reason: ("Evicted" or "OOMKilling" or "Failed")

An Evicted event with the message The node was low on resource: memory would confirm the complete reconstruction.

An important methodological note: this is a hypothesis consistent with the data, not a proven fact. The difference matters. With the measures in point 4, the next incident will allow it to be stated with certainty instead of deduced.

4. Three measures to make the next one investigable.

Measure A — Export the events to the logging stack. It is the gap that caused 80 % of this exercise's difficulty.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-exporter
  namespace: logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: event-exporter
  template:
    metadata:
      labels:
        app: event-exporter
        app.kubernetes.io/part-of: rutas-norte
    spec:
      serviceAccountName: event-exporter   # read-only RBAC over events
      containers:
        - name: exporter
          image: ghcr.io/resmoio/kubernetes-event-exporter:v1.7
          args: ["-conf=/etc/config/config.yaml"]
          volumeMounts:
            - name: config
              mountPath: /etc/config
          resources:
            requests:
              cpu: "20m"
              memory: "64Mi"
            limits:
              memory: "128Mi"
      volumes:
        - name: config
          configMap:
            name: event-exporter-config

With the configuration from section 3, which emits to stdout and therefore inherits the whole 07-05 stack, with its 30 days of retention. Cost: about 64 Mi of memory. Benefit: never losing an incident's timeline again.

Measure B — Keep the Job history and prevent pod collection.

# k8s/base/occupancy-reports/cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: occupancy-reports
  namespace: rutas-norte-pro
spec:
  schedule: "12 4 * * *"
  timeZone: "Europe/Madrid"
  concurrencyPolicy: Forbid

  # Keep the last 7 failures and 3 successes, WITH THEIR PODS.
  # The defaults are 1 and 3: almost everything is lost.
  failedJobsHistoryLimit: 7
  successfulJobsHistoryLimit: 3

  jobTemplate:
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 3600
      # Keep completed pods for 48 h before collecting them:
      # enough time to investigate a weekend failure.
      ttlSecondsAfterFinished: 172800
      template:
        spec:
          restartPolicy: Never
          # Guaranteed QoS: the pod is NO LONGER a preferred eviction
          # candidate when the node is under memory pressure.
          containers:
            - name: reports
              image: registry.rutasnorte.example/occupancy-reports:1.9.2
              resources:
                requests:
                  cpu: "1"
                  memory: "1Gi"
                limits:
                  cpu: "1"
                  memory: "1Gi"
          # Avoid the node bookings-postgres lives on: the report runs
          # heavy queries and should not compete with the database for
          # the same node's memory (06-05).
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    labelSelector:
                      matchLabels:
                        app: bookings-postgres
                    topologyKey: kubernetes.io/hostname

Measure C — Alerts that warn at the time, not 30 hours later.

# We already had this one in 07-04, but we check that the for does not delay it too much
- alert: ReportsCronJobNotRun
  expr: |
    time() - max(kube_job_status_completion_time{job_name=~"occupancy-reports.*"}) > 100000
  for: 30m
  labels:
    severity: warning
    component: occupancy-reports
    team: data

# NEW: warn about the failure at the time, not about the accumulated absence
- alert: ReportsJobFailed
  expr: |
    kube_job_status_failed{namespace="rutas-norte-pro", job_name=~"occupancy-reports.*"} > 0
  for: 5m
  labels:
    severity: warning
    component: occupancy-reports
    team: data
  annotations:
    summary: "Job {{ $labels.job_name }} has failed"
    description: >
      Investigate NOW, while the pod's events and logs still
      exist. Run capture-evidence.sh before touching anything.
    runbook_url: "https://runbooks.rutasnorte.example/reports-cronjob"

# NEW: the probable root cause, warned about in advance (07-04)
- alert: NodeUnderMemoryPressure
  expr: kube_node_status_condition{condition="MemoryPressure", status="true"} == 1
  for: 5m
  labels:
    severity: warning
    team: platform
  annotations:
    summary: "Node {{ $labels.node }} is under memory pressure"
    description: >
      The kubelet is evicting pods. Those with Burstable and BestEffort QoS
      are the first candidates. Check what is running there.

With these three measures, the same incident next Sunday would produce: an alert at 04:17 (the job failing) and another at 04:10 (memory pressure on the node), the complete events in Elasticsearch, the pod's logs kept for 48 hours and the failed pod still in the cluster on Monday morning. From reconstructing with hypotheses to confirming with evidence.

Conclusion

We close module 7. We have gone from a completely opaque platform to one that can be seen, measured and explained.

In this last lesson we have built what was missing: the method.

  • A methodology of five questions in order — does it exist? is it scheduled? did it start? is it ready? is it receiving traffic? — that avoids the most expensive mistake in debugging: looking where the answer is not.
  • Events as the main and least exploited source, with --sort-by=.lastTimestamp as an essential command, and above all with the critical fact that they expire after an hour, which forces you to export them if you want to investigate something that happened last night.
  • The master table of symptom → probable cause → the command that confirms it, covering Pending, ImagePullBackOff, CrashLoopBackOff, OOMKilled and exit code 137, stuck ContainerCreating, endless Terminating, 0/n Ready and the distinction between the Ingress 502 and 503.
  • The inspection tools, with port-forward as the most valuable isolation test and kubectl debug in its three modes: ephemeral containers for debugging a distroless pod without restarting it, --copy-to for experimenting without touching production, and node/ for getting into the machine.
  • And we have finally closed the 502-during-deployments incident that we had been carrying since 02-04: the metric said when and with what pattern, the logs said the 502 was generated by the Ingress and not by the API, the exported events gave the timeline to the millisecond, and the probes from 07-01 gave the framework for understanding it. No single signal was enough.
  • With the advice that saves the most incidents: gather the evidence before restarting, because a kubectl delete pod destroys --previous, the memory state and the events; and the better alternative, isolating by changing a label to restore the service without losing the evidence.

Rutas Norte can be seen. Every component declares whether it is healthy, every unit of consumption is recorded, every request leaves a trail, every alert reaches somebody who can act and every incident can be reconstructed.

And that is where the next problem appears, uncomfortable precisely because we built it ourselves.

Everything we have set up in this module assumes that whoever accesses the cluster is somebody trustworthy. But today, in rutas-norte-pro, anybody with a configured kubectl can read the Secret with the bookings-postgres password and, with it, the name, ID number, phone and email of every Rutas Norte customer. They can deploy a privileged container that mounts the node's filesystem and reads the other pods' secrets. They can run kubectl debug node/ and take control of the machine. They can deploy an image nobody has reviewed that runs whatever it likes as root. And the audit system that would tell us who did what simply does not exist.

In module 8 we close that gap: RBAC so that every person and every ServiceAccount has exactly the permissions they need and not one more; security contexts so that no container runs as root or can escalate privileges; the Pod Security Standards so that those rules are enforced at namespace level without depending on the goodwill of whoever writes the manifest; network security, image security and, at last, auditing. Because a platform that can be seen but that anybody can compromise is not ready for production.

◄ Previous
◄
Next ►
►

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved