The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Deploying a Web Application

Next ►
►

Across ten modules we have got to know the pieces of Kubernetes one at a time: Pods, Deployments, Services, ConfigMaps, Ingress, probes, HPA, network policies, RBAC, Helm, Argo CD. Each one made sense on its own, but none of them is enough by itself to put an application into production. This module changes the approach: instead of studying one piece, we solve a complete scenario end to end.

We start with the most common and, on the face of it, simplest case: taking a stateless web application to production. At Rutas Norte S.L. that means two components, web-store (nginx serving the ticket-sales SPA) and bookings-api (the Node.js REST API that checks availability and confirms bookings). Neither stores data locally, so there are no volumes to manage and no failover to coordinate. And even so, the set of objects needed for that deployment to stand up to a production review runs to thirteen pieces per component.

This lesson is the manifest reference for the whole course. The files that appear here are the ones taken as given in the lessons that follow: when the pipeline in 11-03 updates a digest, it will be the digest of these Deployments; when 11-04 turns bookings-api into a Rollout, it will start from this manifest; when 11-06 writes the runbook for "API latency has gone through the roof", it will refer to these probes and this HPA.

Contents

  1. What must be sorted out before the first manifest
  2. The complete set of objects in a production component
  3. Complete manifests for bookings-api
  4. Complete manifests for web-store
  5. Order of application and why it matters
  6. Layer-by-layer verification, from the inside out
  7. The "production-ready" checklist
  8. The characteristic failure of each layer

  1. What must be sorted out before the first manifest

A common mistake is to treat Kubernetes as the place where application problems get fixed. It is not. Kubernetes amplifies what the application already does: if the application starts slowly, scaling will be slow; if it does not shut down cleanly, every deployment will drop requests; if it keeps sessions in local memory, the replicas will get in each other's way.

Before you write the first line of YAML, these seven points must be resolved in the code and in the image, not in the cluster.

1.1. A reproducible image identified by digest

The image has to be built the same way every time from the same commit, and deployed by digest reference, not by a moving tag.

Reference Example Fit for production?
Moving tag bookings-api:latest No. Two pods of the same Deployment can end up running different code
Semantic tag bookings-api:2.7.0 Only if the registry is immutable; a tag can be rewritten
Digest bookings-api@sha256:3f9c... Yes. It is the only cryptographically stable reference

We already saw in 08-05 how to build with multi-stage and sign with Cosign. The prerequisite here is organisational: the process that deploys must know the digest, and that is guaranteed by the pipeline we will see in 11-03.

Why it matters: without a digest, a RollingUpdate that replaces pods over twenty minutes can pick up two different artefacts if someone rewrites the tag halfway through. It is a failure that is almost impossible to diagnose afterwards.

1.2. Configuration fully externalised

No value that changes between rutas-norte-dev, rutas-norte-pre and rutas-norte-pro can live inside the image. The same image, byte for byte, has to be able to run in all three environments.

  • What changes and is not secret → ConfigMap (internal URLs, timeouts, log level).
  • What changes and is secret → Secret, populated by External Secrets Operator from the corporate store (10-05).
  • What does not change → can stay in the image.

Why it matters: if the image carries configuration inside it, what is tested in pre is not what is deployed to pro, and the whole chain of trust we built in 08-05 stops meaning anything.

1.3. Processes with no local state

Sessions, temporary files that must survive, shared in-memory counters: none of that can live in the pod. At Rutas Norte, the user session is signed with a JWT and the shopping basket is kept in redis-cache.

Why it matters: the HPA creates and destroys replicas continuously over the May bank-holiday weekend. If state lives in the pod, the user loses their basket every time autoscaling reduces the replica count.

1.4. Clean shutdown on SIGTERM

When Kubernetes removes a pod, it sends SIGTERM to the main process and waits terminationGracePeriodSeconds. The application must:

  1. Stop accepting new requests.
  2. Finish the ones already in flight.
  3. Close connections to the database and to Redis.
  4. Exit with code 0.
// Excerpt from the bookings-api startup (server.js)
const server = app.listen(8080);

let shuttingDown = false;

// /ready returns 503 as soon as shutdown begins: this way the Endpoint
// is withdrawn BEFORE we stop accepting connections.
app.get('/ready', (req, res) => {
  if (shuttingDown) return res.status(503).json({ status: 'shutting-down' });
  return res.status(200).json({ status: 'ready' });
});

process.on('SIGTERM', async () => {
  shuttingDown = true;
  // Give kube-proxy and the Ingress time to update their tables.
  await new Promise((r) => setTimeout(r, 5000));
  server.close(async () => {
    await pool.end();          // connections to bookings-postgres
    await redis.quit();
    process.exit(0);
  });
});

That five-second pause looks like a dirty trick, and in a way it is, but it answers something real: the withdrawal of the Endpoint and the sending of SIGTERM happen in parallel, not in sequence. Without the pause, the pod stops accepting connections while the load balancer is still sending them, and the user sees 502 errors on every deployment.

1.5. Structured logs to stdout

As we saw in 07-05, one JSON object per line to stdout, with no log files and no rotation of its own. Minimum fields at Rutas Norte: ts, level, message, trace_id, path, status, duration_ms.

1.6. Distinct health endpoints

Two endpoints with different semantics, not just one:

Endpoint Question it answers What it checks in bookings-api
/health Is the process alive? Only that the event loop responds
/ready Can it serve traffic right now? Connection to bookings-postgres and redis-cache, and that it is not shutting down

Why it matters: if /health checks the database, a PostgreSQL outage causes the kubelet to restart every API pod in a loop, turning a degradation into a total outage.

1.7. Exposed metrics

bookings-api exposes /metrics in Prometheus format, with at least the request counter by route and status code, and the latency histogram. That is what will feed the SLO in 11-06 and the canary analysis in 11-04.

  1. The complete set of objects in a production component

This is the table worth keeping to hand when reviewing any new deployment. Each row is an object, what it contributes, and exactly what happens if it is missing.

Object What it contributes What happens if it is missing
Namespace Isolation boundary, quotas and policies per environment Everything lands in default, with no quota and no policy; impossible to separate environments
ServiceAccount The pod's identity to the API and to the cloud The default one is used, with undetermined permissions and no traceability in the audit log
ConfigMap Non-sensitive configuration, versioned in Git Values baked into the image or into the Deployment; changing one forces a rebuild
Secret (via External Secrets) Credentials synchronised from the corporate store Secrets applied by hand in the cluster, with no rotation and no record of who put them there
Deployment Replicas, controlled updates, securityContext, probes, resources, topology spread Without it there are no managed pods: a bare pod is neither recreated nor updated
Service Stable DNS name and internal load balancing You have to discover pod IPs by hand; every restart breaks the connection
Ingress with TLS Entry from the internet with a certificate The service is not reachable from outside, or it is via an unencrypted NodePort
HPA Replica count matched to real load The May bank-holiday weekend saturates the fixed replicas, or you overpay all year round
PodDisruptionBudget A minimum number of replicas during maintenance and drains A node drain can leave the service at zero replicas
NetworkPolicy Only the authorised conversations Any compromised pod in the cluster can reach the API and the database
ServiceMonitor Prometheus discovers and scrapes the metrics The component appears neither in Grafana nor in alerts: it is blind
topologySpreadConstraints (in the Deployment) Replicas spread across zones and nodes All three replicas can end up on the same node; that node goes down and so does the service
Probes (in the Deployment) Traffic only to healthy pods and restarts for hung ones Traffic is routed to pods that are still starting; hung ones never recover

Thirteen rows. The temptation when deploying something new is to do the first five and leave the rest "for when there is time". Experience says that the ones left out are exactly the ones you need on the day of the incident.

graph TB
  subgraph ns["Namespace rutas-norte-pro"]
    subgraph seg["Security and identity"]
      SA[ServiceAccount]
      NP[NetworkPolicy]
    end
    subgraph cfg["Configuration"]
      CM[ConfigMap]
      SEC[Secret via ESO]
    end
    subgraph carga["Workload"]
      DEP[Deployment<br/>probes + resources + spread]
      HPA[HPA]
      PDB[PodDisruptionBudget]
    end
    subgraph red["Exposure"]
      SVC[Service]
      ING[Ingress + TLS]
    end
    SM[ServiceMonitor]
  end
  SA --> DEP
  CM --> DEP
  SEC --> DEP
  DEP --> SVC
  SVC --> ING
  HPA -.scales.-> DEP
  PDB -.protects.-> DEP
  NP -.filters.-> DEP
  SM -.scrapes.-> SVC

  1. Complete manifests for bookings-api

The following files live in k8s/base/bookings-api/ in the manifest repository, and the Kustomize overlays (10-04) adjust replicas, resources and domain per environment. Here they are shown already resolved for rutas-norte-pro.

3.1. ServiceAccount and configuration

apiVersion: v1
kind: ServiceAccount
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  annotations:
    # Federated identity on EKS (10-06): allows reading from the invoices bucket
    # without any long-lived access key.
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/rutasnorte-pro-bookings-api
# The pod does not need to talk to the Kubernetes API: we do not mount the token.
automountServiceAccountToken: false
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bookings-api-config
  namespace: rutas-norte-pro
data:
  NODE_ENV: "production"
  LOG_LEVEL: "info"
  # Internal DNS: short name because we are in the same namespace.
  REDIS_URL: "redis://redis-cache:6379"
  # The CloudNativePG connection pooler; we will see it in 11-02.
  PG_HOST: "bookings-postgres-pooler-rw"
  PG_PORT: "5432"
  PG_DATABASE: "bookings"
  PG_POOL_MAX: "20"
  PAYMENTS_URL: "https://pagos.proveedorexterno.example/v2"
  PAYMENTS_TIMEOUT_MS: "4000"
  BOOKING_TTL_SECONDS: "900"
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: bookings-api-secrets
  namespace: rutas-norte-pro
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-rutasnorte
    kind: ClusterSecretStore
  target:
    name: bookings-api-secrets   # name of the Secret that will be created
    creationPolicy: Owner
  data:
    - secretKey: PG_USER
      remoteRef: { key: pro/bookings-api/db, property: user }
    - secretKey: PG_PASSWORD
      remoteRef: { key: pro/bookings-api/db, property: password }
    - secretKey: JWT_SIGNING_KEY
      remoteRef: { key: pro/bookings-api/jwt, property: key }
    - secretKey: PAYMENTS_API_KEY
      remoteRef: { key: pro/bookings-api/payments, property: api_key }

Two details that are often overlooked. The first, automountServiceAccountToken: false: bookings-api does not call the Kubernetes API, so mounting the token merely adds attack surface (08-01). The second, refreshInterval: 1h: the ExternalSecret re-reads from the store every hour, so a credential rotation propagates on its own; what it does not do by itself is restart the pods, which is why further down we use a checksum in the annotations.

3.2. Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  labels:
    app.kubernetes.io/name: bookings-api
    app.kubernetes.io/part-of: rutas-norte
    app.kubernetes.io/component: backend
    rutasnorte.example/team: development
spec:
  # replicas is NOT set here: the HPA governs it and Argo CD ignores it
  # with ignoreDifferences (10-05). If we set it, every sync would
  # return the replica count to the value in the file.
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0        # we never drop below the current number of healthy replicas
  selector:
    matchLabels:
      app.kubernetes.io/name: bookings-api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: bookings-api
        app.kubernetes.io/part-of: rutas-norte
        rutasnorte.example/team: development
      annotations:
        # Changes when the ConfigMap changes: forces a rollout on configuration changes.
        rutasnorte.example/config-checksum: "sha256-8f1d2a"
    spec:
      serviceAccountName: bookings-api
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 45    # > the 5 s wait + in-flight requests
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway   # we prefer service over perfect symmetry
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: bookings-api
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: bookings-api
      containers:
        - name: api
          # Digest, not tag: written by the pipeline in 11-03.
          image: registry.rutasnorte.example/rutasnorte/bookings-api@sha256:3f9c1b7e5a04c2d8f61b93ae7c05d2419e8f6ab3c4d5e6f708192a3b4c5d6e7f
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
            - name: metrics
              containerPort: 9090
          envFrom:
            - configMapRef:
                name: bookings-api-config
            - secretRef:
                name: bookings-api-secrets
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              memory: 512Mi        # no CPU limit: avoids throttling (09-06)
          startupProbe:
            # Protects startup: up to 60 s (12 x 5 s) before the pod is given up for dead.
            httpGet: { path: /health, port: http }
            periodSeconds: 5
            failureThreshold: 12
          livenessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet: { path: /ready, port: http }
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2      # leaves the load balancer quickly
            successThreshold: 1
          lifecycle:
            preStop:
              exec:
                # Extra safety net on top of the SIGTERM handling in section 1.4.
                command: ["/bin/sh", "-c", "sleep 5"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

Notes on the less obvious decisions:

  • maxUnavailable: 0 with maxSurge: 25%: during the deployment there is never less capacity than there was before. It costs 25 % in temporary resources; in pro it is worth it.
  • No limits.cpu: as we reasoned in 09-06, a CPU limit causes throttling with latency spikes in the 95th percentile even when the node is idle. With requests set properly and a ResourceQuota on the namespace (03-04), the risk is bounded.
  • readOnlyRootFilesystem: true forces the emptyDir on /tmp. It is a five-minute annoyance that cuts off half a dozen attack techniques at the root (08-02).
  • whenUnsatisfiable: ScheduleAnyway: with DoNotSchedule, over the May bank-holiday weekend the HPA would ask for replicas that cannot be placed if a zone is full. We prefer badly spread replicas to non-existent ones.

3.3. Service, Ingress, HPA, PDB, NetworkPolicy and ServiceMonitor

apiVersion: v1
kind: Service
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  labels:
    app.kubernetes.io/name: bookings-api
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: bookings-api
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: metrics
      port: 9090
      targetPort: metrics
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/proxy-body-size: "1m"
    nginx.ingress.kubernetes.io/limit-rps: "200"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["api.rutasnorte.example"]
      secretName: api-rutasnorte-tls     # filled in by cert-manager (04-05)
  rules:
    - host: api.rutasnorte.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: bookings-api
                port:
                  name: http
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bookings-api
  minReplicas: 4
  maxReplicas: 40          # sized for the May bank-holiday weekend
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 65 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0     # scale up fast
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300   # scale down slowly
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
spec:
  minAvailable: 75%
  selector:
    matchLabels:
      app.kubernetes.io/name: bookings-api
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: bookings-api
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: ingress-nginx }
      ports:
        - { protocol: TCP, port: 8080 }
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: monitoring }
      ports:
        - { protocol: TCP, port: 9090 }
  egress:
    - to:
        - podSelector:
            matchLabels: { cnpg.io/cluster: bookings-postgres }
      ports: [{ protocol: TCP, port: 5432 }]
    - to:
        - podSelector:
            matchLabels: { app.kubernetes.io/name: redis-cache }
      ports: [{ protocol: TCP, port: 6379 }]
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except: ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
      ports: [{ protocol: TCP, port: 443 }]   # pagos.proveedorexterno.example
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: bookings-api
  namespace: rutas-norte-pro
  labels:
    release: kube-prometheus-stack     # the label Prometheus selects on
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: bookings-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

The egress rule towards 0.0.0.0/0 with private exclusions deserves a comment: as we saw in 04-06, NetworkPolicies do not understand DNS names, so you cannot write "allow egress to pagos.proveedorexterno.example". What you can do is allow 443 towards the internet excluding the private ranges, so that a compromised pod cannot use that rule to move laterally inside the corporate network.

  1. Complete manifests for web-store

web-store is simpler: nginx serving static files. It has no secrets, does not talk to the database and its probes are trivial. But it needs exactly the same discipline.

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-store-nginx
  namespace: rutas-norte-pro
data:
  default.conf: |
    server {
      listen 8080;
      root /usr/share/nginx/html;
      # SPA: any unknown path returns index.html
      location / { try_files $uri $uri/ /index.html; }
      location = /health { access_log off; return 200 "ok\n"; }
      location ~* \.(js|css|woff2|png|svg)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-store
  namespace: rutas-norte-pro
  labels:
    app.kubernetes.io/name: web-store
    app.kubernetes.io/part-of: rutas-norte
    rutasnorte.example/team: development
spec:
  revisionHistoryLimit: 5
  strategy:
    rollingUpdate: { maxSurge: 25%, maxUnavailable: 0 }
  selector:
    matchLabels: { app.kubernetes.io/name: web-store }
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web-store
        app.kubernetes.io/part-of: rutas-norte
    spec:
      serviceAccountName: web-store
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 10002
        seccompProfile: { type: RuntimeDefault }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app.kubernetes.io/name: web-store }
      containers:
        - name: nginx
          image: registry.rutasnorte.example/rutasnorte/web-store@sha256:a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90
          ports: [{ name: http, containerPort: 8080 }]
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits: { memory: 128Mi }
          readinessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /health, port: http }
            periodSeconds: 15
          lifecycle:
            preStop: { exec: { command: ["/bin/sh", "-c", "sleep 5"] } }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: conf, mountPath: /etc/nginx/conf.d }
            - { name: cache, mountPath: /var/cache/nginx }
            - { name: run, mountPath: /var/run }
      volumes:
        - name: conf
          configMap: { name: web-store-nginx }
        - { name: cache, emptyDir: {} }
        - { name: run, emptyDir: {} }
---
apiVersion: v1
kind: Service
metadata:
  name: web-store
  namespace: rutas-norte-pro
spec:
  selector: { app.kubernetes.io/name: web-store }
  ports: [{ name: http, port: 80, targetPort: http }]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-store
  namespace: rutas-norte-pro
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts: ["www.rutasnorte.example"]
      secretName: www-rutasnorte-tls
  rules:
    - host: www.rutasnorte.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: web-store, port: { name: http } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-store
  namespace: rutas-norte-pro
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web-store }
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-store
  namespace: rutas-norte-pro
spec:
  minAvailable: 2
  selector:
    matchLabels: { app.kubernetes.io/name: web-store }

Note that readOnlyRootFilesystem: true with nginx forces you to mount emptyDir on /var/cache/nginx and /var/run, and that the listen is on 8080 rather than 80: an unprivileged process cannot open ports below 1024.

  1. Order of application and why it matters

With Argo CD (10-05) the order is resolved by sync waves (argocd.argoproj.io/sync-wave), but it is worth understanding the logic, because it is the same one that applies when deploying by hand in dev or when debugging.

Wave Objects Reason
-2 Namespace, ResourceQuota, LimitRange Nothing exists outside a namespace
-1 ServiceAccount, RBAC, NetworkPolicy The identity and the rules must exist before the workload
0 ConfigMap, ExternalSecret The Deployment fails to start if the ConfigMap is not there
1 Deployment, Service The workload and its stable name
2 Ingress, HPA, PDB, ServiceMonitor They depend on the Service and Deployment existing 
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Kubernetes is eventually consistent: if you apply the Deployment before the ConfigMap, the pods go into CreateContainerConfigError and recover on their own when the ConfigMap appears. The order is not a strict technical obligation, it is a way of keeping the deployment from generating spurious alerts and minutes of confusion.

There are two exceptions where the order really is mandatory: CRDs before their custom resources (applying a ServiceMonitor without the Prometheus operator installed gives a hard error), and the NetworkPolicy before the pod in environments with deny-all, because otherwise the pod starts, its readiness probe against the database fails and it goes into unnecessary restarts.

  1. Layer-by-layer verification, from the inside out

Deployed is not the same as working. This sequence of eight checks goes from the innermost layer to the outermost, and each step only makes sense if the previous one passed. It is the routine whoever deploys should run, and the one that structures the "a deployment has gone wrong" runbook in 11-06.

graph LR
  A[1 Pod starts] --> B[2 Container ready]
  B --> C[3 Endpoints populated]
  C --> D[4 DNS resolves]
  D --> E[5 Service responds<br/>from the cluster]
  E --> F[6 Ingress responds<br/>from outside]
  F --> G[7 Certificate valid]
  G --> H[8 Metrics scraped]
  H --> I[9 Alerts silent]

Layer 1: the pod starts

kubectl -n rutas-norte-pro rollout status deploy/bookings-api --timeout=5m
kubectl -n rutas-norte-pro get pods -l app.kubernetes.io/name=bookings-api -o wide
NAME                            READY   STATUS    RESTARTS   AGE   NODE
bookings-api-7d4b8c9f5d-2xk9p   1/1     Running   0          62s   ip-10-0-2-41
bookings-api-7d4b8c9f5d-8mqrt   1/1     Running   0          58s   ip-10-0-3-17
bookings-api-7d4b8c9f5d-p4vzn   1/1     Running   0          55s   ip-10-0-1-93
bookings-api-7d4b8c9f5d-w6hxl   1/1     Running   0          51s   ip-10-0-2-88

If it fails: kubectl describe pod and look at the events. ImagePullBackOff points to the registry or the digest; CreateContainerConfigError, to a ConfigMap or Secret that does not exist; Pending, to insufficient resources or an impossible topologySpreadConstraint.

Layer 2: the container is ready

READY 1/1 already says so, but it is worth telling "started" apart from "ready":

kubectl -n rutas-norte-pro get pods -l app.kubernetes.io/name=bookings-api \
  -o custom-columns='POD:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,RESTARTS:.status.containerStatuses[0].restartCount'

If a pod is Running but 0/1, the readiness probe is failing: almost always bookings-api cannot connect to bookings-postgres or to redis-cache. You check with kubectl logs and resolve it by looking at the NetworkPolicy or the credentials.

Layer 3: the Endpoints are populated

This is the step most people skip and the one that most often explains a 503.

kubectl -n rutas-norte-pro get endpointslice -l kubernetes.io/service-name=bookings-api -o yaml | grep -A3 addresses
    addresses:
    - 10.244.2.31
      conditions:
        ready: true

Empty Endpoints with healthy pods means a misaligned selector: the labels in the Service's spec.selector do not match those in template.metadata.labels. It is a silent failure: no object reports an error, traffic simply never arrives.

Layer 4: DNS resolves

kubectl -n rutas-norte-pro run debugger --rm -it --restart=Never \
  --image=registry.rutasnorte.example/tools/netdebug:1.4 -- \
  nslookup bookings-api.rutas-norte-pro.svc.cluster.local

If it does not resolve: CoreDNS is down, or a NetworkPolicy does not allow egress on port 53 towards kube-system (the most common failure after rolling out deny-all).

Layer 5: the Service responds from inside the cluster

kubectl -n rutas-norte-pro run debugger --rm -it --restart=Never \
  --image=registry.rutasnorte.example/tools/netdebug:1.4 -- \
  curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' \
  http://bookings-api/health
200 0.004s

If the pod responds but the Service does not, suspect the targetPort or the ingress NetworkPolicy.

Layer 6: the Ingress responds from outside

curl -s -o /dev/null -w 'http=%{http_code} tls=%{ssl_verify_result} t=%{time_total}s\n' \
  https://api.rutasnorte.example/health

If nginx returns 404, check ingressClassName and the host. If it returns 503, the Ingress points to a Service with no Endpoints: you skipped layer 3.

Layer 7: the certificate is valid

kubectl -n rutas-norte-pro get certificate api-rutasnorte-tls
echo | openssl s_client -connect api.rutasnorte.example:443 -servername api.rutasnorte.example 2>/dev/null \
  | openssl x509 -noout -issuer -dates
NAME                 READY   SECRET               AGE
api-rutasnorte-tls   True    api-rutasnorte-tls   184d

issuer=C=US, O=Let's Encrypt, CN=R11
notBefore=Apr 18 09:12:44 2026 GMT
notAfter=Jul 17 09:12:43 2026 GMT

A Certificate stuck at False for more than a few minutes is usually the ACME challenge failing: look at the Order and the Challenge (04-05).

Layer 8: Prometheus scrapes the metrics

kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090 &
curl -sG 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=up{job="bookings-api"}' | jq '.data.result[] | {pod: .metric.pod, value: .value[1]}'
{"pod":"bookings-api-7d4b8c9f5d-2xk9p","value":"1"}
{"pod":"bookings-api-7d4b8c9f5d-8mqrt","value":"1"}

A missing target is almost always the ServiceMonitor's release: label, which has to match the Prometheus serviceMonitorSelector.

Layer 9: the alerts are silent

curl -s http://alertmanager.rutas-norte.example/api/v2/alerts \
  | jq '[.[] | select(.labels.service=="bookings-api")] | length'

Zero active alerts fifteen minutes after the deployment is the real criterion for "it went well". Before then, the rule windows (for: 10m) have not yet closed.

  1. The "production-ready" checklist

This table is pasted verbatim into the description of the pull request that promotes to rutas-norte-pro. Each line is ticked by a person, not by a script.

# Check How it is verified OK
1 Image by digest, signed and scanned with no criticals cosign verify + Trivy report in the pipeline ☐
2 Configuration outside the image ConfigMap diff between pre and pro ☐
3 No local state in the pod Code review; session in JWT and basket in Redis ☐
4 Clean shutdown verified Load test during a rollout restart: 0 5xx errors ☐
5 Distinct probes, including a startupProbe Manifest reviewed ☐
6 requests and limits set, with no CPU limit Manifest reviewed; VPA recommendation consulted ☐
7 securityContext non-root, no escalation, read-only root Kyverno in enforce mode guarantees it (08-03) ☐
8 topologySpreadConstraints by zone and node Manifest reviewed ☐
9 PDB consistent with the HPA's minReplicas minAvailable < minReplicas ☐
10 HPA with min and max sized for the expected peak May bank-holiday calculation documented ☐
11 Explicit ingress and egress NetworkPolicies kubectl describe netpol ☐
12 ServiceMonitor active and Grafana dashboard in place up{job=...} == 1 and a link to the dashboard ☐
13 Alerts with a runbook link in their annotations PrometheusRule rule reviewed ☐
14 Ingress with a valid TLS certificate and automatic renewal Certificate in Ready ☐
15 Layer verification 1-9 run after deploying Output pasted into the pull request ☐

Row 9 is the one that prevents the most incidents and the one most often forgotten: if the HPA can go down to 2 replicas and the PDB demands minAvailable: 3, a node drain blocks indefinitely and cluster maintenance is left hanging.

  1. The characteristic failure of each layer

Each layer fails in a recognisable way. Knowing the characteristic symptom saves half the diagnosis time; the in-depth procedure is in 07-06.

Layer Symptom Most likely cause First command
Pod ImagePullBackOff Non-existent digest or registry credential kubectl describe pod
Pod Prolonged Pending No resources, or topologySpread with DoNotSchedule kubectl describe pod (scheduler events)
Pod CrashLoopBackOff Fails on startup, or a livenessProbe with too tight a threshold kubectl logs --previous
Container Running but 0/1 Dependency unreachable from /ready kubectl logs + test the dependency
Endpoints Empty list with healthy pods Misaligned Service selector kubectl get endpointslice
DNS NXDOMAIN from a pod CoreDNS, or egress on 53 blocked nslookup from a debug pod
Service Connects but times out Wrong targetPort or ingress NetworkPolicy curl the Service from inside the cluster
Ingress nginx 404 Wrong ingressClassName or host kubectl describe ing
Ingress 503 Service with no ready Endpoints go back to layer 3
TLS Browser warning Certificate not Ready, ACME challenge failed kubectl describe certificate
Metrics Target missing in Prometheus The ServiceMonitor's release: label up{job="..."}
Alerts An alert that will not clear Badly written rule or unrealistic threshold Run the expression in Prometheus

Common Mistakes and Tips

  • Deploying without preStop or SIGTERM handling and blaming the Ingress for the 502s. It is the most frequent mistake when moving from dev to production. The definitive test: run a sustained load with k6 and issue kubectl rollout restart halfway through. If a single 5xx appears, the clean shutdown is not right.
  • Putting the livenessProbe on an endpoint that checks dependencies. It turns a database degradation into a mass restart of the entire API. The liveness probe must only look inwards.
  • Setting replicas in the Deployment when you have an HPA. Every Argo CD sync returns the replica count to the value in the file, undoing the scaling. It is fixed with ignoreDifferences (10-05) and removing the field.
  • A PDB with minAvailable equal to the replica count. It blocks every drain. Use percentages and check consistency with the HPA's minReplicas.
  • Forgetting the release: label on the ServiceMonitor. The component deploys perfectly and stays invisible to monitoring. Nobody notices until the first incident.
  • Tip: automate the checklist wherever you can, but keep the human review. Kyverno can enforce rows 6, 7 and 11; nobody can enforce row 10 by policy, because it requires having thought about the May bank-holiday weekend.
  • Tip: keep the layer-verification output in the pull request. When something fails three weeks later, knowing that on deployment day the certificate was valid and the metrics were being scraped narrows the search enormously.

Exercises

Exercise 1: spotting missing objects

A colleague has deployed a new component, promotions-web, in rutas-norte-pro with these objects: Deployment (3 fixed replicas, no probes), Service and Ingress. List which objects are missing from the thirteen in the table in section 2 and describe, for each one, the concrete scenario in which its absence will cause an incident at Rutas Norte.

Exercise 2: layer verification of a real failure

After deploying bookings-api in rutas-norte-pre, this is the situation:

$ kubectl -n rutas-norte-pre get pods -l app.kubernetes.io/name=bookings-api
NAME                            READY   STATUS    RESTARTS   AGE
bookings-api-6c9f7d8b4c-hh2ks   0/1     Running   0          4m
bookings-api-6c9f7d8b4c-tq5rl   0/1     Running   0          4m

$ curl -s -o /dev/null -w '%{http_code}\n' https://api-pre.rutasnorte.example/health
503

State which of the nine layers the failure is in, which three commands you would run and in what order, and what the two most likely causes are.

Exercise 3: consistency between HPA and PDB

notifications-worker has an HPA governed by KEDA with minReplicaCount: 1 and maxReplicaCount: 20, and a PDB with minAvailable: 2. The platform team needs to drain a node in order to upgrade the cluster and kubectl drain hangs. Explain why and propose the fix, justifying the value you choose.

Solutions

Solution 1. Ten objects are missing:

Missing Incident scenario
Its own ServiceAccount Uses the default one; the audit log cannot attribute actions and it inherits unintended permissions
ConfigMap Configuration goes in the image: promoting from pre to pro requires a rebuild, breaking digest traceability
Secret via ESO Credentials applied by hand, with no rotation
Probes Traffic is routed to pods that are still starting: 502s on every deployment
securityContext Kyverno in enforce mode (08-03) will reject the pod; if it is in audit mode, it runs as root
topologySpreadConstraints All 3 replicas can land on the same node; that node is lost and so is the service
HPA The 3 fixed replicas saturate over the May bank-holiday weekend
PDB A drain can take all 3 replicas at once
NetworkPolicy With deny-all in the namespace it will not even start; without it, it is exposed to lateral movement
ServiceMonitor No metrics and no alerts: the component is invisible during the first incident

Solution 2. The failure is in layer 2 (container not ready), and the 503 in layer 6 is a consequence: with no ready pods, there are no Endpoints. Commands, in order:

kubectl -n rutas-norte-pre logs -l app.kubernetes.io/name=bookings-api --tail=50
kubectl -n rutas-norte-pre describe pod -l app.kubernetes.io/name=bookings-api | grep -A10 Events
kubectl -n rutas-norte-pre get endpointslice -l kubernetes.io/service-name=bookings-api

Most likely causes: (a) /ready fails because there is no connection to bookings-postgres (the ExternalSecret credential not synchronised in pre, or a NetworkPolicy missing the egress rule to 5432); (b) the application starts but takes longer than the startupProbe allows, although in that case you would normally see restarts, which here are 0, which reinforces hypothesis (a).

Solution 3. With minReplicaCount: 1, KEDA can leave the worker at a single replica during quiet hours. The PDB requires 2 to remain available, a condition that is impossible with 1 replica in total, so drain cannot evict the pod and waits indefinitely. Fix: use minAvailable: 1 or, better, maxUnavailable: 1, which is expressed in terms of how many replicas can be lost and works correctly across the whole range from 1 to 20. A complementary alternative: raise minReplicaCount to 2 if the business requires notifications never to be left without capacity, accepting the cost of the permanent replica.

Conclusion

We have walked, from start to finish, the path of a stateless web application into production: the seven requirements the application must meet before you touch the YAML, the thirteen objects that make up a complete component and what you lose with each one that is missing, the full manifests for bookings-api and web-store that serve as the reference for the rest of the course, the order of application, the layer-by-layer verification from the inside out with the exact command for each step, the checklist that can be reviewed in a pull request and the characteristic symptom of each layer's failure.

The key lesson is that "it is deployed" and "it is production-ready" are very different statements, and that the difference between them is precisely the objects and the checks that it is tempting to leave for later.

All of this has been relatively comfortable for one reason: web-store and bookings-api store nothing. They can be killed, moved and multiplied without consequences. In the next lesson, Running Stateful Applications, we move into the territory where that stops being true: bookings-postgres holds the personal data of Rutas Norte's customers, you cannot replace a pod light-heartedly, and the first question to answer honestly is whether that database should live in Kubernetes at all.

◄ Previous
◄
Next ►
►

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved