The Project | About Us | Contribute | Donations | License

HOME

Everything we have deployed so far shares one premise: the process must never end. web-store serves pages indefinitely, bookings-api handles requests indefinitely, the log collector from the previous lesson reads files indefinitely. If any of those containers exits, even with code 0, Kubernetes treats it as an anomaly and restarts it.

But an important part of a platform's real work consists precisely of finishing: generating last night's occupancy report, migrating the database schema before a deployment, reprocessing a batch of tickets with a badly calculated price. For those workloads Kubernetes has two objects: the Job, which runs something until it completes successfully, and the CronJob, which creates Jobs on a schedule.

With them we will at last deploy the final outstanding component of Rutas Norte: occupancy-reports, the nightly task that queries bookings-postgres and leaves a report on disk.

Workloads that finish versus workloads that do not
The Job object: anatomy and control fields
restartPolicy: Never versus OnFailure
The three Job patterns
completionMode: Indexed and deterministic work sharing
The CronJob object: schedule and concurrency policy
The central case: occupancy-reports as a nightly CronJob
A second example: a schema migration Job
Debugging failed jobs and cleaning up history

Workloads that finish versus workloads that do not

The difference starts in the container itself. A long-running process blocks in an event loop; a batch process does its work, writes a result and calls exit(0).

Kubernetes distinguishes the two cases by the controller managing the pod, not by the contents of the image.

Aspect	Deployment / StatefulSet / DaemonSet	Job / CronJob
Expected duration	Indefinite	Finite
Exit with code 0	Anomaly; it is restarted	Success; the Job is marked complete
Allowed `restartPolicy`	Only `Always`	Only `Never` or `OnFailure`
Terminal state	Does not exist	`Complete` or `Failed`
What measures success	Staying alive	Having finished successfully N times
Collecting results	Continuous logs	Logs of the finished pod, until they are cleaned up

A detail that often surprises people: the pod of a finished Job does not disappear. It stays in Completed state, consuming neither CPU nor memory, so that you can read its logs. That is also why a badly maintained cluster accumulates thousands of Completed pods, something we will solve in section 9.

graph LR
  subgraph Indefinite
    D[Deployment] --> P1[Pod always Running]
    P1 -->|exits| P1
  end
  subgraph Finite
    C[CronJob] -->|every night| J[Job]
    J --> P2[Pod]
    P2 -->|exit 0| OK[Completed]
    P2 -->|exit != 0| RT[Retry]
    RT --> P2
    RT -->|backoffLimit exhausted| KO[Failed]
  end

The Job object: anatomy and control fields

The simplest possible Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: fare-calculation
  namespace: rutas-norte-dev
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: calculation
          image: busybox:1.36
          command: ["sh", "-c", "echo 'Recalculating fares...'; sleep 5; echo 'Done'"]

Note that there is no selector: the Job generates its own automatically with a unique label (batch.kubernetes.io/controller-uid). It is one of the few workloads where you do not have to declare it, and it is better not to try: a hand-written selector that collides with another Job makes one controller adopt someone else's pods.

The fields that govern the behaviour:

Field	Default	What it does
`completions`	1	How many pods must finish successfully for the Job to count as complete
`parallelism`	1	How many pods may run at the same time
`backoffLimit`	6	How many failures are tolerated before marking the Job as `Failed`
`activeDeadlineSeconds`	no limit	Maximum seconds from the start; on exceeding them the Job is cut short even if it has not failed
`ttlSecondsAfterFinished`	no limit	Seconds after finishing (successfully or not) before the Job and its pods delete themselves
`completionMode`	`NonIndexed`	`NonIndexed` or `Indexed` (section 5)
`suspend`	`false`	If `true`, the Job creates no pods; useful for queueing and releasing later
`podFailurePolicy`	none	Fine-grained rules by exit code (stable since 1.31)

`backoffLimit` and exponential backoff

When a pod fails, the Job creates another, but not immediately: it waits 10 s, then 20 s, 40 s, 80 s… up to a maximum of 6 minutes. That growing wait stops a permanent failure — a wrong password, an unreachable host — from consuming the cluster with thousands of attempts a minute.

On reaching backoffLimit failures, the Job moves to Failed with the reason BackoffLimitExceeded and stops creating pods.

Practical values for Rutas Norte:

Idempotent tasks with network dependencies (querying the payment gateway): backoffLimit: 6, the default, makes sense; a transient failure recovers on its own.
Schema migrations: backoffLimit: 0. If it fails, you want to know and take a look, not to have it retried five times over a half-migrated database.

`activeDeadlineSeconds`

It is a time-based circuit breaker. It counts from the moment the Job starts and includes the retries:

spec:
  backoffLimit: 3
  activeDeadlineSeconds: 1800   # 30 minutes at most, no matter what

If it runs out, the active pods are terminated and the Job is left Failed with reason DeadlineExceeded. It is the protection against the process that does not fail but does not make progress either: a query blocked by a lock in bookings-postgres, for example. activeDeadlineSeconds takes precedence over backoffLimit: whichever is met first wins.

`ttlSecondsAfterFinished`

spec:
  ttlSecondsAfterFinished: 86400   # self-destructs 24 h after finishing

Once that time has passed since the Job reached a terminal state, the TTL controller deletes the Job and its pods. It is the right way to avoid an accumulation of objects. Always set it on Jobs created by automation; leave enough time for someone to read the logs of an overnight failure (24 hours is a good value).

`restartPolicy`: `Never` versus `OnFailure`

The API only accepts these two values in a Job's template. Always is rejected, because it contradicts the very idea of a task that finishes.

The difference between the two is not cosmetic: it changes what gets restarted.

	`restartPolicy: Never`	`restartPolicy: OnFailure`
What happens when the container fails	The pod is left `Failed`; the Job creates a new pod	The kubelet restarts the container inside the same pod
Counter that goes up	The Job's `backoffLimit`	The container's `restartCount` (and `backoffLimit` too)
Trail left behind	One pod per attempt, all of them inspectable	A single pod; the logs of previous attempts are lost except with `--previous`
Node for the retry	May be another one	Always the same one
`emptyDir` volume	Empty on each attempt	Preserved across restarts

Example of what you see with Never:

kubectl get pods -n rutas-norte-dev -l job-name=schema-migration

NAME                     READY   STATUS   RESTARTS   AGE
schema-migration-4kx2p   0/1     Error    0          3m
schema-migration-9dz7w   0/1     Error    0          2m
schema-migration-t6m1c   0/1     Error    0          1m

Three pods, one per attempt, each with its complete logs. With OnFailure you would see:

NAME                     READY   STATUS             RESTARTS      AGE
schema-migration-4kx2p   0/1     CrashLoopBackOff   3 (45s ago)   3m

A single pod with RESTARTS: 3, and to see the log of the previous attempt you would need kubectl logs <pod> --previous.

Recommendation for Rutas Norte: use Never unless there is a specific reason not to. Keeping one pod per attempt makes debugging far more honest, and in a nightly task that failed at 03:00 you will want to be able to read exactly what the first attempt said. OnFailure is preferable when starting the container is very expensive — downloading a model, restoring a cache — and you want to reuse the state of an emptyDir between retries.

An important nuance: if the whole node fails, the pod is recreated in both cases, because the one acting is the Job controller, not the kubelet.

The three Job patterns

With completions and parallelism you build three patterns that cover almost all batch work.

Pattern	`completions`	`parallelism`	When
Single task	1 (or absent)	1 (or absent)	Migration, report, one-off task
Fixed parallelism	N	M (≤ N)	N units of work known in advance
Work queue	absent	M	The workers consume from an external queue until it is empty

Pattern 1: single task

The most common one. One pod, once, successfully.

apiVersion: batch/v1
kind: Job
metadata:
  name: one-off-report-july
  namespace: rutas-norte-dev
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: dev
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 900
  ttlSecondsAfterFinished: 86400
  template:
    metadata:
      labels:
        app: occupancy-reports
        environment: dev
    spec:
      restartPolicy: Never
      containers:
        - name: generator
          image: postgres:16.4
          command: ["sh", "-c", "psql -h bookings-postgres -U rutasnorte -d bookings -c 'SELECT count(*) FROM bookings'"]

Pattern 2: fixed parallelism with a number of completions

Rutas Norte has 12 bus routes and wants to recalculate the historical occupancy of all of them. That is 12 units of work, and it does not want more than 4 simultaneous queries against bookings-postgres:

apiVersion: batch/v1
kind: Job
metadata:
  name: route-occupancy-recalc
  namespace: rutas-norte-dev
spec:
  completions: 12       # 12 pods must finish successfully
  parallelism: 4        # at most 4 at a time
  backoffLimit: 6
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: recalc
          image: busybox:1.36
          command: ["sh", "-c", "echo 'Processing one route'; sleep 10"]

The Job keeps four pods alive at a time: as soon as one finishes, the next starts, until 12 successes have accumulated. parallelism is the load-control mechanism: without it, twelve simultaneous queries could saturate the database.

Watching it progress:

kubectl get job route-occupancy-recalc -n rutas-norte-dev -w

NAME                      COMPLETIONS   DURATION   AGE
route-occupancy-recalc    0/12          3s         3s
route-occupancy-recalc    4/12          15s        15s
route-occupancy-recalc    8/12          27s        27s
route-occupancy-recalc    12/12         39s        39s

The problem with this pattern in its basic form: every pod runs the same command. There is nothing telling each one which route it should handle. That is what section 5 solves.

Pattern 3: work queue

If you omit completions but set parallelism, the Job goes into queue mode: it starts M workers and considers the work finished when any one of them exits successfully with the rest already finished. The workers must coordinate among themselves through an external queue (Redis, RabbitMQ, a table with SELECT ... FOR UPDATE SKIP LOCKED).

apiVersion: batch/v1
kind: Job
metadata:
  name: ticket-reprocess-queue
  namespace: rutas-norte-dev
spec:
  parallelism: 5        # no completions: work-queue mode
  backoffLimit: 10
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: registry.rutasnorte.example/tickets-worker:1.4.2
          env:
            - name: QUEUE_URL
              value: "redis://redis-cache.rutas-norte-dev.svc.cluster.local:6379/3"
          command: ["/app/consume-queue"]

Each worker takes a ticket from the redis-cache queue, reprocesses it and repeats until the queue is empty, at which point it exits with code 0. It is the most flexible pattern and the one that copes worst with design mistakes: if a worker dies halfway through a unit, that unit must go back to the queue, and the application has to guarantee that.

`completionMode: Indexed` and deterministic work sharing

Indexed mode solves the shortcoming of pattern 2: giving each pod a number that tells it which portion of the work is its own.

apiVersion: batch/v1
kind: Job
metadata:
  name: route-occupancy-recalc
  namespace: rutas-norte-dev
spec:
  completionMode: Indexed     # <- the key
  completions: 12
  parallelism: 4
  backoffLimit: 6
  ttlSecondsAfterFinished: 86400
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: recalc
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ROUTE=$(( JOB_COMPLETION_INDEX + 1 ))
              echo "Pod with index $JOB_COMPLETION_INDEX -> recalculating Rutas Norte route $ROUTE"
              sleep 5
              echo "Route $ROUTE recalculated"

With Indexed, Kubernetes assigns each pod a unique index from 0 to completions - 1, and exposes it in two ways:

The environment variable JOB_COMPLETION_INDEX.
The pod annotation batch.kubernetes.io/job-completion-index.

In addition, the pod names include the index, which makes observation trivial:

kubectl get pods -n rutas-norte-dev -l job-name=route-occupancy-recalc \
  --sort-by=.metadata.name

NAME                            READY   STATUS      RESTARTS   AGE
route-occupancy-recalc-0-h2k9x  0/1     Completed   0          62s
route-occupancy-recalc-1-m4p2t  0/1     Completed   0          62s
route-occupancy-recalc-2-w8j5r  0/1     Completed   0          61s
...
route-occupancy-recalc-11-z3q7n 0/1     Completed   0          18s

kubectl logs -n rutas-norte-dev route-occupancy-recalc-7-x1v4b

Pod with index 7 -> recalculating Rutas Norte route 8
Route 8 recalculated

Properties that make Indexed far superior to the default mode when work has to be shared out:

Determinism: index 7 always processes route 8, whenever it runs.
Correct retries: if the pod for index 7 fails, the retry gets index 7 again, not another one. No unit is left unprocessed or processed twice.
Traceability: the pod name tells you which unit of work that pod looked at.

If you need to split a range, the index is enough to work it out:

TOTAL=12000
PER_POD=$(( TOTAL / COMPLETIONS ))
FROM=$(( JOB_COMPLETION_INDEX * PER_POD ))
TO=$(( FROM + PER_POD ))
echo "Processing bookings from $FROM to $TO"

The CronJob object: schedule and concurrency policy

A CronJob does not run anything itself: it creates Jobs on a schedule. And each Job creates its pods. There are three levels, and understanding them avoids a lot of confusion when debugging.

graph LR
  CJ[CronJob<br/>occupancy-reports] -->|03:15 on day 1| J1[Job occupancy-reports-28935120]
  CJ -->|03:15 on day 2| J2[Job occupancy-reports-28936560]
  J1 --> P1[Pod ...-28935120-4kx2p]
  J2 --> P2[Pod ...-28936560-9dz7w]

The `schedule` syntax

Five space-separated fields, in the usual cron format:

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of week (0 - 6, Sunday = 0)
│ │ │ │ │
* * * * *

Operators allowed in each field:

Operator	Meaning	Example
`*`	Any value	`* * * * *` = every minute
`,`	List	`0 8,14,20 * * *` = at 8, 14 and 20
`-`	Range	`0 9-17 * * 1-5` = every hour from 9 to 17, Monday to Friday
`/`	Step	`/15 * * *` = every 15 minutes

Examples that mean something for Rutas Norte:

Expression	When it runs
`15 3 * * *`	Every day at 03:15
`0 * * * *`	On the hour, every hour
`/10 * * *`	Every 10 minutes
`0 4 * * 1`	Mondays at 04:00
`0 5 1 * *`	The 1st of every month at 05:00
`30 2 * * 0`	Sundays at 02:30

`timeZone`

Without this field, the schedule is interpreted in the time zone of the kube-controller-manager, which on most clusters is UTC. For Rutas Norte, which operates in Spain, that means 15 3 * * * would run at 04:15 in summer time and at 03:15 in winter: a different hour depending on the time of year, exactly what you do not want in an overnight window.

spec:
  schedule: "15 3 * * *"
  timeZone: "Europe/Madrid"

The timeZone field is stable since Kubernetes 1.27 and accepts any IANA database identifier. Always use it; it is one of the most useful and least known improvements to the batch API.

`concurrencyPolicy`

What happens if the time for the next run arrives and the previous one has not finished?

Value	Behaviour	When to use it
`Allow` (default)	The new Job is created; they coexist	Independent, lightweight tasks
`Forbid`	The new run is skipped; the miss is recorded	Tasks that must not overlap
`Replace`	The previous one is cancelled and the new one starts	Tasks where only the most recent result matters

The real case that justifies Forbid in Rutas Norte: occupancy-reports runs every night and makes heavy aggregate queries over bookings-postgres. If on a bank-holiday night the booking volume is so large that the report takes 26 hours, with Allow the following night there would be two reports aggregating simultaneously over the same database: twice the load at the worst possible moment, and two files writing to the same PVC. With Forbid, the second run simply is not launched and the event is recorded so that someone can investigate why the report is taking so long.

Replace would be right for, say, a recalculation of seat availability every five minutes: if the 10:00 calculation is still running at 10:05, its result is already stale and it is better to cancel it and start again with fresh data.

`startingDeadlineSeconds` and the cluster being down

Suppose the control plane was down between 03:00 and 05:00 for maintenance, and the CronJob was due to fire at 03:15. What happens when it comes back?

Without startingDeadlineSeconds: the controller sees a missed run and launches it as soon as it can, at 05:00. A report "for 03:15" running at 05:00 may be harmless or disastrous depending on what it does.
With startingDeadlineSeconds: 600: the run is only launched if fewer than 600 seconds have passed since the scheduled time. At 05:00 6,300 s have passed, so it is skipped and the miss is recorded.

spec:
  schedule: "15 3 * * *"
  timeZone: "Europe/Madrid"
  startingDeadlineSeconds: 600

There is also a protection mechanism worth knowing about: if the controller detects more than 100 missed runs within the deadline window, it stops scheduling the CronJob altogether and emits this event:

Warning  FailedNeedsStart  cronjob-controller  Cannot determine if job needs to be started:
too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check
clock skew

It is why a CronJob that has not run for months can end up permanently dead. Setting startingDeadlineSeconds to a reasonable value avoids it.

History and suspension

spec:
  successfulJobsHistoryLimit: 3    # default 3
  failedJobsHistoryLimit: 5        # default 1
  suspend: false

The controller keeps the N most recent Jobs of each kind and deletes the rest along with their pods. Practical advice: raise failedJobsHistoryLimit. The default of 1 means that if the report fails three nights in a row, you only keep the logs of the last one, exactly when you would want to compare all three.

suspend: true stops the scheduling without deleting the object. It is the right manoeuvre during a maintenance window on bookings-postgres:

kubectl patch cronjob occupancy-reports -n rutas-norte-pro -p '{"spec":{"suspend":true}}'
# ... maintenance ...
kubectl patch cronjob occupancy-reports -n rutas-norte-pro -p '{"spec":{"suspend":false}}'

Careful: while it is suspended, missed runs are not recovered on resuming; the controller picks up from the next scheduled time.

The central case: `occupancy-reports` as a nightly CronJob

It is time to deploy the component we have been mentioning for six modules and never creating. Every night occupancy-reports queries the previous day's bookings in bookings-postgres, calculates the occupancy per route and leaves a CSV file on a PVC.

The report PVC

Reports accumulate and are kept for a while. We use the standard class, not the fast one: there are no latency requirements.

# k8s/base/occupancy-reports-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: occupancy-reports-data
  namespace: rutas-norte-pro
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: rutasnorte-standard
  resources:
    requests:
      storage: 5Gi

The ServiceAccount

Following the practice of 03-06: a dedicated account with no token mounted, because the job does not talk to the Kubernetes API.

# k8s/base/occupancy-reports-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: occupancy-reports
  namespace: rutas-norte-pro
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
automountServiceAccountToken: false

The NetworkPolicy

In 04-06 we left rutas-norte-pro with a default deny-all and authorised conversations one at a time. occupancy-reports is a new conversation and without this policy it will not get through the wall: the pods will start, the query will hang and the Job will fail on activeDeadlineSeconds with no message explaining the cause. It is one of the hardest failures to diagnose in the whole course.

Two rules are needed: egress from the report and ingress into the database.

# k8s/environments/pro/occupancy-reports-networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: occupancy-reports-egress-postgres
  namespace: rutas-norte-pro
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  podSelector:
    matchLabels:
      app: occupancy-reports
      environment: pro
  policyTypes: ["Egress"]
  egress:
    # DNS resolution: without this the service name is not resolved
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    # Access to the database
    - to:
        - podSelector:
            matchLabels:
              app: bookings-postgres
              environment: pro
      ports:
        - protocol: TCP
          port: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bookings-postgres-ingress-reports
  namespace: rutas-norte-pro
  labels:
    app: bookings-postgres
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  podSelector:
    matchLabels:
      app: bookings-postgres
      environment: pro
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: occupancy-reports
              environment: pro
      ports:
        - protocol: TCP
          port: 5432

The DNS rule is the one most often forgotten. With deny-all, a pod without egress permission to CoreDNS's port 53 does not resolve bookings-postgres and fails with an unknown-host error that looks like an application configuration problem.

The CronJob

# k8s/base/occupancy-reports-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: occupancy-reports
  namespace: rutas-norte-pro
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: pro
spec:
  schedule: "15 3 * * *"
  timeZone: "Europe/Madrid"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 600
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  suspend: false
  jobTemplate:
    metadata:
      labels:
        app: occupancy-reports
        app.kubernetes.io/part-of: rutas-norte
        environment: pro
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 5400      # 90 minutes as an absolute ceiling
      ttlSecondsAfterFinished: 172800  # self-cleans after 48 h
      template:
        metadata:
          labels:
            app: occupancy-reports
            app.kubernetes.io/part-of: rutas-norte
            environment: pro
        spec:
          restartPolicy: Never
          serviceAccountName: occupancy-reports
          automountServiceAccountToken: false
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            fsGroup: 10001
          containers:
            - name: generator
              image: postgres:16.4
              command:
                - /bin/bash
                - -c
                - |
                  set -euo pipefail
                  YESTERDAY=$(date -d 'yesterday' +%Y-%m-%d)
                  OUTPUT="/reports/occupancy-${YESTERDAY}.csv"
                  echo "Generating occupancy report for ${YESTERDAY}"
                  psql -h bookings-postgres -U "${PGUSER}" -d bookings \
                       --csv --no-psqlrc -o "${OUTPUT}" <<SQL
                  SELECT r.code             AS route,
                         COUNT(b.id)        AS tickets,
                         SUM(b.seats)       AS seats_taken,
                         ROUND(100.0 * SUM(b.seats) / NULLIF(SUM(d.capacity), 0), 2) AS pct_occupancy
                  FROM bookings b
                  JOIN departures d ON d.id = b.departure_id
                  JOIN routes     r ON r.id = d.route_id
                  WHERE d.date = DATE '${YESTERDAY}'
                  GROUP BY r.code
                  ORDER BY pct_occupancy DESC;
                  SQL
                  echo "Report written to ${OUTPUT} ($(wc -l < "${OUTPUT}") lines)"
                  find /reports -name 'occupancy-*.csv' -mtime +90 -delete
                  echo "Reports older than 90 days deleted"
              env:
                - name: PGUSER
                  valueFrom:
                    secretKeyRef:
                      name: bookings-postgres-credentials
                      key: username
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: bookings-postgres-credentials
                      key: password
                - name: PGCONNECT_TIMEOUT
                  value: "10"
              resources:
                requests:
                  cpu: 200m
                  memory: 256Mi
                limits:
                  cpu: "1"
                  memory: 512Mi
              volumeMounts:
                - name: reports
                  mountPath: /reports
          volumes:
            - name: reports
              persistentVolumeClaim:
                claimName: occupancy-reports-data

A review of the decisions, which sum up half a dozen previous lessons:

schedule at 03:15 with timeZone: Europe/Madrid: outside selling hours and at a stable time all year round.
concurrencyPolicy: Forbid: two simultaneous reports over the same database and the same PVC would be harmful.
activeDeadlineSeconds: 5400: if the query gets blocked by a lock, the Job dies after 90 minutes instead of dragging on until morning.
backoffLimit: 2: one retry is useful against a network glitch; five only prolong the load.
ttlSecondsAfterFinished: 172800 plus the history limits: two layers of automatic clean-up.
Credentials through a Secret (03-02), never in the manifest. PGPASSWORD and PGUSER are variables psql recognises directly.
Declared resources: without them the pod would be BestEffort (03-05) and the first node under memory pressure would evict it.
securityContext with a non-root user and fsGroup so it can write to the PVC.
90-day retention applied in the script itself: the PVC is finite and the reports contain aggregated data, not personal data, but accumulating without limit is a guaranteed space leak.

Deploy and test without waiting until three in the morning

kubectl apply -f k8s/base/occupancy-reports-pvc.yaml
kubectl apply -f k8s/base/occupancy-reports-sa.yaml
kubectl apply -f k8s/environments/pro/occupancy-reports-networkpolicy.yaml
kubectl apply -f k8s/base/occupancy-reports-cronjob.yaml

kubectl get cronjob occupancy-reports -n rutas-norte-pro

NAME                SCHEDULE     TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
occupancy-reports   15 3 * * *   Europe/Madrid   False     0        <none>          20s

The indispensable command: firing a manual run from the CronJob.

kubectl create job -n rutas-norte-pro \
  --from=cronjob/occupancy-reports reports-manual-test

job.batch/reports-manual-test created

kubectl wait --for=condition=complete job/reports-manual-test \
  -n rutas-norte-pro --timeout=600s
kubectl logs -n rutas-norte-pro job/reports-manual-test

Generating occupancy report for 2026-08-04
Report written to /reports/occupancy-2026-08-04.csv (13 lines)
Reports older than 90 days deleted

kubectl create job --from=cronjob/... copies the jobTemplate as it is, so it tests exactly what will run in the small hours, including the ServiceAccount, the NetworkPolicy and the volumes. It is the right way to validate a new CronJob.

A second example: a schema migration Job

Before deploying version 2.5.0 of bookings-api a column has to be added to the bookings table. It is a one-off Job with very different requirements from the report.

# k8s/environments/pre/schema-migration-2-5-0.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: schema-migration-2-5-0
  namespace: rutas-norte-pre
  labels:
    app: bookings-api
    app.kubernetes.io/part-of: rutas-norte
    environment: pre
spec:
  backoffLimit: 0                # a migration is NOT retried blindly
  activeDeadlineSeconds: 600
  ttlSecondsAfterFinished: 604800  # 7 days: we want a trail of the migrations
  template:
    metadata:
      labels:
        app: bookings-api
        environment: pre
    spec:
      restartPolicy: Never
      serviceAccountName: bookings-api
      automountServiceAccountToken: false
      containers:
        - name: migration
          image: registry.rutasnorte.example/bookings-api-migrations:2.5.0
          command:
            - /bin/sh
            - -c
            - |
              set -euo pipefail
              echo "Migration 2.5.0 on bookings"
              psql -h bookings-postgres -U "${PGUSER}" -d bookings -v ON_ERROR_STOP=1 <<'SQL'
              BEGIN;
              ALTER TABLE bookings
                ADD COLUMN IF NOT EXISTS sales_channel text NOT NULL DEFAULT 'web';
              CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_bookings_channel
                ON bookings (sales_channel);
              INSERT INTO migrations (version, applied_at)
                VALUES ('2.5.0', now())
                ON CONFLICT (version) DO NOTHING;
              COMMIT;
              SQL
              echo "Migration 2.5.0 applied"
          env:
            - name: PGUSER
              valueFrom:
                secretKeyRef:
                  name: bookings-postgres-credentials
                  key: username
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: bookings-postgres-credentials
                  key: password
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

Deliberate contrasts with occupancy-reports:

Decision	Nightly report	Schema migration	Reason
`backoffLimit`	2	0	Retrying over a half-migrated database can make the state worse
`ttlSecondsAfterFinished`	48 h	7 days	The trail of what migrated and when has audit value
Idempotency	Not applicable	`IF NOT EXISTS`, `ON CONFLICT DO NOTHING`	If somebody relaunches the Job, it must not break anything
Transaction	No	`BEGIN`/`COMMIT` with `ON_ERROR_STOP=1`	Either everything is applied or nothing is

ON_ERROR_STOP=1 is essential: without it, psql carries on after an error and returns code 0, and the Job would show as complete with the migration half done. It is a classic trap.

A design warning: this Job decouples the migration from the deployment. You have to run it, verify it and only then deploy version 2.5.0 of bookings-api. The alternative — chaining the migration to the start-up of every application pod through an init container — is the subject of lesson 06-04, and it has its own advantages and drawbacks.

Debugging failed jobs and cleaning up history

When a nightly job fails, the diagnosis always follows the same path: CronJob → Job → Pod → logs.

Step 1: the state of the Jobs

kubectl get jobs -n rutas-norte-pro

NAME                           STATUS     COMPLETIONS   DURATION   AGE
occupancy-reports-29175300     Complete   1/1           94s        2d
occupancy-reports-29176740     Complete   1/1           88s        1d
occupancy-reports-29178180     Failed     0/1           7m12s      3h

The numeric suffix is the scheduled timestamp, not a random identifier: the Jobs sort themselves chronologically.

Step 2: why the Job failed

kubectl describe job occupancy-reports-29178180 -n rutas-norte-pro

Name:           occupancy-reports-29178180
Parallelism:    1
Completions:    1
Pods Statuses:  0 Running / 0 Succeeded / 3 Failed
Conditions:
  Type     Status  Reason                Message
  ----     ------  ------                -------
  Failed   True    BackoffLimitExceeded  Job has reached the specified backoff limit
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  --------------  -------
  Normal   SuccessfulCreate      3h    job-controller  Created pod: occupancy-reports-29178180-4kx2p
  Normal   SuccessfulCreate      3h    job-controller  Created pod: occupancy-reports-29178180-9dz7w
  Normal   SuccessfulCreate      3h    job-controller  Created pod: occupancy-reports-29178180-t6m1c
  Warning  BackoffLimitExceeded  3h    job-controller  Job has reached the specified backoff limit

The most frequent reasons and what they mean:

Reason	Meaning	Where to look
`BackoffLimitExceeded`	The pods failed more times than allowed	The pods' logs, especially the first one
`DeadlineExceeded`	`activeDeadlineSeconds` ran out	A lock? A slow query? The network cut off?
`FailedCreate`	The pod could not be created	ResourceQuota exhausted, PVC missing, SA without permissions

Step 3: the right pod and its logs

With restartPolicy: Never there is one pod per attempt. Look at the first one, which usually contains the original error without the noise of the retries:

kubectl get pods -n rutas-norte-pro \
  -l job-name=occupancy-reports-29178180 \
  --sort-by=.metadata.creationTimestamp

NAME                               READY   STATUS   RESTARTS   AGE
occupancy-reports-29178180-4kx2p   0/1     Error    0          3h
occupancy-reports-29178180-9dz7w   0/1     Error    0          3h
occupancy-reports-29178180-t6m1c   0/1     Error    0          3h

kubectl logs -n rutas-norte-pro occupancy-reports-29178180-4kx2p

Generating occupancy report for 2026-08-02
psql: error: connection to server at "bookings-postgres" (10.96.144.21), port 5432 failed:
        Connection timed out
        Is the server running on that host and accepting TCP/IP connections?

A timeout towards the database, with the name resolving correctly, almost always points to a NetworkPolicy that does not authorise that conversation: it is exactly what would happen if we had forgotten the manifest from section 7.

Useful shortcuts:

# Logs of every pod of the Job at once
kubectl logs -n rutas-norte-pro job/occupancy-reports-29178180 --all-containers --tail=50

# The container's exact exit code
kubectl get pod occupancy-reports-29178180-4kx2p -n rutas-norte-pro \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}{"\n"}'

# With restartPolicy: OnFailure, the log of the previous attempt
kubectl logs -n rutas-norte-pro <pod> --previous

Exit codes worth recognising:

Code	Usual cause
1	Generic application error
2	Incorrect use of a shell command
126	The command exists but is not executable (permissions)
127	Command not found (typical of a misspelt `command`)
137	`SIGKILL`: almost always `OOMKilled`, `limits.memory` was exceeded
143	`SIGTERM`: orderly termination, often because of `activeDeadlineSeconds`

Cleaning up history

With ttlSecondsAfterFinished and the history limits, clean-up is automatic. For manual maintenance:

# How many finished objects have piled up
kubectl get jobs -A --field-selector status.successful=1 --no-headers | wc -l

# Delete completed Jobs from a namespace
kubectl delete jobs -n rutas-norte-pro --field-selector status.successful=1

# Delete Completed and Error pods across the whole cluster
kubectl delete pods -A --field-selector status.phase=Succeeded
kubectl delete pods -A --field-selector status.phase=Failed

Deleting a Job also removes its pods, because the Job is their ownerReference (the ownership mechanism we saw in 02-02). If you want to keep the pods in order to inspect them:

kubectl delete job occupancy-reports-29178180 -n rutas-norte-pro --cascade=orphan

Common Mistakes and Tips

Putting restartPolicy: Always in a Job. The API rejects it with a clear message (spec.template.spec.restartPolicy: Unsupported value: "Always"). Copying and pasting from a Deployment is the usual cause.

Forgetting the time zone. Without timeZone, a schedule: "15 3 * * *" fires in UTC, and in summer that is 05:15 in Spain: inside other teams' maintenance windows or just as traffic starts picking up. Always set timeZone: "Europe/Madrid".

Trusting the default concurrencyPolicy. Allow is fine for trivial tasks, but for any job that touches the database or writes to a shared PVC it is a source of silent corruption. Actively decide which value you want.

A CronJob that never starts and does not say why. Look for the FailedNeedsStart event with "too many missed start times". It happens after a long outage and is fixed by setting startingDeadlineSeconds.

failedJobsHistoryLimit: 1. The default leaves you with no evidence of the first failure, which is the most informative one. Raise it to 5 on any job with operational importance.

Not setting ttlSecondsAfterFinished. A CronJob every five minutes generates almost 300 objects a day. With no TTL and no history limits, within weeks you will have tens of thousands of Jobs and pods in etcd, and the apiserver will suffer.

Forgetting the NetworkPolicy in rutas-norte-pro. The symptom is a Connection timed out towards a name that resolves fine. And remember to include the egress rule to CoreDNS: without it the failure is a could not translate host name, which is even more baffling.

Ignoring ResourceQuotas. Jobs consume namespace quota (03-04). A Job with parallelism: 20 can exhaust it and block the deployment of bookings-api. The symptom is a FailedCreate event with exceeded quota.

Tip: kubectl create job --from=cronjob/<name>. It is the indispensable validation tool. Never deploy a new CronJob without having fired it manually at least once.

Tip: make your jobs idempotent. They may run twice: because of a retry, a manual run, a recovery after an outage. IF NOT EXISTS, ON CONFLICT DO NOTHING and overwriting instead of appending are the difference between an incident and a non-event.

Tip: always log the beginning and the end. An echo at the start and another at the finish with the result turn the logs into something useful. A Job that only writes when it fails is a Job whose success you cannot verify.

Exercises

Exercise 1: an indexed Job that shares out routes

In rutas-norte-dev, create a Job called occupancy-per-route in Indexed mode with 6 completions and parallelism 2, using busybox:1.36. Each pod must print which Rutas Norte route it was given (route = index + 1), take 5 seconds and finish. Add a ttlSecondsAfterFinished of one hour.

Verify that the 6 pods have been created with their indices and that there were never more than 2 running at a time.

Exercise 2: a CronJob with `Forbid` and a manual test

Create a CronJob sales-summary in rutas-norte-dev that runs every 5 minutes on Madrid time, with concurrencyPolicy: Forbid, startingDeadlineSeconds: 120, a history of 2 successes and 3 failures, and a ttlSecondsAfterFinished of one hour in its template. The content must simulate a summary that takes 30 seconds.

Fire a manual run without waiting for the schedule and check the logs. Then suspend the CronJob.

Exercise 3: diagnosing a failed Job

Deliberately create a Job broken-report in rutas-norte-dev with backoffLimit: 2 and restartPolicy: Never, whose container tries to connect to a non-existent host postgres-nonexistent and fails. Then:

Work out the reason the Job is in Failed.
Find the pod of the first attempt and read its log.
Get the container's exit code.
Clean up.

Solutions

Solution 1

# /tmp/occupancy-per-route.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: occupancy-per-route
  namespace: rutas-norte-dev
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: dev
spec:
  completionMode: Indexed
  completions: 6
  parallelism: 2
  backoffLimit: 3
  ttlSecondsAfterFinished: 3600
  template:
    metadata:
      labels:
        app: occupancy-reports
        environment: dev
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      containers:
        - name: calculation
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              ROUTE=$(( JOB_COMPLETION_INDEX + 1 ))
              echo "index=$JOB_COMPLETION_INDEX -> Rutas Norte route L$ROUTE"
              sleep 5
              echo "route L$ROUTE calculated"
          resources:
            requests:
              cpu: 20m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi

kubectl apply -f /tmp/occupancy-per-route.yaml
kubectl get pods -n rutas-norte-dev -l job-name=occupancy-per-route -w

While it runs you never see more than two pods in Running at once, because parallelism: 2 prevents it:

NAME                          READY   STATUS      RESTARTS   AGE
occupancy-per-route-0-p4m2x   1/1     Running     0          3s
occupancy-per-route-1-r7k9d   1/1     Running     0          3s
occupancy-per-route-0-p4m2x   0/1     Completed   0          9s
occupancy-per-route-2-w3j5t   1/1     Running     0          1s

kubectl get job occupancy-per-route -n rutas-norte-dev
kubectl logs -n rutas-norte-dev occupancy-per-route-4-b8n6q

NAME                  STATUS     COMPLETIONS   DURATION   AGE
occupancy-per-route   Complete   6/6           27s        30s

index=4 -> Rutas Norte route L5
route L5 calculated

Index 4 processes route 5, deterministically and reproducibly. Without completionMode: Indexed the six pods would have run the same command with no idea which portion was theirs.

Solution 2

# /tmp/sales-summary.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: sales-summary
  namespace: rutas-norte-dev
  labels:
    app: occupancy-reports
    app.kubernetes.io/part-of: rutas-norte
    environment: dev
spec:
  schedule: "*/5 * * * *"
  timeZone: "Europe/Madrid"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 300
      ttlSecondsAfterFinished: 3600
      template:
        metadata:
          labels:
            app: occupancy-reports
            environment: dev
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          containers:
            - name: summary
              image: busybox:1.36
              command:
                - sh
                - -c
                - |
                  echo "$(date '+%Y-%m-%d %H:%M:%S') sales summary started"
                  sleep 30
                  echo "tickets sold yesterday: 1842 (made-up figure)"
                  echo "summary completed"
              resources:
                requests:
                  cpu: 20m
                  memory: 32Mi
                limits:
                  cpu: 100m
                  memory: 64Mi

kubectl apply -f /tmp/sales-summary.yaml
kubectl get cronjob sales-summary -n rutas-norte-dev

NAME            SCHEDULE      TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
sales-summary   */5 * * * *   Europe/Madrid   False     0        <none>          8s

# Manual run without waiting for the schedule
kubectl create job -n rutas-norte-dev --from=cronjob/sales-summary summary-manual
kubectl wait --for=condition=complete job/summary-manual -n rutas-norte-dev --timeout=120s
kubectl logs -n rutas-norte-dev job/summary-manual

2026-08-05 18:47:02 sales summary started
tickets sold yesterday: 1842 (made-up figure)
summary completed

# Suspend and check
kubectl patch cronjob sales-summary -n rutas-norte-dev -p '{"spec":{"suspend":true}}'
kubectl get cronjob sales-summary -n rutas-norte-dev

NAME            SCHEDULE      TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
sales-summary   */5 * * * *   Europe/Madrid   True      0        3m              6m

With SUSPEND at True no more Jobs will be created, but the object and its history are still there. It is what you would do before a maintenance window on bookings-postgres.

Solution 3

# /tmp/broken-report.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: broken-report
  namespace: rutas-norte-dev
  labels:
    app: occupancy-reports
    environment: dev
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 300
  template:
    metadata:
      labels:
        app: occupancy-reports
        environment: dev
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      containers:
        - name: generator
          image: postgres:16.4
          command:
            - sh
            - -c
            - |
              echo "Connecting to the bookings database..."
              psql -h postgres-nonexistent -U rutasnorte -d bookings -c 'SELECT 1'
          env:
            - name: PGCONNECT_TIMEOUT
              value: "5"
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi

kubectl apply -f /tmp/broken-report.yaml
# Wait about two minutes for the exponential backoff between attempts
kubectl get job broken-report -n rutas-norte-dev

NAME            STATUS   COMPLETIONS   DURATION   AGE
broken-report   Failed   0/1           2m14s      2m30s

1. Reason for the failure:

kubectl describe job broken-report -n rutas-norte-dev | grep -A5 Conditions

Conditions:
  Type     Status  Reason                Message
  ----     ------  ------                -------
  Failed   True    BackoffLimitExceeded  Job has reached the specified backoff limit

Three attempts (the original plus two retries with backoffLimit: 2) and none succeeded.

2. The pod of the first attempt and its log:

kubectl get pods -n rutas-norte-dev -l job-name=broken-report \
  --sort-by=.metadata.creationTimestamp \
  -o custom-columns=POD:.metadata.name,STATE:.status.phase,CREATED:.metadata.creationTimestamp

POD                   STATE    CREATED
broken-report-2xh4m   Failed   2026-08-05T18:52:03Z
broken-report-8kq7p   Failed   2026-08-05T18:52:18Z
broken-report-v5n1w   Failed   2026-08-05T18:52:49Z

kubectl logs -n rutas-norte-dev broken-report-2xh4m

Connecting to the bookings database...
psql: error: could not translate host name "postgres-nonexistent" to address:
      Name or service not known

The message could not translate host name indicates a DNS resolution failure, not a connectivity one: the name does not exist. If the name existed but a NetworkPolicy blocked the traffic, we would see Connection timed out. Telling those two messages apart saves an enormous amount of diagnosis time.

3. Exit code:

kubectl get pod broken-report-2xh4m -n rutas-norte-dev \
  -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}{"\n"}'

Code 2: psql could not connect. A 137 would have indicated OOMKilled and a 127 a non-existent command.

4. Clean-up:

kubectl delete -f /tmp/broken-report.yaml
kubectl delete -f /tmp/sales-summary.yaml
kubectl delete -f /tmp/occupancy-per-route.yaml
kubectl delete job -n rutas-norte-dev summary-manual --ignore-not-found

Conclusion

Jobs and CronJobs complete the catalogue of Kubernetes workloads with the ones that finish. The Job runs pods until it accumulates completions successes, with parallelism as load control, backoffLimit and activeDeadlineSeconds as circuit breakers, and ttlSecondsAfterFinished as automatic clean-up. The choice between restartPolicy: Never and OnFailure decides whether each attempt leaves its own pod with its own logs — almost always what you want — or whether the container is restarted in place. The three patterns (single task, fixed parallelism, work queue) cover batch work, and completionMode: Indexed with JOB_COMPLETION_INDEX turns the second into a deterministic, retryable division of labour.

The CronJob adds the schedule: five fields, timeZone so as not to depend on UTC, concurrencyPolicy: Forbid when two simultaneous runs would be damaging, startingDeadlineSeconds to decide what to do with missed runs and the history limits so as not to fill etcd.

With this Rutas Norte is complete: web-store, bookings-api, bookings-postgres on a StatefulSet, redis-cache, notifications-worker and now occupancy-reports generating its report on a PVC every night at 03:15, with its ServiceAccount, its resources and the NetworkPolicy that lets it through the deny-all.

Every pod we have written so far has one thing in common: a single container. But deploying the schema migration raised a question we left unanswered: what if, instead of a separate Job, we wanted the migration to run automatically before every bookings-api pod starts? What if bookings-api had to wait until bookings-postgres accepted connections before considering itself started? What if we wanted to export PostgreSQL metrics without modifying its image? All of that is solved by putting more than one container in the same pod, and it is the subject of the next lesson: Init Containers, Sidecars and Multi-Container Patterns.

Jobs and CronJobs

Contents

Workloads that finish versus workloads that do not

The Job object: anatomy and control fields

backoffLimit and exponential backoff

activeDeadlineSeconds

ttlSecondsAfterFinished

restartPolicy: Never versus OnFailure

The three Job patterns

Pattern 1: single task

Pattern 2: fixed parallelism with a number of completions

Pattern 3: work queue

completionMode: Indexed and deterministic work sharing

The CronJob object: schedule and concurrency policy

The schedule syntax

timeZone

concurrencyPolicy

startingDeadlineSeconds and the cluster being down

History and suspension

The central case: occupancy-reports as a nightly CronJob

The report PVC

The ServiceAccount

The NetworkPolicy

The CronJob

Deploy and test without waiting until three in the morning

A second example: a schema migration Job

Debugging failed jobs and cleaning up history

Step 1: the state of the Jobs

Step 2: why the Job failed

Step 3: the right pod and its logs

Cleaning up history

Common Mistakes and Tips

Exercises

Exercise 1: an indexed Job that shares out routes

Exercise 2: a CronJob with Forbid and a manual test

Exercise 3: diagnosing a failed Job

Solutions

Solution 1

Solution 2

Solution 3

Conclusion

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification