The three previous lessons secured identity, the channel and the code. All of that runs on top of a platform — images, pods, secrets, cluster network, API server permissions, pipelines — and a flaw in that layer nullifies the others: an image with a vulnerable library, a container running as root that escapes to the node, a Secret readable by anyone in the namespace, a pod that can talk to whoever it likes, or a CI token with cluster-admin permissions. This lesson closes the module by hardening TechCorp's platform: threat model and the 4Cs, secure images (scanning with Trivy, signing with cosign, digests), a complete securityContext and Pod Security Admission, secrets (Kubernetes Secrets, External Secrets Operator, rotation, OIDC in CI), NetworkPolicy with deny-all by default (closing what 07-02 deferred here), RBAC and one ServiceAccount per service, auditing and detection, supply chain, and a checklist with TechCorp's real status. User authentication (07-01), TLS/mTLS (07-02) and OWASP/code (07-03) are only linked to.

Notice. The policies, versions and values in this lesson are for teaching purposes. Before applying them to a real cluster, review them with a platform security professional and with whoever is responsible for compliance; and test every restriction in staging, because many of them break workloads that used to work.

Contents

  1. Platform threat model and the 4Cs
  2. Secure images: minimal base, scanning, signing and digests
  3. Pod security: complete securityContext and Pod Security Admission
  4. Secrets: Kubernetes Secrets, External Secrets Operator and rotation
  5. Secrets in CI: GitHub Environments and OIDC
  6. Network: NetworkPolicy with deny-all by default
  7. Kubernetes RBAC: one ServiceAccount per service and minimal roles
  8. Auditing and detection
  9. Supply chain: lockfile, SBOM and SLSA
  10. Platform checklist for TechCorp

  1. Platform threat model and the 4Cs

Before hardening anything, what can go wrong and in which layer:

Threat How it happens Layer Measure in this lesson
Compromised image Dependency with a CVE, old base image, malicious npm package Container Minimal base, Trivy in CI, cosign signature, digest
Container escape Process as root + vulnerable kernel or excess capabilities Container / Cluster Restrictive securityContext, seccomp, Pod Security restricted
Leaked secret kubectl get secret by someone who should not, unencrypted etcd, secret in a log or in git Cluster / Code Read RBAC, encryption in etcd, External Secrets, rotation, gitleaks (07-03)
Pod talking to whom it should not Without NetworkPolicy, any pod reaches any port Cluster Deny-all + explicit rules
Stolen CI credentials Long-lived token in a GitHub secret Cloud / Cluster OIDC, Environments, GitOps with no kubeconfig outside
Excessive RBAC cluster-admin for the pipeline or for Argo CD "because that way it works" Cluster Minimal roles per namespace
Misconfigured cloud Nodes with public IPs, open API server, public buckets Cloud Platform/provider responsibility; outside the detailed scope

The 4Cs of cloud native security (Cloud, Cluster, Container, Code) are a reminder that each layer inherits the security of the one outside it: the best securityContext does not save a cluster with its API server open to the Internet, and the best cluster does not save an orders-service with SQL injection. This lesson covers Container and Cluster; Code was 07-01/07-03; Cloud belongs to the provider and to Platform.

  1. Secure images: minimal base, scanning, signing and digests

The Dockerfile from 05-01 already meets the essentials: node:20-alpine, multi-stage (no compilers or devDependencies in the final image), USER node, COPY --chown, HEALTHCHECK and npm ci --omit=dev with the .npmrc as a build secret. What gets added:

  • Minimal base: alpine reduces the surface to a few MB; an even smaller alternative is distroless (gcr.io/distroless/nodejs20-debian12: no shell, no package manager; kubectl exec ... sh stops working, which is a security advantage and a debugging cost — solved with kubectl debug and ephemeral containers). TechCorp keeps alpine in phase 1 and evaluates distroless for Payments.
  • Scanning in CI with Trivy, which 05-03 left configured with severity: CRITICAL. The final policy: fail on CRITICAL and HIGH that have a patch available, ignore those that do not but record them, and rescan the images already deployed every night (CVEs appear after the build):
# .github/workflows/node-service-ci.yml (fragment, replaces the one from 05-03)
      - name: Vulnerability scan
        uses: aquasecurity/[email protected]
        with:
          image-ref: ${{ env.IMAGE }}:sha-${{ github.sha }}
          severity: CRITICAL,HIGH
          ignore-unfixed: true                 # no patch available → warning, not blocking (reviewed weekly)
          exit-code: "1"
          format: sarif
          output: trivy.sarif
      - uses: github/codeql-action/upload-sarif@v3      # findings appear in the repo's Security tab
        with: { sarif_file: trivy.sarif }
  • Image signing with cosign (Sigstore): after the push, the pipeline signs the image with the workflow's OIDC identity (no keys to store, keyless), and the cluster verifies the signature before starting a pod, with an admission controller (Kyverno or Sigstore's policy controller). Mention only, with the commands:
# In CI (permissions: id-token: write), after docker/build-push-action:
cosign sign --yes ghcr.io/techcorp/orders-service@sha256:9f1c…            # sign by digest, keyless (Fulcio + Rekor)
# In the cluster or by hand, verify that TechCorp's workflow signed it:
cosign verify ghcr.io/techcorp/orders-service@sha256:9f1c… \
  --certificate-identity-regexp 'https://github.com/techcorp/.*/.github/workflows/node-service-ci.yml@.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
  • imagePullPolicy and digest: a tag (1.0.1) can be re-pointed; a digest (@sha256:…) cannot. In production, the Kustomize overlay pins newTag and digest (kustomize edit set image ghcr.io/techcorp/orders-service@sha256:9f1c…), imagePullPolicy: IfNotPresent is safe because the digest is immutable, and the ghcr.io/techcorp registry is private with imagePullSecrets (or cloud identity) on the ServiceAccount. latest remains forbidden (05-01).

  1. Pod security: complete securityContext and Pod Security Admission

The Deployment from 05-02 had runAsNonRoot, runAsUser: 1000, allowPrivilegeEscalation: false and readOnlyRootFilesystem: true. The complete version, the one the restricted profile requires:

# k8s/orders-service/base/deployment.yaml (spec.template.spec fragment)
    spec:
      serviceAccountName: orders-service              # section 7: own identity, no token mounted
      automountServiceAccountToken: false
      securityContext:                                # at pod level: inherited by every container
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000                                 # mounted volumes belong to group 1000 (secrets as files, section 4)
        seccompProfile: { type: RuntimeDefault }      # runtime syscall filter (blocks the dangerous and rare ones)
      containers:
        - name: orders-service
          image: ghcr.io/techcorp/orders-service@sha256:9f1c…       # digest in prod (section 2)
          securityContext:
            allowPrivilegeEscalation: false           # no setuid, no gaining capabilities
            readOnlyRootFilesystem: true              # the image's filesystem is read-only
            capabilities: { drop: [ALL] }             # no Linux capabilities at all (Node needs none on 3002 > 1024)
          volumeMounts:
            - { name: tmp, mountPath: /tmp }          # the only writable thing: temporary, empty on every start
          resources:                                  # this is security too: a container without limits can exhaust the node (05-02, 06-04)
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { cpu: 500m, memory: 256Mi }
      volumes:
        - { name: tmp, emptyDir: { sizeLimit: 64Mi } }

What each new line brings: seccompProfile: RuntimeDefault enables the runtime's (containerd) seccomp profile, which blocks dozens of syscalls that no normal application uses and that are the route of many container escapes; capabilities: drop: [ALL] removes even the capabilities Docker grants by default (NET_RAW, CHOWN...): if an attacker runs code inside, they cannot open raw sockets or change owners; emptyDir for /tmp makes readOnlyRootFilesystem compatible with libraries that write temporary files (with sizeLimit so it does not fill the node); fsGroup allows reading secrets mounted as files without being root.

Pod Security Admission (PSA) makes the cluster reject pods that do not meet a profile, instead of trusting that every Deployment writes it correctly. Three profiles: privileged (everything), baseline (without the worst: privileged, hostPath, hostNetwork), restricted (all of the above plus runAsNonRoot, drop ALL, seccomp, no escalation). It is enabled with labels on the namespace:

# k8s/namespace.yaml (extends the one from 05-02)
apiVersion: v1
kind: Namespace
metadata:
  name: techcorp
  labels:
    pod-security.kubernetes.io/enforce: restricted        # rejects non-compliant pods
    pod-security.kubernetes.io/enforce-version: v1.30
    pod-security.kubernetes.io/warn: restricted            # warns on kubectl apply
    pod-security.kubernetes.io/audit: restricted           # records it in the audit log (section 8)
kubectl label ns techcorp pod-security.kubernetes.io/warn=restricted --overwrite   # warn only first: see what would break
kubectl apply -k k8s/orders-service/overlays/dev                                   # "Warning: would violate PodSecurity restricted: ..." if something is missing

The migrations Job from 05-02, the gateway, RabbitMQ and Keycloak must comply too (or live in another namespace with a justified baseline). For finer rules than PSA's (requiring a digest, forbidding latest, requiring resources), a policy engine such as Kyverno or OPA Gatekeeper is used; mention only.

  1. Secrets: Kubernetes Secrets, External Secrets Operator and rotation

A Kubernetes Secret (05-02: orders-db, orders-rabbitmq, and since 07-01 orders-oidc; payments-provider in 07-02) is base64, not encryption: kubectl get secret orders-db -o jsonpath='{.data.ORDERS_DB_URL}' | base64 -d shows it to anyone with read permission. Three layers to make it secure:

  1. Encryption at rest in etcd (the API server's EncryptionConfiguration with aescbc/kms): the responsibility of Platform or of the managed provider (enabled by default on the major ones). Without it, a copy of etcd is a copy of every secret.
  2. Read RBAC (section 7): only the ServiceAccounts of the pods that mount them and a few Platform people can get secrets; the Orders developers cannot. And the Secret's manifest is never in techcorp/platform (GitOps is public inside the company).
  3. External source of truth with the External Secrets Operator (ESO): the secret lives in a manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager...), and ESO creates and updates the Kubernetes Secret from it. That way git contains only the reference:
# k8s/orders-service/base/externalsecret-db.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: techcorp
spec:
  refreshInterval: 1h                                  # ESO rereads the manager every hour: if it rotated, it updates the Secret
  secretStoreRef: { name: techcorp-vault, kind: ClusterSecretStore }   # how to talk to the manager (Vault with Kubernetes auth)
  target:
    name: orders-db                                    # the Secret the Deployment from 05-02 expects: nothing changes there
    creationPolicy: Owner
  data:
    - secretKey: ORDERS_DB_URL                         # key in the Kubernetes Secret
      remoteRef: { key: techcorp/prod/orders/db, property: url }   # path in the manager
kubectl get externalsecret orders-db -n techcorp       # STATUS SecretSynced, READY True
kubectl get secret orders-db -n techcorp               # created by ESO; the manifest in git does not contain the value

Credential rotation. A database password lives for years because rotating it is scary; with this architecture, the procedure is routine (and it runs at least every six months and always after a leak, 07-03):

Step orders-db (PostgreSQL) orders-rabbitmq
1 Create the new svc_orders password in the manager (techcorp/prod/orders/db); PostgreSQL allows two valid ones by creating an svc_orders_b user or by changing it inside a window rabbitmqctl change_password orders <new> (open connections continue; new ones use the new password)
2 ESO updates the Secret within ≤ 1 h (or kubectl annotate externalsecret orders-db force-sync=$(date +%s)) Same
3 Pods do not reread environment variables: kubectl rollout restart deploy/orders-service (rolling, no loss, 05-04); with secrets as files (section 5) and a config.js that rereads them, no restart would be needed Same; the amqplib reconnect (06-03) uses the new URL after the restart
4 Revoke the old password; check in the logs that there is no password authentication failed rabbitmqctl has no "dual password": the window is the rolling restart

A Reloader (Stakater) or the Argo CD annotation can automate step 3 when the Secret changes.

  1. Secrets in CI: GitHub Environments and OIDC

In 05-03 the pipeline had GITHUB_TOKEN, PACT_BROKER_TOKEN, PLATFORM_TOKEN and KUBECONFIG_STAGING, and the mention of OIDC was left pending. What gets closed here:

  • GitHub Environments (staging, production): per-environment secrets, required reviewers for production, and deployment branches so that only main can use them. A workflow from a feature branch does not see KUBECONFIG_STAGING.
  • OIDC instead of long-lived tokens: with permissions: id-token: write, the workflow obtains a JWT signed by GitHub that says "I am workflow X of repo Y on branch Z", and the cloud or the registry exchanges it for credentials that last minutes. It is the same client credentials from 07-01, with GitHub as the issuer. For the registry (ghcr.io) GITHUB_TOKEN already does this; for a managed cluster in the cloud:
# .github/workflows/node-service-cd.yml (fragment: access to AWS/EKS without access keys)
permissions: { id-token: write, contents: read }
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/gha-techcorp-staging-deploy   # role with permission ONLY on the staging cluster
      aws-region: eu-west-1
  - run: aws eks update-kubeconfig --name techcorp-staging && kubectl rollout status deploy/orders-service -n techcorp

With this, KUBECONFIG_STAGING disappears as a secret (the trust is "this repo, this branch, this workflow", configured in the cloud role's policy), and in production there is no credential at all: Argo CD pulls (05-03). PLATFORM_TOKEN (for opening the promotion PR) is replaced by a GitHub App with permissions bounded to the platform repository. PACT_BROKER_TOKEN stays as an environment secret, rotated on a schedule.

Files versus environment variables. Environment variables are visible in kubectl describe pod (not the values from secretRef, but those from env: value are), in /proc/<pid>/environ, and child processes inherit them; files mounted from a Secret (volumes: - secret: { secretName: orders-db } + ORDERS_DB_URL_FILE=/etc/secrets/ORDERS_DB_URL) can only be read by whoever has the right UID/fsGroup, and Kubernetes hot-updates them when the Secret changes. The <VARIABLE>_FILE convention from 04-03 is already implemented: TechCorp migrates the Payments secrets to files first (audit) and the rest as each service is touched.

  1. Network: NetworkPolicy with deny-all by default

Without NetworkPolicy, any pod in the cluster can open a connection to any other: a compromised pod in techcorp reaches inventory-service:3006 with X-User-Roles: admin (07-01) or postgres:5432 to try passwords. NetworkPolicy objects are a firewall by labels; they require a CNI that enforces them (Calico, Cilium; managed offerings usually ship one). Strategy: deny everything in the namespace and open explicitly.

# k8s/network/00-deny-all.yaml — nobody gets in or out, except what is allowed afterwards
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: deny-all, namespace: techcorp }
spec:
  podSelector: {}                         # every pod in the namespace
  policyTypes: [Ingress, Egress]
---
# k8s/network/01-allow-dns.yaml — everyone needs to resolve names (03-05); without this, nothing works
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-dns, namespace: techcorp }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }, podSelector: { matchLabels: { k8s-app: kube-dns } } }]
      ports: [{ protocol: UDP, port: 53 }, { protocol: TCP, port: 53 }]
---
# k8s/network/gateway.yaml — the gateway is the ONLY one that receives from the ingress controller, and it only talks to the public services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: gateway, namespace: techcorp }
spec:
  podSelector: { matchLabels: { app: gateway } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress-nginx } } }]
      ports: [{ port: 8080 }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: catalog-service } } }]
      ports: [{ port: 3001 }]
    - to: [{ podSelector: { matchLabels: { app: orders-service } } }]
      ports: [{ port: 3002 }]
    - to: [{ podSelector: { matchLabels: { app: customers-service } } }]
      ports: [{ port: 3004 }]
    - to: [{ podSelector: { matchLabels: { app: bff-mobile } } }]
      ports: [{ port: 3010 }]
    - to: [{ podSelector: { matchLabels: { app: keycloak } } }]        # JWKS (07-01)
      ports: [{ port: 8080 }]
---
# k8s/network/orders-service.yaml — receives from the gateway (and from Prometheus); talks to Catalog, Customers, RabbitMQ, PostgreSQL and Keycloak
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: orders-service, namespace: techcorp }
spec:
  podSelector: { matchLabels: { app: orders-service } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: gateway } } }]
      ports: [{ port: 3002 }]
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } } }]   # /metrics (06-01)
      ports: [{ port: 3002 }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: catalog-service } } }]
      ports: [{ port: 3001 }]
    - to: [{ podSelector: { matchLabels: { app: customers-service } } }]
      ports: [{ port: 3004 }]
    - to: [{ podSelector: { matchLabels: { app: rabbitmq } } }]
      ports: [{ port: 5671 }]                                          # amqps (07-02)
    - to: [{ podSelector: { matchLabels: { app: postgres } } }]
      ports: [{ port: 5432 }]
    - to: [{ podSelector: { matchLabels: { app: keycloak } } }]
      ports: [{ port: 8080 }]                                          # service token and JWKS
    - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: observability } } }]   # OTLP to Jaeger (06-02)
      ports: [{ port: 4318 }]
---
# k8s/network/inventory-service.yaml — closes what 07-02 left open: ONLY Orders talks to Inventory
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: inventory-service, namespace: techcorp }
spec:
  podSelector: { matchLabels: { app: inventory-service } }
  policyTypes: [Ingress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: orders-service } } }]
      ports: [{ port: 3006 }]
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } } }]
      ports: [{ port: 3006 }]

And payments-service is the only one with egress to the Internet (the payment provider): ipBlock: { cidr: 0.0.0.0/0, except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16] } on port 443, or better, if the CNI supports it (Cilium), a policy by FQDN (api.payment-provider.example). The rest of the services do not leave the cluster: a compromised catalog-service cannot exfiltrate data or download tools.

kubectl apply -f k8s/network/
# Check: from any pod, Inventory does not answer; from Orders, it does
kubectl run test --rm -it --image=curlimages/curl -n techcorp -- curl -m 3 http://inventory-service:3006/health/live   # timeout
kubectl exec deploy/orders-service -n techcorp -- wget -qO- http://inventory-service:3006/health/live               # {"status":"ok"}
flowchart LR
    IN[ingress-nginx] -->|8080| GW[gateway]
    GW -->|3001| CA[catalog-service]
    GW -->|3002| ORD[orders-service]
    GW -->|3004| CUS[customers-service]
    ORD -->|3001| CA
    ORD -->|3004| CUS
    ORD -->|3006| INV[inventory-service]
    ORD -->|5671 amqps| MQ[(rabbitmq)]
    ORD -->|5432| PG[(postgres)]
    PAY[payments-service] -->|443| EXT((external payment provider))
    PR[prometheus] -.->|/metrics| ORD & INV & CA & CUS
    X[any other pod] -. blocked .-x INV

With the policies in place, the warning from 07-01 about X-User-* moves from "any pod" to "only the gateway and Prometheus can reach Orders" — and mTLS with a mesh (07-02) will add, when it arrives, cryptographic identity on top of the network. The policies live in techcorp/platform/k8s/network/ and Platform reviews them; every new service arrives with its own in the template.

  1. Kubernetes RBAC: one ServiceAccount per service and minimal roles

Every pod runs with a ServiceAccount (SA); by default, default, with a token mounted at /var/run/secrets/kubernetes.io/serviceaccount that allows talking to the API server. orders-service does not need to talk to the API server, so: its own SA (a clear identity for NetworkPolicy in some CNIs, for mTLS with a mesh in 07-02, and for imagePullSecrets) and no token mounted:

# k8s/orders-service/base/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: orders-service, namespace: techcorp }
automountServiceAccountToken: false      # and in the Deployment (section 3) too; if one day it needs the API, a minimal Role and a projected token with expiry
imagePullSecrets: [{ name: ghcr-techcorp }]

Those who do talk to the API server, with the minimum:

Identity Needs Role
CD workflow in staging (via OIDC, section 5) apply of Deployment/Service/ConfigMap/Job and rollout status in techcorp Role deployer in techcorp (get/list/watch/create/update/patch on deployments, services, configmaps, jobs, pods read-only); RoleBinding to the federated group/user. Never cluster-admin
Argo CD (production) Apply the manifests from overlays/prod Argo CD brings its own SA; it is given a Role per managed namespace (techcorp, monitoring), not the default admin ClusterRole from the quick install
Prometheus Discover pods and endpoints Read-only ClusterRole (get/list/watch on pods, services, endpoints)
External Secrets Operator Create/update Secret Role in each namespace with secrets (create/update/get), not cluster-admin
People Orders developer: read pods/logs and port-forward in techcorp (view + pods/portforward); Platform: admin in techcorp; nobody uses cluster-admin day to day (audited break-glass account) RoleBinding to identity provider groups (Keycloak/SSO here too)
# k8s/rbac/deployer.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: deployer, namespace: techcorp }
rules:
  - { apiGroups: ["apps"],  resources: ["deployments"],           verbs: ["get", "list", "watch", "create", "update", "patch"] }
  - { apiGroups: [""],      resources: ["services", "configmaps"], verbs: ["get", "list", "create", "update", "patch"] }
  - { apiGroups: ["batch"], resources: ["jobs"],                  verbs: ["get", "list", "watch", "create", "delete"] }
  - { apiGroups: [""],      resources: ["pods", "pods/log"],      verbs: ["get", "list", "watch"] }
  # neither "secrets" nor "delete deployments": secrets are set by ESO; deleting is manual and audited
kubectl auth can-i list secrets -n techcorp --as=system:serviceaccount:techcorp:orders-service   # no
kubectl auth can-i create deployments -n techcorp --as=deployer-staging                          # yes
kubectl auth can-i '*' '*' --as=deployer-staging                                                 # no

  1. Auditing and detection

  • API server audit log: records who did what against the API (kubectl get secret orders-db by a person, a create of a Role, a pod that does not comply with PSA). It is configured with an audit Policy (Metadata level for everything, RequestResponse for secrets, roles and rolebindings) and sent to Loki (06-01) with long retention, separate from application logs (07-03). On managed clusters it is enabled from the provider.
  • Runtime detection: Falco (mention) watches syscalls on the nodes and alerts on anomalous behavior — a shell opened in a production container, a process reading /etc/shadow, an outbound connection from a pod that should not go out — and integrates with Alertmanager (06-05). With readOnlyRootFilesystem, drop ALL and NetworkPolicies, most of those behaviors already fail; Falco reports the attempt.
  • Hygiene: kube-bench (CIS Benchmark) for the cluster configuration and kubectl get events -A --field-selector reason=FailedCreate to see pods rejected by PSA.

  1. Supply chain: lockfile, SBOM and SLSA

The supply chain is everything between Luis's code and the pod in production, and every link can be replaced by something malicious. What TechCorp pins down: package-lock.json and npm ci (05-01/07-03: exactly what was tested is what gets deployed); dependencies and CI actions pinned by version (actions/checkout@v4, better by SHA for the critical ones) and the base image by digest (node:20-alpine@sha256:…, updated by Renovate); an SBOM per image with Syft (syft ghcr.io/techcorp/orders-service:1.0.1 -o spdx-json > sbom.json) attached to the image (cosign attest) to answer within minutes "which services ship libxml2 2.9.x?"; cosign signature and verification at admission (section 2); and SLSA (Supply-chain Levels for Software Artifacts) as the reference framework: TechCorp is at level 2 (build in CI with signed provenance) and aims for 3 (ephemeral, isolated runners). Mention only: the essential thing is that the pipeline from 05-03 is the only path to production, and that it produces verifiable artifacts.

  1. Platform checklist for TechCorp

Status at the close of module 7, kept in techcorp/platform/SECURITY.md:

Area Check Status
Image node:20-alpine base by digest, multi-stage, USER node, no devDependencies Done (05-01)
Trivy CRITICAL/HIGH blocking + nightly scan of deployed images Done / Pending (nightly)
cosign signature + verification at admission Pending (phase 2)
Pod Complete securityContext (runAsNonRoot, drop ALL, seccomp, RO rootfs, emptyDir /tmp), resource limits Done in Orders and Catalog; pending in gateway, Notifications
PSA restricted in techcorp (enforce) warn active; enforce pending until RabbitMQ/Keycloak move to their own namespace
Secrets Encryption in etcd (provider), read RBAC, no Secret manifests in git Done
ESO from the external manager for every application Secret; documented six-monthly rotation Done orders-db, orders-rabbitmq, orders-oidc, payments-provider; rest pending
Secrets as _FILE files Payments in progress
CI Environments with reviewers; OIDC for cloud and registry; no kubeconfig in secrets; GitHub App instead of PLATFORM_TOKEN Done / Pending (GitHub App)
Network Deny-all + DNS + per-service policies; only Payments with external egress Done in techcorp
RBAC SA per service without token; Role deployer; Argo CD bounded; no daily cluster-admin Done / Argo pending
Audit Audit log to Loki with long retention; Falco Done / Pending
Supply chain Lockfile, pinned versions, SBOM with Syft, SLSA 2 Done except SBOM (in progress)

Common Mistakes and Tips

  • Applying deny-all without the DNS policy: nothing resolves anymore and it looks like "Kubernetes is broken". DNS first, always.
  • Forgetting Prometheus scraping, Jaeger or Keycloak in the policies: metrics and traces disappear silently. Review every to/from against the diagrams from 06-01/06-02.
  • readOnlyRootFilesystem without emptyDir on /tmp: the application fails at startup with EROFS on a dependency's first write.
  • PSA enforce all at once in a namespace with RabbitMQ, Keycloak or the ingress controller: rejected pods. warn/audit first, then enforce.
  • Secret in the GitOps repository "because it's base64": it is plain text. Reference with ExternalSecret, value in the manager.
  • Rotating the password and not restarting the pods (environment variables): they fail on reconnect hours later, in the middle of the night. Rotation = change + sync + rollout restart (or hot-reloaded files).
  • cluster-admin for the pipeline or Argo CD "for now": now lasts years. Role per namespace from day one.
  • Tip: every new restriction is tested with a game day (06-03): try to do what is forbidden (kubectl run as root, curl to Inventory from another pod, kubectl get secret as a developer) and check that it fails and that it lands in the audit log.

Exercises

Exercise 1. Write the NetworkPolicy for notifications-service (port 3005): it consumes from RabbitMQ (amqps), sends email to an external provider over HTTPS, exposes /metrics to Prometheus, and nobody in the cluster should be able to call its API except Prometheus. State the risk the external egress introduces and how to bound it.

Exercise 2. A Catalog developer asks to be able to run kubectl exec on the catalog-service pods in staging "to debug Mongo". Design the minimal Role/RoleBinding, say what it does not include and propose a better alternative consistent with this lesson.

Exercise 3. Walk through the threat model from section 1 for the scenario "an attacker manages to run code inside the catalog-service pod thanks to a compromised dependency" and list, layer by layer, which measures from this lesson limit the damage and what they could still do.

Solutions

Solution 1.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: notifications-service, namespace: techcorp }
spec:
  podSelector: { matchLabels: { app: notifications-service } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } } }]
      ports: [{ port: 3005 }]
  egress:
    - to: [{ podSelector: { matchLabels: { app: rabbitmq } } }]
      ports: [{ port: 5671 }]
    - to: [{ podSelector: { matchLabels: { app: keycloak } } }]        # if it validates tokens or requests its own
      ports: [{ port: 8080 }]
    - to: [{ ipBlock: { cidr: 0.0.0.0/0, except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16] } }]
      ports: [{ port: 443 }]

(DNS is covered by allow-dns.) The risk: egress to "any public IP on 443" allows exfiltrating data (emails, names) if the pod is compromised. Bounding it: a policy by the email provider's FQDN with Cilium (toFQDNs: [{ matchName: api.email.example }]), or an outbound proxy (egress gateway) with a domain allowlist that also provides a single audit point; and the minimization from 07-03 (Notifications does not persist personal data) reduces what there is to exfiltrate.

Solution 2. Role catalog-debugger in techcorp with pods (get/list), pods/log (get), pods/exec (create) limited to the Catalog pods — RBAC does not filter by label, so it is done with resourceNames (fragile, names change) or, better, by granting the Role in a techcorp-catalog namespace if things are split by team; RoleBinding to the SSO group shopping-experience-team. It does not include secrets, exec on other pods, or anything write-related. Better alternatives: (1) kubectl debug with an ephemeral container (kubectl debug -it pod/catalog-service-xxx --image=mongo:7 --target=catalog-service) instead of exec into the application container, which also works with distroless; (2) a port-forward to MongoDB (pods/portforward) and the Mongo client locally with the read-only svc_catalog_ro user; (3) whatever needs to be seen should be in logs/metrics (06-01) so nobody ever enters production. And every exec/debug lands in the audit log (section 8).

Solution 3. With code running as the node user (uid 1000) in the Catalog pod: Container: it is not root, drop ALL and seccomp block most escape routes; readOnlyRootFilesystem prevents installing tools (only a 64 MB /tmp); resources.limits prevent taking down the node. Secrets: it sees the process's environment variables (Catalog's MONGO_URL) — with _FILE files it would still see them, it is its own secret —; it cannot read other Secrets (SA without token, and no RBAC). Network: it only reaches Catalog's MongoDB, Keycloak and whatever its policy allows; it does not reach Orders, Inventory, PostgreSQL or RabbitMQ, and it does not get out to the Internet (no external egress): it can read/modify the catalog (its real damage) but cannot easily exfiltrate it or pivot. Identity: no valid headers and no other service's token (the X-User-* headers are useless to it because it cannot reach other services; with a mesh, it would not have a certificate either). Detection: Falco would alert on the shell or the odd process; the audit log would see nothing because it does not touch the API. What it could still do: change prices or descriptions in MongoDB (mitigated by the auditing from 07-03 and the svc_catalog user without dropDatabase), serve fake responses to Orders (mitigated by zod validation in the ACL in Orders and prices frozen in the order) and consume CPU up to the limit. And the underlying lesson: the compromised dependency should have been caught by Trivy/npm audit/SBOM before it arrived (sections 2 and 9).

Conclusion

With this lesson the platform stops being the weak link beneath identity, the channel and the code. TechCorp has a layered threat model (4Cs); minimal images scanned with Trivy (CRITICAL/HIGH blocking) and referenced by digest, with cosign and verification at admission as the next step; pods with a complete securityContext (runAsNonRoot, drop ALL, seccompProfile: RuntimeDefault, readOnlyRootFilesystem with emptyDir on /tmp) and Pod Security Admission restricted in techcorp; secrets encrypted in etcd, restricted by RBAC and provided by the External Secrets Operator (ExternalSecret orders-db from the external manager) with a rotation procedure and rollout restart; CI with Environments and OIDC instead of kubeconfig and long-lived tokens, and secrets as _FILE files where it matters most; deny-all NetworkPolicy with DNS, gateway as the only entry from the Ingress, Orders toward Catalog/Customers/RabbitMQ/PostgreSQL/Keycloak, Inventory only from Orders — closing what 07-02 deferred here — and only Payments with outbound Internet access; one ServiceAccount per service without token, Role deployer and Argo CD bounded, no cluster-admin; API server audit log and Falco as detection; lockfile, pinned versions, SBOM and SLSA as the supply chain; and a checklist with what is done and what is pending. Module 7 thus ends where module 6 began: with a system that is observable, resilient, scalable, operable and now also secure in layers — authentication and authorization with Keycloak and JWT, an encrypted and signed channel, validated and audited code, a hardened platform — and with an honest list of what remains for phase 2 (mTLS with a mesh, cosign at admission, PSA enforce, ESO for every service). Module 8 gathers the whole journey: how the monolith migration was executed step by step, the complete implementation of the services, their end-to-end deployment and operation, and the lessons learned by Marta, Luis and TechCorp's four teams.

Microservices Course

Module 1: Introduction to Microservices

Module 2: Microservice Design

Module 3: Communication between Microservices

Module 4: Implementing Microservices

Module 5: Deployment and Orchestration

Module 6: Monitoring and Maintenance

Module 7: Security in Microservices

Module 8: Case Studies and Practical Examples

© Copyright 2026. All rights reserved