In the previous two lessons we left web-store, bookings-api and bookings-postgres running in production. The manifests are correct, the probes respond, the backups restore. What is missing is the question that makes all of this sustainable: how does the code a developer writes on a Tuesday morning actually get there?
This lesson covers that whole distance: from the git push to the pod serving requests in rutas-norte-pro. It is not a lesson about GitOps, which we already saw in 10-05, nor about image signing, which we saw in 08-05: it is the lesson where all those pieces connect into a pipeline that works, and where we explain why the Rutas Norte pipeline never runs kubectl apply.
At the end we will measure whether all this is worth anything, because an elaborate pipeline that does not reduce the time between writing code and seeing it run is an internal engineering project dressed up as an improvement.
Contents
- The complete Rutas Norte flow
- Continuous integration and continuous delivery: why separate them
- The integration pipeline, step by step
- Integration tests against an ephemeral kind cluster
- Delivery: updating the manifest repository
- Promotion to
preand toprowith human approval - Ephemeral environments per pull request
- Credentials with no long-lived secrets
- Post-deployment verification and automatic rollback
- DORA metrics: Rutas Norte's numbers before and after
- The complete Rutas Norte flow
There are two repositories, and that separation is the axis of everything else:
| Repository | Contains | Who modifies it |
|---|---|---|
rutasnorte/bookings-api |
Source code, Dockerfile, tests, pipeline definition |
Development, by hand |
rutasnorte/manifests |
Helm chart, Kustomize overlays, Argo CD Application |
The pipeline (dev) and people (pre/pro) |
graph TB
DEV[Developer<br/>git push to branch] --> PR[Pull request]
PR --> CI
subgraph CI["Continuous integration · GitHub Actions"]
T1[1 Lint + unit tests]
T2[2 Multi-stage build<br/>with layer cache]
T3[3 Trivy: fails on CRITICAL]
T4[4 Push to registry<br/>tag = commit SHA]
T5[5 Cosign: keyless signing]
T6[6 Obtain digest]
T7[7 Tests on a kind cluster]
T1 --> T2 --> T3 --> T4 --> T5 --> T6 --> T7
end
CI --> EF[Ephemeral environment<br/>ns rutas-norte-pr-1842]
T7 --> MERGE{Merge to main}
MERGE --> BOT[Automatic pull request<br/>in rutasnorte/manifests<br/>digest in overlays/dev]
BOT --> AUTO[Automatic merge]
AUTO --> ACD[Argo CD]
ACD --> DEVENV[rutas-norte-dev]
DEVENV --> HUMO1[Smoke tests]
HUMO1 --> PROMO_PRE[Promotion PR to pre<br/>approval: development]
PROMO_PRE --> ACD2[Argo CD] --> PREENV[rutas-norte-pre]
PREENV --> CARGA[k6 + regression tests]
CARGA --> PROMO_PRO[Promotion PR to pro<br/>approval: platform + product]
PROMO_PRO --> ACD3[Argo CD] --> PROENV[rutas-norte-pro]
PROENV --> VERIF[Post-deployment verification<br/>automatic rollback on failure]
Note something important: the arrow into the namespaces always comes out of Argo CD, never out of the pipeline. The pipeline reaches the manifest repository and stops there.
- Continuous integration and continuous delivery: why separate them
They are named together and constantly confused, but they answer different questions and fail in different ways.
| Continuous integration | Continuous delivery | |
|---|---|---|
| Question it answers | Is this change correct? | Is this artefact in the environments? |
| Input | A commit | A verified artefact (digest) |
| Output | A signed image and its digest | A cluster state |
| Where the logic lives | GitHub Actions | Manifest repository + Argo CD |
| Nature | Imperative: steps in order | Declarative: desired state |
| If it fails | No artefact comes out; nobody outside the team notices | The environments diverge; that does get noticed |
| Frequency | Every push | Every change to the manifest repository |
2.1. Why the pipeline does not run kubectl apply
It is the most important design decision in this lesson, and it is worth understanding properly because it contradicts what a lot of people do.
With kubectl apply from the pipeline:
- The pipeline runner needs write credentials on production. Anyone who can modify the pipeline file can modify production.
- The cluster's real state depends on which runs have passed and in what order. If two overlap, the result is undetermined.
- A manual change with
kubectl editat three in the morning during an incident does not revert itself, and nobody finds out that the cluster no longer matches Git. - Knowing what is deployed means looking at the cluster, not the repository.
- Rebuilding the environment after a disaster means re-running old pipelines, assuming they still work at all.
With GitOps (10-05):
- The pipeline only needs permission to open a pull request in a Git repository. Zero cluster credentials.
- The desired state is a Git revision: reproducible, reviewable, with history and an author.
- Drift corrects itself thanks to
selfHeal, and is visible as an out-of-sync event. - Rolling back is
git revert. - The audit trail for "who changed what in production and when" is the Git log, not a scattered trail of CI jobs.
The rule, in one sentence: integration produces a digest; delivery decides which environment that digest lives in; only Git joins the two.
- The integration pipeline, step by step
The .github/workflows/integration.yml file from the rutasnorte/bookings-api repository, complete and commented.
name: Continuous integration
on:
push:
branches: [main]
pull_request:
# No long-lived secrets: id-token allows requesting an ephemeral OIDC
# token to authenticate against the cloud and sign with Cosign.
permissions:
contents: read
packages: write
id-token: write
pull-requests: write
env:
REGISTRY: registry.rutasnorte.example
IMAGE: rutasnorte/bookings-api
# Cancels older runs on the same branch: we do not burn runners
# validating commits that are already obsolete.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
# ---------------------------------------------------------------
# 1. Unit tests: fast, no network, no real database.
# ---------------------------------------------------------------
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '22'
cache: npm
- name: Install exact dependencies
# npm ci honours the lock file: reproducible, unlike npm install.
run: npm ci
- name: Static analysis
run: npm run lint
- name: Unit tests with coverage
run: npm test -- --coverage --coverageThreshold='{"global":{"lines":80}}'
- name: Dependency audit
run: npm audit --audit-level=high
# ---------------------------------------------------------------
# 2-6. Build, scan, publish and sign.
# ---------------------------------------------------------------
image:
needs: tests
runs-on: ubuntu-latest
outputs:
# The digest is the output consumed by the following jobs
# and, finally, by the manifest repository.
digest: ${{ steps.build.outputs.digest }}
tag: ${{ steps.meta.outputs.tag }}
steps:
- uses: actions/checkout@v4
- name: Derive the tag from the commit
id: meta
run: |
# Short SHA: a unique, traceable and sortable identifier.
TAG="$(git rev-parse --short=12 HEAD)"
echo "tag=${TAG}" >> "$GITHUB_OUTPUT"
- uses: docker/setup-buildx-action@v3
- name: Authenticate to the registry via OIDC
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
# An ephemeral token for this very run, not a stored credential.
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build the multi-stage image
id: build
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ steps.meta.outputs.tag }}
# Layer cache in the registry itself: a typical build drops
# from 4 min to about 50 s by reusing node_modules.
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE }}:cache
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE }}:cache,mode=max
provenance: true # SLSA provenance attestation
sbom: true # software bill of materials
- name: Scan with Trivy
uses: aquasecurity/[email protected]
with:
image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.build.outputs.digest }}
format: table
# Breaks the build only on CRITICAL. HIGH findings are recorded
# and reviewed weekly: if you break on HIGH, the team learns
# to bypass the control rather than to fix it.
severity: CRITICAL
exit-code: '1'
ignore-unfixed: true # with no patch available there is nothing to do today
- name: Install Cosign
uses: sigstore/cosign-installer@v3
- name: Sign keylessly with the workflow identity
# There is no private key anywhere: Cosign obtains an ephemeral
# certificate from Fulcio using this run's OIDC token and records
# the signature in the public Rekor log. The signing identity is
# the workflow itself, verifiable in the admission policy.
run: |
cosign sign --yes \
"${REGISTRY}/${IMAGE}@${{ steps.build.outputs.digest }}"
- name: Run summary
run: |
{
echo "### Image published"
echo "- Tag: \`${{ steps.meta.outputs.tag }}\`"
echo "- Digest: \`${{ steps.build.outputs.digest }}\`"
echo "- Signed by: \`${{ github.workflow_ref }}\`"
} >> "$GITHUB_STEP_SUMMARY"3.1. The decisions that matter
Tagging by commit hash rather than semantic version. The tag a3f91c2b8e04 is unique, immutable and answers "which exact code is this?" without ambiguity. Semantic versions are reserved for releases announced to the business, which are created as an additional tag on an already existing digest.
The digest as the real output. The tag is for people. What travels to the manifest repository is the digest, because a tag can be rewritten and a digest cannot. It is what underpins the by-digest reference in the Deployments in 11-01.
Trivy with severity: CRITICAL and ignore-unfixed: true. It is a calibrated decision. Breaking on HIGH generates so many false alarms that the team ends up adding exceptions as a matter of routine; and failing over a vulnerability with no patch available achieves nothing, because no action is possible. HIGH findings are reviewed in the weekly platform meeting.
Keyless signing. There is no private key to rotate, guard or leak. The certificate is ephemeral and the signing identity is the specific workflow. The admission policy from 08-05 demands exactly that identity:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-rutasnorte-signature
spec:
validationFailureAction: Enforce
rules:
- name: verify-signature
match:
any:
- resources:
kinds: [Pod]
namespaces: ["rutas-norte-*"]
verifyImages:
- imageReferences: ["registry.rutasnorte.example/rutasnorte/*"]
attestors:
- entries:
- keyless:
subject: "https://github.com/rutasnorte/*/.github/workflows/integration.yml@refs/heads/main"
issuer: "https://token.actions.githubusercontent.com"The practical effect is decisive: an image built on somebody's laptop, however correct it may be, cannot run in any Rutas Norte namespace.
3.2. The multi-stage build with caching
# syntax=docker/dockerfile:1.7
# --- Stage 1: dependencies --------------------------------------
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
# The npm cache is mounted, not copied: it does not bloat any layer.
RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev
# --- Stage 2: build ---------------------------------------------
FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY . .
RUN npm run build
# --- Stage 3: final image ---------------------------------------
FROM gcr.io/distroless/nodejs22-debian12:nonroot
WORKDIR /app
# Only the essentials: no compilers, no shell, no package manager.
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY package.json ./
USER 10001:10001
EXPOSE 8080 9090
CMD ["dist/server.js"]Copying package.json and package-lock.json first and the rest of the code afterwards is not a stylistic whim: it means the dependency layer is only invalidated when the dependencies change. Since 95 % of commits touch code only, the cache almost always hits. The resulting final image weighs 118 MB against the 1.1 GB of a single-stage build on node:22, and it contains no shell, which drastically reduces what an attacker can do inside the container.
- Integration tests against an ephemeral kind cluster
Unit tests do not catch what really breaks: a ConfigMap with a misspelt key, a probe pointing at a port that does not exist, a Helm chart with a mandatory value left undefined. Only actually deploying catches that.
The pipeline brings up a kind cluster (10-01) inside the runner itself, deploys the chart and runs smoke tests.
integration:
needs: image
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Bring up a kind cluster
uses: helm/kind-action@v1
with:
cluster_name: ci-rutasnorte
# One node is enough: we test correct behaviour, not high availability.
config: .github/kind/cluster.yaml
wait: 120s
- name: Deploy real dependencies
# A real PostgreSQL and Redis, not fakes: we want to catch
# SQL and connection errors, which is where these things fail.
run: |
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install pg bitnami/postgresql \
--set auth.database=bookings \
--set auth.username=app_bookings \
--set auth.password=testing \
--set primary.persistence.enabled=false \
--wait --timeout 5m
helm install redis bitnami/redis \
--set auth.enabled=false \
--set master.persistence.enabled=false \
--wait --timeout 5m
- name: Load the freshly built image into kind
run: |
IMG="${REGISTRY}/${IMAGE}@${{ needs.image.outputs.digest }}"
docker pull "$IMG"
kind load docker-image "$IMG" --name ci-rutasnorte
- name: Deploy the Rutas Norte chart
run: |
helm upgrade --install bookings-api ./chart \
--values ./chart/values-ci.yaml \
--set image.digest='${{ needs.image.outputs.digest }}' \
--set postgres.host=pg-postgresql \
--set redis.url=redis://redis-master:6379 \
--wait --timeout 5m
- name: Schema migrations
run: kubectl wait --for=condition=complete job/bookings-api-migrations --timeout=3m
- name: Smoke tests
run: |
kubectl port-forward svc/bookings-api 8080:80 &
sleep 5
npm run test:smoke -- --base-url http://localhost:8080
- name: Diagnostics if something fails
if: failure()
run: |
kubectl get pods -o wide
kubectl describe pods -l app.kubernetes.io/name=bookings-api
kubectl logs -l app.kubernetes.io/name=bookings-api --tail=200 --all-containers
kubectl get events --sort-by=.lastTimestamp | tail -40The smoke tests are deliberately few and highly meaningful:
// tests/smoke.spec.js — run on kind and also after every real deployment
describe('bookings-api smoke', () => {
test('responds to the readiness probe', async () => {
const r = await fetch(`${base}/ready`);
expect(r.status).toBe(200);
});
test('exposes metrics in Prometheus format', async () => {
const t = await (await fetch(`${base}/metrics`)).text();
expect(t).toMatch(/^bookings_api_requests_total/m);
});
test('queries timetables against the real database', async () => {
const r = await fetch(`${base}/schedules?line=BIL-SAN&date=2026-08-14`);
expect(r.status).toBe(200);
expect(Array.isArray((await r.json()).departures)).toBe(true);
});
test('creates a booking and is idempotent', async () => {
const body = { line: 'BIL-SAN', date: '2026-08-14', seats: 2 };
const headers = { 'Idempotency-Key': 'smoke-test-001', 'Content-Type': 'application/json' };
const r1 = await fetch(`${base}/bookings`, { method: 'POST', headers, body: JSON.stringify(body) });
const r2 = await fetch(`${base}/bookings`, { method: 'POST', headers, body: JSON.stringify(body) });
expect(r1.status).toBe(201);
expect((await r2.json()).id).toBe((await r1.json()).id); // no duplicate
});
});This job costs about three minutes per run. In exchange, over the past year it has caught before merging: two ConfigMap keys renamed without updating the code, a migration that was not reversible, a wrong targetPort after a port change, and a readiness probe that returned 200 before the database pool was ready. Any of those four would have been an incident in dev at the very least, and the last one could have reached production.
- Delivery: updating the manifest repository
When the merge to main finishes with everything green, the final job opens a pull request in rutasnorte/manifests.
update-manifests:
needs: [image, integration]
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
steps:
- name: Clone the manifest repository
uses: actions/checkout@v4
with:
repository: rutasnorte/manifests
# A GitHub App token scoped to this repository,
# not a personal token with access to the whole organisation.
token: ${{ secrets.MANIFESTS_TOKEN }}
- name: Write the new digest into the dev overlay
run: |
cd overlays/dev
kustomize edit set image \
"bookings-api=${REGISTRY}/${IMAGE}@${{ needs.image.outputs.digest }}"
- name: Open a pull request and merge it automatically
uses: peter-evans/create-pull-request@v7
with:
token: ${{ secrets.MANIFESTS_TOKEN }}
branch: auto/bookings-api-${{ needs.image.outputs.tag }}
commit-message: "dev: bookings-api ${{ needs.image.outputs.tag }}"
title: "dev: bookings-api ${{ needs.image.outputs.tag }}"
body: |
Automatic update from `rutasnorte/bookings-api`.
- Commit: ${{ github.sha }}
- Digest: `${{ needs.image.outputs.digest }}`
- Trivy: no critical vulnerabilities
- Signature: verified (keyless)
- Integration tests on kind: passed
labels: automatic,devIn dev, that pull request merges itself because a repository rule allows it when the only change affects overlays/dev/kustomization.yaml. As soon as it merges, Argo CD detects the change and syncs. Between the developer's git push and the new pod running in rutas-norte-dev, about eleven minutes go by.
Why a pull request and not a direct commit: it leaves a reviewable trail, it lets branch protection rules apply in the same way, and it makes reverting a git revert with context rather than an orphaned commit from a bot.
- Promotion to
pre and to pro with human approval
pre and to pro with human approvalPromotion is changing the same digest in the overlay. Nothing is rebuilt: the artefact validated in dev is exactly the one that reaches production, byte for byte.
# Run by a person (or by a button that does exactly this)
cd manifests
DIGEST=$(yq '.images[] | select(.name=="bookings-api") | .digest' overlays/dev/kustomization.yaml)
git switch -c promotion/bookings-api-pre
cd overlays/pre
kustomize edit set image "bookings-api=registry.rutasnorte.example/rutasnorte/bookings-api@${DIGEST}"
git commit -am "pre: promote bookings-api ${DIGEST:0:19}"
gh pr create --title "pre: promote bookings-api" --body-file ../.github/templates/promotion.mdEach gate checks different things, and that is what makes promotion more than pressing a button:
| Gate | Who approves | What they must check first |
|---|---|---|
dev → pre |
A member of the development team |
Application deployed in dev with no restarts for 30 min; smoke tests green; no new alerts; migrations applied without error |
pre → pro |
A member of platform and one from product |
Full regression tests in pre; k6 load test with the May bank-holiday profile within thresholds; migrations tested against a restored copy of production; rollback plan written; change window open (no freeze in force) |
The pull request template for promotion to pro requires filling in:
## Promotion to production — bookings-api
- **Digest:** `sha256:...`
- **Changes included:** (links to the code pull requests)
- **Does it include a schema migration?** Yes / No — if yes, is it backwards compatible?
- **k6 result in pre:** p95 = ___ ms (threshold 300 ms), errors = ___ % (threshold 0.1 %)
- **Estimated risk:** low / medium / high — justification:
- **Rollback plan:** `git revert <sha>` + estimated time ___ min
- **Window:** is there a freeze in force? Yes / No
- **Approvals:** platform @____ · product @____The question about the migration's backwards compatibility is not rhetorical: it is what makes rollback possible. A migration that drops a column makes reverting the code impossible without restoring the database. In 11-04 we will see the expand-and-contract pattern that solves it.
- Ephemeral environments per pull request
Every pull request opened in rutasnorte/bookings-api gets a namespace of its own, with its database, its Ingress and its URL, and it disappears when the request is closed.
ephemeral-environment:
if: github.event_name == 'pull_request'
needs: image
runs-on: ubuntu-latest
environment:
name: pr-${{ github.event.number }}
url: https://pr-${{ github.event.number }}.dev.rutasnorte.example
steps:
- uses: actions/checkout@v4
- name: Authenticate to the development cluster via OIDC
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::111122223333:role/ci-rutasnorte-ephemeral
aws-region: eu-west-1
- run: aws eks update-kubeconfig --name rutasnorte-dev
- name: Deploy the environment
env:
NS: rutas-norte-pr-${{ github.event.number }}
run: |
kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -
# Labels that govern quotas, policies and automatic cleanup.
kubectl label namespace "$NS" --overwrite \
rutasnorte.example/ephemeral=true \
rutasnorte.example/pr=${{ github.event.number }} \
rutasnorte.example/created=$(date +%Y-%m-%d) \
pod-security.kubernetes.io/enforce=restricted
helm upgrade --install bookings-api ./chart \
--namespace "$NS" \
--values ./chart/values-ephemeral.yaml \
--set image.digest='${{ needs.image.outputs.digest }}' \
--set ingress.host=pr-${{ github.event.number }}.dev.rutasnorte.example \
--wait --timeout 8m
- name: Comment the URL on the pull request
uses: peter-evans/create-or-update-comment@v4
with:
issue-number: ${{ github.event.number }}
body: |
Test environment ready: https://pr-${{ github.event.number }}.dev.rutasnorte.example
Namespace: `rutas-norte-pr-${{ github.event.number }}`
cleanup-ephemeral:
if: github.event.action == 'closed'
runs-on: ubuntu-latest
steps:
- run: kubectl delete namespace "rutas-norte-pr-${{ github.event.number }}" --ignore-not-foundThis is the only point in the whole pipeline where kubectl runs against a cluster, and it is deliberately fenced in: only the development cluster, only namespaces with the rutas-norte-pr- prefix, and with a role that cannot touch anything else.
The safety net: a nightly CronJob deletes ephemeral namespaces older than seven days, because there are always pull requests left open and cleanup jobs that fail.
apiVersion: batch/v1
kind: CronJob
metadata:
name: cleanup-ephemeral-namespaces
namespace: platform
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: ephemeral-cleaner
restartPolicy: OnFailure
containers:
- name: cleanup
image: registry.rutasnorte.example/tools/kubectl:1.30
command:
- /bin/sh
- -c
- |
LIMIT=$(date -d '7 days ago' +%Y-%m-%d)
kubectl get ns -l rutasnorte.example/ephemeral=true \
-o jsonpath='{range .items[*]}{.metadata.name} {.metadata.labels.rutasnorte\.example/created}{"\n"}{end}' |
while read -r NS CREATED; do
[ "$CREATED" \< "$LIMIT" ] && kubectl delete ns "$NS"
doneWhy they are the best investment a team can make. It takes about two days to set this up properly, and it changes the entire dynamic of the work:
- Code review goes from reading a diff to using the feature. Product and design comment on something real, not on a screenshot.
- Integration errors are caught in the pull request, not in
devafter merging. - The queue for the shared environment disappears: five changes in parallel get five environments.
- The manifests get tested, not just the code. A badly parameterised chart fails here.
The cost has to be bounded: a strict quota per namespace (03-04), a single replica of everything, a small database with no persistence, and automatic deletion. At Rutas Norte, with an average of six live ephemeral environments, this comes to about 130 euros a month: less than the cost of half an hour's meeting to coordinate who is using the test environment.
- Credentials with no long-lived secrets
The goal is for no permanent credential to exist in the pipeline's secrets. Every access authenticates with an ephemeral token issued for that specific run.
sequenceDiagram participant W as Workflow participant G as GitHub OIDC issuer participant A as AWS STS participant K as EKS dev cluster W->>G: Requests an OIDC token (permissions.id-token) G-->>W: JWT with sub=repo:rutasnorte/bookings-api:ref:refs/heads/main W->>A: AssumeRoleWithWebIdentity(JWT) A->>A: Validates issuer, audience and the condition on sub A-->>W: Temporary credentials (1 h) W->>K: kubectl with those credentials K->>K: RBAC of the mapped role: only ns rutas-norte-pr-*
The role's trust policy is where the real security lives:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com" },
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" },
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:rutasnorte/bookings-api:pull_request"
}
}
}]
}The condition on sub is what stops another repository in the organisation, or any branch at all, from assuming this role. Writing it as repo:rutasnorte/*:* — a very frequent mistake — is equivalent to handing the role to the whole organisation.
And the RBAC on the cluster side, minimal (08-01):
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ci-ephemeral
rules:
# It can create and delete namespaces (needed for the ephemeral ones)...
- apiGroups: [""]
resources: [namespaces]
verbs: [get, list, create, delete, patch]
# ...and manage workloads inside them.
- apiGroups: ["", apps, networking.k8s.io, batch, autoscaling]
resources: ["*"]
verbs: [get, list, create, update, patch, delete]
# Explicitly NOT: nodes, clusterroles, clusterrolebindings,
# persistentvolumes or secrets in other namespaces.A comparison of the risk surface:
| Approach | What leaks if the runner is compromised | Expiry |
|---|---|---|
| Kubeconfig stored as a secret | Permanent access to the cluster | None |
| Cloud access key stored | Permanent access to the account | None |
| A developer's personal token | Everything that person can do | Months |
| Federated OIDC | A 1-hour token, bounded by the sub condition |
1 hour |
- Post-deployment verification and automatic rollback
Argo CD saying Synced and Healthy means the pods started, not that the application works. Post-deployment verification is the missing layer.
# Argo CD sync hook: runs AFTER syncing.
apiVersion: batch/v1
kind: Job
metadata:
name: smoke-bookings-api
namespace: rutas-norte-pro
annotations:
argocd.argoproj.io/hook: PostSync
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
backoffLimit: 2
activeDeadlineSeconds: 300
template:
spec:
restartPolicy: Never
containers:
- name: smoke
image: registry.rutasnorte.example/rutasnorte/smoke-tests@sha256:e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4
env:
- name: BASE_URL
value: https://api.rutasnorte.example
- name: MODE
value: read-only # in pro we do not create real bookingsIf the Job fails, Argo CD marks the sync as failed and, with this policy, rolls back on its own:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: bookings-api-pro
namespace: argocd
spec:
source:
repoURL: https://github.com/rutasnorte/manifests
path: overlays/pro
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: rutas-norte-pro
syncPolicy:
automated:
prune: true
selfHeal: true
retry:
limit: 2
backoff: { duration: 30s, factor: 2, maxDuration: 3m }
revisionHistoryLimit: 20Automatic rollback has two levels and it is worth telling them apart:
- Workload rollback:
kubectl rollout undoon the Deployment. It is immediate but leaves the cluster out of sync with Git, so Argo CD withselfHealwould put the bad version straight back. It only works as an emergency patch alongside step 2. - Rollback in Git:
git revertof the commit that changed the digest. This is the correct one, it takes as long as the sync takes (seconds if forced) and leaves the system consistent.
Automation at Rutas Norte: if the verification hook fails in pro, a workflow automatically opens and merges the git revert of the last commit on overlays/pro, and notifies the on-call channel. The mean time from failure to the previous version running is 3 minutes and 40 seconds.
# The automatic rollback flow, in essence
gh workflow run rollback.yml -f environment=pro -f reason="PostSync smoke failed"
# ...which does:
git revert --no-edit "$(git log -1 --format=%H -- overlays/pro)"
git push origin main
argocd app sync bookings-api-pro --prune
argocd app wait bookings-api-pro --health --timeout 300One important nuance: automatic rollback only kicks in if the promotion pull request declared that there is no incompatible schema migration. If there is one, the rollback has to be human, because reverting code on top of an already migrated schema can corrupt data.
- DORA metrics: Rutas Norte's numbers before and after
All this engineering has to be justified with numbers, or it is a hobby. The four DORA metrics are the standard way of measuring it.
| Metric | What it measures | How it is obtained at Rutas Norte |
|---|---|---|
| Deployment frequency | How often something reaches production | Commits on overlays/pro per week |
| Lead time for changes | From merged commit to being in production | Commit timestamp → Argo CD sync event |
| Change failure rate | What percentage of deployments causes a problem | Deployments followed by a rollback or incident / total |
| Time to restore service | How long it takes to recover after a failure | Incident opened → resolved |
10.1. The numbers
The starting point, in November 2025: images built manually from laptops, the latest tag, kubectl apply from the machine of whoever was deploying, deployments on Thursday afternoons "when it is due".
| Metric | Before (Nov 2025) | After (Jul 2026) | High-performance benchmark |
|---|---|---|---|
Deployment frequency to pro |
1 every 2 weeks | 4-6 per week | On demand |
Lead time (merge → pro) |
9 days | 4 h 20 min | < 1 day |
| Change failure rate | 31 % | 7 % | 0-15 % |
| Time to restore | 3 h 40 min | 12 min | < 1 hour |
10.2. What moved each number
- Frequency: rose thanks to the ephemeral environments and the automatic promotion to
dev. When deploying stops hurting, people deploy more. - Lead time: from nine days to a little over four hours. The bulk of those nine days was not technical: it was waiting for the shared environment to free up and for a slot in the Thursday window.
- Failure rate: fell from 31 % to 7 % mainly because of two things, the integration tests on kind and the fact that what is tested in
preis exactly the same digest that goes topro. Most of the earlier failures were differences between what was tested and what was deployed. - Time to restore: from nearly four hours to twelve minutes, thanks to automatic rollback and to rolling back being
git revertinstead of rebuilding an old image that nobody remembers how to build.
10.3. How to measure them without launching a project
# Deployment frequency to pro over the last 30 days
git log --since='30 days ago' --oneline -- overlays/pro | wc -l
# Lead time: difference between the code commit and the promotion commit
git log --since='30 days ago' --format='%H %ct' -- overlays/proThe first two come out of Git. The failure rate and the time to restore come out of the incident log, which we will see in 11-06. The classic trap is to measure only the two easy ones: speed without stability is not performance, it is risk piling up.
Common Mistakes and Tips
- Running
kubectl applyfrom the pipeline. It forces you to store production credentials in the CI system, makes state dependent on the order of runs and removes traceability. With GitOps the pipeline only writes to Git. - Deploying by moving tag instead of by digest. The tested artefact and the deployed one stop being the same, and the whole verification chain loses its meaning.
- Rebuilding the image when promoting from
pretopro. It is the most expensive conceptual mistake: you promote an artefact, not a commit. Rebuilding invalidates every test that came before. - Breaking the build on
HIGHor on unpatched vulnerabilities. The team learns to add exceptions as routine and the control stops catching anything. - An OIDC condition that is too broad (
repo:org/*:*). It grants the role to the whole organisation. The condition must pin the repository and, where possible, the branch or event type. - Ephemeral environments with no quota and no automatic deletion. They turn into the cost line nobody understands. Quota, a single replica, no persistence and nightly cleanup.
- Automatic rollback with incompatible migrations. It can corrupt data. Every migration must be declared backwards compatible or block automatic rollback.
- Tip: make the smoke tests run on kind and also after every real deployment. The same code, two moments: it validates before merging and verifies after deploying.
- Tip: publish the four DORA metrics on a visible dashboard. If only the speed ones are watched, quality degrades without anyone noticing until the incident.
Exercises
Exercise 1: finding the flaws in a pipeline
A Rutas Norte team proposes this simplified pipeline for a new service:
on: [push]
permissions: write-all
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: docker build -t registry.rutasnorte.example/rutasnorte/promotions:latest .
- run: docker push registry.rutasnorte.example/rutasnorte/promotions:latest
- run: echo "${{ secrets.KUBECONFIG_PRO }}" > /tmp/kc
- run: KUBECONFIG=/tmp/kc kubectl -n rutas-norte-pro apply -f k8s/
- run: KUBECONFIG=/tmp/kc kubectl -n rutas-norte-pro rollout restart deploy/promotionsIdentify at least five serious problems and propose a fix for each one.
Exercise 2: designing the promotion gate
bookings-api incorporates a change that adds the promo_code column to the bookings table and starts using it. Design the promotion sequence to production, stating: how many deployments it splits into, what each one contains, in what order the migration and the code are applied, and what makes rollback possible at each point.
Exercise 3: interpreting the DORA metrics
Three months after the improvements, Rutas Norte measures: deployment frequency 11 per week (up), lead time 2 h 10 min (down), change failure rate 22 % (up from 7 %) and time to restore 14 min (stable). Interpret the set as a whole, propose two hypotheses about the cause and say what you would measure to tell them apart.
Solutions
Solution 1. Problems and fixes:
| # | Problem | Fix |
|---|---|---|
| 1 | permissions: write-all |
Minimal permissions: contents: read, packages: write, id-token: write |
| 2 | The latest tag |
Tag by short SHA and deploy by digest |
| 3 | No tests, no lint, no scanning, no signing | Add a test job, Trivy on CRITICAL and keyless Cosign signing |
| 4 | Production kubeconfig stored as a secret | Remove it; the pipeline must not touch pro. Replace it with a pull request to the manifest repository |
| 5 | kubectl apply straight to pro on every push to any branch |
GitOps: write the digest into overlays/dev; pre and pro by approved promotion |
| 6 | rollout restart as the deployment mechanism |
Unnecessary if the digest changes; the digest change already triggers the deployment |
| 7 | on: [push] with no branch filter |
Restrict what publishes to main, and use pull_request to validate |
Solution 2. Three deployments, applying expand and contract:
- Deployment 1 (expand): a migration that adds
promo_codeas a nullable column defaulting to null. The deployed code ignores it entirely. Rollback possible: reverting the code breaks nothing, the column is surplus but harmless. - Deployment 2 (use): new code that writes and reads the column, with reads tolerant of nulls for the older rows. No schema changes. Rollback possible: the previous version still works because it ignores the column.
- Deployment 3 (contract), weeks later and only if needed: add a
NOT NULLconstraint, indexes, or drop the old columns that have been replaced. From here on, rolling back to version 1 is no longer safe, which is why it is separated in time and done only when confidence is complete.
General order: the migration is applied before the code that needs it, and always in such a way that the previous version of the code keeps working with the new schema. That is what makes RollingUpdate (two versions coexisting) and rollback possible.
Solution 3. Interpretation: speed has improved but stability has degraded seriously; almost one deployment in four causes a problem. The stable time to restore indicates that automatic rollback still works well, which cushions the impact but does not remove it. Taken together, this suggests deployments are going out faster than verification can validate them.
Hypothesis A: the quality gates have been relaxed (fewer regression tests in pre, approvals given as routine, k6 thresholds not reviewed). Hypothesis B: the increase in frequency comes from smaller but more numerous changes in areas of the system poorly covered by tests, or from a new component with less maturity.
To tell them apart: (a) break the failure rate down by component — if it concentrates in one, it is hypothesis B; if it is spread out, hypothesis A; (b) measure the coverage and duration of the pre tests over the last three months and check whether any gate has been switched off; (c) review the failed promotion pull requests and classify the root cause of each. Likely action: not to reduce the frequency, but to reinforce the verification that was skipped, and to consider the canary deployment from 11-04 so that a defective change only affects a small percentage of users.
Conclusion
We have walked through the complete Rutas Norte pipeline, from git push to production. We saw why integration and delivery are separated, and why with GitOps the pipeline never runs kubectl apply: it produces a signed, verified digest and writes that digest into the manifest repository, leaving Argo CD to do the rest. We went through the complete integration file — tests, multi-stage build with caching, tagging by commit, Trivy, keyless signing, digest — the real integration tests against a kind cluster brought up inside the pipeline itself, the promotion through environments with gates that check different things, the ephemeral environments per pull request as the best investment a team can make, the federated credentials with no long-lived secrets, the post-deployment verification with automatic rollback in under four minutes, and the DORA metrics that show all of this achieved something real.
The central idea: you promote an artefact, not a commit. The same digest that passed the tests in dev and the load test in pre is the one serving customers in pro, with nothing rebuilt along the way.
Even with all of this, every deployment to production is still a leap: the new version replaces the previous one in every pod and, if something slipped past the tests, every user discovers it at once. In the next lesson, Deployment Strategies: Blue-Green and Canary, we will see how to eliminate that leap: deploying without committing, validating with real traffic, exposing the new version to a small percentage of users and letting Prometheus decide automatically whether to promote or abort.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
