In 08-02 TechCorp's system was completed as code: six services, a gateway, a library and their tests. This lesson takes it to a cluster and operates it. The first half answers "how do I bring all of this up from scratch?": the complete techcorp/platform repository, the local compose.yaml with the six services and all the infrastructure, the startup order on a fresh cluster, the script that applies it, the migration Jobs and the smoke test with a Keycloak token. The second half answers "and what do I do every day?": a normal end-to-end deployment of orders-service, the Black Friday campaign, an incident in a DLQ step by step, a secret rotation, the Node upgrade and a contract evolution applied to the whole system, plus a table of frequent operations and the approximate monthly cost.

We do not re-explain any YAML from 05-02 or any code: when a Deployment, a ServiceMonitor or an ExternalSecret shows up, we will point to the lesson where it was written and show only what changes when going from one service to six.

1. The complete techcorp/platform repository
2. compose.yaml: the whole system on a laptop
3. Startup order on a fresh cluster
4. scripts/deploy-all.sh and the migration Jobs
5. Verification and smoke test
6. Operations: a normal deployment of orders-service
7. Operations: the Black Friday campaign
8. Operations: an incident in payments.stock.dlq, step by step
9. Operations: rotating a secret, upgrading Node and evolving a contract
10. Quick runbook of frequent operations
11. Approximate monthly cost and how to reduce it

It is the Platform team's repository and the source of truth for production (Argo CD reads it, 05-03 §6). Everything we have been leaving in it throughout the course, organized:

Two decisions hold the tree together: one base per service, one overlay per environment (05-02 §12: Inventory's base is Orders' with different names, port 3006 and its ScaledObject instead of an HPA), and third parties through Helm with values versioned here, never installed by hand. When the Payments team needs a new Secret, it opens a PR against this repository; nobody runs kubectl create secret in production.

The compose.yaml from 05-01 §8 had PostgreSQL, MongoDB, RabbitMQ, the seed, the Orders migrations, Catalog, Orders, the Customers stub and the gateway. The additions for the complete system, summarized (the new services follow exactly the orders-service pattern from 05-01: image + build, the variables from the config.js table in 08-02, depends_on with conditions, stop_grace_period: 15s):

With this, docker compose up -d --wait brings up some twenty containers (seventeen running plus the one-shot tasks) in about 90 seconds on an ordinary laptop, and the four E2E tests from 08-02 §9 pass against http://localhost:8080 with a token obtained from http://localhost:8180. It is the environment with which the learner can reproduce the whole course (08-04).

Whether it is a kind cluster (platform/local/kind.yaml, 05-02 §4) or a managed one, the order matters: each layer depends on the previous one, and several pieces (ESO, cert-manager, KEDA, the Prometheus operator) install CRDs that later manifests use.

About step 8: the "saga order" is not strictly necessary —services start even if a collaborator is missing (/health/ready does not check Catalog or Customers, 04-04 §9) and durable queues hold messages— but deploying the consumers first and Orders and the gateway last prevents the first orders of the smoke test from waiting for someone to declare inventory.orders.

In dev and in a test cluster everything is applied with a script; in production steps 8-9 are done by Argo CD (app-of-apps.yaml), and the script is only used for the infrastructure of steps 1-7 (or Terraform/Helmfile, if Platform prefers).

Every service with a database carries its migrations Job in base/ (05-02 §6) with argocd.argoproj.io/hook: PreSync for production (05-03 §6): orders-service-migrations, inventory-service-migrations, payments-service-migrations, notifications-service-migrations, customers-service-migrations (Catalog has catalog-service-seed only in dev; Notifications does have a DB for deliveries). All of them run node scripts/migrate.js from the service image and are idempotent, which is why the delete + apply is safe. And all are expand-only (05-04 §3): the script never needs to "wait until no old pod remains" to apply a migration.

The smoke test is the E2E from 04-05 done by hand through the Ingress, with a real token. scripts/token-keycloak.sh obtains one for the test user with the password grant (enabled only on the e2e-tests client of the dev/staging realm; in production that client does not exist, 07-01 ex. 1):

And the three observability checks, which are the reason modules 6 and 7 were built: (1) in Jaeger (kubectl -n observability port-forward svc/jaeger-query 16686), search service=gateway, tag orderId=$ORD: one trace with the spans of the gateway, Orders, Customers, Catalog, PostgreSQL, and —linked via links from the outbox traceparent (06-02 §6)— Inventory, Payments (with the payment provider span) and Notifications; (2) in Grafana, "Orders saga" dashboard: orders_created_total +1, saga_duration_seconds with one observation of ~3 s, outbox_pending at 0 in the four services with an outbox; (3) in Loki, {namespace="techcorp"} | json | orderId="ord-…" returns the lines from the six services sorted by time, with the same requestId in the synchronous part.

If the status gets stuck at STOCK_RESERVED, the table in section 10 says where to look (almost always: payments.stock with no consumer, or a wrong PAYMENT_PROVIDER_URL in the Payments ConfigMap).

Luis merges a PR that adds customer cancellation (exercise 1 of 08-02). What happens, with each piece in its lesson:

Total: about 50 minutes of wall-clock time, of which about 12 are machine time; the rest is human review and canary observation, and they are deliberate. Compared with the Thursday night from 01-05 (a two-hour deployment, four rollbacks out of twelve), this happens at 11 in the morning on a Tuesday, eleven times a day across the set of services, and if the burn rate rises during the canary, canary-weight: 0 leaves 100% on the stable one in seconds: an MTTR of minutes without anyone typing a kubectl by hand.

The campaign is prepared with a checklist that Platform and the four teams go over three weeks ahead (the first version was written for the 2025 Black Friday, 08-01 §3; today it is a templated issue in platform):

What you see on D-day in a good year (like 2025): Catalog between 6 and 16 replicas following the browsing curve; Inventory at 4-7 by inventory.orders length; saga_duration_seconds p95 at 4 s (the payment provider is slower too); error budget consumed on the day: 6% of the monthly. And what the checklist prevents: a manual kubectl scale that Argo CD reverts three minutes later (05-03 §6, selfHeal: true), which is why campaign scaling also goes through a PR.

A Tuesday in September 2026, 15:20. Different from INC-2031 (poison message due to a bug): this time the cause is external and the tools from 06-03/06-05 already exist. Follow the timeline as if you were the on-call person:

1. 15:31 — Alert DlqHasMessages{queue="payments.stock.dlq"} (warning: 10 min with messages, 06-05 §12) reaches the Payments team channel; it wakes nobody up (it is not critical), but it is business hours and on-call acknowledges it and opens #inc-2047-payments-dlq.
2. 15:33 — Runbook common/dlq: "how many, since when, what type?". kubectl -n infra exec rabbitmq-0 -- rabbitmqctl list_queues name messages | grep dlq → payments.stock.dlq 14. In the RabbitMQ console, all 14 are stock.reserved with x-attempts: 5: they exhausted their retries, they are not poison.
3. 15:36 — Loki: {app="payments-service"} | json | level="error" | line_format "{{.eventId}} {{.err.code}} {{.message}}" → DEPENDENCY_UNAVAILABLE payment-provider: timeout since 15:12; CircuitOpen{dependency="payment-provider"} has also been in warning since 15:15 (06-03 §4). Jaeger: the POST payment-provider/charges span with error=true and exactly 5,000 ms: the payment provider is not responding, not declining.
4. 15:38 — Impact: "Orders saga" dashboard: 41 orders in STOCK_RESERVED for more than 5 minutes; the watchdog will start cancelling them for PAYMENT_TIMEOUT at 10 (06-03 §9). Severity SEV2 (part of the flow broken, no workaround for the customer); a notice goes to #platform-status: "since 15:12 charges are not completing due to a problem at the payment provider; orders are being held; next update at 16:00".
5. 15:40 — Stabilize before understanding: the payment provider's status page confirms an incident. Payments' decision: temporarily raise the watchdog limit from 10 to 30 minutes (Orders' WATCHDOG_LIMIT_MIN ConfigMap + rollout restart; express PR in platform with the incident label, so that Argo does not revert it) and thus not cancel 41 orders with reserved stock because of someone else's 20-minute outage. Reservations expire at 15 min via expires_at… unless RESERVATION_TTL_S is raised too; it is raised to 2,400 s the same way. Two configuration changes, zero code.
6. 15:52 — The payment provider recovers. The breaker goes half-open and closes; the messages in payments.stock.retry (the ones that still had attempts left) get charged on their own within 2 minutes. The 14 in the DLQ remain.
7. 15:55 — Reprocess: kubectl -n techcorp run reprocess --rm -it --image=ghcr.io/techcorp/payments-service:1.4.2 --env-from=secret/payments-rabbitmq -- node scripts/reprocessDlq.js --queue payments.stock --max 50 (06-03 §8): 14 messages go back to the queue with x-attempts: 0 and x-reprocess; chargeOrder finds them IN_PROGRESS (step 1 of 08-02 §4) and looks up by idempotency key before charging: 3 had already been charged on the first attempt (the payment provider charged and the timeout fired before the response) and are marked CAPTURED without charging twice; 11 get charged now. payment.confirmed ×14, sagas closed.
8. 16:02 — Dashboard: 0 old orders in STOCK_RESERVED; the two configuration changes are reverted (PR back); the incident is closed. Duration: 50 min from the first failure, 31 from the alert; 0 orders cancelled, 0 duplicate charges.
9. Brief postmortem (SEV2, 06-05 §10): external cause; everything designed worked (retries, breaker, DLQ, alert, runbook, payment provider idempotency); actions: (a) the DlqHasMessages alert took 10 min by design, but CircuitOpen fired at 3: add the step "raise the watchdog" to the CircuitOpen{payment-provider} runbook; (b) make WATCHDOG_LIMIT_MIN and RESERVATION_TTL_S hot-reloadable flags (04-03 §7) to avoid restarts; (c) ask the payment provider about its SLA. And the standing question: "which alert would have caught it earlier?" —CircuitOpen already did; what was missing was the link between that alert and the action.

Compare with INC-2031 (40 min, 61 orders affected, 38 cancelled): same symptom ("the saga is not advancing"), half the impact and no destructive action, because the system already had deferred retries, DLQ, symptom-based alerts, a runbook and a script. It is exactly what 06-05 was written for.

Rotating orders-db. It is the procedure from 07-04 §4, run on the first Monday of each semester by Platform with the owning team: (1) new password for svc_orders in Vault (techcorp/prod/orders/db), with the svc_orders_b user so both are valid; (2) kubectl -n techcorp annotate externalsecret orders-db force-sync=$(date +%s) → the Secret changes; (3) kubectl -n techcorp rollout restart deploy/orders-service (rolling maxUnavailable: 0, 05-04) and rollout status; (4) orders-migrations is not needed; (5) revoke the old one and watch for password authentication failed in Loki for an hour. Fifteen minutes, no maintenance window; it is repeated for the *-db and *-rabbitmq of the six services on the same morning with a loop in scripts/rotate-secrets.sh. The audit_log of who rotated what is the PR's git log and Vault's audit log.

Node 20 → 22. Node 20 reaches end of maintenance in April 2026; the upgrade is done once in the template and propagates through Luis's rule: (1) node-service-template: FROM node:22-alpine in both stages of the Dockerfile (05-01 §3), engines.node: ">=22", @types/lint; (2) node-service-ci.yml@v2: actions/setup-node with node-version: 22 and a temporary [20, 22] matrix so that each service checks both before switching; (3) @techcorp/common-http is published and tested on 22 (05-03 §10); (4) each team, whenever it wants within a deadline (one month), opens a PR in its service with two lines: uses: …/node-service-ci.yml@v2 and the FROM; CI, Testcontainers, pacts and staging validate; production with the usual strategy (canary in Orders and Payments, rolling elsewhere); (5) after the month, @v1 is retired. Six two-line PRs instead of a coordinated migration; and if Notifications is a week late, nobody else waits.

Products v2 with price: { amount, currency }. It is the breaking change from 03-06 §10, now executed end to end:

No step cuts anything: at every moment two versions of the contract, of the event and of the schema coexist, and each team deploys when it wants within the deadline. It is the same discipline as in section 6, applied to a change that in the monolith would have been "search for price across the whole repository and cross your fingers on Thursday".

Fictional, indicative figures for a managed production cluster with 3,000 orders/day (no campaign), rounded up. They serve to give an order of magnitude and for the FinOps argument in 08-04:

Compared with the monolith (three large servers + PostgreSQL + ten servers for one month a year ≈ €1,900/month on average), it is 45% more expensive in pure infrastructure… and 30% cheaper during a campaign, and it does not include what is saved in deployment hours and incidents (08-01 §10). How to reduce it, in order of return: (1) real requests measured with Prometheus (06-04 §4) and node right-sizing: most TechCorp services request 250 m of CPU and use 60 m; (2) shut down staging outside working hours (already done) and use KEDA's minReplicaCount: 0 for sporadic consumers such as analytics; (3) trace sampling at 10% and 30-day log retention (already), sample_limit and a quarterly cardinality review (08-01 §11); (4) reserve base node capacity for a year (−30%); (5) review whether Redis, under real load, pays off versus the gateway's HTTP cache; (6) do not save on backups, TLS or replicas of the Orders and Payments databases.

• Installing the infrastructure by hand and the services through GitOps. Three months later nobody knows which RabbitMQ version is running or with which values. Everything in k8s/infra/ and applied by script or by Argo, even if it is Helm.
• Skipping the CRD order. An ExternalSecret applied before ESO or a ServiceMonitor before the operator fails with "no matches for kind": the script waits with --wait for each operator before continuing.
• A smoke test without a token or with the admin token. It must use a user with the customer role and their customerId: it is the only way to check the owner rule and the 404 for someone else's order.
• Scaling by hand in production with Argo CD in selfHeal. It reverts it within minutes and, worse, in the middle of a campaign. Through a PR, always.
• Reprocessing the DLQ before understanding the cause. If the message is poison, it goes back to the DLQ and burns five retries; if the dependency is still down, same thing. First Loki/Jaeger, then the script.
• Rotating a secret without rollout restart and believing it worked because kubectl get secret shows the new one. The old pods keep the old password until they restart (or until it is read as a file).
• Tip: run deploy-all.sh dev on a clean kind cluster once a month. If it takes more than 15 minutes or fails at some step, something in the platform tree has gone stale; better to find out on a Tuesday than during disaster recovery.

The production cluster is lost entirely (region down). With the platform repository, the backups of the managed databases and the images in ghcr.io, write the recovery sequence on a new cluster in another region, indicating which steps from section 3 change, what must be restored before what, what data could be lost (think about the outbox and the queues) and a time estimate. What would need to be prepared in advance for it to take an hour instead of a day?

During the 10% canary of orders-service 1.6.0 the burn rate is perfect, but when moving to 100% saga_duration_seconds p95 rises from 3 to 40 s. Explain what kind of failure a canary by HTTP traffic weight does not detect in a service that also consumes events, how you would have detected it during the canary (specific metrics and labels), and what you would change in the procedure from section 6.

Marta asks to bring the monthly bill down 25% without touching the availability of Orders or Payments. Using the table in section 11, propose a concrete list of measures with their estimated savings and their risk, and say which ones you would not accept even if they save money.

Exercise 1. Sequence: steps 1-2 unchanged (namespaces, operators) on the new cluster; step 3 changes: restore the managed databases from backup (PostgreSQL for Orders and Payments first, with point-in-time recovery to the closest instant; the catalog MongoDB; RabbitMQ is not restored: it is created empty with definitions.json, and in-flight messages are lost); step 4 Keycloak from its restored DB; 5-6 unchanged; 7 the ExternalSecrets point at the same Vault (which must be outside the region or replicated); 8-9 unchanged, switching the api.techcorp.example DNS to the new load balancer and waiting for the certificate. Data at risk: the events that were in RabbitMQ unconsumed (durable queues of a lost cluster) and the seconds between the last backup and the outage. The outbox is the partial salvation: on startup, the relays of the four services republish everything with published_at IS NULL, so events generated but not published are recovered; those already published and not consumed are lost → orders in STOCK_RESERVED or PENDING that do not advance are closed by the watchdog (PAYMENT_TIMEOUT) and the reservation reconciliation (06-03 §9), and chargeOrder looks up the payment provider before charging (IN_PROGRESS), so there are no double charges. Time: with everything at hand and rehearsed, 2-4 hours; without practice, a day. For one hour: a "warm" secondary cluster with the infrastructure of steps 1-7 already applied and Argo CD pointing at the same repository (only the target changes), replicas of the managed DBs in the other region, multi-region Vault, and a quarterly recovery drill (the tip from the previous section).

Exercise 2. The Ingress weight-based canary splits HTTP requests, but the canary replicas also consume orders.saga on equal terms with the stable ones (competing consumers, 06-04 §9): at 10% of HTTP traffic, the canary may already be processing 33% of the events (1 replica out of 3), and if its consumer is slow (an unindexed query on orders added in 1.6.0), the effect is diluted among the stable ones and shows up "all at once" at 100%. Detection during the canary: look at metrics labeled by version (the service.version label from 06-02, which the common-http metrics include): histogram_quantile(0.95, sum by (le, version) (rate(saga_duration_seconds_bucket[5m]))), the duration of processOnce per version, pg_stat_statements or the latency of the consumer's queries, and x-attempts/un-acked messages on orders.saga per pod. Procedure change: add to section 6 a "canary vs stable" panel that also compares consumer and database metrics by version, not just HTTP RED; and, for services with consumers, lengthen the canary or do a per-replica canary with a test queue (or a mesh, when it arrives) —it is one of the signals 02-05 set for reconsidering saga orchestration and that 05-05 left as an argument in favor of the mesh.

Exercise 3. Measures: (1) real requests and nodes from 6 to 4 outside campaigns (autoscaling continues): −€250, low risk if the PDBs and the HPA are right; (2) staging at 1/4 and only during working hours with on-demand startup: −€200, low risk (the cd.yml E2E takes 5 more minutes if it is off); (3) one-year reserved capacity for 4 base nodes: −€180, commitment risk; (4) traces at 5% with error-tail sampling at 100%, Catalog logs at info (today debug in a forgotten overlay), 15 days of logs: −€80, medium risk for debugging; (5) MongoDB down one instance size with the Redis cache (which already absorbs 85%): −€60, low risk, measure with k6; (6) KEDA to zero for analytics and bff-mobile at night: −€30. Total ≈ −€800 (29%). I would not accept: removing the replica or the backups of the Orders/Payments PostgreSQL, moving RabbitMQ to a single node, removing internal TLS from RabbitMQ/DBs, or reducing Orders/Payments minReplicas to 1 (a single pod breaks the zero-downtime rolling from 05-04 and the PDB): they save €100-200 and put the 99.9% SLO and the security from 07-02 at stake.

TechCorp's system is no longer just code: it lives in a cluster and is operated with written procedures. We have walked through the complete techcorp/platform repository (bases and overlays per service, k8s/network/, k8s/infra/ with the Helm values for RabbitMQ, PostgreSQL, MongoDB, Redis, Keycloak and observability, SLOs, Argo CD, dashboards, runbooks, E2E and load tests, the reusable workflow and SECURITY.md), the compose.yaml that brings up the six services with Keycloak, Jaeger, Prometheus, Grafana and Loki on a laptop, the startup order of a fresh cluster (namespaces → operators and CRDs → data and messaging → identity → observability → network → secrets → services in saga order → Ingress) captured in deploy-all.sh with its idempotent migration Jobs, and the verification with a real smoke test —a Keycloak token, an order through the Ingress that reaches CONFIRMED, its trace in Jaeger, its mark on the "Orders saga" dashboard and its logs in Loki—. And we have operated: an orders-service deployment from PR to 100% through CI, pacts, signed image, staging, can-i-deploy, human review, Argo CD and an observed canary; the Black Friday checklist with campaign overlay, KEDA, error budget and freeze; incident INC-2047 in payments.stock.dlq resolved with an alert, a runbook, Loki, Jaeger, two configuration changes and reprocessDlq.js without a single duplicate charge; the rotation of orders-db with ESO, the Node 22 upgrade propagated through the template and the reusable workflow, and products v2 with price {amount, currency} applied end to end with expand/contract; plus the quick runbook and a monthly bill with its levers.

With this, TechCorp's story is complete: monolith, migration, implementation, deployment and operations. What remains is the most valuable part of a case study, which is distilling it: what we did, what went wrong and what we would do differently in each dimension, which antipatterns we learned to recognize, a consolidated list of best practices with their reference lesson, what TechCorp would do in its phase 2 and how you can apply all of this to your own context. It is the last lesson of the course.