The previous lesson ended with a tally: six services, three data stores, Kafka, Redis, MinIO, Kong, Keycloak, Vault, Prometheus, Loki, Tempo, Patroni, etcd. Dozens of containers, hundreds of parameters, and a docker-compose.yml that no longer fits on one screen. As long as all of that is deployed the way the monolith of 01-06 was (on Tuesdays and Thursdays, with an ssh session, a git pull and a list of steps in a document that somebody follows by hand), the platform has two problems that no resilience pattern will fix: it is not reproducible (nobody knows exactly what is on inventory-2 or why it differs from inventory-1) and it depends on one person not making a mistake on a Thursday at 18:00. This lesson is about taking that person off the critical path: first by describing the infrastructure as code (Ansible to configure machines; immutable images for the services), and then by delegating to an orchestrator, Kubernetes, the decisions that somebody makes by hand today: where each replica runs, how many are needed, what to do when one dies, and how to take a new version to production without Anna noticing. The tests that give you the confidence to automate are lesson 07-06; managed cloud services are 08-03.

1. Why automate
2. Infrastructure as code: configuration versus provisioning
3. Ansible: inventory, playbooks, roles and idempotency
4. Immutability: images, registry and per-commit tags
5. Kubernetes: why an orchestrator and how it is built
6. Essential objects
7. Kilometre Zero's manifests: k8s/
8. Deployment strategies: rolling, blue/green, canary
9. Namespaces, RBAC, service mesh and operators
10. GitOps and CI/CD
11. docker-compose versus Kubernetes
12. Common mistakes and tips
13. Exercises and solutions
14. Conclusion

Three reasons, and all three have already appeared in the course without being named:

• Reproducibility. In 07-03 we said that redundancy only protects against independent failures, and that three "identical" replicas configured by hand never quite are. If inventory-2 was created by copying inventory-1 and editing "what was needed", there are differences nobody remembers, and one of them will be the cause of an incident. Automating means the description is the only source: a node can be destroyed and recreated identically.
• Scale. With 3 replicas you can do it by hand; with 30, during Grape Harvest Week, you cannot. And "by hand" includes deciding how many are needed: autoscaling is the automation of a decision, not just of a task.
• Human error. The DELETE of 07-03, the Saturday deployment of 07-01 with 8% errors, the certificate that expired on a Sunday: what the three have in common is that somebody did something by hand, or failed to. Automation does not eliminate mistakes (a wrong script is wrong 30 times on 30 nodes), but it makes them reviewable beforehand (code in a pull request), testable (07-06) and reversible (rollout undo).

The monolith of 01-06 was deployed inside a maintenance window; the goal here is for a deployment of orders to be such a routine event that it happens several times a day without anyone noticing, Anna included.

Infrastructure as code (IaC) means treating servers, networks, configuration and deployments as files in a repository: versioned, reviewed and applied by a tool. Two families that should not be confused:

Ansible is the one this lesson covers, because it configures what already exists (Kilometre Zero's physical or virtual machines) and because its central concept, task idempotency, is the same one that governs the retries of 02-05 and 07-04. Terraform comes in 08-03, once the infrastructure becomes a cloud API.

Ansible needs no agent: it connects over SSH to every machine in the inventory, runs modules (small Python programs that know how to install packages, write files, manage services) and reports whether anything changed. A playbook is a YAML file saying which tasks to apply to which group of machines; a role is a packaged, reusable playbook (tasks, templates, default variables, handlers).

The Kafka playbook delegates to a role and passes in the cluster variables:

And the role, with carefully idempotent tasks:

Idempotency is in every detail: creates: avoids repeated downloads and, above all, avoids re-formatting a broker that holds data (a kafka-storage.sh format on a production broker destroys it); template only notifies the handler if the content changed, so running the playbook ten times in a row restarts Kafka zero times; serial: 1 with a wait_for at the end makes a configuration rollout walk the brokers one at a time, honouring the ISRs of 07-03. Running ansible-playbook -i inventory.ini kafka.yml --check --diff shows what would change without changing anything: the prior review that the Tuesday ssh never had. The same structure (a cassandra role with nodetool drain before restarting, a patroni role with patronictl switchover if the node is the leader) applies to the rest of the stateful machines.

Ansible configures machines that change. For the services under km0/ we take the opposite path: immutable infrastructure. A service is not "updated": a new container image is built with everything inside (interpreter, pinned dependencies, code, default configuration), pushed to a registry, and deployed by replacing the old containers with new ones. Nobody ever sshes into a container to fix something; if something is wrong, another image is built.

The image tag is the key to reproducibility: registry.km0.internal/km0/orders:1.14.2-3f9a1c7 (semantic version + short commit hash). Never latest: latest is a moving pointer, and "which version is in production" must have an exact answer. The same image travels through tests, staging and production; what changes between environments is the injected configuration (environment variables, ConfigMap, Secret), not the image. That is the "build once, deploy many" principle.

With immutable images, what remains is deciding where each container runs, how many there are, what happens when one dies or a node goes down, how they find each other and how they are replaced by the new version. In docker-compose a person decides all of that, on a single node. An orchestrator decides it continuously, across a cluster:

The idea that explains everything is the reconciliation loop: the user declares the desired state in the API ("3 replicas of orders:1.14.2"), etcd stores it (the same etcd as in 03-03: Kubernetes is, at bottom, a set of controllers on top of a consensus store), and every controller continuously compares the desired state with the actual one and acts to bring them closer: if there are 2 Pods and 3 are wanted, it creates one; if node 1 disappears, its Pods are recreated elsewhere. There is no "deploy command"; there is "change the desired state" and wait for the controllers to converge. It is the same philosophy as Ansible's idempotency, but running non-stop.

Every block answers something seen earlier: the probes are those of 07-03 with their parameters; requests feeds the scheduler and the HPA; limits is a per-Pod resource bulkhead (07-04); maxUnavailable: 0 guarantees that a rollout never reduces capacity; topologySpreadConstraints spreads across zones; preStop + terminationGracePeriodSeconds are the graceful shutdown (a 400 ms request is not cut off halfway). And envFrom: secretRef gives the Pod only the bare minimum to introduce itself to Vault, which is what hands out dynamic credentials and certificates.

Kong (inside the cluster) routes /api/v1/orders to http://orders.km0-prod.svc:8000. When a Pod fails readiness, it disappears from the Service's endpoints within seconds; when a new version is rolled out, the new Pods only join once they are ready. Nobody edits an IP.

The HPA takes the maximum of what the metrics ask for: if CPU says 5 replicas and lag says 8, it scales to 8. The external metric comes from the kafka_exporter of 07-01 through prometheus-adapter, which publishes it in the Kubernetes metrics API. The stabilisation windows are the equivalent of the alerts' for:: scale up fast, scale down slowly.

A StatefulSet provides what a database needs and a Deployment does not: stable names (inventory-db-0 is always the same one, with the same disk), ordered start-up and shutdown, and one volume per replica. In production, however, the usual practice is to delegate to an operator (section 9).

The Ingress only carries traffic as far as Kong; everything from 06-05 (JWT, rate limiting, OpenAPI, X-Request-Id) still happens in Kong. With cert-manager, the certificate that used to expire on a Sunday now renews itself, and the CertificateExpiringSoon alert of 07-01 keeps watch that it does.

A canary with plain Kubernetes is done with two Deployments behind the same Service; the split is proportional to the number of Pods (1 canary out of 10 = 10%):

For fine-grained weights (1%) independent of the number of Pods, you use the capabilities of the mesh or the ingress (Istio VirtualService with weight: 1/99; Kong with weighted upstreams) and tools such as Argo Rollouts or Flagger that automate the analysis: they query Prometheus for the error rate and the p99 of the canary version (the version label on the metrics of 07-01) and advance or revert on their own according to the SLO. On the Saturday of 07-01, with a 10% canary analysed automatically, the 8% error rate would have affected less than 1% of orders for two minutes.

Namespaces separate environments and teams within the same cluster: km0-prod, km0-staging, observability, mesh. Each with a ResourceQuota (total CPU and memory) and a LimitRange (defaults for Pods without resources), and with Kubernetes RBAC (distinct from the application RBAC of 06-01, same model): who may do what on which objects.

Pods have an identity too (ServiceAccount), and with it they authenticate to Vault (06-04) and to the API if they need to; the principle is the same least privilege.

Service mesh. In 07-04 we saw Envoy as a sidecar with timeouts, retries and outlier detection; in 06-04, mTLS between services with Vault certificates managed by each service. A service mesh (Istio, Linkerd) injects that sidecar automatically into every Pod in the namespace and configures it from Kubernetes objects: mandatory mTLS with automatic certificate rotation (PeerAuthentication: STRICT), service-to-service authorization policies, per-route timeouts and retries (VirtualService, DestinationRule with outlierDetection), and uniform RED telemetry for Prometheus without instrumenting anything. It is the way to get 06-04 and 07-04 without code for services that are not Python (or so as not to depend on every team doing it right), in exchange for 1-2 ms of latency, memory per sidecar and one more piece to operate. The architectural decision of when it is worth it is taken up again in 08-01.

Operators. An operator is a controller that knows one specific system: it knows how to fail over PostgreSQL, run a nodetool drain before restarting a Cassandra node, or reassign Kafka partitions when a broker is added. It is installed in the cluster and spoken to through its own objects (kind: Kafka, kind: PostgresCluster). For Kilometre Zero: Strimzi for Kafka, K8ssandra for Cassandra, CloudNativePG or the Zalando operator (Patroni) for PostgreSQL. With them, the StatefulSet of section 7 is replaced by a ten-line declaration and the operator takes care of what 07-03 did by hand (and of the backups to MinIO). Here it is enough to know that they exist and that they are the recommended way to run stateful systems on Kubernetes.

With manifests in k8s/, the last piece is who applies them and when. GitOps is the answer: the Git repository is the single source of truth for the desired state; nobody runs kubectl apply by hand in production; an agent in the cluster (Argo CD or Flux) watches the repository and continuously reconciles the cluster with whatever is on the main branch. Deploying is merging a pull request that changes the image tag; rolling back is reverting the commit; auditing who deployed what is git log. And a manual change in the cluster ("drift") is detected and undone.

Two repositories: the code one (which produces images) and the deployment one (k8s/, which Argo CD watches). That way, "which version is in production" is one line in a YAML file with Git history, and the pipeline can promote to staging automatically and to prod with an approval. Image signing (cosign) and an admission policy that only allows signed images close the loop with 06-05: nothing runs in production that has not gone through the pipeline.

The practical rule: compose for developing (the docker-compose.yml in km0/ is still how the platform is brought up on a laptop), Kubernetes for production from the moment you need more than one node or zero-downtime deployments. Jumping to Kubernetes with a team of two and one service is paying the complexity without collecting the benefit; staying on compose with six services, three zones and daily deployments is Tuesdays and Thursdays under another name.

• Ansible tasks that are not idempotent. A command: without creates: that formats a broker on every run. Use declarative modules; for command/shell, always creates:/removes: or changed_when:; test with --check --diff.
• Restarting all the stateful nodes at once. serial: 1 and a health wait between nodes; for Kafka and Cassandra, it is the difference between maintenance and an incident.
• image: orders:latest. Nobody knows what is running in production and a rollout undo may "go back" to the same image. Version + commit hash, always.
• Deployments without resources. Without requests the scheduler places blindly and the CPU-based HPA does not work; without limits a Pod with a memory leak takes the node down with it. Both, measured with the metrics of 07-01.
• Liveness probes that check dependencies. Already in 07-03: cascading restarts. Minimal liveness, readiness with dependencies, generous startup.
• A high maxUnavailable "to deploy faster". It reduces capacity during the rollout, exactly when the new version may be slower. maxUnavailable: 0, maxSurge: 1.
• kubectl apply by hand in production. History is lost and drift appears. GitOps: everything goes through a PR; Argo CD/Flux apply it.
• Scaling down as fast as scaling up. The HPA flaps and every scale-down cuts capacity in a spike that comes back. A long stabilisation window on scaleDown.
• Secrets in ConfigMaps or in the repository. A Kubernetes Secret is just base64; the truly sensitive material is delivered by Vault at runtime (06-04); in Git, never.
• Kubernetes for its own sake. A cluster for one service with two people is a burden with no benefit. Compose until the reasons (nodes, scaling, zero-downtime deployments) are real.

Exercise 1. Jordan runs ansible-playbook -i inventory.ini kafka.yml a second time without having changed anything, and sees changed=2 on kafka-2: the tasks "Broker certificate and key from Vault" and "Write the broker keystore" show as changed, and Kafka has restarted on kafka-2. (a) What is going on, and why did it not happen on kafka-1 or kafka-3? Hint: look at changed_when and at what copy does when the content differs. (b) Is this an idempotency problem in the role or in the certificate design of 06-04? Propose a change to the role that avoids restarting the broker on every run while still renewing the certificate when it is due. (c) What would have happened without serial: 1 and with all three brokers in the same situation?

Exercise 2. During Grape Harvest Week, orders scales up to 12 replicas (the HPA's maximum) and still has a p99 of 700 ms. Martha looks: the Pods' CPU is at 35%, the lag of orders-saga is low, km0_bulkhead_in_use{dependency="inventory"} is at 24 on every Pod and km0_circuit_state is 0. (a) Why is the HPA not helping, and what does a full bulkhead with low CPU indicate? (b) What should be scaled, and which Kubernetes object governs that component? Is an HPA any use there? (c) Design an HPA metric for orders that reflects real saturation better than CPU, using something from 07-01.

Exercise 3. orders 1.15.0 is about to be deployed; it changes the format of the order.confirmed event on orders.events by adding a mandatory field that analytics 2.3 does not understand yet. (a) Why is a rolling update of orders dangerous here even if the probes are fine, and which strategy from section 8 mitigates it only in part? (b) Describe a step-by-step deployment plan (with the corresponding commands or GitOps PRs) that never breaks analytics, building on the schema compatibility of 02-05. (c) Which automatic check in the CI pipeline would have blocked the PR before reaching this situation? (It is developed in 07-06; it is enough to name it and say which job it would go in.)

Exercise 1.

(a) The Vault task generates a new certificate on every run (with changed_when: false it is not flagged as a change, but the content registered in cert is different every time); the copy task compares the new content with the existing file, sees that it differs (another certificate, another key) and writes it: changed=true and notify: restart kafka. On kafka-1 and kafka-3 it did not happen because... it did happen, or it would have: if it was not seen, it is because the playbook with serial: 1 was interrupted or because on those nodes the previous run failed before writing the keystore; with a role like this, every broker would be restarted on every run. (b) It is an idempotency defect in the role, not in the certificate design: the short-lived certificates of 06-04 are correct, but the role must generate one only when the current one is about to expire: add a prior task that reads the expiry date of the existing certificate (community.crypto.x509_certificate_info) and run the generation and the copy with when: current_cert.expired or (current_cert.not_after | to_datetime - now()) < 7 days; or, better, take the renewal out of the playbook and delegate it to the Vault agent on each node (Vault Agent with templates, which renews and reloads without restarting the broker through dynamic kafka-configs). (c) All three brokers restarting at the same time leaves every partition without ISRs: with min.insync.replicas=2 and acks=all, orders.events rejects writes (NotEnoughReplicas) during the restart, the orders outbox piles up and KafkaLagDelivery and the alerts of 07-01 fire; without unclean.leader.election.enable=false there could even be message loss. serial: 1 with wait_for turns that into three successive restarts with no loss of availability.

Exercise 2.

(a) The HPA scales orders by CPU, and orders is not CPU-bound: it is waiting. km0_bulkhead_in_use at 24 (the maximum) with CPU at 35% and the circuit closed means that every call to inventory succeeds but is slow: the 24 permits are held by requests waiting on inventory, and the next ones are rejected (BulkheadFull, fallback to "pending confirmation") or wait for a thread. Adding orders replicas multiplies the bulkheads (12 × 24 = 288 concurrent calls) and makes the load on inventory worse. (b) inventory (or its database) is what needs scaling: inventory is a Deployment (stateless, the database is separate) and does support an HPA, with a saturation metric of its own (requests in flight or the p99 of ReserveStock); if the bottleneck is km0_inventory (lock contention on crianza-wine, as in the trace of 07-02), adding inventory replicas does not help: the PostgreSQL StatefulSet/operator does not scale horizontally for writes, and the solution is a design one (04-01: partition the stock by market, or batch the reservations). (c) A Prometheus metric via prometheus-adapter: km0_requests_in_flight{service="orders"} averaged per Pod with a target of, say, 20 (with 32 threads, an average of 20 in flight indicates an incipient queue); or latency directly: histogram_quantile(0.99, ...) of POST /orders with a target of 0.4 s. The first is better for the HPA because it responds almost linearly to the number of replicas; the second is more useful as an alert. In both cases, with a slow scaleDown.

Exercise 3.

(a) The rolling update replaces orders Pods without downtime, but as soon as the first new Pod publishes an order.confirmed in the new format, analytics 2.3 fails to deserialise it: the orders probes are perfect; the damage is in another service, via Kafka, and shows up as messages in the analytics DLQ (02-05) and lag. A canary reduces the fraction of events in the new format, but a single message is enough to break the consumer (or send it to the DLQ): it mitigates the volume, not the problem. (b) Forward and backward compatibility: 1) a PR in analytics (2.4) that tolerates the new field (ignores it if it does not understand it, uses it if present), deployed first (kubectl rollout status deployment/analytics), with the schema registered as compatible; 2) deploy orders 1.15.0 with a 10% canary (a PR in the deployment repo with orders-canary, or Argo Rollouts), watching the analytics DLQ and lag and the orders error rate; 3) promote to 100%; 4) only when no producer emits the old format any more (and the old messages have aged out of the topic's retention, 7 days), a PR in analytics that makes the field mandatory. Making the field optional in the schema rather than mandatory also removes step 4. At no point is there a consumer that does not understand what is on the topic. (c) A contract test between the orders producer and the analytics consumer on the order.confirmed schema (checking the schema's compatibility against the registered ones, and the consumer-provider contract): it would go in the test job of the orders pipeline (python tests/contract/verify.py orders), and it would have failed on detecting a new mandatory field that a registered consumer does not accept. It is developed in 07-06.

Automating means taking the person off the critical path and leaving in their place a description that can be reviewed, tested and applied as many times as needed. For the stateful machines, Ansible: an inventory, playbooks and roles whose tasks are idempotent (creates:, template with handlers, --check --diff), and which walk the Kafka brokers one at a time, honouring the ISRs. For the services, immutable images tagged by version and commit, built once and deployed to every environment with injected configuration. And on top of them, Kubernetes: a consensus store with controllers that endlessly reconcile the desired state with the actual one; Deployments with probes, resources, zone spreading and graceful shutdown; Services that balance only towards ready Pods; an HPA driven by CPU and by Kafka lag; StatefulSets or operators for Cassandra, Kafka and PostgreSQL; an Ingress towards Kong; rolling, blue/green and canary with automatic analysis against the SLOs; namespaces and RBAC; the service mesh that provides mTLS and resilience policies without code; and GitOps with a pipeline in which deploying is merging and rolling back is reverting a commit. Compose stays on the laptop.

But this whole machine rests on an assumption that has not yet been examined: that the version 1.15.0 the pipeline builds, signs and deploys works. The pipeline runs pytest and verifies contracts, and the canary watches the SLOs, but which test proves that the saga compensates correctly when inventory dies halfway through? How do you know that the circuit breaker of 07-04 opens in time, or that Patroni fails over in under 30 seconds, before it happens in production on a Saturday? Testing a distributed system is harder than testing a program: non-determinism, partial failures and time mean that unit tests are not enough. The last lesson of the module covers testing distributed systems (integration with real dependencies, contracts, load tests simulating Grape Harvest Week) and chaos engineering: deliberately provoking the failures of 07-03 and 07-04, with a hypothesis and a controlled blast radius, to check that the platform responds as designed.