Docker Compose brings TechCorp's system up on a laptop, but it falls short as soon as there is more than one machine: if the server running orders-service goes down, nobody starts it somewhere else; if Black Friday demands ten replicas of catalog-service, someone has to decide where to put them; if version 1.0.1 has to be deployed without interrupting the service, the startup of the new one and the shutdown of the old one have to be coordinated by hand. An orchestrator does all of that declaratively: you tell it what you want ("two replicas of this image, healthy, reachable under this name") and it takes care of the how, continuously. Kubernetes is the orchestrator on the shortlist of 04-01. In this lesson we understand its architecture and its objects, set up a local cluster, and write and apply the complete manifests of orders-service with the names that 03-05 (Service, /health/*) and 04-03 (orders-service-config, orders-db, orders-rabbitmq) left settled. The pipeline that will apply these manifests automatically (05-03) and the strategies for changing version without downtime (05-04) come afterwards.

1. Why an orchestrator is needed
2. Kubernetes architecture: control plane and nodes
3. The basic objects
4. Local environment with kind and essential kubectl
5. Namespace, ConfigMap and Secret for orders-service
6. A Job for the migrations
7. The orders-service Deployment, line by line
8. The Service and the DNS name orders-service:3002
9. Ingress: exposing the gateway at api.techcorp.example
10. Manual scaling and deployment status
11. RabbitMQ and PostgreSQL: inside the cluster or managed
12. Helm and Kustomize to avoid duplicating YAML across six services

What an orchestrator solves, compared with "containers on machines":

Everything is declared in YAML and stored in git; the cluster continuously reconciles the actual state with the desired one. That is the central idea: you do not run imperative commands, you describe the outcome.

• API server: receives every request (from kubectl, from the controllers, from the kubelets), validates them and persists them in etcd, the cluster's key-value database.
• Scheduler: watches pods with no node assigned and picks one for them according to requested resources, affinities and constraints.
• Controller manager: runs the control loops. The Deployment controller creates ReplicaSets; the ReplicaSet controller creates or deletes pods until replicas matches; the Endpoints controller maintains the list of ready pods for each Service (the "registry" of 03-05).
• kubelet (on each node): agent that talks to the API server, starts the containers of the pods assigned to its node through the runtime (containerd, the same container technology as 05-01) and runs the probes.
• kube-proxy: programs the network rules so that a Service's ClusterIP spreads traffic across its pods.

A TechCorp developer only talks to the API server (kubectl), and not even that in production: the pipeline or Argo CD will (05-03).

kind (Kubernetes in Docker) creates a complete cluster inside containers; minikube is the equivalent alternative. TechCorp uses kind because it is the same one that runs in CI. The configuration file publishes the node's ports 80/443 on the laptop so the Ingress works:

The kubectl commands used every day (kubectl config set-context --current --namespace=techcorp avoids repeating -n techcorp; from here on we omit it):

The manifests live in techcorp/platform/k8s/orders-service/base/ (section 12 explains the structure). First the namespace and the non-sensitive configuration, which is exactly the "production" column of the 04-03 table:

Secrets do not go in YAML inside the repository. They are created with kubectl (or injected by an external manager, 07-04):

That value is base64, not encryption: echo cG9z... | base64 -d gives back the password. A Secret is only "secret" because access to it is restricted with permissions (RBAC) and because etcd can be encrypted at rest; both belong to 07-04. That is why secrets are not versioned in plain text and why the .dockerignore of 05-01 excluded .env. The Secret keys are named like the environment variables (ORDERS_DB_URL) so they can be injected with envFrom.

scripts/migrate.js (04-04) must run before the new version of Orders starts, once, and fail loudly if it cannot. It is a Job:

The alternative is an init container inside the service's pod; it works, but it runs the migrations on every replica at startup (with two replicas, twice, and migrate.js has to cope with the concurrency). The Job runs them once per version and shows up in kubectl get jobs. In 05-03 the pipeline launches it and waits before applying the Deployment.

The points that raise the most questions:

• selector and labels: the Deployment (through its ReplicaSet) "owns" the pods whose app: orders-service label matches; the Service of section 8 uses the same selector. Changing selector on an existing Deployment is not allowed: choose well from the start.
• readinessProbe versus livenessProbe (03-05): readiness looks at the service's own dependencies (PostgreSQL and RabbitMQ in the /health/ready of 04-04) and its effect is to stop receiving traffic; liveness only checks that the process responds and its effect is to restart. Putting the database check in liveness is the classic mistake: if PostgreSQL goes down, Kubernetes restarts every Orders pod in a loop without fixing anything.
• terminationGracePeriodSeconds: 30 closes the loop of the graceful shutdown: when a pod is deleted, Kubernetes removes it from the Endpoints and sends SIGTERM to PID 1 (the node of 05-01); the service sets /health/ready to 503, finishes in-flight requests and exits in under 10 s; if it did not, SIGKILL would arrive at 30 s. With two replicas and this contract, a deployment does not lose a single request (05-04 shows it step by step).
• securityContext: runAsNonRoot verifies what the image already does (USER node); readOnlyRootFilesystem forces any write to go to an explicit emptyDir. The rest of the hardening (capabilities, seccomp, admission policies) belongs to 07-04.

With this, the gateway's ORDERS_URL=http://orders-service:3002 and the CATALOG_URL=http://catalog-service:3001 in the Orders ConfigMap resolve exactly as in Compose (05-01) and as 03-05 promised, without any service knowing how many replicas there are or which node they are on.

Apply and check all of the above:

If a pod gets stuck in CrashLoopBackOff, kubectl logs --previous will almost always show the config.js message (04-03) saying which variable is missing: the fail-fast validation was designed for this very moment.

The gateway (03-04) is deployed like any other service (Deployment + Service gateway:8080, with its ConfigMap of URLs). The only thing that leaves the cluster is an Ingress that the ingress controller (NGINX in kind and in production; Traefik would be equivalent with ingressClassName: traefik) turns into proxy rules:

The ingress controller balances at layer 7 towards the gateway's Service, and the gateway towards the internal services (03-05). The only entry point is still the gateway's 8080; no *-service has an Ingress.

kubectl scale is manual: someone decides the number. Automatic scaling by CPU or by metrics (HorizontalPodAutoscaler) belongs to 06-04, once we have metrics to decide with. What matters here is that scaling means changing a number and that the Service spreads traffic only across the ready replicas.

TechCorp's decision: in dev and in the CI kind cluster, PostgreSQL and RabbitMQ inside the cluster with Helm (two commands, disposable data); in staging and production, managed, with the URL in the Secrets (orders-db, orders-rabbitmq) and the name rabbitmq resolved by a Service of type ExternalName if RABBITMQ_URL needs to stay stable. The Platform team has four people and its priority is the gateway, the cluster and the pipeline, not being DBAs.

Helm installs a chart (a package of parameterized manifests) that creates the StatefulSet, the Service (rabbitmq, pg-orders-postgresql) and the volumes; RABBITMQ_URL=amqp://orders:dev-rabbit@rabbitmq:5672 works in kind without touching the services' manifests.

The four Orders files would be repeated almost identically for Catalog, Inventory, Payments, Notifications, Customers and the gateway, and per environment as well (dev, staging, prod). Two tools avoid copy-and-paste:

TechCorp uses Kustomize for its services and Helm for third parties. Structure in techcorp/platform/k8s/:

There is one base per service, and the overlays only state how each environment differs: replicas, image tag, two configuration keys. When the Inventory team creates its service, it will copy orders-service/ and change names, port (3006) and ConfigMap: that is the "template" part of Luis's rule; the "automate" part belongs to 05-03.

• Checking the database in the livenessProbe. PostgreSQL goes down for 30 s and Kubernetes restarts every Orders pod in a loop. Dependencies in readiness; in liveness, only the process.
• No resources. Without requests the scheduler piles pods onto one node; without limits the first one whose memory spikes takes the others down. The starting values are tuned by measuring (06-04).
• selector labels that don't match template.metadata.labels. The apply fails with a clear message; in the Service, on the other hand, a misspelled selector simply leaves Endpoints empty and the service "doesn't respond". kubectl get endpoints is the first check.
• Secrets in YAML inside the repository "because they are base64". It is not encryption. kubectl create secret or the tools of 07-04 (Sealed Secrets, External Secrets).
• Reapplying a Job with the same name and changes: Kubernetes rejects it ("field is immutable"). Version in the name, or kubectl delete job first.
• Tip: kubectl describe pod and its Events section answer 90% of the "it doesn't start" cases; kubectl get events --sort-by=.lastTimestamp gives the namespace-wide view.

Exercise 1. Write the Deployment and the Service for catalog-service (image ghcr.io/techcorp/catalog-service:1.4.2, port 3001, ConfigMap catalog-service-config with PORT, MONGO_DB, LOG_LEVEL, NODE_ENV; Secret catalog-mongo with MONGO_URL) stating only the lines that differ from the Orders ones. How many replicas would you set in the prod overlay and why?

Exercise 2. An orders-service pod shows READY 0/1 for minutes but STATUS Running and no restarts. kubectl logs shows "listening on 3002" and no error trace. List, in order, the three commands you would run to diagnose it and the two most likely causes.

Exercise 3. Marta asks why the migrations Job carries the version in its name and what happens if the pipeline deploys 1.0.1 without any migration having changed. Answer and propose how to prevent the Job from failing in that case.

Solution 1. What differs: metadata.name, labels/selector (app: catalog-service), image: ghcr.io/techcorp/catalog-service:1.4.2, containerPort: 3001, envFrom with configMapRef: catalog-service-config and secretRef: catalog-mongo, and in the Service port: 3001 (everybody's CATALOG_URL=http://catalog-service:3001). The probes point at the same /health/live and /health/ready (contract of 03-05); terminationGracePeriodSeconds, securityContext and resources can be the same. There is no migrations Job (MongoDB has no schema; the seed is dev-only). Replicas in prod: more than Orders (for example 4), because Catalog takes the ×20 campaign peaks (01-05) and is read-only, cheap to replicate; the definitive number will come from the HPA of 06-04.

Solution 2. (1) kubectl describe pod <name>: Events will show "Readiness probe failed: HTTP probe failed with statuscode: 503" (or a timeout). (2) kubectl port-forward pod/<name> 3002:3002 and curl localhost:3002/health/ready: the body says which dependency is wrong (postgres or rabbitmq). (3) kubectl get secret orders-db -o jsonpath='{.data.ORDERS_DB_URL}' | base64 -d (and the same for RabbitMQ) to see where it points. Most likely causes: the URL in the Secret points at a wrong host or has bad credentials (the service starts because config.js only validates the format, but /health/ready cannot run SELECT 1), or the dependency is not reachable from the namespace (RabbitMQ not yet installed, rabbitmq Service missing). This is the desired behavior: not ready, no traffic, no restart loop.

Solution 3. A Job is immutable and its name unique: if orders-service-migrations were reapplied with another image, the API server would reject it. With the version in the name, every deployment creates a new Job, a record remains (kubectl get jobs) and ttlSecondsAfterFinished cleans it up. If 1.0.1 brings no new migrations, the Job starts, migrate.js queries applied_migrations, sees that 001-004 are already there and exits with code 0 without doing anything: it is idempotent by design (04-04), so it does not fail; it is simply a two-second Job. The alternative of skipping the Job when "there are no changes" requires someone to decide it, and Luis's rule prefers that it always be the same pipeline.

Kubernetes runs the images of 05-01 declaratively and continuously: a control plane (API server, etcd, scheduler, controller manager) that reconciles the desired state, and nodes with kubelet, kube-proxy and containerd that materialize it. For orders-service we have written and applied in the techcorp namespace the orders-service-config ConfigMap, the orders-db and orders-rabbitmq Secrets created with kubectl create secret (base64, not encryption), a versioned Job that runs scripts/migrate.js with the same image, a two-replica Deployment with envFrom, resources, readinessProbe on /health/ready, livenessProbe on /health/live, terminationGracePeriodSeconds: 30 and runAsNonRoot, and the Service that provides the name http://orders-service:3002; the gateway reaches the outside through an NGINX Ingress at api.techcorp.example; RabbitMQ and PostgreSQL go via Helm in dev/kind and managed in production; and Kustomize (base + dev/prod overlays, with images: as the spot the pipeline will change) avoids duplicating YAML across the six services. Everything has been applied by hand with kubectl apply, and that is precisely what the next lesson removes: a CI/CD pipeline per service that tests, verifies pacts, builds the image and updates these manifests without anyone typing a command.