The previous lesson closed with a promise: you know what Kubernetes does —continuously maintain the state you declare to it— and now it is time to see how it does it. A cluster is not a magic box: it is six or seven processes with tightly defined responsibilities that always communicate through a single central point. Understanding that architecture is what will later let you diagnose problems instead of restarting things at random: when a pod is stuck in Pending, when a deployment makes no progress or when the cluster stops responding, the cause is almost always in one specific piece, and this lesson teaches you which. We will finish by following, step by step, everything that happens from the moment you run kubectl apply -f bookings-api.yaml until the bookings-api container is serving requests on a node.

1. The big picture: control plane and worker nodes
2. The control plane components
3. The worker node components
4. The reconciliation loop: the central idea
5. A complete walkthrough of kubectl apply
6. What happens when each piece fails
7. High availability of the control plane

A Kubernetes cluster is split into two halves with very different roles:

• The control plane: the brain. It decides, records and watches. It does not run user applications (except on single-node clusters such as minikube).
• The worker nodes: the muscle. They run your application containers and report their state.

Note the most important property of the diagram: every arrow goes through kube-apiserver. No component talks directly to another. The scheduler does not call the kubelet; the kubelet does not query etcd. They all read from and write to the API, and that API is the only thing that touches etcd. This star topology is what makes the system extensible: to add a new capability all you need is a process that watches the API and acts on it (that is a controller, and it is exactly what the operators in module 6 do).

It is the REST front end of the entire cluster and the only component that writes to etcd. Everything that happens in Kubernetes is an HTTP request against this server.

Its responsibilities, in the order they are applied to each request:

1. Authentication: who are you? (client certificate, token, OIDC…).
2. Authorization: are you allowed to do this? This is where RBAC comes in (module 8).
3. Admission control: modify or reject the object before storing it (for example, injecting default values, applying quotas or rejecting unsigned images).
4. Validation of the object schema.
5. Persistence in etcd.
6. Notification of every subscribed component through the watch mechanism.

It is stateless and horizontally scalable: you can run three replicas behind a load balancer.

It is a distributed, consistent key-value database based on the Raft consensus algorithm. It stores absolutely all the cluster state: objects, configuration, secrets and observed state.

Points to remember:

• It is the single source of truth. If you lose etcd with no backup, you have lost the cluster (not the applications already running, but certainly their whole definition).
• It is deployed with an odd number of members (3 or 5) so that quorum can be formed.
• It is sensitive to disk latency: a dedicated SSD is recommended. It is the usual bottleneck in large clusters.
• Its backup (etcdctl snapshot save) is the cluster's backup, and it is examinable material in the CKA (module 12).

Its job is one single thing, and it does it very well: deciding which node each newly created pod runs on. It continuously watches the API for pods with no nodeName assigned and, for each one, runs two phases:

The outcome is not "start the container": it is simply writing the pod's spec.nodeName field through the API. The scheduler never contacts the node. The advanced criteria (affinity, taints, tolerations) are covered in module 6.

It is a single binary that runs dozens of controllers in parallel, each with its own reconciliation loop. The ones that matter most for this course:

When in module 2 you write a Deployment with replicas: 3 and see three pods appear, this is the process that created them.

It isolates everything that depends on the cloud provider, so that the Kubernetes core carries no AWS, Azure or GCP code. It takes care of:

• Nodes: asking the provider whether a virtual machine has really been deleted.
• Load balancers: when you create a Service of type LoadBalancer, this is what requests the real load balancer from the cloud (module 4).
• Routes: configuring routing between nodes on the provider's network.

On minikube or kind it does not exist, and that is why a Service of type LoadBalancer keeps its external IP at <pending> unless you use minikube tunnel. It is a classic source of confusion that we will come back to.

It is the agent that runs on every node and the only one that actually starts containers. Its loop is:

1. Ask the API which pods have been assigned to it (spec.nodeName == this node).
2. Compare that with what is running on the machine.
3. Ask the container runtime, over CRI, to start or stop whatever is needed.
4. Mount volumes, inject ConfigMaps and Secrets, apply resource limits.
5. Run the health probes (module 7) and restart failed containers.
6. Continuously report the state of the node and its pods to the API.

Important: the kubelet only obeys the API, it takes orders from nobody else. And it only manages pods, not loose containers: if you start a container with docker run on a node, the kubelet ignores it.

It implements the Service concept at network level on each node. It watches Services and their EndpointSlices and programs network rules (iptables or, on large clusters, IPVS) so that traffic sent to a service's virtual IP is spread across the healthy pods.

In its usual mode it is not a proxy in the data path: it writes kernel rules and steps aside. Some modern network plugins (Cilium in eBPF mode) replace it entirely. It is covered in module 4.

The kubelet does not know how to create containers: it delegates to a runtime through the CRI (Container Runtime Interface), a standard gRPC API. The common runtimes:

Below the runtime there is another level (runc) that actually talks to the kernel (namespaces and cgroups). You do not need that detail to operate a cluster, but it explains the phrase "Kubernetes no longer uses Docker": the images you build with Docker keep working perfectly because they follow the OCI standard.

Two more interfaces worth naming now:

• CNI (Container Network Interface): the plugin that assigns IPs to pods and lets them see each other across nodes. Calico, Cilium, Flannel. With no CNI installed, nodes stay NotReady. Module 4.
• CSI (Container Storage Interface): the plugin that provisions real disks for persistent volumes. Module 5.

If you take only one idea away from this lesson, make it this one. Every Kubernetes controller runs the same pattern:

Practical consequences of this design, which explain many behaviours you will run into:

• You write spec; the system writes status. Never edit a status by hand.
• It is level-based, not event-based. The controller does not react to "a pod was deleted": it compares the current picture with the desired one. That is why the system recovers even if a message is lost or a controller restarts.
• It is idempotent. Applying the same manifest ten times produces the same result as applying it once.
• Manual fixes do not last. If you delete a pod managed by a ReplicaSet, it comes back within seconds. To make it go away you must change the desired state.
• Failures are "eventually consistent". Time passes between declaring something and it being satisfied; that is why objects have conditions and events, and why kubectl get sometimes shows intermediate states.

Let us follow a real Rutas Norte request. Assume this minimal file in the repository's k8s/ directory (the details of the format are covered in the lesson Objects, YAML Manifests and the Declarative Model):

And you apply it:

Behind that single line of output, twelve steps have taken place.

Step by step, in detail:

1. kubectl reads your kubeconfig to work out which cluster to talk to, with which credentials and in which namespace.
2. It converts the YAML to JSON and sends it as a POST (or PATCH if the object already existed) over HTTPS to the resource endpoint.
3. Authentication. The apiserver validates your client certificate or token. If it fails: error: You must be logged in to the server (Unauthorized).
4. Authorization. RBAC checks whether your user may create pods in the rutas-norte-dev namespace. If it fails: Error from server (Forbidden).
5. Admission control. The admission controllers run in two rounds: first the mutating ones (they add defaults, the ServiceAccount, limits from a LimitRange…) and then the validating ones (ResourceQuota, Pod Security Standards, your own webhooks). Any of them can reject the request.
6. Write to etcd. The object is stored. At this moment the pod already exists as an object, even though no container is running. Its state is Pending and its spec.nodeName is empty.
7. etcd acknowledges the write through Raft consensus.
8. The apiserver replies to kubectl with 201 Created. Your command ends here: kubectl apply does not wait for the container to start.
9. The scheduler is notified through its watch on pods with no node assigned.
10. The scheduler decides. It filters out nodes without enough resources or otherwise incompatible, scores the rest, picks node-2 and writes the binding to the API. The pod is still Pending, but it now has a destination.
11. The kubelet on node-2 finds out through its own watch and takes over.
12. The kubelet executes: it asks the runtime, over CRI, to pull registry.rutasnorte.example/bookings-api:2.4.0 (state ContainerCreating, or ImagePullBackOff if the registry refuses the pull), creates the network sandbox by requesting an IP from the CNI plugin, mounts volumes and secrets, and starts the container. It then reports to the API: status.phase = Running.

From then on, the kubelet watches the container permanently and the reconciliation loop stays alive until the object is deleted.

This table is pure gold for troubleshooting incidents, and a recurring question in interviews and certifications.

The operational takeaway is reassuring: the control plane is the brain, not the heart. If it goes down, the cluster stops changing, but it does not stop serving. This is a deliberate design and it explains why a control plane outage is serious but rarely a service outage.

A practice cluster has a single control node. A production cluster, like the one Rutas Norte will want for its rutas-norte-pro namespace, must tolerate losing one control plane machine.

The standard layout:

• 3 control plane nodes (an odd number, because of etcd quorum).
• kube-apiserver replicated across all three, behind a load balancer or a keepalived setup with a virtual IP. Being stateless, it scales active-active.
• etcd with 3 members. It tolerates losing 1. With 5 members it tolerates 2. With 2 members it tolerates none: never use an even number.
• scheduler and controller-manager active-passive. All three run, but only one acts: they elect a leader through a Lease object in the API. Only one must reconcile, otherwise they would create duplicate pods.
• Stacked or external etcd: stacked (etcd on the same control machines) is the usual and simplest option; external isolates the failure and is used on large clusters.

On top of that, in a managed cluster (EKS, AKS, GKE, module 10-06) the provider operates all of this for you and you never even see the control plane machines: you pay a fee and only administer the worker nodes. For most companies, Rutas Norte included, that is the sensible option.

• Believing the scheduler starts containers. It does not: it only writes a node name. The kubelet is what starts things. If a pod is Pending, the problem belongs to the scheduler (it fits on no node); if it is ContainerCreating or ImagePullBackOff, the problem belongs to the kubelet or the runtime.
• Thinking kubectl apply waits for the application to be ready. It returns as soon as the object is stored in etcd. To wait for real, use kubectl rollout status or kubectl wait.
• Changing resources by hand on a node. Stopping a container with crictl or docker achieves nothing: the kubelet recreates it. You always act through the API.
• Forgetting the etcd backup. It is the only component whose data is irreplaceable. Without a snapshot, a disaster in etcd means rebuilding the cluster from the manifests (one more reason to keep them all in Git).
• Running 2 etcd members "for redundancy". It makes availability worse: with 2 members quorum is 2, so losing one blocks the cluster. Always an odd number.
• Troubleshooting tip: faced with any problem, walk the chain in order — does the object exist in the API? (kubectl get) → does it have a node assigned? (kubectl get pod -o wide) → what does the kubelet say? (kubectl describe pod, Events section). That order replicates exactly the walkthrough in section 5.

For each symptom observed in the Rutas Norte cluster, state which component is the prime suspect and why:

1. The bookings-api pod has been Pending for 10 minutes and describe says 0/3 nodes are available: insufficient memory.
2. kubectl get pods returns Unable to connect to the server: dial tcp ... connection refused.
3. A web-store pod managed by a ReplicaSet is deleted by hand and is not recreated, but the rest of the cluster responds fine.
4. The notifications-worker pod has been ContainerCreating for 5 minutes with the event failed to pull image ... unauthorized.
5. A node shows up as NotReady and a few minutes later its pods appear on other nodes.

Put these eight steps, which are out of order, back in sequence, and point out at which of them the pod goes from Pending to ContainerCreating:

• a) The kubelet asks the runtime to create the container over CRI.
• b) The apiserver writes the object to etcd.
• c) RBAC checks the user's permissions.
• d) The scheduler writes spec.nodeName.
• e) kubectl sends the HTTPS request.
• f) The admission controllers run.
• g) The apiserver replies 201 Created.
• h) The kubelet receives the watch event.

Rutas Norte wants a self-managed cluster for rutas-norte-pro that tolerates losing an entire control plane machine. State: how many control nodes you would use, how many etcd members, how the apiserver is reached, in which mode the scheduler and controller-manager work, and explain in two lines what would happen to the web store during the 3 minutes when the control plane was completely down.

Solution 1

1. kube-scheduler (together with node capacity). The pod was created correctly in the API, but no node passes the filtering phase for lack of memory. The fix is to reduce the pod's requests or add capacity.
2. kube-apiserver (or the network to it). connection refused means kubectl does not even get as far as authenticating: the process is not listening or the kubeconfig endpoint is wrong.
3. kube-controller-manager. The ReplicaSet controller is what recreates replicas; if everything else works but nothing reconciles, the suspect is the controller-manager (or its leader election).
4. kubelet and container runtime, with the root cause in the registry credentials. The pod already has a node assigned; the failure is in pulling the image (the imagePullSecret is missing).
5. The kubelet on that node (or the node's network). Once it stops reporting, the node controller marks it NotReady and, after the tolerance period, evicts its pods so they are recreated elsewhere. Expected behaviour, not a fault.

Solution 2

Correct order: e → c → f → b → g → d → h → a.

• e) HTTPS request
• c) authentication and RBAC authorization
• f) admission (mutation and validation)
• b) write to etcd → the pod exists in Pending
• g) 201 Created reply to kubectl
• d) the scheduler assigns a node → still Pending, but now with a destination
• h) the kubelet receives the event
• a) the kubelet creates the container → here the pod moves to ContainerCreating and then to Running

Solution 3

• 3 control plane nodes and 3 etcd members (stacked topology): with a quorum of 2, losing 1 machine is tolerated.
• Access to the apiserver through a load balancer or virtual IP (keepalived + HAProxy) spreading requests across the three replicas; the kubeconfig points at that single endpoint, not at a specific machine.
• kube-scheduler and kube-controller-manager active-passive, with leader election through a Lease: all three processes run, but only the leader reconciles, to avoid duplicating actions.
• During those 3 minutes without a control plane, the web store keeps working normally: the pods are already running on the nodes and kube-proxy keeps its network rules. What does not happen in those 3 minutes is change: you cannot deploy versions, scale, recreate failed pods or run kubectl.

A Kubernetes cluster is a set of processes with very strict responsibilities that communicate exclusively through the kube-apiserver, with etcd as the single source of truth. The control plane decides (apiserver, scheduler, controller-manager, cloud-controller-manager) and the nodes execute (kubelet, kube-proxy, runtime over CRI). The conceptual glue is the reconciliation loop: compare the desired state you declare with the observed actual state and act until they match, continuously and idempotently. With that mental map you can already reason about incidents instead of guessing, and you know why a control plane outage stops changes but not the service.

We have the architecture; what we lack is the vocabulary. In the coming lessons, terms such as pod, ReplicaSet, Service, PVC or namespace will appear constantly, and it pays to have a clear map of what each one is and how they relate to each other before using them. That is what the next lesson sets out: Key Concepts and Terminology.