The previous lesson ended with an unsolved problem: bookings-api and web-store are running in rutas-norte-dev, but as bare pods. If the node goes down, if the kubelet evicts them for lack of memory or if somebody deletes them, they vanish and nobody brings them back to life. We need something that watches permanently and acts. That something is a controller, and the first one we are going to take apart is the ReplicaSet: the object whose sole mission is to guarantee that at all times a specific number of pods matching a set of labels exists. In this lesson you will see what a controller is exactly and how the ReplicaSet controller embodies the reconciliation loop you studied in the architecture, you will write your first ReplicaSet manifest for web-store, you will verify self-healing by deleting pods on purpose, you will understand how cascading deletion relies on ownerReferences, you will discover the adoption mechanism for orphaned pods and why an overly broad selector is a time bomb, and you will learn to scale with kubectl scale. And you will finish knowing why, despite all of this, you will almost never write a ReplicaSet by hand.

1. What a controller is
2. The ReplicaSet controller and the reconciliation loop
3. Anatomy of the manifest: replicas, selector and template
4. The golden rule: the template must match the selector
5. Self-healing live with web-store
6. ownerReferences and cascading deletion
7. Adoption of orphaned pods and the danger of broad selectors
8. Manual scaling with kubectl scale
9. ReplicaSet versus ReplicationController
10. Why a ReplicaSet is almost never created by hand

In Kubernetes Architecture we defined the reconciliation loop as the heart of the system. A controller is simply a program that runs that loop for one type of object:

Traits shared by every Kubernetes controller that are worth internalising:

• They never finish. They are infinite loops, not one-shot scripts.
• They are idempotent. Running the loop a thousand times with the state already correct changes nothing.
• They only talk to the kube-apiserver. They never contact nodes or containers directly. They request changes from the API and the kubelet makes them real.
• They work by level, not by event. They do not react to "a pod has been deleted"; they constantly compare how many there are against how many there should be. That is why they are robust: if the controller was down for ten minutes, on coming back it corrects the accumulated difference with no need to replay history.

Most of them live inside the kube-controller-manager: the Deployment one, the ReplicaSet one, the Job one, the Namespace one, the endpoints one, the ServiceAccount one... All with the same anatomy and different responsibilities.

Let's make the generic loop concrete for the case at hand. The ReplicaSet controller runs, non-stop, for every ReplicaSet in the cluster:

1. It reads spec.replicas: how many pods must exist (desired state).
2. It reads spec.selector: which labels identify its pods.
3. It asks the API how many pods in that namespace match the selector and are not terminating (actual state).
4. It compares:
   • If pods are missing, it creates as many as are needed using spec.template as the mould.
   • If there are too many, it picks victims and deletes them.
   • If they match, it does nothing and updates the status.

One decisive nuance, which explains everything that follows: the ReplicaSet does not keep a list of the pods it has created. It has no memory. On every turn of the loop it asks again, "which pods are there with these labels?". Its relationship with the pods is purely by label. This has two enormous consequences:

• If you delete one of its pods, on the next turn it sees one is missing and creates a new one. Self-healing.
• If a pod with those same labels appears out of nowhere, created by somebody else, the ReplicaSet counts it as its own. Adoption. And if that makes one too many, it will delete somebody.

How it picks who to delete when there are too many: the controller prioritises removing Pending pods over Running ones, those that have been ready for less time, those with more restarts and those on nodes with more replicas. In other words, it always sacrifices the least valuable.

Let's turn web-store into something that looks after itself. Create the file in the project repository:

A field-by-field read, paying attention to what changes compared with the bare pod:

• apiVersion: apps/v1. Here there is a group. The Pod is in the core group (v1), but ReplicaSet, Deployment, StatefulSet and DaemonSet live in the apps group. If you write plain v1, the API will reject the manifest with no matches for kind "ReplicaSet" in version "v1".
• metadata (the top-level one): it identifies the ReplicaSet, not the pods. Its labels are there so you can find the ReplicaSet itself with kubectl get rs -l ....
• spec.replicas: 3: the desired state. It is a number, not a range. Automatic scaling by load arrives in module 9.
• spec.selector.matchLabels: which pods are mine. Here we demand two labels: a pod must have app=web-store and environment=dev to count. The selector also accepts richer expressions (matchExpressions), which we will see in Labels, Selectors and Annotations. It is a mandatory and immutable field.
• spec.template: the pod template. It is exactly a pod manifest without apiVersion or kind, with its own metadata (the labels of the child pods) and its spec (the containers). It is the mould the controller uses to manufacture replicas.

Notice the Russian-doll structure, the point where everybody gets stuck at first:

Before applying it, remove the bare web-store pod so it does not interfere with the experiment (in section 7 you will see exactly what would happen if we did not):

Two observations about that output:

• The pods are called web-store-<random suffix>. The controller generates the name from the ReplicaSet's name plus five random characters, because there cannot be three objects with the same name in a namespace.
• The ReplicaSet columns: DESIRED is spec.replicas; CURRENT, how many pods exist; READY, how many are ready to take traffic. When all three match, the loop is in equilibrium.

This is the constraint that causes the most rejected manifests:

The labels in spec.template.metadata.labels must satisfy spec.selector. If they do not, the API rejects the object.

The reason is pure logic: if the ReplicaSet manufactured pods that its own selector does not recognise, it would count zero pods of its own, create three more, fail to recognise those either, create another three… an infinite loop that would fill the cluster. Kubernetes prevents this at the root by validating the manifest.

Check it by triggering the error on purpose:

Note the asymmetry, which is valid and very useful: the template may have more labels than the selector. In our manifest the selector asks for app and environment, but the pods also carry app.kubernetes.io/part-of. That is allowed; what is forbidden is missing any of the labels the selector demands.

As a project rule: keep the selector as small and stable as possible (one or two labels that unambiguously identify the component and the environment) and add the rest of the metadata only in the template.

The time has come to check what module 1 left pending. Open two terminals.

In the first one, start watching:

In the second one, kill a pod behind its back (use one of the real names from your cluster):

In the first terminal you will see something like this:

Under two seconds. And something very revealing: the replacement pod (web-store-t8m4r) appears in Pending before the deletion of the previous one has finished. The controller does not wait for the old one to disappear entirely; as soon as the API marks the pod as terminating, it stops counting it and acts.

The ReplicaSet's events tell the same story in writing:

Compare that with what happened in module 1 when we deleted the bare pod: nothing. Silence. Here, by contrast, there is a replicaset-controller signing off every creation. This is, in one line, the difference between a container and a platform. For Rutas Norte it means that March's four-and-a-half-hour night-time outage would have fixed itself in two seconds, in the small hours, without anybody noticing.

An even more forceful experiment: delete them all at once.

Three new pods, with new names and new IPs. And there the course's next problem peeks out: if the IPs change every time, which address does web-store connect to in order to talk to bookings-api? That is the job of Services.

How does Kubernetes know that those pods "belong" to the ReplicaSet, if the relationship is by labels? Because, when creating them, the controller stamps a reference to their owner on every pod:

Field by field:

The garbage collector, another kube-controller-manager controller, works on top of those references: it walks the objects, and when it finds one whose owner no longer exists, it deletes it. This is cascading deletion, and it is why kubectl delete rs web-store takes the three pods with it.

Kubernetes offers three propagation policies:

The third is a very useful emergency resource, and it also sets the scene for the next section:

The ReplicaSet no longer exists, but the three pods keep serving traffic. They are now orphans: nobody is watching them. If you delete one, it does not come back.

We have three orphaned pods with the labels app=web-store, environment=dev. Create the ReplicaSet again:

Look closely at the ages: 8 minutes. The ReplicaSet has just been born and has not created a single pod. It has adopted the three orphans because they matched its selector and had no owner. On adopting them, it has written its ownerReferences into them:

The exact rules of adoption:

1. The pod must be in the same namespace.
2. Its labels must match the selector.
3. It must not already have an ownerReferences with controller: true. A pod with an owner is not stolen: the ReplicaSet ignores it.

Here is the trap. Suppose somebody, in a hurry, defines the web-store ReplicaSet with a lazy selector:

That selector matches every pod on the platform: web-store, bookings-api, redis-cache, the worker... The consequences, in a chain:

1. The ReplicaSet counts every Rutas Norte pod in the namespace. Say 5.
2. Its desired state is 3. There are 2 too many.
3. It picks two victims and deletes them. It could perfectly well delete bookings-api and the worker.
4. Those components' controllers recreate them, the ReplicaSet again sees too many and deletes them once more. A war of controllers.

It is a real and fairly common incident in young clusters, and the symptom —pods deleting themselves with no explanation— is baffling until you understand the mechanism. Let's simulate it safely to see it with our own eyes:

The ReplicaSet has adopted intruder, counted 4 where there should have been 3 and executed it on the spot (it was the youngest, the preferred victim). Had it been an important pod instead of a test nginx, it would have gone just the same.

The project rules that follow from this:

• The selector must identify the component unambiguously: app: <component> plus environment: <environment>, never part-of alone.
• No component shares a selector with another.
• Never create bare pods with the labels of a governed component. To debug, use different labels or ephemeral pods with --rm.

Changing the number of replicas means changing spec.replicas. There are three ways.

Imperative, fast (for incidents):

Conditional, very useful in scripts to avoid trampling on somebody else's changes:

If there are not exactly 5 replicas at that moment, the command fails instead of applying the change.

Declarative, the correct one according to the project conventions: edit replicas: 5 in the file, commit and apply.

The rule you already know from module 1 still stands: if a change is not in Git, it does not exist. Scale with kubectl scale to put out a fire, but move the change into the file straight afterwards.

Go back to 3 before moving on:

And watch what happens when scaling down:

The ones that go are the youngest, exactly as we predicted: the controller sacrifices whatever is least established.

You will come across old documentation and forum answers talking about ReplicationController. It is the ancestor of the ReplicaSet, from the Kubernetes 1.0 era, and it still exists for compatibility, but you should not use it.

The relevant functional difference is the set-based selector. Thanks to matchExpressions, a Deployment can issue queries such as "bookings-api pods whose pod-template-hash is one of these two", which is exactly what it needs in order to manage two simultaneous versions during a rolling update. With the ReplicationController's flat selector that was not expressible, and that is why Deployments were built on top of ReplicaSets.

Operational summary: if you see kind: ReplicationController, it is legacy code. Migrate to a Deployment.

And here comes the lesson's twist: after all of this, in your professional life you will write very few ReplicaSets. What you will write are Deployments.

The reason is that the ReplicaSet lacks the essential thing for operating a service: it does not know how to change version. It is superb at keeping N copies of a template, but if you modify the image in its template:

The pods still run the old image. The ReplicaSet only uses the template when it creates a pod; the existing ones are left alone because the desired state ("three pods with these labels") is still satisfied. To roll out the new version you would have to delete them by hand, one by one, waiting in between. No pacing control, no verification, no way back.

That is precisely what the Deployment brings: it manages ReplicaSets, not pods. When you change the image, it creates a new ReplicaSet with the new version and shifts replicas from one to the other in a controlled way.

That is why the project rule is:

At Rutas Norte, ReplicaSets are not created by hand. Deployments are created, and they manage their ReplicaSets.

So what is this lesson for? For understanding the middle layer. When in the next lesson you see two ReplicaSets of the same Deployment coexisting during an update, or when in production you have to work out why there are pods from two versions at once, the explanation will lie in what you have just learned. ReplicaSets do not go away: they become invisible because somebody else manages them.

Leave the cluster ready for the next lesson:

• Using apiVersion: v1 for a ReplicaSet. It is in apps/v1. The error is no matches for kind "ReplicaSet" in version "v1".
• template labels that do not match the selector. The API rejects the object with selector does not match template labels. Check that you are looking at spec.template.metadata.labels and not the top-level metadata.labels: they are different places.
• Overly broad selectors. This is the most destructive mistake in the lesson: a ReplicaSet can adopt and delete another component's pods. Selector = component + environment, always.
• Creating bare pods with the labels of a governed component. They will be adopted and, most likely, executed within seconds.
• Expecting a change to the template to update the existing pods. It does not. It only affects pods created from that moment on. That is the gap the Deployment fills.
• Trying to change the selector of an existing ReplicaSet. It is immutable. You have to delete and recreate (or, in practice, let the Deployment handle the change by creating a new ReplicaSet).
• Confusing CURRENT with READY. CURRENT counts pods that exist; READY, pods that can serve traffic. A 3/3 in CURRENT with 0 in READY is a broken rollout.
• Tip: kubectl get rs -o wide adds the CONTAINERS, IMAGES and SELECTOR columns. It is the fastest way to see at a glance which version each ReplicaSet serves.
• Tip: whenever something behaves inexplicably, always look at kubectl get pod <name> -o jsonpath='{.metadata.ownerReferences}'. Knowing who owns a pod solves half the mysteries.

Write k8s/base/bookings-api-replicaset.yaml with a ReplicaSet of 2 replicas of bookings-api in rutas-norte-dev, reusing the container from the Pods lesson (the node:20-alpine image with the minimal server) and respecting the project's three labels. The selector must use app and environment.

1. Apply it and check that there are 2 pods.
2. Delete one and measure how long the replacement takes to appear.
3. Find out, with a single command, who owns the new pod.
4. Show the ReplicaSet events that document the creation.

Starting from the previous exercise's ReplicaSet:

1. Delete it with the policy that leaves the pods alive and check that they are still there.
2. Verify that the pods no longer have an owner.
3. Apply the same manifest again and prove, by looking at the pods' ages, that no new pods have been created.
4. Explain in three lines why this is a real operational advantage and what would have happened if the new ReplicaSet had asked for 1 replica instead of 2.

A colleague has deployed this manifest in rutas-norte-dev and ever since "pods delete themselves":

1. Explain exactly what is happening and why.
2. Predict what would happen if exercise 1's 2 bookings-api pods were running when it is applied.
3. Propose the corrected manifest.
4. State which command you would use, in a real cluster, to identify the culprit behind an unexpected deletion.

Solution 1

The replacement appears in 1-2 seconds.

Solution 2

The empty output confirms that they no longer have ownerReferences.

The pods are 5 and 6 minutes old, while the ReplicaSet is 3 seconds old: nothing new has been created, the existing ones have been adopted.

4. The operational advantage is that it lets you replace the controller without touching the service: if you need to recreate the ReplicaSet (for instance, because its selector has to change, and that is immutable), you delete it with --cascade=orphan, apply the new one and the pods keep serving requests without a single second of downtime. Had the new ReplicaSet asked for 1 replica, it would have adopted both orphans, counted 2 where there should have been 1 and deleted one immediately, picking the youngest.

Solution 3

1. The selector app.kubernetes.io/part-of: rutas-norte matches every pod on the platform, because that label is common to all six components by project convention. The platform-monitor ReplicaSet adopts everything it finds in the namespace, compares it against its replicas: 1 and deletes everything in excess. Since the other controllers recreate their pods, you enter a cycle of continuous creation and deletion: the "war of controllers".

2. With the 2 bookings-api pods running, applying the manifest would make the monitor adopt them alongside its own pod (3 in total), see 2 too many and delete two of them, most likely the bookings-api ones if they are the youngest. The bookings-api ReplicaSet would recreate them, the monitor would delete them again, and the API would fill up with SuccessfulDelete and SuccessfulCreate events. In all likelihood, bookings-api would go down intermittently.

3. Corrected manifest: a selector that is exclusive to the component.

4. To identify the culprit behind an unexpected deletion:

You now have the course's first controller fully taken apart. You know that a controller is an infinite, idempotent, level-driven loop that only talks to the kube-apiserver, and you have seen how the ReplicaSet controller makes it concrete: it reads spec.replicas, counts the pods that match spec.selector and creates or deletes until the numbers add up, using spec.template as the mould. You know the manifest's Russian-doll structure and the non-negotiable rule that the template's labels must satisfy the selector.

You have verified self-healing first-hand: you deleted a web-store pod and it came back in two seconds; you deleted them all and all three came back. That is, literally, the problem that cost Rutas Norte four and a half hours of sales one night in March, solved. You understand that the owner-child relationship is materialised in ownerReferences, that the garbage collector uses it for cascading deletion, and that the three propagation policies —Background, Foreground and Orphan— give you fine control, including the manoeuvre of replacing a controller without interrupting the service. You have seen the friendly face of orphan adoption and its dangerous one: an overly broad selector turns a ReplicaSet into a destroyer of other people's pods. And you know how to scale with kubectl scale, without forgetting that the change must always end up in Git.

But you have also discovered its limit: changing the template's image does not update the existing pods. A ReplicaSet maintains, it does not roll out. It does not know how to change version, or go back, or keep a history. All of that comes from the layer above, which is exactly the next lesson: Deployments. There we will finally turn the bare web-store pod into a 3-replica Deployment, we will create the bookings-api one, and we will watch the very ReplicaSet we have just learned to read appear under the bonnet, managed automatically.