The previous lesson ended by pointing out the hidden assumption behind everything we have built so far: we have taken it for granted that, when the HorizontalPodAutoscaler asks for thirty replicas of bookings-api, there is somewhere to put them. There is not. The Rutas Norte cluster has four nodes, and thirty bookings-api pods plus fifteen web-store plus twenty notifications-worker do not fit in them, not by a long way.

That situation has a name and a very concrete appearance: pods in Pending, with a FailedScheduling event saying Insufficient cpu. We already know how to read it from 06-05 and 07-06. What we do not yet know is what to do about it automatically.

This is the third and final level of scaling. The HPA changes the number of pods. The VPA changes the size of each pod. Cluster autoscaling changes the number of nodes. Without it, the other two have a hard ceiling they cannot break through.

We are going to look at the Cluster Autoscaler in detail: how it decides to scale up, the delicate business of removing nodes (which is where all the real problems live), how long a node genuinely takes to become available, the over-provisioning pattern that compensates for those times, and Karpenter as the modern alternative. We will finish with Rutas Norte's concrete capacity plan for the May bank-holiday weekend, with its numbers and its cost.

1. The hard ceiling: when the HPA asks and there is no room
2. The exact symptom: Pending and FailedScheduling
3. What the Cluster Autoscaler is and where it lives
4. How it decides to scale up: simulation and expanders
5. Node groups and their relationship with the provider
6. Removing nodes: the delicate part
7. The reasons a node cannot be drained
8. Diagnosis: the status ConfigMap and the events
9. The real timings: how long a node genuinely takes
10. Over-provisioning with filler pods
11. Karpenter: the modern alternative
12. How the three autoscalers fit together
13. What can be practised on minikube
14. The Rutas Norte capacity plan and its cost
15. Common Mistakes and Tips
16. Exercises
17. Conclusion

Let's set the scene with real numbers. The Rutas Norte production cluster:

And the idle consumption, with every component at minReplicas:

That leaves about 6.2 free cores. Now the May bank holiday arrives:

Fifteen bookings-api pods will be left in Pending. The HPA will have done its job: it asked for 30 replicas and the Deployment created them. The ReplicaSet has them registered. But the scheduler cannot find anywhere to put them, and there they stay.

And here is the worst of it: the HPA does not know. In kubectl get hpa you will see REPLICAS 30. In Grafana, the replicas panel will say 30. The platform will have 15 pods ready and 15 ghosts. And since the HPA computes the CPU average only across the pods that do exist and are ready, it will see them at 90% and will ask for... nothing more, because it is already at maxReplicas.

The result is a platform that falls over under the illusion of being scaled.

Let's learn to recognise it precisely, because it is the trigger for everything else.

Pending with 0/2 and no restarts: the pod exists in the API but no node has accepted it. There are no containers running because there is nowhere to run them.

Two golden lines:

Node: <none> — confirmation that it is not assigned to any node.

0/4 nodes are available: 4 Insufficient cpu — the scheduler evaluated the four nodes and all four failed on insufficient CPU. The message always has the shape X/Y nodes are available: <reasons>, and the reasons are grouped by cause. The ones you will meet in practice:

The NotTriggerScaleUp line is the Cluster Autoscaler's, and in this example it says max node group size reached: the CA is installed, it has seen the pending pod, and it can do nothing because the node group is already at its maximum size. We will come back to this message in section 8, because it is the main diagnostic tool.

A very useful command to see every pending pod at a glance:

And to see the real occupancy of the nodes:

Note that the column that matters is Requests, not Limits. The scheduler places pods according to requests. A node at 97% of requests is full for scheduling purposes, even if real consumption is 30%. It is the lesson of module 3, now with capacity consequences.

The Cluster Autoscaler (CA) is a component that adjusts the number of nodes in the cluster. Like the VPA, it lives in the kubernetes/autoscaler repository and is not part of the Kubernetes core.

How it works, in one sentence: it watches the pods that cannot be scheduled and adds nodes; it watches the underutilised nodes and removes them.

It runs as a Deployment inside the cluster itself, normally in kube-system, with a ServiceAccount that holds broad permissions (read pods and nodes, create events, evict pods) and cloud-provider credentials to manipulate the node groups.

On managed Kubernetes (10-06) the provider installs and configures it for you: on EKS, AKS and GKE you switch autoscaling on with a toggle and the provider takes care of the rest. That is the most common scenario, and it is why you should understand its behaviour even if you never install it by hand.

That third requirement matters more than it looks. The CA assumes that every node in a group is identical. When it simulates whether a pod would fit on a new node from group A, it uses group A's template. If the group has heterogeneous machines, the simulation lies and the CA makes the wrong decisions. Hence the universal recommendation: one machine type per node group.

And the fourth requirement connects with the whole of module 3: a pod without requests is invisible to the scheduler for capacity purposes, and to the CA as well. Here, once again, requests is the foundation of everything.

The CA loop runs every 10 seconds by default (--scan-interval).

1. It lists the unschedulable pods. The ones that have been Pending with FailedScheduling for at least --max-pod-provisioning-time.
2. It groups equivalent pods. Pods from the same controller with the same requirements are treated as one group, so as not to simulate the same thing twenty times.
3. For each node group, it simulates. "If I add a node from group A, how many of these pending pods would fit?" The simulation uses the real scheduler, with all its predicates: resources, taints, affinities, topology spread, host ports.
4. It discards the groups that do not help. If adding a node from group B would not allow a single pod to be scheduled (because it has a taint the pod does not tolerate, for example), that group is discarded.
5. It chooses among the viable groups using the expander.
6. It computes how many nodes are needed and asks the provider to grow the group.

An important detail of step 6: the CA can add several nodes at once. If there are 15 pending pods of 520m each and 6 fit on a 4-core node, it will ask for 3 nodes in one go, not one at a time. This is bounded by --max-nodes-total and by the group's maximum.

When several node groups would do the job, the expander decides which one.

A concrete example of least-waste. There are 3 pending pods needing 500m and 512Mi each (total: 1.5 cores and 1.5 GiB), and two groups available:

least-waste picks group A: it fits better and you do not pay for 6 idle cores. most-pods would pick B (many more future pods would fit). price would pick A if it is cheaper.

Configuring the expander:

The --balance-similar-node-groups option deserves a mention because it solves a real availability problem: if you have one node group per availability zone, this flag makes the CA spread new nodes evenly across zones instead of piling them into one. Without it, you can end up with twelve nodes in zone A and none in zone B, and a zone outage leaves you with no platform. It connects directly with what we will see in 09-05.

A very profitable pattern in the cloud. Spot (interruptible) instances cost a fraction of the price, but the provider can reclaim them with two minutes' notice.

The CA tries the priority-100 group first. If the provider has no spot capacity available, it falls back to the priority-50 one. For notifications-worker, which tolerates interruptions without trouble, it is the obvious choice. For bookings-postgres, never.

The CA does not create machines: it manipulates the provider's node groups. Every provider has its own name for the same thing:

A node group has a minimum size, a maximum and a current one. The CA only changes the current one, within those bounds. Everything else (the machine type, the image, the initial taints and labels) is defined by the group.

The Rutas Norte groups:

Three design decisions worth understanding:

One group per zone. Necessary so that --balance-similar-node-groups can spread across zones and so that zone affinity works. With a single multi-zone group, the CA cannot guarantee which zone the new node will appear in, and that breaks the topology spread of 09-05.

A dedicated data group with a taint. bookings-postgres and redis-cache have tolerations for role=data (06-05) and node affinity towards that group. That way the databases live on big machines with fast disks and do not share a node with the avalanche of bookings-api pods. The taint stops the application pods from landing there.

The data group has max=3, which is very low. That is deliberate: we do not want the CA booting big, expensive machines by mistake. bookings-postgres does not scale horizontally (09-01), so that group almost never needs to grow.

When the CA simulates whether a pod would fit on a node from group A, it uses a template. If the real node, on boot, receives labels or taints the template did not know about, the simulation lies.

A typical and very frustrating case: a pod with nodeSelector: disk=ssd is Pending. The CA simulates with the group template, which does not carry that label, concludes that the pod would not fit even on a new node, and does not scale up. But the real nodes in the group do receive that label on boot, through a start-up script.

The fix is to declare the labels and taints in the provider's own node-group definition (on AWS, with k8s.io/cluster-autoscaler/node-template/label/... tags on the ASG), so that the CA knows about them before booting the node.

Scaling up is easy: there are pending pods, you add nodes. Scaling down is where all the problems are, because removing a node means evicting the pods it holds and trusting that they will reappear somewhere else without breaking anything.

Every 10 seconds, for each node:

1. Is it below the utilisation threshold? By default, --scale-down-utilization-threshold=0.5: the sum of its pods' requests is less than 50% of its allocatable capacity.
2. Has it been that way long enough? --scale-down-unneeded-time=10m: ten consecutive minutes below the threshold.
3. Can it be drained? It simulates whether all its pods would fit on other existing nodes. If they do not fit, it is left alone.
4. Is there any pod that prevents eviction? The list in section 7.
5. If everything passes: it cordons the node, evicts its pods respecting the PDBs, waits, and asks the provider to delete it.

Main parameters:

--scale-down-delay-after-add is the key parameter for avoiding node flapping. Without it, the CA could add a node, see that the cluster is now underutilised and remove it immediately. Ten minutes of grace break that cycle.

About the utilisation threshold, an important calculation:

Note that DaemonSets count towards utilisation but do not prevent deletion (they are ignored in step 4, because they disappear with the node). With many DaemonSets, the baseline utilisation of an empty node is already high, and that makes the CA scale down less than you would expect. In clusters with many agents (logging, security, service mesh, monitoring), it is common to have to lower the threshold to 30-40% for scale-down to work at all.

A node that only holds DaemonSets is considered empty and is deleted via a fast path, without the full simulation. That is why, after a traffic peak, the nodes that end up completely free disappear fairly quickly, while the ones with a stray pod take far longer.

This explains a phenomenon that puzzles a lot of people: after the May bank holiday, eight nodes go away in twenty minutes and two stay for hours with a single pod each. The real fix is not to fiddle with the CA: it is topology spread and consolidation (Karpenter, section 11).

This is the list to know by heart, because it explains 90% of the cases of "I have empty nodes the autoscaler will not remove and I am paying for them".

It is the most direct control you have:

When "false" makes sense:

• A long batch job that would lose hours of compute if restarted. occupancy-reports is a candidate: if it takes 40 minutes and the CA kills it at minute 35, the night's work is lost.
• A database migration in flight.
• A pod with irreproducible local state.

And the opposite value, "true", serves to unblock situations:

It is the clean way to resolve reason 2 without changing the CA's global flag. In Rutas Norte, web-store uses an emptyDir for the nginx cache: marking it as safe-to-evict: "true" lets the CA remove its nodes with no trouble, because that cache regenerates itself.

The most frequent case and the most dangerous. A PDB like this:

With a single replica of bookings-postgres and minAvailable: 1, that pod can never be evicted: doing so would leave 0 available. The node hosting it is pinned permanently. The CA will try, will fail, and will record it in its events.

In this particular case the block is desirable (we do not want the CA moving the database on its own), but the same pattern applied by mistake to an ordinary service produces zombie nodes that nobody can explain. We develop this fully in 09-05.

The CA publishes its internal state in a ConfigMap. It is the first place to look.

How to read it:

In the output above: group A has 3 ready and 1 booting. The CA has already requested the node; all you have to do is wait. This information is exactly what you need during an incident to know whether the CA is working or stuck.

ScaleDown states you will see:

When the CA decides not to do anything, it says so in an event on the pending pod:

This message is a gift: it tells you exactly why each node group was discarded. Here: one group is at its maximum and another has a taint the pod does not tolerate.

Common NotTriggerScaleUp messages and what they mean:

And for blocked scale-down:

The CA logs are the definitive source when the ConfigMap and the events are not enough. Keep them in EFK (07-05): they are indispensable for the postmortem of a capacity incident.

Here is the figure that changes the strategy, and the one almost nobody mentions.

Adding it up: between 3 and 4 minutes in the good case. In the bad case (a large uncached image, a saturated region, heavy DaemonSets) it can exceed 6 minutes.

Typical breakdown in a table:

Let's go back to the May bank holiday. The sale opens at 10:00. The real traffic profile:

Five minutes of degraded latency and errors at the most commercially valuable moment of the year. Thousands of users abandoning their purchase. The Cluster Autoscaler, however well configured, cannot solve a spike that rises in forty seconds, because the physics of booting a virtual machine does not allow it.

Two strategies come out of this, and you need both:

1. Over-provisioning: having capacity already booted and waiting (section 10).
2. Anticipatory scaling: scaling before the peak using a time-based trigger rather than a reactive one. That is KEDA's cron trigger, which we will see in 09-04.

The pattern that solves the problem of the previous section, and which makes brilliant use of the PriorityClass and preemption from 06-05.

We deploy pods that do nothing — they sleep — but that reserve CPU and memory, with a negative priority. The effects:

1. The CA counts those requests as occupancy, so it keeps nodes booted to host them.
2. When a real pod arrives (priority 0 or higher) and there is no room, the scheduler instantly evicts a filler pod and places the real one in its slot. Preemption takes seconds, not minutes.
3. The evicted filler pods go to Pending, which triggers the CA to boot new nodes... which will refill the cushion for next time.

It is a self-regenerating capacity cushion. You buy idle capacity in exchange for response time.

Step 1: the negative PriorityClass.

Two important details:

• value: -10 guarantees that any pod without a priorityClassName (priority 0) outranks them.
• preemptionPolicy: Never means these pods, once Pending, will not try to evict others. It would be absurd for the cushion to throw out a real pod.

Step 2: the filler Deployment.

The pause image is the canonical choice: it is the same one Kubernetes uses internally for each pod's infrastructure container, it weighs less than a megabyte, it is already cached on every node, and its only job is to sleep.

Step 3: checking that it works.

Now we force a spike:

And in the events of the bookings-api pods:

Three seconds from the pod being created to being scheduled, instead of three and a half minutes. That is the difference between absorbing the opening of the sale and falling over.

Nothing is free:

Is it worth it? At Rutas Norte the answer is clear: five minutes of outage at the opening of the May bank-holiday sale cost far more than €1,940 in unsold tickets. But it is a business decision, not a technical one, and it must be taken with the numbers on the table.

An obvious optimisation: the cushion does not have to be constant. A CronJob (06-03) that scales the filler Deployment from 2 to 6 replicas on Friday afternoons and returns it to 2 on Mondays reduces the annual cost to a fraction:

And its counterpart on Monday at 6:00 with --replicas=2. The scaling-operator ServiceAccount needs a Role with permission over deployments/scale, exactly as we saw in 08-01.

The Cluster Autoscaler has a design limitation: it works with predefined node groups. Someone has to decide in advance which machine types exist, create a group for each of them, and maintain them.

Karpenter turns the approach around: it looks at the pending pods, works out which machine would be best for them, and boots it directly, with no groups.

Instead of node groups, you define constraints on which machine types are acceptable:

With this definition, when 15 bookings-api pods turn up pending, Karpenter works out that the best machine to host them is, say, a c6i.4xlarge spot instance in zone B, and boots it. Without anyone ever having created a c6i.4xlarge node group.

This is the most practical difference day to day. Karpenter continuously checks whether the current set of pods would fit on a cheaper node configuration, and if so, it migrates the pods and replaces the nodes.

The scenario it solves: after the May bank holiday, the CA leaves five nodes with one pod each, because none drops below the utilisation threshold in a way that allows draining. Karpenter sees that those five pods would fit on a single node, boots that node, migrates the pods and deletes the five.

It is exactly the problem we mentioned at the end of section 6, solved at the root.

When to choose each one:

• Cluster Autoscaler if you are outside AWS, if you have regulatory requirements about which machine types are admissible, if your provider already gives it to you configured and working, or if you value predictability over savings.
• Karpenter if you are on AWS, if cost is a genuine priority, if your workload is heterogeneous (small and large pods living together), and if the reduction in operational work pays off for you.

For Rutas Norte, with four nodes and a homogeneous workload, the CA is more than enough. If the platform grew to fifty nodes with varied workloads, Karpenter would be the obvious decision. We will come back to this kind of decision in 10-06 and 11-06.

An important note the comparison must not hide: Karpenter does not fix the forty-second spike either. One or two minutes is still too long. Over-provisioning is still necessary.

We now have all three pieces. Let's see the full flow in time order.

The three routes and their timings:

The strategic conclusion is direct: the over-provisioning cushion is not a luxury, it is what turns a five-minute incident into a thirty-second one.

And there is still a better route, which does not appear in the diagram because it is not reactive: scaling before the traffic arrives. If at 9:30 you already have 25 replicas and eight nodes because a cron trigger ordered it, at 10:00 there is no incident at all. That is 09-04.

All three are necessary and none replaces another. The HPA without the CA has a hard ceiling. The CA without the HPA never fires. The VPA makes both more accurate, because correct requests are the foundation of both the HPA's calculation and the CA's simulation.

Time for honesty: the Cluster Autoscaler cannot really be tested on minikube. There is no cloud provider to boot machines. The rutas-norte profile has the nodes you asked for when you created it, and that is all of them.

1. Producing the symptom and learning to read it.

You will see exactly the FailedScheduling with Insufficient cpu. It is the same message as in production.

2. The over-provisioning pattern with preemption. This works perfectly, because preemption is a function of the scheduler, not of the cloud provider.

Seeing the Preempted with your own eyes and measuring that it happens in seconds is the most valuable part of the exercise, and you can practise it without a cloud.

3. Adding and removing nodes by hand, simulating what the CA would do.

That cordon → drain → delete sequence is exactly what the CA does internally. Practising it by hand teaches you more about node removal than reading any documentation, and it is the basis of the maintenance procedure we will see in 09-05.

4. Practising the safe-to-evict annotation and seeing how it blocks a drain.

To practise the CA for real you need: a managed cluster (10-06) with autoscaling enabled, or kind with the CA's test provider, or a personal cloud environment. A practical recommendation: a trial account with a provider for one afternoon, with a small cluster and a low maxSize, costs a few euros and teaches an enormous amount.

We close with the concrete numbers. This is the work that has to be done before a May bank holiday, and that was not done last year.

From the maxReplicas of 09-01 and the requests recalibrated with the VPA in 09-02:

The 33% margin is not a whim: it covers estimation errors, a deployment in progress during the peak (which temporarily doubles the pods of one component), and the loss of an entire zone.

Fictitious prices, but of a realistic order of magnitude:

And the comparison that justifies the whole module:

The last row is the goal: practically the same cost as the insufficient fixed sizing, with the capacity of the peak sizing. That is the entire value of module 9 summed up in one line.

And a reminder of intellectual honesty: the reduced off-season cushion only works if the CronJob that expands it actually runs. Put an alert on it: if at 16:05 on Friday the filler Deployment does not have 6 replicas, somebody has to find out.

Mistake 1: believing the HPA can take you further than the capacity allows, and not noticing. The symptom is silent: kubectl get hpa shows 30 replicas and everything looks fine. Only kubectl get pods reveals that 15 are Pending. A mandatory alert on pending pods:

Mistake 2: HPA maxReplicas that is inconsistent with the node groups' maxSize. A maxReplicas: 100 with groups that top out at 8 nodes is a lie: you will never reach 100 replicas. Always compute Σ(maxReplicas × requests) and compare it with Σ(maxSize × allocatable capacity). Document that calculation next to the manifests.

Mistake 3: pods without requests in a cluster with a CA. A pod without requests is invisible to the CA's simulation: the autoscaler believes a node has room to spare when in fact it is saturated with pods consuming without declaring anything. The LimitRange of module 3 is the defence: it forces a defaultRequest onto everything that gets created.

Mistake 4: expecting the CA to save you from a sudden spike. Three to six minutes. If your traffic rises in forty seconds, the CA arrives late. Over-provisioning (section 10) and anticipatory scaling (09-04). There is no third way.

Mistake 5: impossible PDBs that pin nodes forever. A minAvailable equal to the replica count means no pod can ever be evicted, and the node hosting them will never be removed. You will be paying for underutilised nodes without knowing why. We develop this in 09-05, but the diagnosis is in the CA logs: pdb-blocked.

Mistake 6: a single node group for everything. Without separating the data from the application, an avalanche of bookings-api pods can land on the same node as bookings-postgres and compete for its CPU. Taints and separate groups (06-05).

Mistake 7: heterogeneous nodes inside the same group. The CA uses one template per group. If the group mixes 2-core and 8-core machines, the simulation lies and the decisions are erratic. One machine type per group, always.

Tip 1: --balance-similar-node-groups if you have one group per zone. Without it, the CA can pile all the new nodes into one zone and leave you exposed to a zone outage. It is one line of configuration with a direct impact on availability.

Tip 2: monitor cluster_autoscaler_unschedulable_pods_count. The CA exposes Prometheus metrics. The ones that matter:

That third indicator is especially valuable: a failed_scale_up means the provider has said no. Quota exhausted, an instance type unavailable in the zone, an account limit. These are problems you only discover when it is too late, unless you watch for them.

Tip 3: rehearse the scaling before the peak. A week before the bank holiday, run a load test against rutas-norte-pre (09-06) and time how long the cluster takes to reach maximum capacity. That number, measured rather than estimated, is what tells you whether the cushion is enough.

Tip 4: raise the groups' maxSize before high season and lower it afterwards. It is a one-minute change that avoids max node group size reached at the worst possible moment. Put it on the pre-event checklist.

Tip 5: keep the CA logs. They are the only source that explains why it did not scale up or why a node was not removed. Without them, the postmortem of a capacity incident is guesswork. EFK (07-05).

Tip 6: review the scale-down threshold if you have many DaemonSets. With Fluentd, Falco, the CNI, the node exporter and perhaps a service-mesh proxy, the baseline utilisation of an empty node can be around 25%. With the default 50% threshold, scale-down almost never fires. Lowering it to 35% can save entire nodes.

The Rutas Norte team has configured this for the May bank holiday:

HPAs:

Fixed components: bookings-postgres (1 replica, 2 cores, 4Gi), redis-cache (1 replica, 300m, 2Gi), Ingress (4 replicas, 200m, 256Mi).

Node groups:

DaemonSets: 450m and 512Mi per node. System reservation: 500m and 2Gi per node.

Namespace ResourceQuota: requests.cpu: 40, requests.memory: 80Gi.

Compute: (a) the maximum CPU and memory the application can demand; (b) the real maximum allocatable capacity of the groups; (c) whether the configuration survives the peak; (d) the percentage of bookings-api's maxReplicas that is achievable in practice; (e) what you would fix and in what order.

After the May bank holiday, the cluster still has 9 nodes even though traffic returned to normal 5 hours ago. The bill is a concern. This is what you see:

Analyse each blocked node, say whether the block is legitimate or a mistake, and propose the fix for each case. Compute how many nodes could be removed after the fixes and the monthly saving (€95/node/month).

Rutas Norte is launching a Black Friday campaign unlike anything before:

• Duration: exactly 2 hours, from 20:00 to 22:00 on a Friday.
• Expected traffic: x25 compared with a normal day (far more than the May bank holiday).
• The peak arrives in 20 seconds: it is announced on the radio and everyone comes in at once.
• Tickets are limited: if the platform does not respond in the first 10 minutes, the campaign has failed.
• Budget: up to €400 for that night.
• It is known three weeks in advance.

Design the complete capacity strategy. What combination of HPA, CA, over-provisioning and scheduled scaling would you use? How many nodes, how much cushion, how far in advance? What would it cost? Write the key manifests and a checklist with concrete timings for that night.

Solution 1

(a) Maximum demand from the application:

(b) Real maximum allocatable capacity:

(c) Does it survive the peak? NO.

And there is a second problem many people would overlook: the ResourceQuota.

(d) Percentage of bookings-api's maxReplicas that is achievable:

The maxReplicas: 40 is fiction: the reality is 25 replicas. And worse: when the HPA asks for replicas 26 to 40, they will sit in Pending and nobody will find out unless there is an alert.

(e) Fixes, in order of priority:

Priority 1 (essential): raise the maxSize of the general groups.

Priority 2: reduce the maxSize of the data group.

Priority 3: review bookings-api's maxReplicas: 40.

With 12 general nodes there is room for 25 replicas... wait, let's recompute with the new capacity:

Priority 4: add the over-provisioning cushion. Without it, replicas 5 to 40 will take between 3 and 6 minutes to appear. With 6 filler pods of 1 core, the first 11 replicas are instantaneous.

Priority 5: the pending-pods alert. Without it, this whole calculation could be wrong and nobody would know until it was too late.

Solution 2

Node-by-node analysis:

Case 1: ip-10-0-1-88 — occupancy-reports with safe-to-evict: false.

Legitimacy: conditional. If the Job is running right now, the block is correct: killing it would lose the work. But the bank holiday ended 5 hours ago and occupancy-reports is a nightly CronJob.

If it shows as Running and has been for hours, it is a stuck Job: it probably failed and got wedged. The fix:

And a structural prevention in the CronJob:

Node recoverable: YES, after cleaning up the Job.

Case 2: ip-10-0-2-73 — redis-cache-0.

Legitimacy: yes, completely. redis-cache is a single-replica StatefulSet with a volume. The CA cannot move it safely, and it certainly must not do so on its own initiative: restarting the cache dumps the whole load onto bookings-postgres.

Fix: none in the short term. In the medium term, the options are:

• Pin redis-cache to the data node group with affinity and a taint (06-05), so that it does not occupy a general node.
• Accept it: a node dedicated to the cache is not waste if it is sized properly.

The best fix is the first: redis-cache should not be on a general node. It is an architectural decision we should already have taken.

Node recoverable: NOT directly, but yes once redis-cache is moved to the data group.

Case 3: ip-10-0-1-15 — the PDB blocking bookings-api.

Legitimacy: no, it is a configuration error. bookings-api is a Deployment with many replicas: moving one pod should be trivial. That the PDB prevents it means the PDB is wrong.

The likely suspect:

After the bank holiday, the HPA has come down to 4-6 replicas. With minAvailable: 30 and 6 live replicas, no pod can ever be evicted: we are already below the minimum.

The fix:

This is the central lesson of 09-05, previewed: with an HPA, absolute-number PDBs are a trap. The replica count changes on its own, and a minimum that was reasonable at the peak becomes impossible in the trough.

Node recoverable: YES, immediately after fixing the PDB.

Case 4: ip-10-0-3-22 — load-generator with no controller.

Legitimacy: no, it is junk. It is the load-test pod from section 12 of 09-01, created with kubectl run without a controller and then forgotten. It has spent days there pinning an entire node.

Prevention:

• Always use --rm with kubectl run for tests.
• A Kyverno policy (08-03) that rejects pods without ownerReferences in rutas-norte-pro.
• A periodic review of orphan pods:

Node recoverable: YES, immediately.

Summary and saving:

Verification after the fixes:

It should go from NoCandidates to CandidatesPresent and then to InProgress.

The underlying lesson of the exercise: zombie nodes are never the Cluster Autoscaler's fault. They are always the consequence of something misconfigured in the workloads: an impossible PDB, a stuck Job, an orphan pod, a stateful component in the wrong place. The CA is the messenger; the logs are the message.

Solution 3

Analysing the scenario. This case is qualitatively different from the May bank holiday:

The key conclusion: reactive scaling is NO USE here. Neither the HPA (15 s of detection + 30 s of start-up) nor the CA (3-6 min) arrives in time for a rise of 20 seconds. All the capacity has to be booted and warm before 20:00.

Strategy: total pre-scaling.

Step 1: compute the capacity needed.

Step 2: translate into nodes.

Step 3: the manifests.

Node groups for that night:

Note the important detail: the groups' min is raised to 3, we do not rely on the CA. By setting the minimum high, the provider boots the machines and the CA cannot remove them. The capacity is guaranteed by construction, not by reaction.

Step 4: the cost.

The result is surprising and it is the lesson of the exercise: sizing generously for two hours is dirt cheap. The cost of cloud infrastructure is proportional to time, and two hours of over-provisioning cost less than a coffee per node. With a €400 budget, there is no excuse whatsoever for skimping on capacity that night.

With the spare margin you can afford improvements:

• Going to 12 nodes from the start instead of 8: +€5.20.
• Doubling the redis-cache replicas as a read replica: +€2.
• Keeping the capacity until 2:00 to drain the notifications queue without rushing: +€15.

All of that fits comfortably.

Step 5: the checklist for the night.

Critical notes on the plan:

1. The 18:30 verification is the most important one. 88 pods created is not the same as 88 pods Ready. If at 18:30 there are 20 in Pending, there are two hours to fix it. If you find out at 20:01, there is nothing to be done.

2. The scaleDown: Disabled is non-negotiable. Without it, a one-minute trough between 20:15 and 20:16 could destroy 30 replicas that would take minutes to come back.

3. notifications-worker is not scaled during the campaign. It is a deliberate decision: the confirmation emails can wait two hours, the sale cannot. All the capacity goes to sales. This is the kind of explicit trade-off that distinguishes a capacity plan from a wish list.

4. The "touch nothing" rule during the campaign. Almost every serious incident at events like this is caused by somebody trying to fix something under pressure. If the dashboards are green, you look and you do not touch.

5. Pre-warming the caches at 19:30 saves the first 5,000 users from paying the cost of cold queries against PostgreSQL. It is half an hour of work with an enormous impact on the first few minutes.

Cluster autoscaling is the level that holds up the other two. Without it, the HorizontalPodAutoscaler has a hard, invisible ceiling: it asks for replicas, the Deployment creates them, and they sit in Pending while the dashboards show numbers that do not correspond to reality.

The essentials:

• The symptom is always the same: pods in Pending with FailedScheduling and Insufficient cpu. Recognising it and alerting on it comes first.
• The Cluster Autoscaler watches unschedulable pods, simulates which node group would host them, and chooses with an expander. least-waste is the most sensible in general.
• Scaling up is easy; scaling down is where the problems are. A utilisation threshold, a waiting time, and a list of blockers you have to know: pods without a controller, local storage, restrictive PDBs, system pods, and the safe-to-evict: "false" annotation.
• Diagnosis means reading three places: the cluster-autoscaler-status ConfigMap, the NotTriggerScaleUp events on the pending pods, and the CA logs with their not suitable for removal.
• A node takes between 3 and 6 minutes to go from being requested to serving traffic. That does not save you from a spike that rises in forty seconds, however well configured everything is.
• Over-provisioning with negative-priority pods turns three and a half minutes into three seconds, in exchange for paying for idle capacity. It is the most elegant use of the PriorityClass and preemption of 06-05, and it regenerates itself.
• Karpenter picks the machine according to the pending pods instead of working with fixed groups, and consolidates continuously. Better on AWS and with heterogeneous workloads; the CA remains the solid, predictable option everywhere else.
• The three autoscalers fit together as a cascade: the HPA creates pods → if they do not fit, Pending → the CA adds nodes. And the cushion cuts out the long path.
• The Rutas Norte capacity plan costs practically the same as last year's insufficient fixed sizing, but with the capacity of the peak sizing. That is the entire value of the module.

But we still have an underlying problem that none of the three lessons has solved. Everything we have built is reactive: it waits for CPU to rise before acting. And there are two cases where that simply does not work.

The first is notifications-worker. During the May bank holiday it can have forty thousand emails waiting in the queue and CPU at 20%, because its job is to wait for responses from the mail server, not to compute. A CPU-based HPA would see that 20% against a 70% target and conclude that there are replicas to spare: it would scale down precisely when the most consumers are needed.

The second is the very moment the sale opens. We know it happens at 10:00 sharp. It is information we have months in advance. And yet our entire system waits until 10:00:15 to discover, through CPU, something we already knew in March.

In the next lesson, Event-Driven and Custom-Metric Scaling with KEDA, we solve both. We will see how the HPA can consume business metrics through the aggregation API, what KEDA is and why it does not replace the HPA but feeds it, how to scale notifications-worker by the real length of its queue — scale-to-zero included —, how to scale bookings-api by requests per second using the metrics we instrumented in 07-03, and how to schedule a trigger that pre-warms the whole platform half an hour before the May bank-holiday sale opens.