We closed module 8 with a secure, auditable platform, but with an uncomfortable confession: the Rutas Norte components still have a fixed, hand-written replica count. bookings-api has replicas: 4 and web-store has replicas: 3 because someone, months ago, took a measurement on an ordinary Tuesday and thought it was enough. That number knows nothing about the May bank-holiday weekend. Last year the platform fell over eleven minutes after the sale opened, and the team's answer was a hurried kubectl scale from a laptop, on a station platform, with the train pulling out.

This lesson removes that scramble. The HorizontalPodAutoscaler (HPA) is the Kubernetes object that automatically adjusts a Deployment's replica count according to observed metrics. It is not magic: it is one more controller, with its reconciliation loop — the same one we studied in module 1 —, a very concrete piece of arithmetic and a handful of parameters you need to understand properly if the result is to be stability rather than a pendulum of pods starting up and dying.

We are going to look at which workloads can be scaled horizontally and which cannot, the full autoscaling/v2 API, the exact formula used to compute the desired replicas, the behavior block that governs how fast it scales up and down, and the two definitive manifests for bookings-api and web-store. We will finish with a conflict that bites almost every team the first time round: the HPA and the replicas field of the versioned YAML fighting over the same number.

1. Scaling out versus scaling up
2. Which workloads support horizontal scaling and which do not
3. Anatomy of the HorizontalPodAutoscaler in autoscaling/v2
4. The four metric types: Resource, Pods, Object and External
5. The controller's exact formula, step by step
6. The tolerance band and replica flapping
7. Non-negotiable requirements: requests and metrics-server
8. Diagnosing the dreaded <unknown>
9. The behavior block: policies, periods and the stabilisation window
10. Several metrics at once: the greediest one wins
11. The Rutas Norte HPAs, complete manifests
12. Simulating a load spike and watching it live
13. The conflict between the HPA and the replicas field
14. What not to scale with an HPA, and the real bottleneck
15. Common Mistakes and Tips
16. Exercises
17. Conclusion

There are exactly two ways to give a service more capacity, and it is worth keeping the names straight because the rest of the module leans on them.

Scaling up (scale up, vertical scaling) means giving each instance more resources: raising the CPU requests/limits from 200m to 800m, the memory from 256Mi to 1Gi. The number of pods does not change; each pod is bigger. In Kubernetes this means recreating the pod, because a container's resources are part of its immutable specification (with the caveat of in-place resizing, which we will see in 09-02).

Scaling out (scale out, horizontal scaling) means adding more instances of the same size: going from 4 bookings-api pods to 12 identical pods. Each pod is still exactly as big; there are simply more of them. In Kubernetes this is changing one integer: the Deployment's replicas.

The rule of thumb: vertical scaling solves the problem of a badly sized instance; horizontal scaling solves the problem of traffic volume. They are not alternatives, they are different axes. Rutas Norte needs both: getting each pod's size right (09-02) and multiplying the number of pods when the May bank holiday arrives (this lesson).

One important detail many people overlook: horizontal scaling also improves availability, whereas vertical scaling does not. Twelve bookings-api pods spread across nodes survive the loss of a node; a single enormous pod does not. We will return to this idea in depth in 09-05.

Horizontal scaling only works if any replica can serve any request. This is called being stateless, and it is a property of the application, not of Kubernetes. Kubernetes will happily let you scale anything; whether the result is correct depends on how the application is written.

Let's go over the Rutas Norte components:

The bookings-postgres case deserves a paragraph. It is a StatefulSet with one persistent volume per replica. If you put an HPA on it and the HPA decided to go from 1 to 5 replicas, Kubernetes would create five pods bookings-postgres-0 through bookings-postgres-4, each with its own empty PVC, each starting an independent, empty database. The Service would spread requests across all five. The result would be silent data corruption: some bookings would go to one database and some to another. Scaling a relational database is a data-architecture problem (read replicas, partitioning, an operator that knows how to do it — 06-07), not a problem of counting pods.

A rule worth memorising: if adding a replica does not make the system answer more correct requests per second, the HPA is not your tool.

The HPA is an API object like any other. Its stable, current version is autoscaling/v2; the v2beta1 and v2beta2 versions have been removed for several releases now, and autoscaling/v1 only supports CPU (the server still serves it for compatibility, but do not use it: you lose behavior and every metric that is not CPU).

Here is the complete skeleton, with comments on each field:

Three starting points worth nailing down before we go on:

• scaleTargetRef points at an object that exposes the /scale subresource. Deployments, ReplicaSets and StatefulSets expose it. A DaemonSet does not, because its pod count is determined by the number of nodes (06-02), and it would make no sense.
• minReplicas and maxReplicas are hard barriers. maxReplicas is your financial and operational safety net: if an application bug drives CPU to 100% with no real traffic, the HPA will try to scale forever; the ceiling stops it. Always set it, and set it thinking about what would happen if it were reached.
• minReplicas can be 0 from Kubernetes 1.30 onwards if the HPAScaleToZero feature gate is enabled, but that requires an external or object metric and is not enabled by default in most clusters. Real scale-to-zero we will do with KEDA in 09-04.

The HPA is driven by the horizontal-pod-autoscaler that lives inside the kube-controller-manager. Its default cycle is 15 seconds (--horizontal-pod-autoscaler-sync-period). On each cycle:

Note the end of the flow: the HPA does not create pods. It modifies the Deployment's replicas field through the /scale subresource, and from there the Deployment controller and the ReplicaSet controller do their usual work (module 2). It is chained reconciliation, exactly the pattern we saw in 01-02.

The metrics block accepts four types. You will use two of them today; the other two are defined here and put to work in 09-04.

This is the most common form. It expresses the target as a percentage of the container's requests:

This means: "keep the average CPU consumption of the bookings-api pods at around 65% of what each one has reserved in requests.cpu". If requests.cpu is 500m, the target is 325m per pod.

It is essential to understand that Utilization is computed against requests, not against limits nor against the node's CPU. A pod can show 180% utilisation if it consumes 900m with a requests of 500m; that is perfectly possible because requests is a minimum reservation, not a cap (module 3).

This expresses the target in absolute units, ignoring requests:

"Keep the average memory consumption per pod at 700Mi."

When should you use each one?

A warning about memory as an HPA metric: it is almost never a good idea. Many runtimes (the JVM, the Node.js garbage collector, PostgreSQL) do not return memory to the system even when they no longer need it. Consumption goes up and stays up. An HPA driven by memory would scale out and never come back down, because the memory of the old pods never drops below the target. Use CPU, or better still, a business metric (09-04).

A variant of Resource that looks at one specific container instead of adding up all of the pod's. It is very useful in Rutas Norte, where several pods carry sidecars:

Without this, a sidecar burning a constant 50m distorts the percentage for the whole pod, and in small pods the distortion is large. If your target has sidecars, prefer ContainerResource.

We define them now so that you recognise the syntax, but their real use arrives in 09-04.

The key difference between Value and AverageValue in Object and External: with Value the HPA compares the raw value against the target; with AverageValue it divides the value by the number of replicas before comparing. For a queue you always want AverageValue ("each worker takes charge of 500 messages"), because that is what makes the replica count grow with the queue.

The Pods, Object and External types do not work out of the box: they need a metrics adapter registered in the API aggregation layer. Without one, the HPA will report <unknown> forever. That is exactly what KEDA solves in 09-04.

Here is the heart of the lesson. All of the HPA's behaviour comes out of a single expression:

Where ceil() is rounding upwards to the next integer. Let's apply it to bookings-api with real numbers.

bookings-api is deployed with these resources (we set them in module 3):

And its HPA has averageUtilization: 65, minReplicas: 4, maxReplicas: 30.

Absolute target per pod: 65% of 500m = 325m.

There are 4 replicas. kubectl top pods shows:

Average: (140 + 155 + 132 + 149) / 4 = 144m.

The formula asks for 2 replicas. But minReplicas is 4, so it stays at 4. Correct: we do not want to go below 4 in production for availability reasons.

There are still 4 replicas, but traffic goes through the roof:

Average: (910 + 940 + 895 + 925) / 4 = 917.5m. Note: they are brushing against their 1000m limit, which means they are being throttled (module 3).

The HPA wants 12 replicas. That is below maxReplicas: 30, so it is applied (subject to behavior, which we will see in section 9).

Notice the elegance of the formula: it multiplies the current count by the excess factor. If you are at three times the target with 4 pods, it asks for 12 pods. It is a rule of three that implicitly assumes that load is spread evenly across the replicas, which is true if the Service balances reasonably and the requests are homogeneous.

Now there are 12 replicas and the average drops to 120m:

It asks for 5 replicas. It will go from 12 down to 5, but not all at once: the stabilisation window and the scaleDown policies control the pace (section 9).

These nuances explain behaviours that otherwise look erratic:

1. Pods without metrics. If a pod has no metrics yet (it has just started), the HPA excludes it from the average but counts it conservatively: when scaling up it assumes 0 consumption for that pod (so as not to overestimate), and when scaling down it assumes consumption equal to the target (so as not to underestimate). The result is that the HPA is cautious in both directions while pods are starting.

2. Not-ready pods. Pods that have not passed their readinessProbe are excluded from the calculation when scaling up. This avoids the classic vicious loop: new pods that are not yet receiving traffic, with high CPU because they are starting, would make the HPA believe even more scaling is needed.

3. --horizontal-pod-autoscaler-initial-readiness-delay (30 s by default): during a pod's first 30 seconds of life, its metrics are ignored entirely. The rationale is clear: Node.js start-up burns a lot of CPU compiling and loading modules, and we do not want that spike to contaminate the decision.

4. --horizontal-pod-autoscaler-cpu-initialization-period (5 min by default): an extra margin for the CPU metrics of freshly ready pods.

The practical consequence of points 3 and 4 is that the HPA has a start-up latency of at least half a minute per decision, and that is why the over-provisioning of 09-03 and the cron-based pre-warming of 09-04 make sense.

If the HPA applied the formula literally every 15 seconds, the replica count would oscillate endlessly. With 4 pods at 320m and a target of 325m the formula gives ceil(4 × 0.984) = 4; with 330m it gives ceil(4 × 1.015) = 5. A 10m fluctuation would send one pod up and down indefinitely. That is called flapping or thrashing, and it is expensive: every start-up burns CPU, warms caches from scratch and pollutes the metrics.

Kubernetes prevents it with a tolerance band: if the ratio currentValue / targetValue lies between 0.9 and 1.1 (that is, within ±10%), the HPA does nothing.

With our target of 325m, the dead zone is:

This threshold is cluster-wide (--horizontal-pod-autoscaler-tolerance) and not configurable per HPA... until recently. Since Kubernetes 1.33 there is the HPAConfigurableTolerance feature gate, which lets you set the tolerance per metric inside behavior:

Being an optional feature gate, do not take it for granted: check your cluster's version and configuration before using it in production. On the course's minikube cluster we will work with the default 10% tolerance.

Two requirements, and neither is negotiable.

An HPA using Resource metrics reads from the metrics.k8s.io API, which metrics-server serves. We installed it in 07-02 with the minikube addon. Verify it:

The AVAILABLE column must say True. If it says False (MissingEndpoints) or similar, the HPA will not work.

If kubectl top works, the HPA will have data. If it does not, no HPA is going to help. This is always the first check.

On minikube, if you do not have it yet:

This is the most frequent trap. The Utilization target is computed as a percentage of requests. If a container does not declare requests.cpu, the HPA has no denominator, and its metric stays <unknown> forever. There is no spectacular error message; it simply does not scale.

And beware of sidecars: if the pod has a container without requests.cpu, the metric for the whole pod becomes invalid for the HPA when you use type: Resource. In Rutas Norte, bookings-api carries a metrics-exporter sidecar; if that sidecar does not declare requests, the HPA for the entire Deployment goes blind. Two solutions:

1. Declare requests on the sidecar too (recommended, and also necessary for the QoS class of module 3).
2. Use type: ContainerResource pointing only at the api container.

We apply both in Rutas Norte: belt and braces.

Sooner or later you will see this:

<unknown> means: "the HPA could not obtain the current value of this metric". It does not scale, full stop. Diagnosis always follows the same order:

Look at the Conditions block and at the events at the bottom. The causes, ordered by frequency:

An example of real output with the missing-requests error:

The message names the guilty container in full: metrics-exporter. Adding requests.cpu to it fixes the problem on the next 15-second cycle.

A quick diagnostic trick: query the metrics API directly to see what the HPA is seeing.

If that call returns data and the HPA is still <unknown>, the problem is with requests, not with metrics-server.

So far we have seen how many replicas the HPA wants. The behavior block decides how fast it gets to that number. It is the difference between autoscaling that works and autoscaling that frightens you.

Field by field:

policies — each policy limits how much the replica count can change within a window of periodSeconds:

• type: Percent with value: 100 and periodSeconds: 30 means "in any 30-second window you may at most double the replica count" (increase it by 100% over the base).
• type: Pods with value: 4 and periodSeconds: 30 means "in any 30-second window you may add at most 4 pods".

selectPolicy — when there are several policies, it decides which one rules:

An example of what selectPolicy: Max does in scaleUp with our policies and 4 current replicas:

• Percentage policy: 100% of 4 = allows adding 4 → 8 replicas.
• Pods policy: allows adding 4 → 8 replicas.
• Max takes the larger: 8 replicas.

With 20 current replicas:

• Percentage: 100% of 20 = allows adding 20 → 40 replicas.
• Pods: allows adding 4 → 24 replicas.
• Max takes 40 (later capped by maxReplicas: 30 → 30).

With selectPolicy: Min it would have picked 24. This illustrates why Max on the way up makes sense: once you already have many replicas, the percentage gives you room to grow fast.

stabilizationWindowSeconds — this is the most important parameter and the worst understood. It works like this: the HPA keeps a history of the recommendations from the last N seconds and uses the most conservative value in that window.

• In scaleUp, "most conservative" means the minimum of the recent recommendations. With a 60 s window, if over the last minute the HPA recommended 12, 8 and 14 replicas, it will scale to 8. Effect: it ignores isolated spikes.
• In scaleDown, "most conservative" means the maximum of the recent recommendations. With a 600 s window, if over the last 10 minutes it recommended 12, 5 and 6, it will keep 12. Effect: it does not scale down until the calm has held for ten full minutes.

Default values if you do not define behavior:

In other words, by default the HPA can double every 15 seconds and can drop straight to minReplicas after 5 minutes of calm. The first is usually fine; the second is far too abrupt for Rutas Norte.

Our policy is deliberately asymmetric, and it is worth understanding why:

Up fast. The cost of over-scaling for a few minutes is a few pennies of compute. The cost of under-scaling for a few minutes is that the May bank-holiday sale falls over and Rutas Norte loses thousands of euros in unsold tickets, plus the reputational damage. The costs are radically asymmetric, and therefore so must be the response. stabilizationWindowSeconds: 0 in scaleUp: the moment load rises, we act.

Down slowly. Traffic on a ticketing site is irregular by nature: a spike at 10:00, a trough at 10:03, another spike at 10:05. If we scaled down quickly we would be destroying pods that have to be recreated thirty seconds later, with the cost of a cold start (PostgreSQL connections, empty route cache) and worse latency for the user. stabilizationWindowSeconds: 600 in scaleDown: we do not scale down until the calm has lasted ten minutes. And even then, we drop 20% every two minutes, not all at once.

The mental framing: scaling up is an availability mechanism; scaling down is a cost mechanism. Availability is urgent; savings can wait ten minutes.

An advanced and very practical use of selectPolicy: Disabled. During the three days of the May bank holiday, the Rutas Norte team may decide it wants no reduction in replicas at all, not even a slow one:

It is applied on Friday morning and reverted on Monday. It is a legitimate emergency switch, and far safer than deleting the HPA.

The metrics field is a list. When there are several, the HPA computes the formula for each one separately and then applies a very simple rule:

It keeps the HIGHEST replica count of all the calculations.

It is an OR of need: it is enough for one single metric to ask for more replicas for scaling to happen. They are never averaged or combined.

An example with two metrics on bookings-api (5 current replicas):

CPU says there are pods to spare; requests per second says a lot are missing. 12 wins. This asymmetry is intentional and correct: the HPA prefers to err on the side of too much capacity rather than too little.

A practical consequence worth keeping in mind: if one of the metrics goes <unknown>, the HPA cannot guarantee that scaling is not needed on its account. Its behaviour in that case is conservative: if the valid metric asks to scale up, it scales up; if it asks to scale down, it does not, because it does not know whether the broken metric would justify keeping the replicas. This explains the usual bewilderment of "my HPA has a metric on <unknown> and it is stuck at the top".

Let's write the two definitive manifests. They go in k8s/environments/pro/, because the minReplicas values differ by environment.

A calculation that justifies maxReplicas: 30: each pod reserves 500m of CPU and 512Mi of memory. Thirty pods are 15 cores and 15 GiB for bookings-api alone. Adding web-store, notifications-worker and the databases, that defines the cluster size we will need in 09-03. maxReplicas is not a number you pick at random: it is a capacity commitment.

Another calculation, this time about how fast it scales up. Starting from 4 replicas with Max between "double" and "+6 pods":

From 4 to 30 replicas in 90 seconds. Adding some 20-30 seconds of start-up for each Node.js pod, the platform goes from its Tuesday capacity to its maximum capacity in about two minutes. That is what you have to compare against the speed at which the traffic arrives.

web-store is nginx serving static assets. Its profile is different: it burns very little CPU per request, starts in a second, and its real limit is bandwidth and connection count, not compute.

A reasoned comparison of the two HPAs:

The TARGETS column shows currentValue/target. If you see numbers instead of <unknown>, everything is wired up correctly.

Let's provoke a spike in the development environment and watch the HPA at work. We will use the course's minikube profile.

For the demonstration, a development HPA with low thresholds so that it reacts quickly:

An ephemeral pod that hammers the route search endpoint in a loop:

An explanation of the command:

• --rm -it creates the pod, attaches to it and deletes it when you leave with Ctrl+C.
• http://bookings-api uses the cluster's internal DNS (04-03): it resolves to the Service in the same namespace.
• The infinite wget loop generates requests as fast as it can.

For a more serious spike, several replicas of the generator:

Read the sequence with the formula in hand:

1. 3%/40% with 1 replica: ceil(1 × 0.075) = 1. Nothing to do.
2. 187%/40% with 1 replica: ceil(1 × 4.675) = 5. But the Pods: 3 every 15s policy caps it at 4. There is behavior in action.
3. 142%/40% with 4 replicas: ceil(4 × 3.55) = 15. The policy caps it at 4+3 = 7.
4. 94%/40% with 7 replicas: ceil(7 × 2.35) = 17. Capped at 10 by maxReplicas.
5. 38%/40% with 10 replicas: ratio 0.95, inside the tolerance band. Stable. Target reached.

Now stop the generator (Ctrl+C or by deleting the Deployment) and watch the way down:

Notice: CPU drops to 2% immediately, but the replicas stay at 10 for two full minutes. That is stabilizationWindowSeconds: 120. Once the window has elapsed, it drops to 1 in one go (in this demo we did not define scaleDown policies, so the default applies: 100% every 15 s).

These events are pure gold for the postmortem of an incident: they tell you exactly when it scaled, to what and why. Remember that events expire (one hour by default); for a long history you need the centralised logging of module 7.

This is the mistake that bites every team exactly once, and it had better not be during the May bank holiday.

The bookings-api Deployment we have been writing since module 2 says:

And now an HPA also governs that same field. We have two authorities over the same number.

As long as nobody applies the YAML, nothing happens: the HPA modifies the field on the server and the object lives happily with 22 replicas. The problem shows up when someone applies the manifest again.

A real scenario:

Fifteen seconds of total outage at the year's peak sales moment. All because of a field that should not be there.

And it is worse with GitOps: if Argo CD (10-05) has the repository as the source of truth and detects that the cluster has 22 replicas when Git says 4, it will mark the resource as OutOfSync and, with automatic sync, will "correct" it to 4. Every single time. You will have a permanent tug of war between the HPA and the GitOps operator, with the platform oscillating between 4 and 22 replicas indefinitely.

By omitting replicas, the API's default value is 1, but only on initial creation. On subsequent applies the field is simply left alone: kubectl apply compares against the kubectl.kubernetes.io/last-applied-configuration annotation, sees that replicas was not there before and is not there now, and leaves it as it is. The HPA remains in charge.

One consequence to keep in mind: the first time you create the Deployment without replicas, it will start with 1 pod until the HPA brings it up to minReplicas on the next cycle (less than 15 seconds). That is acceptable on a fresh deployment; if it bothers you, create the HPA first.

That explicit comment in the YAML is not decoration: it is documentation placed exactly where somebody will make the wrong decision. Write it.

Beware of one Server-Side Apply trap: if you apply with SSA and the replicas field is in your manifest, you will take ownership of the field and the HPA will get a conflict. The API will tell you so with an explicit ownership-conflict error, which is better than the silent failure of classic kubectl apply, but it is still a failure.

If you see kube-controller-manager among the managers, the HPA is writing the field. That is the confirmation that the chain works.

We finish with the most valuable lesson of all, and the one that saves the most money.

Picture this sequence during the May bank holiday:

1. The flood of traffic arrives.
2. The bookings-api HPA scales from 4 to 30 replicas in two minutes. Perfect.
3. Each of the 30 replicas opens its pool of 20 connections to bookings-postgres. Total: 600 connections.
4. bookings-postgres has max_connections = 200.
5. The 400 excess connections are rejected. The bookings-api pods return 500 errors.
6. bookings-api CPU rises (retries and error handling consume CPU), so the HPA wants to scale further. maxReplicas caps it at 30.
7. The platform is down with 30 replicas where before it went down with 4. It has cost more money and achieved nothing.

This pattern has a name: moving the bottleneck without solving it. The HPA does not detect it because the HPA only knows about CPU: it has no idea the database is saturated.

The layer that does not scale determines the capacity of the entire system. It is the weakest-link law applied to architecture.

With bookings-postgres as the limit, the real levers are:

None of them is an HPA. The lesson: before adding an HPA, identify the scarce resource. If it is not the CPU of the pods you are going to scale, the HPA is not going to help you.

Ask yourself:

1. Is the application genuinely stateless? Can any replica serve any request with no local memory and no session affinity?
2. Is CPU the scarce resource? Or is it the database, the disk, the network, an external service or a global lock?
3. Is the load spread evenly? If one endpoint concentrates 90% of the cost and a single client calls it, adding replicas does not help.
4. How long does a new pod take to become useful? If it takes three minutes, the HPA will arrive late to fast spikes. You will need to work on start-up (09-06) or pre-warm (09-04).
5. Is there room in the cluster for the new replicas? If not, the HPA will only produce Pending pods (09-03).
6. Can the layer below take N times more load? The connection pool, the external API quota, the rate limit.

Only if all six answers are satisfactory will the HPA do what you expect.

Mistake 1: leaving the replicas field in the manifest. We already covered this in section 13, but it bears repeating because it is mistake number one and its consequences are immediate and visible. Remove it and leave a comment in its place.

Mistake 2: scaling on memory. Garbage-collected runtimes do not return memory to the system. The HPA scales up and never comes back down. Use CPU, or a business metric (09-04). If you genuinely need memory, use AverageValue rather than Utilization and keep your expectations realistic.

Mistake 3: setting the utilisation target too high. An averageUtilization: 90 looks efficient, but it leaves only 10% of headroom. By the time the HPA detects the rise, the new pods take 30 seconds to become ready, and during those 30 seconds the existing pods are at 130% and being throttled. The target must leave room for the start-up time. Between 50% and 70% is the healthy range; the slower your application starts, the lower the target.

Mistake 4: maxReplicas with no capacity plan. A maxReplicas: 200 on a three-node cluster does not give you 200 replicas: it gives you a pile of Pending pods and a false sense of security. Compute maxReplicas × requests and compare it against the real capacity (09-03).

Mistake 5: forgetting requests on the sidecars. A single container without requests.cpu blinds the HPA for the whole Deployment. In Rutas Norte, the bookings-api metrics exporter is the usual suspect.

Mistake 6: not checking that the new pods are useful. A pod that starts and passes readiness but has an empty connection pool and a cold cache serves worse than a warm one. During the first few seconds, scaling can worsen average latency. Tune the probes (07-01) so that a pod only declares itself ready when it really is.

Tip 1: put the HPA in observation mode first. Create it with minReplicas equal to maxReplicas (for example, both at 4). The HPA will not be able to scale, but it will compute and publish the metrics in kubectl get hpa. Leave it for a week, see what it would have done, and only then open up the range. It is free and it avoids nasty surprises.

Tip 2: alert when the HPA is pinned to the ceiling. If bookings-api has been at 30 replicas for half an hour, maxReplicas is capping capacity and nobody knows. An alert in Alertmanager (07-04):

Tip 3: build a dashboard with replicas and metric side by side. In Grafana, overlaying "current replicas" and "average CPU" on the same graph makes it obvious whether the HPA is reacting well or lagging behind. It is the most useful visual diagnosis there is for an HPA.

Tip 4: test the HPA before you need it. A load rehearsal against rutas-norte-pre a week before the May bank holiday costs an afternoon and prevents the outage of the year. We will do it formally with k6 in 09-06.

Tip 5: HPA events expire. One hour by default. If you want the history of the incident, you need EFK (07-05) or the kube-state-metrics metrics in Prometheus.

web-store is deployed with requests.cpu: 200m and an HPA with averageUtilization: 50, minReplicas: 3, maxReplicas: 15. Right now there are 6 replicas and kubectl top gives:

Work out: (a) the absolute target per pod; (b) the average consumption; (c) the ratio and whether it is inside the tolerance band; (d) the desired replicas; (e) what the HPA will do with this behavior:

A colleague has deployed notifications-worker with this HPA and complains that it never scales:

The Deployment is:

kubectl get hpa shows <unknown>/70%. Identify all the problems, write the diagnostic commands you would run and the fixes. Also explain why, even after fixing all of it, this HPA will still be a bad idea for this particular component.

Rutas Norte is launching drivers-dashboard, an internal web application used by the fleet's 40 drivers. Load profile:

• Monday to Friday, from 5:30 to 7:00, every driver logs in to check the day's route: load goes from zero to its maximum in fifteen minutes.
• The rest of the day there is sporadic, very low usage.
• At night there is nobody.
• The application is Node.js and takes about 25 seconds to become ready.
• It is internal: a two-minute outage is annoying but does not cost money.
• The team is under cost pressure: the cluster is shared and there is a ResourceQuota.

Write the complete HPA (autoscaling/v2) justifying every decision: minReplicas, maxReplicas, metric, target and the whole behavior. Compare at least three decisions with those of the bookings-api HPA and explain why they differ.

Solution 1

(a) Absolute target per pod:

(b) Average consumption:

(c) Ratio and tolerance band:

(d) Desired replicas:

13 replicas, below maxReplicas: 15.

(e) What the behavior does:

Current replicas: 6.

• Policy Percent 100% / 15 s: allows adding 100% of 6 = 6 → up to 12 replicas.
• Policy Pods 4 / 15 s: allows adding 4 → up to 10 replicas.
• selectPolicy: Max takes the higher ceiling: 12.

The HPA wants 13 but can only reach 12 in this step. It will scale to 12 replicas now.

On the next cycle (15 s later), with 12 replicas and assuming total load does not change, average consumption will drop to around 103m per pod. The new ratio would be 1.03, inside the tolerance band, so it will stay at 12 and never reach 13. This is normal and correct behaviour: the tolerance absorbs the last step.

An additional observation: the pods are at 103% of their requests (206m against 200m). They are not being throttled because their limit is higher, but they are consuming more than they reserved, which means they depend on the node's spare CPU. It is a sign that web-store's requests is undersized and that the VPA of 09-02 would have something to say about it.

Solution 2

There are four problems, one after another.

Problem 1: the worker container does not declare requests.cpu.

It only has limits. Without requests, the HPA has no denominator for the percentage.

An important nuance many people are unaware of: when you declare limits without requests, Kubernetes automatically fills requests with the value of limits. So this container does end up with requests.cpu: 500m. As a side effect, the pod's QoS would be Guaranteed if every container were like this (module 3). But...

Problem 2: the metrics-exporter container declares no resources at all.

This one really is fatal. With neither limits nor requests, there is no automatic fill-in, and the HPA cannot compute the pod's utilisation. This is the one producing the <unknown>. It also makes the pod's QoS Burstable instead of Guaranteed.

Problem 3: we do not know whether metrics-server is available.

That has to be checked first of all.

Problem 4 (a design one, and the most important): CPU is the wrong signal for this component.

Diagnostic commands, in order:

The output of step 3 would give the problem away:

The corrected Deployment:

And, defensively, switch the HPA to ContainerResource so that the sidecar does not distort the percentage:

Why it is still a bad idea even once it works:

notifications-worker consumes messages from an email queue. Its work per message is mostly input/output waiting: talking to the SMTP server, waiting for the network reply. That burns very little CPU.

During the May bank holiday there may be forty thousand emails waiting in the queue while the worker sits at 20% CPU, because it is blocked waiting for SMTP most of the time. A CPU-driven HPA would see 20% against a target of 70% and conclude that there are replicas to spare: it would scale down exactly when consumers are needed most.

The correct metric is the queue length: "I want one worker per 500 pending messages". The HPA cannot read that out of the box; you need an external metric, and that is precisely KEDA's reason for existing in 09-04.

Solution 3

Reasoned comparison with the bookings-api HPA:

An additional consideration worth noting: this load profile — zero at night, an avalanche at a fixed time — is the perfect candidate for a cron trigger that pre-warms the service at 5:15, instead of waiting for CPU to rise and reacting late. That is no longer something the stock HPA does: it is KEDA, and we will see it in 09-04. A reactive HPA will always lag behind a load that rises in a step; only an anticipatory signal really solves that case.

The HorizontalPodAutoscaler turns the replica count from a fixed, hand-written figure into an observed consequence of real traffic. It is the difference between sizing for an ordinary Tuesday and surviving the May bank holiday without anyone having to open a laptop on a station platform.

The essentials you take away from this lesson:

• Scaling out and scaling up are different axes. Horizontal solves traffic volume and improves availability as well; vertical solves the sizing of each instance. Rutas Norte needs both.
• Only what is genuinely stateless can be scaled horizontally. bookings-api and web-store yes; bookings-postgres never, and the attempt would produce data corruption, not capacity.
• The formula is a rule of three: ceil(currentReplicas × currentValue / targetValue), with a ±10% tolerance band that prevents flapping.
• Without requests on every container and without metrics-server there is no HPA. The <unknown> in the TARGETS column almost always points at a forgotten sidecar.
• behavior is where it is decided whether autoscaling works. The Rutas Norte asymmetry — up in zero seconds, down after ten minutes of calm — reflects a real asymmetry of costs: losing sales is far more expensive than having spare capacity for a while.
• With several metrics, the one asking for the most replicas wins. An OR of need, deliberately conservative.
• The replicas field must disappear from the versioned manifest. Otherwise an innocent kubectl apply will bring the platform down at the sales peak, and with GitOps it will be a permanent war (10-05).
• Before adding an HPA, find the scarce resource. Scaling the web layer when the limit is the database multiplies the cost without adding a single ticket sold.

Rutas Norte now has bookings-api and web-store responding to traffic on their own. But we are still dragging along a problem we have mentioned several times without fully solving: the HPA leans on the CPU requests as the reference for its entire calculation, and that requests is still a number we picked by eye. In 07-02 we recalibrated it by comparing against real consumption, but we did it by hand, once, staring at kubectl top for an afternoon. If requests is wrong, the HPA's percentage lies, and with it every one of its decisions.

In the next lesson, Vertical Pod Autoscaling, we will look at the component that solves exactly that problem: the VPA observes real consumption over days, computes percentiles and tells you with data what requests and what limits your containers should have. We will also find out why the VPA and the HPA cannot govern the same metric without fighting, and what workflow serious teams use: the VPA as an adviser, not as an autopilot.