The previous lesson ended on an uncomfortable note. The bookings-api HorizontalPodAutoscaler computes every one of its decisions as a percentage of requests.cpu, and that requests.cpu: 500m is a number someone picked by eye months ago. In 07-02 we recalibrated it by comparing against real consumption, but we did it by hand, one afternoon, staring at kubectl top and jotting figures into a spreadsheet. If requests is wrong, the HPA's percentage lies and all of its decisions rest on a false reference.

And the problem is wider than the HPA. requests and limits govern scheduling (where a pod fits), the QoS class (who dies first when memory runs short), OOMKilled, CPU throttling, the namespace's ResourceQuota consumption and, in the cloud, the bill. All of that rests on numbers that in most organisations were written once, copied from another service's manifest, and never looked at again.

The VerticalPodAutoscaler (VPA) exists for exactly that. It does not scale by load: it scales by knowledge. It observes what your containers actually consume over days, computes percentiles and tells you with data what they ought to be asking for. This lesson covers what it is, how it works inside, how to read its recommendations, why most serious teams use it as an adviser rather than an autopilot, and why it cannot coexist with the HPA over the same metric.

1. The real problem: nobody knows what to ask for
2. What the VPA is and what it is not
3. It is not part of the core: installation
4. The three components and how they collaborate
5. The VerticalPodAutoscaler object field by field
6. The update modes: Off, Initial, Recreate and Auto
7. In-place resizing and its current state
8. resourcePolicy: bounding, excluding and controlling
9. Reading a recommendation: target, lowerBound, upperBound, uncappedTarget
10. Off mode as an adviser: the real workflow
11. The classic incompatibility: VPA and HPA over the same metric
12. Applying it to Rutas Norte, component by component
13. Interaction with LimitRange and ResourceQuota
14. The limits of the VPA
15. Common Mistakes and Tips
16. Exercises
17. Conclusion

Let's put numbers on the problem. These are the requests Rutas Norte has been dragging along since module 3, alongside the real consumption measured over two weeks with Prometheus:

Each row tells a different story and each one costs money or availability:

• bookings-api short on memory. It asks for 512Mi and uses 690Mi at p95. It works because the limit is 1Gi and there is free memory on the nodes... until there is not. The day a node fills up, the kubelet evicts pods, and a Burstable pod consuming above its requests is one of the first candidates (module 3). It is a time bomb.
• web-store oversized threefold. With 15 replicas at peak that is 3 reserved cores of which 0.8 are used. Those 2.2 cores are blocked for the scheduler: no other pod can use them even while they sit idle. It translates into extra nodes.
• occupancy-reports badly undersized. It asks for 500m and needs 1750m. Consequence: brutal throttling. A report that should take 8 minutes takes 40, and some nights it does not finish before the maintenance window.
• The metrics exporter. 20m of requests for a process that uses 4m. It looks like a trifle, but multiplied by every pod on the platform it is whole cores wasted.

Now the uncomfortable question: how do you fix this? The usual answer — "we look at Grafana and adjust" — has three problems. It requires somebody to remember. It requires that somebody to know which percentile to look at. And it has to be repeated every time the application changes, which in practice means never.

The VPA automates exactly that work.

The VerticalPodAutoscaler is a component that:

1. Observes the historical CPU and memory consumption of the containers in a set of pods.
2. Computes a requests (and optionally limits) recommendation based on percentiles of that history, with exponential decay so that recent data weighs more.
3. Optionally applies that recommendation, recreating the pods with the new values.

And, very importantly, what it is not:

This distinction is the key to understanding the relationship with the HPA. The HPA is a load-response mechanism, with a time constant of seconds. The VPA is a sizing-correction mechanism, with a time constant of days. They do not compete on the same ground... except when both look at the same metric, which is the conflict of section 11.

A useful way to remember it: the HPA answers the question "how many?"; the VPA answers the question "how big?".

Unlike the HPA, which lives inside the kube-controller-manager and is available on any cluster, the VPA is an external component you have to install. It lives in the kubernetes/autoscaler repository, alongside the Cluster Autoscaler we will see in 09-03.

That has practical consequences worth knowing before you begin:

• You must check the version compatibility between the VPA and your Kubernetes. The compatibility table is in the repository; using a mismatched version produces obscure errors.
• It installs three Deployments and a handful of CRDs, RBAC and an admission webhook.
• On managed Kubernetes (10-06), many providers offer it as an addon you enable with a switch (GKE has had it built in for years). If you are in the cloud, look first at whether your provider gives it to you ready-made.

An important prerequisite: the VPA needs metrics-server (the same one from 07-02) for the updater and the admission-controller, and its recommender can additionally read from Prometheus if you configure it. On minikube:

And the registered CRDs:

The second CRD, VerticalPodAutoscalerCheckpoint, deserves a note: that is where the recommender persists the consumption histogram every few minutes. Without it, a restart of the recommender would lose all the accumulated history and the observation period would have to start again. When somebody tells you "I restarted the VPA and now the recommendations are odd", the checkpoint is the first thing to look at.

To uninstall:

The VPA is not one process, it is three, with clearly separated responsibilities. Understanding that separation explains almost all of its behaviour.

It is the brain. Its job:

• Reads the CPU and memory consumption of every pod matching some VPA object's targetRef.
• Keeps, per container, a histogram with exponential decay: old samples weigh less than recent ones. The default half-life is 24 hours, which means a sample from a day ago is worth half of one from now.
• Computes percentiles over that histogram and writes the recommendation into the VPA object's status.recommendation.
• Persists the histogram in a VerticalPodAutoscalerCheckpoint so it survives restarts.

The percentiles it uses by default:

The asymmetry is deliberate and very sensible. Falling short on CPU produces slowness; falling short on memory produces OOMKilled. That is why memory is computed from the peak and CPU from a high but not extreme percentile. A container can survive perfectly well through CPU spikes above its requests (the node lends it more if there is any); a memory spike above its limit, no.

An additional detail: the recommender watches OOMKilled events and, when it detects one, raises the memory recommendation in one step (roughly 20%) without waiting for the histogram to catch up. It is direct feedback from the failure.

The recommender always runs, whatever the updateMode. Even in Off mode it is working and publishing recommendations. That is precisely what makes the adviser mode useful.

It is the enforcing arm. Every minute:

• Compares each pod's current resources against the current recommendation.
• If the difference exceeds a threshold (by default, if the current value is outside the [lowerBound, upperBound] range), it decides the pod must be recreated.
• Evicts the pod using the eviction API, the same one kubectl drain uses. This is fundamental: the updater respects PodDisruptionBudgets (09-05). If your PDB does not allow the eviction, the VPA waits patiently.
• It does not modify the pod: it kills it. The Deployment recreates it, and the new pod goes through the admission controller.

Two safeguards that prevent disasters:

• It does not evict a pod unless at least 12 hours have passed since its creation (avoiding recreation cycles).
• It never evicts all the pods of the same controller at once.

The updater only acts in Recreate and Auto modes. In Off and Initial it is idle.

It is a MutatingAdmissionWebhook registered with the API. When a pod creation request arrives:

• It checks whether that pod matches an active VPA.
• If it does, it rewrites the pod's requests (and limits, keeping the ratio) before it is persisted in etcd.
• The pod is born with the recommended values. The scheduler places it using those numbers.

This is the component that makes recreation work. Without it, the updater would kill a pod and the Deployment would recreate it with the manifest's original values: an infinite loop of pointless evictions.

An important and often surprising consequence: the VPA does not modify the Deployment. If you run kubectl get deployment bookings-api -o yaml you will still see requests.cpu: 500m. The real pods will have different values. The mutation happens at the pod level, in admission. To see the effective values:

A derived risk: if the admission webhook is down or misconfigured, depending on its failurePolicy it can either block the creation of every pod in the cluster or let them through unmutated. The official installer sets it to Ignore by default, which is the safe option.

A note on targetRef: unlike the HPA, it does not require the object to expose the /scale subresource. That is why the VPA can be applied to a DaemonSet, whereas the HPA cannot. This is genuinely useful: the Fluentd and Falco agents we deployed in modules 7 and 8 are DaemonSets, and their requests are just as arbitrary as everyone else's.

The updateMode is the object's most important decision.

The recommender observes and publishes to status.recommendation. Nothing more. No pod is touched, no manifest changes. It is pure information.

Operational cost: zero. Risk: zero. Value: high. That is why it is, by a wide margin, the most used mode in serious production.

The admission controller mutates pods when they are born for some other reason: a new deployment, a restart after a node failure, an HPA scale-up. The updater never evicts anything.

It is an interesting middle ground: the values really are applied, but without causing a single extra restart. The trade-off is slowness: if bookings-api is deployed every two weeks, recommendations take two weeks to materialise. And it creates a disconcerting effect: two pods of the same Deployment, created at different times, can have different resources.

The updater actively evicts the pods whose resources have drifted. It is genuine vertical autoscaling, with the inevitable consequence: every adjustment is a pod restart.

For notifications-worker that is acceptable (it processes messages from a queue, a restart loses at most the in-flight message, which the queue redelivers). For bookings-postgres it is a different story, and we discuss it in section 12.

minReplicas: 2 is the essential protection: the updater will not evict if the controller has fewer replicas than that number. With a single replica, an eviction is a complete service outage.

Semantically it means "use the best mechanism available". Today, in practice, Auto behaves exactly like Recreate in most installations. The project's stated intention is that, once in-place resizing is stable, Auto will use it and stop recreating pods.

Operational advice: use Recreate explicitly if that is what you want. That way, when the meaning of Auto changes with a VPA upgrade, your behaviour will not change by surprise.

For years, the fundamental limitation of vertical scaling in Kubernetes was that a container's resources were immutable: changing them required recreating the pod.

That is changing. The feature is called In-Place Pod Vertical Scaling, and it allows a running container's requests and limits to be modified without recreating the pod, through the /resize subresource.

Each container can declare a restart policy per resource:

And the change is applied with:

The state of the process is reflected in the pod:

This is the part that matters for decision-making:

• The feature moved to beta in Kubernetes 1.33 and is still maturing. In 1.30-1.32 it is alpha and requires enabling the InPlacePodVerticalScaling feature gate on the API server and on the kubelets.
• Shrinking memory in place is not generally solved: lowering the memory limit below what the process has already mapped can trigger an immediate OOMKilled. That is why the usual policy for memory is RestartContainer.
• The VPA's integration with this feature exists but is neither complete nor the default path.
• On managed Kubernetes (10-06), the feature gate may not be available: you do not control the control plane.

An explicit warning: do not build your production capacity strategy on in-place resizing yet. It is the right direction and in a couple of releases it will be the natural way to do things, but today:

1. It may not be available on your cluster.
2. Its behaviour with memory has rough edges.
3. The VPA does not fully exploit it.

Get to know it, try it out in rutas-norte-dev, keep an eye on it. In rutas-norte-pro, use the workflow of section 10.

Letting a recommender write numbers with no guard rails is a bad idea. resourcePolicy is where you put the human judgement.

Why each guard rail:

minAllowed. The recommender relies on observed consumption. If a service spends the night without traffic, its CPU p90 is nearly zero, and without minAllowed the recommendation would drop to 15m. When the morning traffic arrives, the pod chokes while starting. minAllowed encodes "this is the minimum needed to start and respond decently, whatever the metrics say".

maxAllowed. Two reasons. First, scheduling: an 8-core pod only fits on large nodes, and if your nodes have 4 cores that pod stays Pending forever. Second, anomaly detection: if the recommendation reaches the ceiling, there is something to investigate (a memory leak, a pathological query). maxAllowed turns a silent problem into a signal.

A rule of thumb for maxAllowed: never above what fits on a real node, with room for the system. On nodes of 4 cores and 16 GiB, a reasonable maxAllowed sits around 3 cores and 12Gi.

mode: "Off" at container level takes it out of the VPA entirely: it is neither recommended for nor mutated. When to do it:

• Sidecars with well-known, stable resources. A Prometheus exporter uses 5m and 20Mi today, tomorrow and always. It does not need a recommender.
• Service mesh sidecars (the Istio proxy, 08-04). They usually have their own resource management and the VPA treads on it.
• Containers with deliberately high requests for reasons the metrics cannot see: reserving CPU for critical latency spikes, for example.

Watch out for one detail: if you exclude every container of the pod, the VPA will do nothing, which is correct, but the object still exists and confuses people. Better to delete it.

controlledResources is the key piece for coexisting with the HPA: if the HPA scales on CPU, the VPA must confine itself to memory (section 11).

controlledValues decides whether the VPA touches the limits too:

The ratio detail deserves an example. If the manifest has requests.cpu: 500m and limits.cpu: 1000m (a 1:2 ratio) and the VPA recommends requests.cpu: 800m, with RequestsAndLimits it will set limits.cpu: 1600m to preserve the 1:2. It is sensible behaviour but a surprising one: the VPA can raise your limits higher than you expected, and that interacts with the namespace's ResourceQuota (section 13).

Here is the VPA's real value in Off mode. Let's look at a complete recommendation.

The lowerBound and upperBound limits are not consumption percentiles: they are the recommender's confidence interval. The less history it has, the wider they will be. A VPA with two hours of data will give an enormous upperBound; with two weeks, a narrow interval. The width of the interval is a direct indicator of how much you can trust the recommendation.

They are also the updater's action criterion: in Recreate/Auto modes, a pod is evicted when its current value leaves the [lowerBound, upperBound] range. Inside the range, it is left alone. This band is the vertical equivalent of the HPA's tolerance band.

Let's contrast it against what we have in the manifest:

The critical row is api's memory: 512Mi is below the lowerBound of 573Mi. The VPA is saying, with two weeks of data, that this container has less memory reserved than it needs to run safely. It is exactly the risk we sensed in section 1, now quantified.

And the exporter confirms the other intuition: 20m reserved for something that needs 6m. With 30 bookings-api replicas at peak, that is 420m of CPU reserved and never used: nearly half a core blocked for the scheduler.

Suppose we set maxAllowed.memory: 512Mi in the resourcePolicy. The recommendation would come out as:

When target and uncappedTarget differ, your maxAllowed (or minAllowed) is biting. And here you have to think twice, because there are two opposite interpretations and both are plausible:

1. The guard rail is set wrong. It was a conservative number from months ago and the application has legitimately grown. Raise it.
2. The guard rail is doing its job. There is a memory leak and consumption grows without pause. maxAllowed is the only thing stopping that pod from eating a whole node.

Telling them apart requires looking at the trend in Grafana (07-04): if memory grows monotonically and never comes down, it is a leak; if it settles on a higher plateau, it is legitimate growth. An uncappedTarget that drifts away from target month after month is a memory leak until proven otherwise.

To automate the workflow:

Clean output, ready to drop into a weekly report or into an automatic comment on a pull request.

Here is the practice serious teams use, and it deserves a proper explanation because it runs against the grain of what the name "autoscaler" suggests.

Most organisations with Kubernetes in production use the VPA exclusively in Off mode. Not as an autopilot, but as an adviser whose recommendation a person reviews before it reaches production.

Step 2, two weeks and why. The recommender's histogram has a 24-hour half-life, so two or three days already give you signal. But two weeks cover two full weekly cycles, weekends included. At Rutas Norte this matters a lot: Saturday's traffic looks nothing like Wednesday's. And if there is a monthly close or a periodic event, that has to be covered too.

Step 6, the 20% threshold. Changing a Deployment's resources means deploying it. It is not worth doing for 5%. The 20% threshold is a reasonable convention: below it, noise outweighs signal.

Step 7, the pull request. Here is the heart of the method. The change is documented:

That note about the HPA is exactly the kind of reasoning that gets lost when you let a controller change the numbers on its own.

A very profitable practice: a VPA in Off mode over every Deployment, StatefulSet and DaemonSet on the platform, even if you never apply them automatically. The cost is one object per controller and a little memory in the recommender. The benefit is having, at any moment, the answer to "is this properly sized?".

And a weekly report comparing recommendation against manifest. That report, reviewed every Monday in fifteen minutes, is probably the highest-return-per-minute activity in operating a cluster.

This is the trap you absolutely must know about.

The VPA and the HPA cannot govern the same resource (CPU or memory) on the same workload. If you try, they feed back on each other destructively.

Let's see why, with bookings-api and concrete numbers. Suppose an HPA on CPU with averageUtilization: 65 and a VPA in Recreate mode controlling CPU.

The loop is obvious: the HPA lowers per-pod consumption by adding replicas; the VPA reads that lower consumption as the pods being too big and lowers requests; lowering requests lowers the HPA's absolute target, and the HPA scales again. The total load has not changed by a single watt, but we have gone from 4 replicas of 500m to 10 replicas of 263m, and the process continues.

Each iteration means recreating pods (VPA) and creating/destroying pods (HPA). It is a spiral of instability that is also very hard to diagnose, because each component in isolation appears to be behaving correctly.

The recommended configuration for bookings-api, if you did want an active VPA:

controlledResources: ["memory"] is the line that makes coexistence possible. Without it, you get the loop.

The most elegant configuration appears when the HPA scales on a metric that is neither CPU nor memory: requests per second, queue length. Then:

• The HPA decides how many pods, according to business demand.
• The VPA decides how big, according to observed consumption.
• There is no feedback: the business metric does not depend on requests.

That is exactly what KEDA enables in 09-04. When notifications-worker scales on queue length, a full VPA over it will be perfectly safe and very useful.

And this is worth underlining: the conflict only exists when the VPA acts. In Off mode the VPA changes nothing, so it can coexist with any HPA on any metric with no risk whatsoever. Its CPU recommendation will be somewhat pessimistic (it measures pods whose load the HPA has already relieved), but it will still be valid information.

This is another argument in favour of Off mode: it wipes out a whole class of problems in one stroke.

Concrete, justified decisions for each piece of the platform.

Both have a CPU-driven HPA from 09-01. We put a VPA in Off mode, as advisers.

With the measurements from section 1, web-store is a clear candidate to come down from 200m to around 70m of CPU. With 15 replicas at peak, that frees almost two cores: fewer nodes to start (09-03) and a smaller bill.

Here we do switch it on, and it is the best candidate on the platform.

The most profitable case of all. Recall: it asks for 500m and needs 1750m. It gets throttled, takes five times longer than it should and some nights does not finish.

Initial is the right choice for batch jobs, and it is worth understanding why: a Job evicted ten minutes into a thirty-minute run loses all the work and starts again. With Initial, each overnight run is born with the resources the recommender has learnt from previous runs. The VPA learns from Monday night for Tuesday night.

Expected effect: from 40 minutes down to about 9. And since it runs at 3 in the morning, with the cluster empty, those extra resources take nothing from anybody.

Here we have to think. The facts:

• It is a StatefulSet with a single primary replica.
• It has Guaranteed QoS (requests = limits), a deliberate decision from module 3 so that it is the last eviction candidate.
• A restart means cutting every connection, losing PostgreSQL's page cache and a few seconds of total platform unavailability.
• It holds customers' personal data: any risk of corruption is unacceptable.

Decision: VPA in Off mode, with controlledValues: RequestsAndLimits.

Point 4 of the comment deserves emphasis, because it is the underlying argument: sizing a database is a configuration decision, not an observational one. Raising the container's memory without adjusting shared_buffers in harmony with it does not improve performance: it only wastes memory. The VPA knows nothing about postgresql.conf. A PostgreSQL operator (06-07) does, and that is the right path for scaling a database.

The VPA does not live alone. In module 3 we put namespace guard rails in place that interact with it in ways worth anticipating.

Let's recall the rutas-norte-pro LimitRange:

Interactions:

1. The LimitRange's max wins over the VPA's maxAllowed. If the VPA recommends 6 cores and the LimitRange caps at 4, the mutated pod will be rejected in admission. The VPA is aware of LimitRanges and adjusts its recommendation, but if you change them after creating the VPA, there can be a lag.
2. The min can raise a low recommendation. If the VPA recommends 20m and the LimitRange demands a minimum of 50m, the pod ends up with 50m.
3. The default values drop out of the equation when the VPA mutates the pod, because the pod already arrives with explicit values.

A rule of thumb: keep the VPA's maxAllowed below the LimitRange's max. That way the clipping shows up in uncappedTarget (useful information) instead of producing an admission rejection (a confusing failure).

The ResourceQuota is more treacherous, because the failure appears late and somewhere else.

The problem arises from the interaction between HPA and VPA over the quota:

The failure does not appear when the VPA is applied: it appears weeks later, at the worst possible moment, and the error message is in the ReplicaSet's events, not in the HPA nor in the VPA. It is a deferred, badly signposted failure.

A mandatory check when applying any VPA recommendation:

The 0.8 factor leaves room for ephemeral pods, jobs and deployments (which during a rolling update consume quota for both the old and the new version at once).

The calculation for Rutas Norte with the recommendation:

18.29 against a quota of 40, with an 80% margin (32). It fits comfortably. And notice that the VPA's recommendation reduces the maximum consumption compared with the current scenario (which would be 15 + 3 + ... ≈ 21), because web-store and the exporter were oversized. Getting requests right does not only save money: it gives you room to scale further.

As we saw, the VPA preserves the requests:limits ratio. If your ratio is 1:2 and the VPA raises requests by 50%, the limit goes up by 50% too. And the ResourceQuota has a separate limits.cpu line that can also run out. Watch both.

We finish with what the VPA cannot do. Knowing the limitations stops you expecting from it what it does not give.

A freshly created VPA knows nothing. Its first recommendations, in the first hours, are unreliable: the upperBound will be enormous and the target will reflect only what it has seen. Rule: do not take decisions with less than a week of data, and prefer two.

An uncomfortable corollary: for a new service, the VPA does not help. You have to estimate by hand, deploy, and let the VPA correct you afterwards.

With a 24-hour half-life, a real profile change takes days to be fully reflected. This is deliberate and correct — we do not want a five-minute spike to change the resources of the whole platform — but it means the VPA is no use for responding to events:

• It is no use for the May bank holiday. By the time the VPA notices that more CPU is needed, the bank holiday will be over.
• It is no use for a deployment that radically changes the consumption profile. There you have to estimate by hand and let the VPA validate afterwards.

The VPA is for the steady state, not for the transient. The transients are covered by the HPA (09-01) and KEDA (09-04).

In Recreate and Auto modes, every adjustment is a restart. The cost depends on the component:

The VPA respects PDBs (09-05), which mitigates the risk, but does not eliminate it: a PDB guarantees that everything does not go down at once, not that the restart is painless.

The VPA sees CPU and memory consumption. It does not see:

• That the May bank holiday starts on Friday.
• That this pod serves the payment process and cannot restart mid-transaction.
• That last night's high memory was a maintenance run and not the normal regime.
• That PostgreSQL's shared_buffers must change in harmony with the container's memory.

All of that a person supplies. And that is, summed up in one sentence, the justification for Off mode.

• It does not manage resources other than CPU and memory. Ephemeral storage, GPUs and extended resources are out of scope.
• With an active updateMode, it should not be used alongside aggressive continuous deployment: the VPA's evictions and rolling updates overlap unpredictably.
• One VPA per workload. Two VPA objects with a targetRef to the same Deployment produce undefined behaviour. Kubernetes does not prevent it; you must.

Mistake 1: putting the VPA in Auto mode over CPU when there is already a CPU-driven HPA. The catastrophic mistake of this lesson. The feedback loop of section 11 takes hours to manifest and is hard to diagnose because each component looks fine in isolation. If you have a CPU-driven HPA: controlledResources: ["memory"] or Off mode. No exceptions.

Mistake 2: applying the recommendation of a two-day-old VPA. The outsized upperBound gives away the lack of history. Wait two weeks. If you are in a hurry, look at the lowerBound: if your current value is below it, that is a reliable signal even with little data, because it means the container has already run short in real observations.

Mistake 3: not setting maxAllowed. With no ceiling, a memory leak makes the recommendation grow without end. In active mode, you will end up with a pod asking for 30Gi that fits on no node and stays Pending forever. maxAllowed turns that leak into a visible signal (uncappedTarget drifting away from target) instead of into an outage.

Mistake 4: believing that the VPA modifies the Deployment. It does not. kubectl get deployment will keep showing the manifest's values. The mutation happens on the pod, in admission. To see reality you have to look at the pods, not the controller. This confusion generates a huge number of "the VPA does not work" tickets.

Mistake 5: an active VPA over a single-replica workload. An eviction is a total outage. The default minReplicas: 2 protects you, but if you lower it to 1 "so it works", you are asking for service interruptions. If you have a single replica, either make it two or use Off mode.

Mistake 6: forgetting the ResourceQuota. Section 13. Raising requests without recomputing requests × maxReplicas against the quota produces a deferred failure that shows up at the traffic peak, weeks later, with an error message in an unexpected place.

Tip 1: start with the CronJobs. They are the most profitable case and the lowest risk. Nobody is waiting for a response, Initial mode is safe, and the runtime saving is usually spectacular. occupancy-reports going from 40 to 9 minutes is a visible win that buys credibility for the rest.

Tip 2: export the recommendations to Prometheus. The VPA exposes metrics, and kube-state-metrics publishes kube_verticalpodautoscaler_status_recommendation_containerrecommendations_target. With that you can build a "recommended versus configured" dashboard in Grafana (07-04) for the whole platform, and an alert when the deviation exceeds 30%:

Tip 3: a VPA in Off mode over absolutely everything. Almost no cost, permanent information. Include it in the base template of every new component.

Tip 4: use lowerBound as a risk alert. A container whose requests are below the lowerBound is in real danger of eviction or OOMKilled. It is the most actionable signal the VPA gives, and it deserves an alert.

Tip 5: review uncappedTarget monthly. If it grows month after month while target stays pinned to maxAllowed, you have a memory leak. The VPA detects it before it becomes an incident.

Tip 6: document in the manifest where the numbers come from. A comment like # requests.memory: 800Mi -- VPA recommendation 2026-05-11, real p95 690Mi turns a magic number into a traceable decision. Whoever reads it next will know whether they can touch it.

This is the describe of the redis-cache VPA after three weeks in Off mode:

The current manifest says:

And the VPA's resourcePolicy:

Answer: (a) what is going on with the CPU; (b) what is going on with the memory and why target and uncappedTarget differ; (c) what it means that upperBound is exactly 2048Mi; (d) what QoS the pod has now and what that implies; (e) what you would do, in what order of priority, and what you would check first.

web-store has been behaving strangely for three days. The team reports:

• The replica count oscillates between 3 and 12 several times a day with no traffic to justify it.
• The pods restart frequently, with reason Evicted.
• The pods' requests.cpu do not match the manifest and change on their own.

The configuration is:

Explain exactly what is happening, with the timeline, and give two alternative solutions with their manifests and their trade-offs.

Rutas Norte is adding accounting-exporter, a component that generates the invoicing files for the accountancy firm. Profile:

• It is a CronJob that runs on the 1st of each month at 2:00.
• Each run takes between 20 and 90 minutes, depending on the month's booking volume.
• It uses a lot of memory: it loads all of the month's bookings into memory to aggregate them.
• If it fails, it has to be relaunched by hand and the administration team goes without the files that day.
• It has been OOMKilled in two of the last four months.
• Currently: requests.memory: 1Gi, limits.memory: 2Gi, requests.cpu: 500m, limits.cpu: "1".
• The namespace has a ResourceQuota of 40 cores and 80Gi.

Design the complete strategy: VPA yes or no? Which mode? What resourcePolicy? How much observation time do you need and why is this a special case? What would you do in addition to the VPA? Write the manifest and justify every decision.

Solution 1

(a) The CPU is oversized.

The manifest asks for 500m; the upperBound is 390m. The current value is above the upperBound, which means the recommender is confident there is CPU to spare. The target is 245m: less than half of what is configured. Since redis-cache is a single replica, the absolute saving is modest (255m), but it frees scheduling capacity on a node.

(b) The memory: target 1980Mi, uncappedTarget 3410Mi.

This is the critical observation of the exercise. maxAllowed.memory is 2Gi = 2048Mi, and the uncappedTarget of 3410Mi is far above it. The recommender wants 3410Mi and the guard rails are clipping it to 1980Mi.

On top of that, the current requests (1Gi = 1024Mi) is below the lowerBound (1420Mi). A double alarm: reserved memory is short AND the recommender wants far more than we allow it.

Memory leak or legitimate growth? With Redis there is a third, very specific possibility, and it is the most likely one: the cache has grown until it filled the available space. Redis, with no eviction policy configured, keeps accepting keys until memory runs out. A growing uncappedTarget does not indicate a software fault: it indicates that maxmemory is not configured in Redis and the cache is growing unchecked.

Check:

If maxmemory is 0, that is the underlying problem, and no requests adjustment solves it: it only delays the next OOMKilled.

(c) upperBound exactly 2048Mi.

It is no coincidence: it is the maxAllowed. The recommendation's bounds are clipped by the guard rails too. When you see an upperBound that matches your maxAllowed exactly, the recommender is telling you its confidence interval is truncated, and that the real recommendation is outside what you allow it.

(d) The pod's QoS.

Since not every resource has requests equal to limits, the QoS is Burstable, not Guaranteed (module 3).

A serious implication: under node memory pressure, a Burstable pod consuming above its requests is a priority eviction candidate. And redis-cache is consuming 1980Mi with a requests of 1024Mi: nearly twice what it reserved. It is among the first to fall when the node fills up, and its fall dumps all the load on bookings-postgres exactly when the node is already under pressure. A textbook domino effect.

(e) Action plan, by priority.

Priority 1 (today): configure maxmemory in Redis. This is the root-cause fix.

With that, Redis stops growing unchecked and starts behaving like a real cache. The uncappedTarget should settle over the following weeks.

Priority 2 (this week): raise requests.memory to 2Gi and match it to the limit.

This achieves three things: it covers the lowerBound, promotes the pod to Guaranteed QoS (it stops being a priority eviction candidate, consistent with its role as a buffer in front of PostgreSQL), and frees 200m of CPU.

Priority 3 (in a month): review again. With maxmemory in place, check that the uncappedTarget has settled below 2Gi. If it keeps growing, there is something else.

Checks before applying:

That last point is the one always forgotten: changing redis-cache's resources means restarting it, and restarting it empties the cache. For the following minutes, all the availability query load goes straight to bookings-postgres. It has to be done on a Tuesday at 4 in the morning, not on a Friday at 10.

Solution 2

What is happening: the HPA-VPA feedback loop of section 11, in its purest form.

The HPA scales web-store on CPU. The VPA is in Auto mode with no controlledResources, which means it controls CPU and memory. Both govern the CPU.

Timeline:

All three symptoms are explained: the replica oscillation (the HPA reacting to moving targets), the Evicted events (the VPA updater evicting), and the requests that do not match the manifest and change on their own (the admission controller mutating the pods).

Commands that would confirm it:

Solution A (recommended): VPA in Off mode.

Advantages: it eliminates the conflict entirely, no surprise restarts, the manifest is still the truth, compatible with GitOps. Trade-off: somebody has to apply the recommendation by hand.

Solution B: an active VPA over memory only.

Advantages: memory adjusts itself, with no intervention. Trade-offs: there are still unplanned restarts; requests.cpu is left unadjusted (you would have to do it by hand anyway); during the May bank holiday you would have to switch to Off to avoid restarts at the worst moment.

Which to choose. For web-store, A. The memory saving on nginx is small (we are talking tens of Mi per pod) and does not offset the risk of unplanned restarts on the platform's front door. Solution B would make sense on a workload with large, variable memory consumption that tolerates restarts: notifications-worker is a better candidate.

Immediate action while the decision is being made:

This stops the oscillation on the updater's next cycle, without deleting anything.

Solution 3

VPA yes or no? Yes, emphatically. It is a textbook case: a batch job with variable consumption that has already been OOMKilled twice. The cost of a restart is zero (the Job is ephemeral) and the cost of the current failure is high (manual intervention, the accountancy firm left without files).

Which mode? Initial. Just like occupancy-reports:

• Recreate makes no sense: evicting a Job 40 minutes into a 90-minute run loses all the work and starts from scratch. It would be worse than the OOMKilled.
• Off wastes the opportunity: here we do want automatic application, because consumption varies each month with the booking volume and nobody is going to review the recommendation monthly.
• Initial applies the recommendation on every new run. Perfect.

The special observation case: the monthly frequency.

This is the interesting point of the exercise. The recommender's histogram has a 24-hour half-life. A monthly run means that, when the next one comes round, the previous sample is thirty half-lives old: its weight is 2⁻³⁰, that is, zero.

The VPA, as it stands, is practically useless for a monthly job. Every run starts almost from scratch.

Three ways to solve it, from least to most advisable:

1. Adjust the recommender's half-life. The vpa-recommender accepts --cpu-histogram-decay-half-life and --memory-histogram-decay-half-life. Raising them to 720h (30 days) would make the previous month's sample weigh half. Trade-off: it is a global recommender parameter, it affects every VPA in the cluster and it would make bookings-api's recommendations glacially slow. Only viable with a second, dedicated recommender instance, which is serious complexity.

2. Run the job more frequently in pre. Schedule it weekly on rutas-norte-pre with representative data, let the VPA learn there, and take the recommendation to production by hand. It works, but it requires realistic test data.

3. VPA in Initial + a generous minAllowed + explicit measurement. The pragmatic option and the one I recommend.

Beyond the VPA, three measures that matter more:

And the VPA:

ResourceQuota check:

Follow-up plan:

The underlying lesson of the exercise: when the execution frequency is far lower than the histogram's half-life, the VPA loses almost all of its automatic usefulness. In those cases, the guard rail (minAllowed) matters more than the recommendation, and manually measuring the first few runs remains irreplaceable. The VPA does not exempt you from thinking: it speeds up thinking well.

The VerticalPodAutoscaler does not respond to load: it responds to ignorance. It turns requests and limits from numbers somebody copied off another manifest into values derived from real consumption observed over days.

The essentials:

• The VPA answers "how big?"; the HPA answers "how many?". Different axes, different time constants: days versus seconds.
• It is not in the core. You have to install it from kubernetes/autoscaler, check version compatibility, and in the cloud look first at whether the provider offers it.
• Three components: the recommender computes (percentiles with exponential decay, p90 for CPU, the peak for memory), the updater evicts while respecting PDBs, and the admission controller mutates the pods as they are born. The Deployment never changes.
• Off is the mode of serious teams. The manifest is still the truth, there are no surprise restarts, it works with GitOps and every resource change goes through a human review. Initial for batch jobs; Recreate only for workloads that tolerate restarts.
• target, lowerBound, upperBound, uncappedTarget. target is what you take to the manifest; being below lowerBound is an alarm; an uncappedTarget that drifts away from target month after month is a memory leak until proven otherwise.
• VPA and HPA over the same metric destroy each other. The loop is real and takes hours to manifest. controlledResources: ["memory"] when the HPA scales on CPU, or Off mode. The optimal combination is an HPA on a business metric (09-04) and a full VPA.
• In-place resizing will arrive and change the rules, but today it is no basis for a production strategy.
• Beware of the ResourceQuota: recommended requests × the HPA's maxReplicas must fit, or the failure will appear at the traffic peak weeks later.

At Rutas Norte we have left the VPA in Off mode over bookings-api, web-store, bookings-postgres and redis-cache, active in Recreate over notifications-worker and in Initial over occupancy-reports. And we have discovered, with data, that bookings-api had its memory below its lowerBound and that web-store was reserving three times the CPU it uses.

With that, each pod asks for what it needs and the number of pods adapts to traffic. But there is a hidden assumption in everything we have done: we have taken for granted that, when the HPA asks for thirty bookings-api replicas, there is somewhere to put them. The Rutas Norte cluster has four nodes. Thirty bookings-api pods plus fifteen of web-store plus the twenty of notifications-worker do not fit. The scheduler will leave them Pending and the HPA will be able to do nothing about it: it will have asked for the replicas, the Deployment will have created them, and there they will stay, queued, while the May bank-holiday sale falls over exactly as it did last year.

In the next lesson, Cluster Autoscaling, we climb the last step: how to make the cluster add nodes when the pods do not fit, why removing them is far more delicate than adding them, how long a node really takes to be ready (and why that does not save you from a sudden spike), and the over-provisioning trick with filler pods that we now have almost all the pieces to understand.