The previous lesson put quotas on the three Rutas Norte environments and left two very specific loose ends. The first is practical and gets in the way every day: ever since rutas-norte-dev got a compute quota, a simple kubectl run fails with must specify requests.cpu, and any third-party manifest that does not declare resources is unusable. The second is deeper: we know that when a node runs short of memory somebody has to die, but we do not know who, and that decision is not random at all. This lesson closes both: the LimitRange injects default values and sets per-container ceilings within the namespace, and the quality of service classes — Guaranteed, Burstable and BestEffort — determine the exact order in which the kubelet evicts and the kernel kills. By the end you will have assigned a reasoned class to every Rutas Norte component and triggered a real eviction in your minikube to see it with your own eyes.

1. What a LimitRange solves
2. The fields: default, defaultRequest, min, max, maxLimitRequestRatio
3. Demonstration: a pod with no resources comes out with resources
4. The types: Container, Pod and PersistentVolumeClaim
5. The exact interaction between LimitRange and ResourceQuota
6. The three QoS classes and the rule that determines them
7. Checking a pod's class
8. Consequence 1: oom_score_adj and the kernel's OOM killer
9. Consequence 2: the kubelet's eviction order
10. The QoS class of each Rutas Norte component
11. The LimitRange for the three environments
12. Experiment: trigger an eviction and see who falls first

The LimitRange is a namespaced object that does two distinct things to every container created in it:

1. It injects default values into containers that do not declare requests or limits.
2. It validates ceilings: it rejects containers asking for less than a minimum or more than a maximum.

The difference from a ResourceQuota is the scope, and it is worth being crystal clear about it because the two get confused constantly:

An analogy with Rutas Norte: the ResourceQuota is the department's annual budget (you cannot spend more than X euros between all of you). The LimitRange is the per-person expenses policy (nobody can book a ticket costing more than Y euros, and if no class is specified, economy is assumed).

The three concrete problems it solves:

Problem 1: the quota breaks everything convenient. We already saw it:

Problem 2: a single container can hog the entire quota. Without a LimitRange, somebody can deploy a pod with requests.memory: 4Gi in rutas-norte-dev and exhaust the environment's whole memory quota in one go, leaving the five components with nowhere to run.

Problem 3: absurd values. A container with requests.memory: 4Mi will never start reliably, but nothing stops it.

Field by field:

Resolution rules you need to know precisely:

Rule 1: what is declared always wins. The LimitRange never overwrites a value the container declares. It only fills gaps.

Rule 2: if you declare limits but not requests, the request takes the value of the limit, not of defaultRequest. This behaviour catches a lot of people out:

With the LimitRange above (defaultRequest.memory: 128Mi) you would expect requests: 128Mi. But the result is:

The Kubernetes rule of "if there are only limits, the request equals the limit" takes precedence. Practical consequence: you reserve 1 GiB of the namespace budget without meaning to.

Rule 3: if you declare requests but not limits, default is applied. Here it does work as you would expect... unless default is lower than your request, in which case the pod is rejected as inconsistent.

Rule 4: maxLimitRequestRatio limits individual overcommitment. With memory: "4", a container with requests.memory: 128Mi cannot have a limits.memory greater than 512Mi. It is the tool for stopping somebody reserving little and then consuming a lot, which is exactly the pattern that causes evictions.

A table of the six possible scenarios, with the example LimitRange:

Nothing is as convincing as seeing it. We start from rutas-norte-dev with its quota and we apply the LimitRange:

Now the same kubectl run that failed before:

It works. And the interesting part is seeing what resources it actually has:

The manifest we sent had no resources and the object stored in etcd does have them. The LimitRange injected them.

Who performs that injection? An admission controller called LimitRanger, which is part of the apiserver. Remember the flow from module 1: a request goes through authentication, authorization and admission before being written to etcd. Admission has two phases:

The order is crucial and explains everything in section 5: the LimitRange mutates first, the ResourceQuota validates afterwards. By the time the quota looks at the pod, it already has its resources injected.

And a consequence to keep in mind: the object in the cluster is no longer identical to the YAML file in your repository. It is the first time in the course that this happens, and it is worth knowing so you are not baffled by a kubectl diff. The pod also records it as an annotation:

Let us also check the ceilings. A container asking for too much:

And one asking for too little:

All three behaviours verified: it injects, it sets a ceiling and it sets a floor.

A LimitRange can have several entries in spec.limits, each with its own type:

Differences between the three:

The Pod type becomes useful when module 6 arrives with sidecars and init containers: a pod can have three or four containers, each within the Container max, but adding up to an outrageous total. The Pod limit cuts that off.

The PersistentVolumeClaim type will make sense in module 5. With max: storage: 200Gi nobody can ask for a 2 TiB disk through a typo, something that in the cloud translates into an unpleasant bill at the end of the month.

This is the point where everything fits together. The sequence, for a pod arriving with no resources at a namespace holding both objects:

The four consequences of this order:

Consequence 1: the LimitRange saves the ResourceQuota from its own strictness. Without a LimitRange, the quota forces you to declare resources by hand everywhere. With a LimitRange, the default value is injected and the quota tallies it without anyone writing anything.

Consequence 2: injected values consume quota. This is not free and has to be sized. If defaultRequest.memory is 128Mi and somebody launches 20 debugging pods with no resources, they eat 2.5 GiB of the namespace budget. That is why the default values must be modest: they are for pods nobody has calibrated.

Consequence 3: a pod can pass the LimitRange and fail on the quota. They are independent validations and both must be satisfied:

That pod met the LimitRange max (2Gi), but it did not fit in what was left of the quota.

Consequence 4: changing a LimitRange does not affect existing pods. Admission acts only at creation. If you raise defaultRequest, the pods already running keep what was injected into them. To apply the new values you have to recreate them:

And a consistency rule between the two objects that avoids an unpleasant problem: the LimitRange max must never exceed the namespace quota. If max.memory is 8Gi but the requests.memory quota is 4Gi, somebody can write a manifest that passes the LimitRange and is impossible to deploy. It is better for the rejection to come with the LimitRange's clear message ("maximum memory usage per Container is 2Gi") than with the quota's, which is harder to interpret.

We change subject and reach the most interesting part. When a pod is created, Kubernetes automatically assigns it a quality of service class in status.qosClass. It is not declared: it is deduced from how the resources are set.

The exact rule, in evaluation order:

In precise words:

Guaranteed — assigned if and only if, for every container in the pod (init containers included):

• CPU and memory limits are declared.
• CPU and memory requests are declared (or omitted, in which case they equal the limits).
• requests is exactly equal to limits for both resources.

BestEffort — assigned if no container in the pod declares any request or any limit, for any resource.

Burstable — everything else. It is the default bucket: at least one container declares something, but the strict Guaranteed condition is not met.

Examples that clarify the borderline cases:

Three traps worth memorising:

Trap 1: one container is enough to ruin the class. If a pod has the application with requests == limits and a logging sidecar with no resources, the whole pod is Burstable. The class belongs to the pod, not to the container.

Trap 2: with a LimitRange active, BestEffort is almost impossible. The LimitRanger injects default values, so no pod ever reaches BestEffort. It is an important and generally desirable side effect.

Trap 3: defaultRequest different from default prevents Guaranteed. If the LimitRange injects requests: 100m and limits: 300m, every pod with no resources will be Burstable. To make a pod Guaranteed you have to declare it explicitly.

For every pod in the namespace at a glance:

A quick count by class:

Those six BestEffort pods deserve a look: in a production cluster they are the first candidates to die, so it is worth knowing who they are and whether that is intentional.

describe shows it too:

Here is the first real consequence of the class, and it is literally a matter of life and death.

When the Linux kernel runs out of memory, it invokes the OOM killer, which picks a victim. The choice is based on a per-process score, oom_score, which combines the memory being consumed with a manual adjustment called oom_score_adj, a value between -1000 and 1000. The higher it is, the sooner you die.

The kubelet sets that adjustment according to the QoS class:

The formula for Burstable is where all the nuance lies:

Read it slowly: the larger your memory request, the lower your score and the later you die. A Burstable pod that asks for a lot of memory is almost as protected as a Guaranteed one; one that asks for very little is almost as exposed as a BestEffort.

Let us check it on a node with 8 GiB allocatable. For bookings-api with requests.memory: 512Mi:

Exactly as predicted. And the business conclusion is direct:

When the node runs short of memory, the kernel will kill bookings-api long before bookings-postgres. And that is precisely what we want: losing an API pod means losing a few requests the customer can retry; losing the database means the whole platform stops selling tickets and, in the worst case, that a half-finished transaction corrupts data.

You have to distinguish two different situations that are often confused:

The second case is the baffling one: a pod dies and its metrics show it was well below its limit. The cause is a neighbour that went out of control, and the QoS class is what determined that the victim was this one and not another.

Before the kernel gets to killing processes, the kubelet has its own, more orderly mechanism: resource pressure eviction.

The kubelet monitors the node's resources every 10 seconds. When a threshold is crossed, it flags the corresponding condition (MemoryPressure, DiskPressure, PIDPressure) and starts evicting pods to reclaim resources.

The default thresholds:

And the victim selection criteria, in this strict order:

1. First, whether the pod exceeds its requests. Those consuming more than they reserved are candidates before those staying within.
2. Second, the QoS class: BestEffort first, then Burstable, and Guaranteed last.
3. Third, the PriorityClass, if one is defined.
4. Fourth, how far usage exceeds the request. Between two equal candidates, the one that went furthest over falls.

Put operationally:

The first pod to fall is a BestEffort. If there is none, the Burstable that has gone furthest over its request falls. A Guaranteed pod staying within its resources is practically untouchable.

An evicted pod looks like this:

That message says it all: request is 0 because it was BestEffort, and therefore any consumption exceeds its reservation. It was the perfect candidate.

Two important practical notes:

Evicted pods do not delete themselves. They stay in Failed, taking up space in etcd. It is worth cleaning them up:

An evicted pod belonging to a Deployment is recreated. The ReplicaSet from module 2 sees a replica missing and creates another, possibly on a different node. A loose evicted pod, by contrast, never comes back. One more reason not to deploy loose pods.

Now for the design decision, which is what gives meaning to everything above. The question, for each component, is: what happens if this pod suddenly dies?

The concrete manifests. bookings-postgres as Guaranteed:

bookings-api as Burstable:

The ad-hoc analysis as BestEffort:

Careful: with a LimitRange active in rutas-norte-dev, that resources: {} will not produce BestEffort, because the LimitRanger will inject the default values. To get a genuinely BestEffort pod you need a namespace with no LimitRange, and that is why the experiment in section 12 uses a separate one.

One case deserves comment: redis-cache is Burstable even though it has state. Should it not be protected like bookings-postgres? No, and the difference is the value of the data. redis-cache holds cached seat availability: if it is lost, it is recomputed by querying the database. The impact is a spike in latency and load on bookings-postgres, not a loss of information. What determines the class is not whether the component has state, but what is lost if it dies.

We close the configuration part with the three objects, consistent with the quotas from the previous lesson.

A comparison of the decisions:

Time to check the theory. We are going to create a namespace with no LimitRange (so we can have genuinely BestEffort pods), deploy three pods of the three classes, and squeeze the node's memory until the kubelet starts evicting.

Warning: do this in your practice minikube, never in a shared cluster. We are going to deliberately cause memory pressure on a node.

Step 1: prepare the ground.

About 7.7 GiB allocatable.

Step 2: the three pods.

The three classes confirmed. Let us also check their oom_score_adj:

The 975 of victim-burstable comes from the formula in section 8: 1000 - (1000 × 200Mi / 7751Mi) = 1000 - 25 = 975. It asks for very little, so it is almost as exposed as the BestEffort one.

Step 3: squeeze until the pressure appears.

Step 4: observe.

The order is exactly as predicted: first the BestEffort, then the Burstable that had gone furthest over its request, and the Guaranteed is still standing.

Step 5: read the eviction messages.

Compare the last two sentences: the BestEffort was exceeding a reservation of 0 (anything it uses exceeds it), and the Burstable was exceeding its 200Mi reservation by 505Mi. Both were candidates; the BestEffort went first by class.

Step 6: the node condition and the events.

Step 7: clean up.

What this experiment demonstrates, translated into Rutas Norte terms: if an August bank holiday causes memory pressure on the node where bookings-postgres lives, the database will not be the one to fall. The ad-hoc analysis pods will fall first, then the bookings-api replicas that have gone over their reservation — and the Service will keep spreading traffic among the ones left — and the platform will keep selling tickets. That chain of survival is no accident: you designed it when you assigned the classes.

Tips:

1. A quota and a LimitRange always go together. Treat them as a single decision when creating a namespace.
2. Modest defaults, generous max. The default value is for the uncalibrated; the max is a safety net against extra zeros.
3. Guaranteed only for the genuinely critical. It costs real capacity: you reserve the maximum all the time. In Rutas Norte, only bookings-postgres.
4. Watch the Burstable pods with a high ratio. A pod with a request of 200Mi and a limit of 4Gi is a permanent eviction candidate. maxLimitRequestRatio prevents it from the namespace.
5. Add the QoS class to your manifest reviews. It is one line in a pull request review and it stops you discovering during an incident that the database was Burstable.

With the limits-dev LimitRange from section 11 applied in rutas-norte-dev:

1. Create six pods, one per row of the table in section 2 (nothing at all, CPU requests only, memory limits of 1Gi only, both complete, requests.memory: 16Mi, and requests: 128Mi with limits: 1Gi).
2. For those that get created, show their effective resources and their QoS class.
3. For those that fail, copy the exact error message.
4. Explain in particular why the third one ends up with requests.memory: 1Gi instead of 128Mi, and what consequence that has on the namespace quota.
5. Work out how much quota the six would consume between them if they had all been created.

1. List every pod across the three Rutas Norte namespaces with its QoS class in a single table.
2. Check whether bookings-postgres is Guaranteed in all three environments. If it is not in one of them, identify why.
3. Modify the bookings-postgres manifest so that it is Guaranteed in rutas-norte-pro, verify the change and check its oom_score_adj.
4. Work out by hand the expected oom_score_adj of bookings-api with requests.memory: 512Mi on a node with 7751Mi allocatable, and compare it with the real value.
5. Add a sidecar with no resources to bookings-postgres and observe what happens to the pod's class. Explain the result and revert the change.

1. Reproduce the experiment from section 12 in your minikube.
2. Note the exact order of the evictions and the describe messages.
3. Modify victim-burstable so that its requests.memory is 1Gi (instead of 200Mi) while keeping the same consumption, and repeat the experiment. Does the order change? Explain why using the oom_score_adj formula.
4. Explain what would have happened if the Rutas Norte bookings-postgres had been on that node as Burstable with requests.memory: 256Mi and a real consumption of 1.8 GiB.
5. Propose two further measures, beyond the QoS class, to protect bookings-postgres from this scenario, and say in which lesson of the course each one is covered.

Solution 1

The p3 case is the interesting one: we declared only limits.memory: 1Gi and the result is requests.memory: 1Gi, not the 128Mi of the defaultRequest. The reason is the resolution order: Kubernetes first applies its general rule of "if there is a limit and no request, the request equals the limit", and the LimitRange's defaultRequest only fills in what is still empty after that.

The consequence for the quota is serious: this pod, which probably consumes 4 MiB of real memory, has committed 1 GiB of the namespace's 4 GiB budget. With four pods like that, rutas-norte-dev runs out of memory quota without anybody using anything. It is a very expensive and very invisible mistake.

Total consumption if all six had been created (only the four valid ones count):

p3 is only 25% of the pods and consumes 67% of the committed memory.

Solution 2

bookings-postgres is not Guaranteed in any environment. The reason:

The memory matches but the CPU does not (500m against 2). One unequal resource is enough to lose Guaranteed. It is a classic mistake: the memory gets equalised with the OOM in mind and the CPU is forgotten.

The bookings-api calculation:

It matches. The 1931-point difference from bookings-postgres is enormous: in practice, the kernel will sacrifice every API replica before touching the database.

The sidecar:

The pod has lost Guaranteed because of a sidecar with no resources. In rutas-norte-pro the LimitRange injects requests: 200m/256Mi and limits: 500m/512Mi, which differ from each other, and that is enough. The fix is to give the sidecar requests == limits too:

With that the pod regains Guaranteed. Lesson: in a pod that must be Guaranteed, every one of its containers must be too, sidecars and init containers included.

Solution 3

The observed order is BestEffort → Burstable → the Guaranteed survives, with the messages from section 12.

With requests.memory: 1Gi in victim-burstable and the same 700Mi consumption:

The Burstable no longer falls. Two reasons acting at once:

1. Its oom_score_adj dropped from 975 to 868, so the kernel prioritises it less as a victim.
2. And above all, it no longer exceeds its request: it consumes 700Mi and it reserved 1Gi. The kubelet's first selection criterion (section 9) is precisely "pods exceeding their requests", and this one is no longer in that group.

The operational moral is emphatic: declaring a realistic memory request is the cheapest protection there is against eviction. It does not change what you consume; it changes whether you are a candidate or not.

4. If bookings-postgres had been Burstable with requests.memory: 256Mi and a real consumption of 1.8 GiB:

A 967, almost as exposed as a BestEffort, and exceeding its request by more than 1.5 GiB, which puts it in the first group of candidates. It would have been among the first to fall. For Rutas Norte that means: the database goes down at the peak sales moment of a bank holiday, bookings-api starts returning connection errors, web-store shows errors to customers, and in the worst case a half-finished booking transaction leaves seats blocked with no ticket issued. A PostgreSQL Recreate also implies WAL recovery at start-up, with minutes of unavailability.

All of that is avoided by one line: requests == limits.

5. Two further measures:

And a third, the most important in the medium term: turning bookings-postgres into a StatefulSet with a persistent volume and verified backups, so that the survival of the data does not depend on the survival of the pod. That is the work of modules 5 and 6.

You have closed the two loose ends left by the previous lesson. The LimitRange turns a strict quota into something workable: it injects default and defaultRequest into containers that declare nothing, it sets a floor and a ceiling with min and max, and it limits individual overcommitment with maxLimitRequestRatio. You know that it acts as a mutating admission controller before the ResourceQuota validates, that therefore the injected values consume quota, that it is not retroactive, and that its three types — Container, Pod and PersistentVolumeClaim — cover everything from the single container to the cost of a disk. And you know the rule that costs the most money when ignored: if you declare only limits, the request copies the limit, not the defaultRequest, and with that you reserve far more than you think.

You have also mastered the quality of service classes and the exact rule that determines them: Guaranteed demands requests == limits for CPU and memory on every container in the pod, BestEffort demands that none of them declares anything, and Burstable is everything else. You know how to check it with -o jsonpath='{.status.qosClass}' and, above all, you know what consequences it has: the oom_score_adj the kubelet writes into each process (-997 for Guaranteed, up to 1000 for BestEffort, and a formula proportional to the memory request for Burstable) and the kubelet's eviction order, which looks first at who exceeds their requests and then at the class. You have distinguished the cgroup OOM — you die from your own limit, whatever the class — from the node OOM, where the class decides everything.

Rutas Norte now has a reasoned design: bookings-postgres is Guaranteed because it holds the bookings and the customers' personal data and its death stops sales; bookings-api, web-store, redis-cache and notifications-worker are Burstable because they have replicas, are recoverable and need to burst on bank holidays; and exploratory analysis is BestEffort because it must be the first thing to fall. All three environments have a LimitRange consistent with their quota. And you have verified it by triggering a real eviction in minikube, watching the BestEffort fall first, then the Burstable that had gone over its reservation, and the Guaranteed survive; and discovering along the way that raising a request to a realistic value is the cheapest protection against eviction there is.

One last piece of the module remains, and it is of a different nature. We have given our components their configuration, their credentials and their resources, but we have not given them an identity. Every Rutas Norte pod is right now using the default ServiceAccount of its namespace, with a mounted token none of them needs, and none of them has any way of telling the Kubernetes API who it is. The next lesson, ServiceAccounts and API Access from Pods, solves it: you will see the difference between users and ServiceAccounts, why using the default one is a bad idea, how tokens have changed (from eternal secrets to short-lived projected tokens the kubelet rotates), what exactly lives in /var/run/secrets/kubernetes.io/serviceaccount/, why automountServiceAccountToken: false should be your default choice, and you will talk to the API from inside a pod with curl to see both an authorised response and a well-deserved 403 Forbidden.