Ever since module 2 we have been writing resources blocks in every Rutas Norte manifest — cpu: 250m, memory: 256Mi — without ever explaining what they mean exactly or where those numbers come from. The previous lesson even exposed them to the container with resourceFieldRef so that bookings-api could size itself. It is time to take them seriously, because they are the piece that decides which node each pod fits on, how much CPU it gets when there is competition and which process dies when memory runs short. And we are still carrying the last module 2 debt: no namespace has a quota, so right now a wrong deployment in rutas-norte-dev can eat the cluster capacity and leave the production pods with nowhere to go. In this lesson you will understand the complete resource model and you will put a ResourceQuota on each of the three environments.

1. The resource model: requests and limits
2. Units: millicores, Mi versus M
3. Who uses requests: the scheduler
4. Who uses limits: the kubelet and cgroups
5. Compressible CPU versus incompressible memory
6. Diagnosing CPU throttling
7. Diagnosing an OOMKilled
8. ephemeral-storage and disk eviction
9. Cluster overcommitment
10. ResourceQuota: what it limits and how to read it
11. The effect that forces you to declare resources
12. The quotas for the three Rutas Norte environments
13. How to choose sensible values

Every container in a pod can declare two numbers for each type of resource:

The conceptual difference, which is the basis of everything:

A useful analogy, and one that fits Rutas Norte: think of a coach in the fleet. The request is the reserved seat: it is paid for and nobody else can take it, whether you use it or not. The limit is the maximum luggage allowance: you can carry less, but if you try to go over the maximum they stop you at the door.

And the two consequences to fix in your mind from the outset:

1. requests is what "costs" in the cluster. A pod with requests: 4Gi blocks 4 GiB of node capacity even if its process uses 100 MiB. The scheduler treats that node as having 4 GiB less available for everything else.
2. limits reserves nothing. A pod with limits: 8Gi does not reserve 8 GiB; it simply will not be able to go past that if it tries to use them.

You can declare only requests, only limits, both or neither:

The CPU unit is the core (more precisely, a thread of execution: a vCPU in the cloud, a hyperthread on bare metal). It can be expressed in two ways:

The m stands for milli: 1000m = 1 core. The m form is preferred because it avoids floating-point errors and because 0.1 can suffer representation problems. Always write millicores.

CPU is fractional and elastic: if you ask for 250m you are not given a quarter of a physical processor but a quarter of the CPU time available in a time window. And if the node is idle, you can use more (up to your limit).

Memory is measured in bytes and accepts two families of suffixes that are not equivalent:

256M is 6.9% less memory than 256Mi. And 1G is 74 MiB less than 1Gi. In a memory limit, that 6.9% is exactly the difference between a container that just about copes and one that dies with OOMKilled under load.

Rule for Rutas Norte and for any serious project: always use Mi and Gi, never M or G. It is what every Kubernetes tool does and what anyone reading your manifest expects.

And a typo that costs dearly:

Kubernetes accepts it without complaint: m is a valid suffix (milli). The container gets a limit of half a byte and dies instantly. The symptom is a baffling CrashLoopBackOff. m only makes sense for CPU.

If you see that, you already know what the problem is.

The kube-scheduler you studied in module 1 has one job: pick a node for each new pod. And the decision is based exclusively on requests, not on actual usage.

The calculation the scheduler makes for each node is:

Two critical nuances:

Nuance 1: allocatable is not the node's total capacity. Kubernetes reserves a portion for the operating system and for its own components (kubeReserved, systemReserved, evictionHard).

In this minikube the difference is small, but on a managed cloud node it can be 10-15% of the memory. Always plan against Allocatable, not against Capacity.

Nuance 2: requests are summed, not actual usage. This is the source of the most common confusion. A node can be at 8% actual CPU usage and still reject a new pod because the sum of the requests of what is already assigned leaves no room.

Read it carefully: the CPU Requests add up to 58% of allocatable, but the Limits add up to 130%. That is the overcommitment of section 9, and that note in brackets warns about it explicitly.

When there is no room on any node, the pod stays Pending:

Insufficient memory with the pod in Pending always means the same thing: the requests do not fit. The fixes are lowering the requests (if they were inflated), adding a node, or waiting for the cluster autoscaler to do it for you.

A warning about inflated requests: it is tempting to ask for more "just in case". But every MiB of request you do not use is cluster capacity nobody can use and that you are paying for. In real clusters it is normal to find 30-40% of capacity committed and unused purely because of badly calibrated requests.

The scheduler never sees limits. They are applied by the kubelet on the node, and not on its own: it asks the Linux kernel through cgroups (control groups), the mechanism that allows the resources of a set of processes to be limited.

We can see it from inside the container. With limits: cpu: "1" and memory: 512Mi on cgroups v2:

Let us decode the three lines, because they explain the whole behaviour:

• memory.max = 536870912: that is exactly 512 MiB. It is a hard ceiling. If the process tries to allocate byte 536870913, the kernel fires the OOM killer.
• cpu.max = 100000 100000: these are quota and period in microseconds. It means "100,000 µs of CPU every 100,000 µs", that is, a whole core. With limits: cpu: 500m it would be 50000 100000.
• cpu.weight = 10: it derives from the CPU request (250m). It is the relative weight with which the kernel scheduler shares out CPU time when there is competition. A container with twice the request receives twice the time.

That last line is important and little known: the CPU request is not only for the Kubernetes scheduler; it also determines the relative priority within the node. Two pods on the same node competing for CPU share the time in proportion to their requests.

This is the most important distinction in the lesson and the one you really have to understand.

CPU: throttling. If your container has limits: cpu: 500m and tries to use more, the kernel simply gives it no more CPU time in that 100 ms period. The process does not die: it waits. The visible effect is latency. For bookings-api this means that a request that took 80 ms starts taking 400 ms, and if the TIMEOUT_MS of web-store is set to 2000, with enough throttling timeout errors start appearing in the customer's browser.

Memory: death. There is no way to "throttle" memory. If a process has allocated 512 MiB and asks for more, the kernel cannot take anything back: either it gives it memory or it kills somebody. With limits: memory: 512Mi, it kills the process that went over, inside the container's cgroup. The container dies with exit code 137 and the kubelet restarts it according to the restartPolicy.

The practical design consequence:

Memory limits have to be calibrated carefully, because falling short kills the process. CPU limits are far less dangerous, but also far more debatable.

And here there is a genuine debate in the community that is worth knowing about: should you set CPU limits or not?

The Rutas Norte position, which is the recommendable one for a team that is starting out:

• CPU and memory requests: always, on every container. Not negotiable.
• Memory limits: always. A pod with no memory limit can bring down the whole node and cause cascading evictions.
• CPU limits: yes in dev and pre (to spot excessive consumption early) and generous in pro, between 2 and 4 times the request, so that bank-holiday and school-holiday peaks can be absorbed.

Throttling is treacherous because it does not show up in any pod state. The pod is Running, READY 1/1, with no restarts, and it still responds badly.

The signal is in the cgroup itself:

How to read it:

• nr_periods: how many 100 ms periods have elapsed.
• nr_throttled: in how many of them the container used up its quota and was throttled.
• throttled_usec: total microseconds spent waiting.

The key indicator is the ratio:

Almost a quarter of the periods have been throttled. Interpretation:

A command to review every pod of a component:

The proper way to monitor this continuously is with metrics: the container_cpu_cfs_throttled_periods_total series from cAdvisor, which you will see in Monitoring with Prometheus. The command above is for a one-off diagnosis.

The fix: raise the CPU limit. If bookings-api has requests: 250m and limits: 500m with 23% throttling, raising the limit to "1" usually solves it at no real cost, because the limit reserves no capacity.

The opposite case is loud and easy to identify:

The detail is in describe:

The three unmistakable clues: Reason: OOMKilled, Exit Code: 137 (which is 128 + 9, the SIGKILL signal) and a Restart Count that keeps growing.

Careful with Last State: it describes the previous container. If the pod is Running right now but has RESTARTS 4, Last State tells you why it died last time. It is the first thing to look at when facing a pod with restarts.

And a warning about kubectl logs: when the container restarts, the logs start from scratch. To see those of the dead container, which are the ones holding the clue:

There is the cause: an availability query for a bank-holiday weekend returned 8,412 results and the application loaded them all into memory.

The five typical causes of an OOMKilled and how to treat them:

The fourth deserves a note, because it is the most frequent and the most invisible: a Java process without MaxRAMPercentage on a 32 GiB node with a limit of 512 MiB sizes its heap for the node's 32 GiB and dies as soon as it starts working. The previous lesson already gave you the tool to fix it.

There is a third resource that almost nobody declares and that causes baffling incidents: ephemeral storage, that is, the node disk the container uses for:

• The write layer of its file system (anything it writes outside a volume).
• emptyDir volumes.
• The container logs (whatever it writes to stdout/stderr, which the kubelet stores on the node's disk).

It is declared just like the others:

What happens when the limit is exceeded: the kubelet evicts the pod, with Reason: Evicted:

A relevant nuance: unlike an OOMKilled, disk eviction is not immediate. The kubelet checks usage roughly every 10 seconds, so a process can fill the disk before it gets evicted.

And the bigger problem: if the node's disk fills up, the kubelet goes into pressure and evicts pods even when their limits are in order. It is a failure that affects the whole node. The signals:

DiskPressure: True explains apparently random evictions. The order in which the kubelet picks its victims depends on the QoS class, which is the subject of the next lesson.

In Rutas Norte, the natural candidate to fill the disk is notifications-worker: if it produces one log line per email sent and tens of thousands are sent over a bank-holiday weekend, without log rotation the node's disk fills up. The practical rule: declare ephemeral-storage on any component that writes bulky logs or uses emptyDir, and keep an eye on DiskPressure on the nodes.

Let us go back to that earlier line:

The sum of the CPU limits is 130% of the node's allocatable. Is the cluster misconfigured? No: that is normal and even desirable.

The logic of overcommitment: pods do not consume their maximum simultaneously. notifications-worker works in bursts, bookings-api peaks at specific hours, web-store serves static assets with little consumption. Allowing the sum of limits to exceed capacity takes advantage of those gaps.

Where the danger lies, depending on the resource:

The operational conclusion is asymmetric and worth engraving:

Overcommitting CPU is acceptable and normal. Overcommitting memory aggressively is dangerous.

For Rutas Norte, the recommended ratios:

That last case, with requests == limits, has a name and very concrete consequences for who dies first: it is the Guaranteed class, and it is the subject of the next lesson.

And a memory warning that surprises many people: Kubernetes disables swap by default (although since 1.30 there is configurable beta support). With no swap, the kernel has no buffer: when memory runs out, it runs out. It is one more reason not to overcommit it.

Everything above is per container. The ResourceQuota acts at another level: it puts a ceiling on a whole namespace. It is the object that settles the last module 2 debt.

The three families of limits:

Points that define its behaviour:

• It is a namespaced object and only affects its namespace. There are no cluster-wide quotas.
• It is applied at creation time. A pod that would exceed the quota is rejected; those that already exist are untouched.
• There can be several ResourceQuotas in a namespace. All of them apply and all of them must be satisfied.
• Terminated pods do not count. Those in Succeeded or Failed are excluded from the tally.
• services.loadbalancers: "0" is a very useful cost-control tool: every cloud LoadBalancer costs money.

Reading a quota:

Two columns: Used (what is consumed right now) and Hard (the ceiling). This namespace is at 57% of its CPU requests and 40% of its memory. It is the view that answers "how much headroom do I have left in development?".

When somebody tries to go over:

Pay attention to a detail that is very important for diagnosis: the kubectl scale command succeeded. The Deployment now says 12 replicas. What fails is the creation of the pods by the ReplicaSet, and that failure does not appear in kubectl get deploy other than as a READY 8/12 that lasts for ever.

Faced with a Deployment stuck at READY x/y, always look at the ReplicaSet events and the namespace quota. It is one of the most frequent and least obvious causes, and it connects directly with the diagnosis of stuck rollouts you saw in module 2.

A quota can be applied to only a subset of pods:

Available scopes: Terminating and NotTerminating (depending on whether they have activeDeadlineSeconds), BestEffort and NotBestEffort (depending on the QoS class from the next lesson), and PriorityClass. They let you give more headroom to the critical than to the experimental within the same namespace.

Here is the most useful and at the same time most disconcerting behaviour of ResourceQuotas:

If a namespace has a quota limiting requests.cpu or limits.memory, then EVERY pod created in it must declare that resource. If it does not, it is rejected.

The logic is obvious as soon as you think about it: to tally the sum of the namespace's requests.cpu, Kubernetes needs every pod to declare requests.cpu. A pod that did not declare it would make the tally impossible.

Demonstration. With the dev quota active, we try to create a pod with no resources:

This is, by some distance, the most valuable side effect of quotas: it turns the discipline of declaring resources into something mandatory at platform level, rather than a recommendation people forget.

But it has an immediate cost: it breaks everything convenient. An ephemeral pod for debugging, a quick kubectl run, a one-off Job... they all fail. And this is where the missing piece appears, which is the subject of the next lesson: a LimitRange in the namespace injects default values into containers that declare nothing, so the pod passes the quota validation without anyone writing resources by hand.

ResourceQuota and LimitRange are inseparable. Setting a quota without a LimitRange is a defensible decision (it forces everyone to think about their resources), but it turns day-to-day work into a constant nuisance. The combination of the two is the standard configuration of any serious namespace.

Now for real: we apply the quotas and the debt is settled. The criterion is that dev must not be able to do harm and pro must have headroom for bank-holiday and school-holiday peaks.

Comparison and justification:

Applying and verifying:

The debt is settled. Now, if somebody scales bookings-api to 50 replicas in development by mistake, or if a restart loop keeps creating pods, the damage stays inside rutas-norte-dev: the pods simply will not be created and production never notices. Remember from module 2 that the namespace isolates neither the network nor DNS; the quota is one of the few things a namespace really does isolate, and that is why it is so valuable.

A final note on the total sum. If you add up the requests.cpu of the three quotas you get 30 cores, and your cluster may have fewer. That is intentional: the quota is a per-environment ceiling, not a reservation. Nobody guarantees you can reach all three ceilings at once; what it guarantees is that no single one can go past its own.

The numbers above do not come from intuition. This is the method.

Step 1: measure, do not guess. You need metrics-server, which we enabled as an addon in module 1:

The detail of this command and its limitations (it is a snapshot, not a history) is the subject of Metrics Server. To calibrate properly you need history with percentiles, which is what Prometheus gives you.

Step 2: apply the formulas.

Note the asymmetry: memory from the 95th percentile, CPU from the median. It is the direct application of section 5.

Step 3: work it out for bookings-api in production. With two weeks of history, including a bank-holiday weekend:

Step 4: iterate. The initial values are wrong by definition. Review them:

• After every significant change to the application.
• After every high season (in Rutas Norte, after each bank holiday and after the summer).
• Whenever throttling goes above 5% or any OOMKilled appears.
• Quarterly, to reclaim committed and unused capacity.

Four rules that save you grief:

1. Start generous and come down. A limit that is too high costs capacity; one that is too low causes outages in production. On the first iteration, prefer the waste.
2. Multiply by replicas before looking at the quota. requests.cpu: 300m × 8 replicas = 2400m of the pro budget.
3. Stateful components are different. bookings-postgres must have requests == limits for memory: that is the Guaranteed class from the next lesson, and a database must never be the first candidate to die.
4. Document where the numbers come from. A comment in the YAML with the measurement date and the percentile used turns the manifest into something reviewable:

Tips:

1. Every manifest template must ship with resources. Put it in your base template so nobody has to remember.
2. Check the quota before scaling. kubectl describe quota -n <ns> before a kubectl scale.
3. Watch out for unused requests. It is the biggest invisible cost in a cluster: committed capacity nobody takes advantage of.
4. Add an alert at 80% of quota. So you find out before a deployment fails, not during.
5. ephemeral-storage is not optional in production. A node in DiskPressure evicts healthy pods.

1. Create in rutas-norte-dev a pod called memory-hog with limits.memory: 128Mi that runs a process trying to allocate 300 MiB. Use the polinux/stress image or busybox with a file in /dev/shm.
2. Observe its state, identify the Reason and the Exit Code, and explain what 137 means.
3. Create a pod called cpu-hog with limits.cpu: 100m running an infinite loop.
4. Check with kubectl top how much CPU it actually consumes and read /sys/fs/cgroup/cpu.stat to work out the throttling percentage.
5. Explain in a table why one died and the other did not, even though both exceeded their limit.

1. Apply the three ResourceQuotas from section 12.
2. Show the current consumption of each namespace with kubectl describe.
3. Try to scale bookings-api in rutas-norte-dev to a replica count that exceeds the quota. Which command fails and which succeeds? Find the exact error message.
4. Try to create a pod with kubectl run without declaring resources in rutas-norte-dev. Copy the error and explain it.
5. Work out how many replicas of bookings-api (with requests: cpu 250m, memory 256Mi) fit at most in rutas-norte-dev according to each of the four compute limits, and say which one governs.

You have these two weeks of production measurements, including a bank-holiday weekend:

1. Work out the CPU and memory requests and limits by applying the formulas from section 13, and justify why memory uses the p95 and CPU uses the median.
2. Decide a value for ephemeral-storage and reason it out.
3. Write the complete resources block, with the traceability comment.
4. With 3 replicas in rutas-norte-pro, work out what percentage of the production quota this component consumes.
5. If the CPU peak is 780m and you set limits.cpu: 700m, what will happen to the component during the bank holiday? Is that acceptable for a mail worker? Compare with the answer if it were bookings-api.

Solution 1

The 137 is 128 + 9. By Unix convention, a process terminated by a signal returns 128 + signal number, and signal 9 is SIGKILL. It is the unmistakable signature of a violent death: the kernel gave the process no chance to clean up. (Its relative, 143 = 128 + 15, is SIGTERM: graceful termination, the one from module 2.)

The CPU hog:

The infinite loop would like to consume a whole core (1000m), but kubectl top shows exactly 100m: the limit is enforced to the millicore. And the throttling is 601/604 = 99.5%: practically every period has been cut short. Even so, the pod is Running with no restarts.

Solution 2

kubectl scale succeeds and the pod creation fails. The Deployment sits at READY 3/12 indefinitely. That asymmetry is the key to the diagnosis: the error is never on the object you touched, but in the ReplicaSet events.

The quota controls requests.cpu, requests.memory, limits.cpu and limits.memory. To be able to tally those four sums it needs every pod to declare them, so it rejects the one that does not. The elegant fix, without forcing anyone to write them by hand, is the LimitRange from the next lesson.

The maximum replica calculation, with requests: cpu 250m, memory 256Mi and limits: cpu 500m, memory 512Mi:

requests.cpu governs (tied with limits.cpu): 8 replicas. The effective limit is always the most restrictive one, and here it is the CPU. And careful: those 8 replicas would be the whole namespace, so you have to subtract what web-store, bookings-postgres, redis-cache and notifications-worker already consume.

Solution 3

1. Applying the formulas:

The asymmetry between memory and CPU is exactly the one from section 5. Memory uses the p95 because falling short does not mean "running slowly", it means OOMKilled: the confirmation email being sent is lost and the customer never gets their ticket. CPU uses the median because falling short only means the worker takes longer to drain the queue; the generous limit lets it speed up and catch up over a bank holiday.

2. ephemeral-storage: 400 MiB/day of logs plus up to 600 MiB of attachments emptyDir. Assuming daily log rotation and a safety margin:

The factor of 2 covers the case where log rotation fails for a day or where a bank holiday generates twice as many attachments. Without this limit, a rotation failure would fill the node's disk and cause DiskPressure, evicting the healthy pods sharing the node too, bookings-postgres included.

3. The complete block:

4. With 3 replicas in rutas-norte-pro:

A very modest consumption: there is plenty of headroom for bookings-api to scale during the peaks.

5. With limits.cpu: 700m and a peak of 780m, the worker is throttled during the bank holiday. It does not die: it processes more slowly. The concrete effect is that the notification queue grows and the confirmation emails are delayed, perhaps from 30 seconds to several minutes.

Is that acceptable? For notifications-worker, yes, with caveats. It is an asynchronous process: the customer already has their booking confirmed on screen when the email goes out. A delay of minutes is annoying but does not break the business, and the queue drains itself once the peak passes. You would need to watch that the delay does not grow without bound: if the arrival rate exceeds the processing rate, the queue never recovers and then you really do have an incident.

For bookings-api the answer would be no. It is synchronous: the customer is sitting in front of the screen waiting. Throttling of 20% turns an 80 ms response into a 400 ms one, web-store starts hitting its TIMEOUT_MS of 2000, and the customer sees errors at the very moment of the highest sales of the year. In a user-facing component, the CPU limit must cover the peak comfortably.

That difference — between what merely runs slower and what the customer suffers — is what you must keep in mind when calibrating each component.

You no longer write resources by eye. You understand the whole model: requests are what the scheduler reserves when picking a node and what determines relative CPU priority within the node; limits are the ceiling the kubelet enforces through cgroups at runtime. You know that the scheduler only looks at requests, never at actual usage, which explains why a node at 10% usage can reject a pod. And you have mastered the units, including the two traps that cause real incidents: the 6.9% difference between M and Mi, and the lower-case 512m that sets a limit of half a byte.

You are clear about the fundamental distinction: CPU is compressible and exceeding it only produces throttling — diagnosable with the nr_throttled/nr_periods ratio from cpu.stat, invisible in the pod state — whereas memory is incompressible and exceeding it produces OOMKilled with code 137, with its five typical causes and the reflex of looking at kubectl logs --previous. You know about ephemeral-storage and the disk eviction that can bring down healthy pods across a whole node, and you understand why overcommitting CPU is normal and overcommitting memory is dangerous.

And you have settled the last module 2 debt: the three Rutas Norte environments have a ResourceQuota. rutas-norte-dev cannot go past 2 committed cores and 4 GiB, nor create a single LoadBalancer; rutas-norte-pro has 24 cores and 48 GiB of headroom for bank-holiday and school-holiday peaks. You can read kubectl describe quota, recognise the exceeded-quota error, and diagnose the treacherous case where kubectl scale succeeds and the Deployment sits at READY 8/12 for ever. And you have a method for choosing values: measure with kubectl top, apply the p95 to memory and the median to CPU, document the date and the percentile in the manifest itself, and iterate after every high season.

One loose end remains, and it has come up twice in this lesson. Setting a compute quota forces every pod to declare resources, and that breaks any quick kubectl run and any third-party manifest. The missing piece is the LimitRange, which injects default values into containers that declare nothing. And there is another, even more interesting loose end: we know that when a node runs short of memory somebody dies, but not who. The answer is not random: it depends on a classification Kubernetes assigns to each pod according to how it declared its resources. The next lesson, LimitRanges and Quality of Service (QoS) Classes, covers both things: the default, min, max and maxLimitRequestRatio fields, their exact interaction with the ResourceQuota, and the Guaranteed, Burstable and BestEffort classes with their effect on the kernel's oom_score_adj and on the kubelet's eviction order. By the end you will know why bookings-postgres must be Guaranteed and you will be able to trigger an eviction in your own minikube to see with your own eyes which pod falls first.