In the previous lesson we taught Kubernetes to tell a live container apart from a healthy application. Rutas Norte already knows whether each component works. What it still does not know is how much it consumes. And that figure is not a luxury: in 03-04 we set the requests and limits of each component practically by eye, promising that we would calibrate them by observing real consumption. Without that data, half our cluster is reserving memory nobody uses while the other half suffers CPU throttling in silence.

This lesson introduces the first source of real consumption data in the cluster: metrics-server, a lightweight component that gathers CPU and memory usage from nodes and pods and publishes it inside the Kubernetes API itself. It is the piece that makes kubectl top work, and also the one that will make module 9's autoscaling work. We are going to understand exactly what it measures, where it gets the data from, and — just as importantly — what it cannot do, because those limitations are precisely the reason we will install Prometheus in the next lesson.

1. Where metrics-server sits in the observability ecosystem
2. Where the data comes from: kubelet, cAdvisor and the Summary API
3. How it is published in the API: the aggregation layer and the APIService
4. Installation: the minikube addon and the official manifest
5. kubectl top nodes: reading each column correctly
6. kubectl top pods: --containers, -A, --sort-by and their subtleties
7. The essential limitations and why Prometheus is needed
8. The central use case: recalibrating the Rutas Norte requests and limits
9. Why the HPA and the VPA depend on this component
10. Diagnosing the typical failures
11. Common mistakes and tips
12. Exercises

Before installing anything it is worth placing the piece, because it is the component that causes the most confusion in the module. Many people install it expecting a monitoring system and end up disappointed.

metrics-server is not a monitoring system. It is an internal cluster infrastructure component with a very specific and very narrow purpose:

To provide the Kubernetes API with the current CPU and memory consumption of nodes and pods, so that Kubernetes' own autoscaling mechanisms can make decisions.

Everything else — the fact that you can look at it with kubectl top — is a useful side effect.

The two coexist in any serious cluster: metrics-server because the HPA needs it, Prometheus because you need to know what happened yesterday.

A third common confusion: metrics-server does not replace the probes from 07-01. Probes answer "is it healthy?"; metrics-server answers "how much does it spend?". A pod can consume 5 millicores and be completely hung.

So as not to treat metrics-server as a black box, we need to follow the data's path from the kernel to your terminal.

When the container runtime creates the bookings-api container, the Linux kernel puts it in a control group (cgroup). The kernel accounts there, among other things:

• cpuacct.usage: nanoseconds of CPU accumulated by that cgroup since it started.
• memory.current: bytes of memory in use right now by that cgroup.

Those are the primary data. Everything else is transport layers.

cAdvisor (Container Advisor) is a Google component that reads the cgroups and translates them into meaningful container metrics. For years it has been built into the kubelet binary: it is not a separate pod and does not have to be installed. Every kubelet carries its own cAdvisor that examines the containers on its node.

The kubelet exposes what cAdvisor gathers on an authenticated HTTPS endpoint:

It returns a JSON with the consumption of the node and of every pod and container. A simplified excerpt:

Two fields deserve special attention, because they explain everything you will see later:

• usageNanoCores: nanocores consumed. One core = 1,000,000,000 nanocores. The 187,000,000 in the example are 187 millicores, that is, 187m in Kubernetes notation.
• workingSetBytes: the "working set". It is resident memory minus the file cache pages the kernel could reclaim without trouble. It is the metric the kubelet uses to decide who to evict under memory pressure, and it is the one you will see in kubectl top. It is not the same as the RSS that ps gives you, which is why the numbers sometimes do not match what you see inside the container.

metrics-server is a single Deployment (one replica in small clusters) that:

1. Discovers the nodes by querying the Kubernetes API.
2. Queries the Summary API of every kubelet every 15 seconds (the --metric-resolution parameter).
3. Keeps the last two points of each series in memory.
4. Calculates the CPU rate by dividing the increment in accumulated nanocores by the elapsed time.
5. Serves those values through the Kubernetes API.

Key point: it writes nothing to disk and keeps no history. If the metrics-server pod restarts, kubectl top stops working for about 30 seconds and then comes back, with no data about the past.

That split also explains why CPU takes a while to appear: since it is a rate, two samples are needed. A newly created pod shows no CPU until metrics-server has two readings of it.

Here is the elegant part of the design. metrics-server does not expose a port of its own for you to connect to. Instead, it registers itself inside the Kubernetes API by means of the API Aggregation Layer.

The mechanism is the same one we saw in 06-06 for CRDs, but through a different route: instead of the API Server storing the objects in etcd, it delegates the requests of a particular API group to an external service.

This means that when you run:

what happens underneath is a perfectly ordinary REST request to the Kubernetes API:

The API Server sees that the path belongs to the metrics.k8s.io group, consults its APIService table, and forwards the request to the metrics-server Service in kube-system. The response comes back to the client as if the API Server itself had produced it.

The practical consequences of this design are important:

• Unified authentication and authorization: the RBAC we will study in 08-01 applies to kubectl top exactly as it does to kubectl get pods. Anyone without get permission on pods.metrics.k8s.io will see nothing.
• A single entry point: no need to open ports or expose additional services.
• A specific fragility: if the APIService is False (for example, because the metrics-server pod will not start), some API discovery operations slow down for the whole cluster. You will see warnings such as couldn't get resource list for metrics.k8s.io/v1beta1. It is an annoying side effect worth recognising.

Checking the state of the APIService:

AVAILABLE: True is the sign that everything is fine. If it says False (MissingEndpoints) or False (FailedDiscoveryCheck), go straight to section 10.

Since module 1 we have been working with minikube on the rutas-norte profile, and we already listed metrics-server among the required addons. If you have not enabled it yet:

The addon deploys the Deployment in kube-system with the configuration already adapted to minikube (including the flag from the next section).

Wait between 30 and 60 seconds before expecting data: metrics-server needs at least two collection cycles.

In a cluster that is not minikube you install it by applying the project's official manifest:

That manifest creates, in kube-system: a ServiceAccount, a ClusterRole with read permissions on nodes and pods, the corresponding ClusterRoleBindings (including the delegation of authentication to the API Server), the Service, the Deployment and the APIService we saw earlier.

This is the point that blocks an installation most often. metrics-server connects over HTTPS to port 10250 of each kubelet. By default it verifies the kubelet's certificate against the cluster CA. The problem: in many installations (minikube, kubeadm without kubelet server certificate rotation, some managed clusters) the kubelet uses a self-signed certificate that is not signed by the cluster CA, or whose name does not match the node's IP.

The result: metrics-server cannot connect and kubectl top returns nothing. The solution that appears on every forum:

Security warning. --kubelet-insecure-tls disables verification of the kubelet's certificate. The connection remains encrypted, but metrics-server no longer checks that the other end really is the node's legitimate kubelet. An attacker able to get in the middle of the control plane network could impersonate a kubelet and return fake metrics. In a cluster with an HPA, that is a real attack vector: made-up CPU metrics can trigger massive scaling (cost) or prevent necessary scaling (denial of service).

It is acceptable in minikube and in development environments. It is not in rutas-norte-pro. The correct solution in production is to enable kubelet server certificate rotation (--rotate-server-certificates=true on the kubelet and approval of the kubernetes.io/kubelet-serving CSRs), so that certificates are signed by the cluster CA and verification works. In a managed cluster (EKS, AKS, GKE, which we will see in 10-06), the provider has already solved this and the flag is not needed.

The remaining arguments, briefly:

With metrics-server working, we can start looking.

How to read this correctly, column by column:

• CPU(cores): millicores consumed right now by everything running on the node (system pods included). 1834m means 1.834 cores busy continuously.
• CPU%: the percentage of the node's allocatable capacity (node.status.allocatable.cpu), not of total capacity nor of the sum of requests. This is an important detail: allocatable = total capacity minus what is reserved for the operating system and for the kubelet.
• MEMORY(bytes): the node's workingSetBytes. Repeating the subtlety from section 2: it does not include reclaimable page cache, so it is almost always lower than what free -m shows inside the node.
• MEMORY%: percentage of allocatable.memory.

Checking a node's allocatable to verify the calculation:

With 4 allocatable cores and 2701m consumed: 2701 / 4000 = 67.5 %. It adds up.

An operational reading of the table above for Rutas Norte:

• rutas-norte-worker-2 is at 89 % memory. Dangerous: when the kubelet detects memory pressure it starts evicting pods, beginning with those with BestEffort and Burstable QoS that exceed their requests (go back over 03-05). We need to investigate what runs there and probably rebalance.
• rutas-norte-worker-3 is practically empty at 7 % and 14 %. That points to a scheduling problem: perhaps it has a taint we had not anticipated, or the pods have affinity rules that exclude it (06-05).

Useful flags:

A critical difference from top nodes: here there are NO percentage columns. kubectl top pods shows no % at all, because it would not know what to calculate it against (against the limit? against the request? against the node?). You have to make that comparison yourself, and in section 8 we will.

With the arrival of sidecars in 06-04, a pod is no longer a container. bookings-postgres-0 has the PostgreSQL container and the metrics exporter; notifications-worker has the worker and the log adapter. Adding them together hides valuable information:

Now we know that the exporter sidecar costs 13m of CPU and 22Mi: cheap, as we expected. And that the log adapter costs 4m and 15Mi. These numbers matter because, multiplied by the number of replicas, an apparently harmless sidecar can consume a whole node in a large cluster.

--sort-by only accepts cpu or memory. Watch out for one counter-intuitive behaviour: the sorting is done on the numeric values, but the output is printed in Kubernetes units, and 1024Mi comes after 2Gi alphabetically even though it is smaller. Trust the order --sort-by gives you, not the visual reading.

That last question is more useful than it looks. If out of six bookings-api replicas one consumes 800m and the rest 190m, you have a traffic distribution problem or a pathological request being served by that particular pod.

This section is the most important in the lesson, because it defines when you should not use this tool.

metrics-server keeps the last two points of each series in memory. There is no database, no file, no configurable retention. Questions you cannot answer with kubectl top:

• "How much memory was bookings-api using last night at 03:00, when it went down?"
• "Has bookings-postgres consumption been growing over the last two weeks?"
• "What was the CPU peak during the May bank-holiday weekend?"

All of them are exactly the questions asked after an incident, and for all of them metrics-server's answer is silence.

With --metric-resolution=15s and CPU rate calculation windows of tens of seconds, a 3-second CPU spike does not show up. It is averaged out and disappears. For bookings-api, whose latency spikes are measured in seconds, that resolution is not enough to diagnose anything.

It also means that two consecutive runs of kubectl top can give the same value: if both fall inside the same collection cycle, you are reading the same sample.

Nothing about disk, nothing about network, no application metrics. You will not know how many requests per second bookings-api serves, nor how many bookings have been confirmed, nor how many connections PostgreSQL has open. Nothing about the business.

There are no queries, no thresholds, no notifications. kubectl top is an interactive tool: somebody has to be sitting there typing it. At three in the morning nobody is sitting there.

In 03-04 we saw that a container reaching its limits.cpu is not killed: it is throttled. The symptom is high latency with CPU consumption stuck at the limit. kubectl top shows you the millicores, but not the throttled periods counter (container_cpu_cfs_throttled_periods_total), which is the figure that confirms the diagnosis. You will only see that counter in Prometheus.

The honest conclusion: metrics-server is a photograph; Prometheus is the film. You need both, for different reasons. The photo is free and always there; the film costs money, disk and maintenance, but it is the only one that tells you what happened.

This is the legitimate and most valuable use of kubectl top, and the one that fulfils the promise made in 03-04.

• requests: what the scheduler reserves on the node. It determines where the pod fits and what QoS it gets. Over-reserving wastes cluster (nodes at 30 % real usage and "full" as far as the scheduler is concerned); under-reserving sends pods to nodes where they do not really fit and makes them suffer.
• limits: the ceiling. Exceeding it on CPU causes throttling; exceeding it on memory causes OOMKilled.

You do not recalibrate requests by looking once. The honest procedure:

1. Observe over a representative period: at least a week, including a peak. For Rutas Norte you have to include a bank-holiday weekend or the start of the holidays.
2. Take periodic samples, not a single reading. Since metrics-server keeps no history, this means a script that runs kubectl top every few minutes and accumulates the results.
3. Calculate percentiles, not averages. The average lies when there are spikes.
4. Apply the criteria:

The asymmetry between CPU and memory is fundamental and must be understood: if you go over on CPU, the system slows you down and you survive. If you go over on memory, the kernel kills you. That is why memory is sized by the high percentile and CPU by the middle one.

An important note for beginners: the command above with -A returns the namespace as the first column, which is why read takes five fields. If you run it without -A, drop the ns variable.

And the analysis of the data collected:

After two weeks of observation (including the 1 May bank-holiday weekend), these are the figures and the decisions:

Three findings worth commenting on, because they are the typical ones:

1. bookings-api was requesting 100m and using 185m on average. With Burstable QoS, that works while the node has room, but the scheduler was placing pods as if each one took 100m: it was overselling the node. And the limits.cpu of 500m caused throttling at the 640m peaks, which explains the latency spikes nobody could attribute to anything. This is the most valuable finding of the whole exercise.
2. web-store was reserving 4 times what it uses. Multiplied by its replicas and by three environments, that is several cores and several GB of cluster reserved for nothing, and counting against the namespace's ResourceQuota too (03-04).
3. The occupancy-reports CronJob was suffering silent throttling. Its limits.cpu of 1000m against a desired usage of 1100m meant the nightly report took 40 minutes instead of 18. Nobody complained because nobody was awake.

A word of warning about the process: recalibrate in rutas-norte-pre first, with representative synthetic load, and only then take the values to rutas-norte-pro. Lowering a requests.memory all at once in production can make the scheduler put more pods on a node than really fit, with evictions afterwards.

And do not do it by hand indefinitely. In 09-02 we will see the VPA (Vertical Pod Autoscaler), which performs exactly this analysis continuously and can recommend or apply the values automatically.

Although module 9 develops autoscaling, the dependency must be made clear here, because it is metrics-server's whole reason for being.

When in 09-01 we write a HorizontalPodAutoscaler like this one:

the HPA controller, which lives inside the kube-controller-manager, queries exactly the same API as kubectl top:

Every 15 seconds by default. With the result, it calculates the average CPU usage of the Deployment's pods, divides it by the sum of their requests to get the utilisation percentage, and decides how many replicas there should be.

Two consequences that must be burned into your memory:

Consequence 1: without metrics-server, the HPA does not work. It does not fail noisily: it sits in the unknown state and does not scale. The symptom:

That <unknown> is the signature of metrics-server being unavailable. And kubectl describe hpa confirms it:

During the May bank-holiday weekend, with the HPA "installed" but no metrics-server, bookings-api would have stayed at 4 replicas while traffic multiplied sixfold.

Consequence 2: the HPA percentage is calculated against the requests. An averageUtilization: 70 with requests.cpu: 100m means scaling when it reaches 70 millicores. If you recalibrate the request to 200m, the same HPA will scale when it reaches 140 millicores. Changing the requests changes the autoscaling behaviour. That is why the correct order is: calibrate with kubectl top first (this lesson), configure the HPA afterwards (09-01).

The VPA (09-02) uses the same source, but with a history of its own that it maintains itself, precisely because metrics-server does not have one.

It means the API Server cannot serve the metrics.k8s.io group. The diagnostic sequence:

0/1 with restarts: the pod starts but its own readiness fails. Off to the log:

This is the case described in section 4. The development fix:

Remember the warning: in rutas-norte-pro, the correct solution is kubelet server certificate rotation, not this patch.

This is not a failure. It is the expected behaviour during the first 30–60 seconds after starting metrics-server or after creating a new pod: two samples are needed to calculate the CPU rate. Wait a minute and try again. An awful lot of people reinstall metrics-server unnecessarily because they did not wait.

metrics-server is trying to reach the node by its hostname and DNS does not resolve it. The fix: force the use of the internal IP.

The usual causes, in order of frequency:

1. The pod has been alive for less than 30 seconds.
2. The pod is Pending: it has no running containers, so there is nothing to measure.
3. The node the pod lives on is unreachable for metrics-server (check the logs).
4. You do not have RBAC permissions on that namespace.

1. Expecting metrics-server to be a monitoring system. This is confusion number one. It keeps no history, it does not alert and it measures only two things. Installing metrics-server and considering observability solved leaves the platform almost as blind as before.

2. Confusing kubectl top with kubectl describe node. They are completely different data and they answer different questions:

These are the declared requests and limits, that is, what the scheduler has reserved. kubectl top shows real consumption. A node can be at 80 % of reservations and 20 % of real usage: that is precisely what we are correcting in section 8.

3. Recalibrating from a single reading. A kubectl top on a Tuesday at 11:00 tells you nothing about the May bank-holiday weekend. Measure for at least a week and use percentiles.

4. Lowering --metric-resolution below 10 s. It saturates the kubelets' Summary API, which in large clusters is an expensive operation. It does not improve data quality and it can degrade the node.

5. Leaving --kubelet-insecure-tls in production. I repeat it because it happens constantly: it gets added to unblock the installation and nobody ever removes it. In rutas-norte-pro it is a real risk, not a theoretical one.

6. Running more than one metrics-server replica without adjusting the configuration. For high availability you need to add --enable-aggregator-routing=true on the API Server and tune the anti-affinity. With one replica, the only consequence of it going down is losing kubectl top for a few seconds: acceptable in most cases.

7. Interpreting MEMORY(bytes) as the process RSS. It is workingSetBytes, which excludes reclaimable page cache. It will not match ps or free inside the container, and that is normal.

8. Using kubectl top to investigate a past incident. There is no past. If the incident is over, kubectl top shows you a healthy present and contributes nothing. That is literally the reason for the next lesson.

9. Forgetting that requests affect the HPA. Changing requests.cpu without reviewing the HPA's averageUtilization silently changes the scaling threshold.

10. Not checking the sidecars with --containers. Since 06-04 we have sidecars in several components. Adding them to the main container hides their cost and complicates sizing.

On your rutas-norte minikube:

1. Enable the metrics-server addon and verify that the APIService ends up AVAILABLE: True.
2. Obtain the metrics of bookings-api without using kubectl top, talking directly to the aggregated API.
3. Explain exactly where that number came from: which component read it, from where, and by what route it reached your terminal.

These are two weeks of data for notifications-worker in rutas-norte-pro:

Its current configuration:

1. What QoS class does this pod have now? (go back over 03-05)
2. Identify two serious problems in this configuration in the light of the data.
3. Propose a new configuration and justify every value.

During a traffic peak, bookings-api stays at 4 replicas despite having an HPA configured. A colleague passes you this information:

1. What is the causal relationship between the three symptoms?
2. Write the sequence of commands you would run to get to the root cause.
3. Assuming the log shows x509: cannot validate certificate ... because it doesn't contain any IP SANs, what would you do in rutas-norte-dev and what would you do in rutas-norte-pro?

Solution 1

1. Enabling and verifying.

2. Querying the aggregated API directly.

You can also query a specific pod:

3. The data's path, step by step.

1. The Linux kernel accounts for the usage of the api container's cgroup in cpuacct.usage and memory.current.
2. cAdvisor, built into the kubelet binary on the node where that pod lives, reads those cgroup files.
3. The kubelet exposes the aggregate at https://<node-ip>:10250/stats/summary.
4. metrics-server queries that endpoint every 15 seconds, keeps the last two samples in memory and calculates the CPU rate as (nanocores_t2 - nanocores_t1) / (t2 - t1).
5. metrics-server serves the result in the metrics.k8s.io/v1beta1 API group, exposed through a Service in kube-system.
6. The APIService v1beta1.metrics.k8s.io makes the API Server delegate to that Service any request to that group.
7. kubectl asks the API Server for /apis/metrics.k8s.io/v1beta1/..., which acts as a proxy, and returns the JSON.

The essential detail: as far as kubectl is concerned this is indistinguishable from a kubectl get pods. Same endpoint, same authentication, same RBAC.

Solution 2

1. QoS class.

For a pod to be Guaranteed, requests and limits must match on every resource of every one of its containers. Here memory does match (256Mi = 256Mi), but CPU does not (200m ≠ 250m). The QoS class is therefore Burstable: under node memory pressure, this pod is a candidate for eviction before any Guaranteed pod.

2. The two serious problems.

Problem A: severe CPU throttling. The limits.cpu is 250m, but the 95th percentile of desired usage is 380m and the 99th percentile reaches 520m. That means 5 % of the time the worker is throttled, and that 5 % is precisely when it sends the bursts of confirmation emails. The real business effect: purchase confirmation emails are delayed exactly when there are most purchases. On top of that, kubectl top would show consumption stuck at 250m without saying there is throttling: Prometheus would be needed to confirm it.

Problem B: risk of OOMKilled and an inflated CPU request. The limits.memory is 256Mi and the observed maximum is 291Mi. The limit has already been exceeded: that worker is being OOM-killed periodically, probably without anybody connecting it to the emails that never arrive. And in the other direction, the requests.cpu: 200m is well above the p50 of 43m: 200m are reserved of which 43m are used on average, wasting cluster.

3. Proposed configuration.

Justification in brief:

• requests.cpu drops from 200m to 100m → 100m per replica are freed in the namespace's ResourceQuota.
• limits.cpu rises from 250m to 600m → the throttling disappears during the bursts.
• requests.memory rises from 256Mi to 320Mi → above the observed maximum of 291Mi: the OOMKilleds stop.
• limits.memory at 384Mi → 20 % headroom that absorbs an unexpected peak but cuts off a memory leak before it affects the node.

A note on process: apply it in rutas-norte-pre first and confirm with kubectl top pods --containers over a week that consumption stays within the expected range. Also confirm that the restart counter (RESTARTS) stays at zero, which is the proof that the OOMKilleds have stopped.

Solution 3

1. Causal relationship.

The three symptoms are the same problem seen from three places, and the chain runs bottom-up:

An important detail: the HPA does not fail noisily. It generates no visible error event on the Deployment and does not leave the object in an obviously broken state. It simply does nothing. That is why in 07-04 we will configure a specific alert on the HPA's ScalingActive condition.

2. Diagnostic sequence.

3. Different action per environment.

In rutas-norte-dev — unblock quickly, acceptable risk:

In rutas-norte-pro — the flag is not acceptable. The correct solution is for the kubelet certificate to be signed by the cluster CA:

1. Enable server certificate rotation on each kubelet:

2. Restart the kubelet on each node (systemctl restart kubelet), which generates one CSR per node.

3. Approve the pending CSRs of type kubernetes.io/kubelet-serving:

In a real cluster this is automated with an approver (for example, kubelet-csr-approver), because the CSRs are renewed periodically.

4. Verify that metrics-server works without the insecure flag:

An essential preventive measure: the real problem is that this failure went unnoticed for 18 minutes during a traffic peak. In 07-04 we will create an alert that fires when the metrics APIService is unavailable or when an HPA has spent more than five minutes with ScalingActive=False. Nobody should find out about this because traffic collapses.

In this lesson we have installed Rutas Norte's first source of real consumption data and learned to read it with judgement:

• metrics-server is not a monitoring system but an infrastructure component whose purpose is to feed the HPA and the VPA. The fact that kubectl top works is a pleasant consequence.
• The data travels from the kernel cgroups → cAdvisor inside the kubelet → the Summary API on port 10250 → metrics-server every 15 seconds → the aggregation layer's APIService → your terminal. Understanding that path turns every error message into a diagnosis.
• kubectl top nodes gives percentages of the node's allocatable; kubectl top pods gives no percentages and the per-container breakdown with --containers has been essential ever since we got sidecars.
• Its most valuable use today has been settling the debt from 03-04: we have recalibrated the requests and limits of the six components with real data, discovering that bookings-api was undersized and suffering throttling, that web-store was reserving four times what it uses, and that the nightly CronJob was taking twice as long as necessary because of a limit set too low.
• And we have seen its hard limits: no history, no queries, no alerts, CPU and memory only, with a resolution of tens of seconds. For "what happened last night at three?" it is useless.

That last limitation is exactly the hook for the next lesson. Rutas Norte needs a system that remembers, that lets you ask and that knows how to warn. In 07-03 we will deploy Prometheus with the operator we announced in 06-07, we will finally connect the bookings-postgres metrics exporter sidecar we added in 06-04, we will instrument bookings-api with business metrics, and we will learn PromQL to answer questions we cannot even formulate today: how many bookings per minute are confirmed, what the 95th percentile of latency is and how long until the database disk fills up.