In the previous lesson we installed metrics-server and at last found out how much every Rutas Norte component consumes right now. We also discovered its hard limit: it remembers nothing. The question "what happened last night at three in the morning?" is still unanswered, and with it all the ones that really matter: how many bookings per minute are we confirming? has latency gone up since the last deployment? how long until the bookings-postgres disk fills up?

This lesson deploys Prometheus, the de facto standard for monitoring Kubernetes and the second CNCF graduated project after Kubernetes itself. We are going to understand its data model, where a cluster's metrics really come from (one of the biggest sources of confusion for newcomers), how to instrument bookings-api with business metrics, how to deploy the stack with the operator we announced in 06-07, and how to write PromQL that answers real Rutas Norte questions. By the end, the platform will have a memory.

1. What Prometheus adds compared with metrics-server
2. The pull model and the /metrics endpoints
3. The data model: series, labels and the four metric types
4. Where a cluster's metrics come from: the table that clears up the confusion
5. Instrumenting bookings-api with business metrics
6. Deploying the kube-prometheus-stack with Helm
7. The Prometheus Operator and its custom resources
8. The Rutas Norte ServiceMonitor and target diagnostics
9. PromQL from scratch and with purpose
10. The four golden signals applied to Rutas Norte
11. Storage, retention and the limits of Prometheus
12. Common mistakes and tips
13. Exercises

Three things, and all three are what we were missing.

History. Prometheus stores every sample in a time-series database on disk. With the stack's default retention we will have several days; by configuring storage, weeks or months. The three-in-the-morning question has an answer.

Queries. PromQL is a complete language for aggregating, filtering, deriving and comparing series. It is not "look at a number": it is "give me the 5xx error rate of bookings-api in the production environment, grouped by route, over the last hour".

Alerts. Prometheus evaluates expressions periodically and, when they hold for a sustained period, fires alerts that Alertmanager routes to whoever needs them. That is what makes somebody find out at three in the morning without staring at a screen.

And a fourth one, less obvious but decisive: anything can expose metrics. metrics-server only knows about CPU and memory. Prometheus collects whatever an HTTP endpoint offers it: requests per route, confirmed bookings, the email queue depth, PostgreSQL's active connections, certificates about to expire.

The two live side by side. We are not replacing metrics-server: we are complementing it.

Here is the design decision that shapes everything else.

Most traditional monitoring systems work by push: the application sends its metrics to a central server. Prometheus does the opposite: pull. Prometheus keeps a list of targets and, at a set interval (typically 30 seconds), issues a GET to each one.

Why is pulling better than receiving?

• Prometheus knows whether a target is down. If the GET fails, the synthetic metric up is 0. With push, the absence of data is ambiguous: is the application down or has it simply not sent anything?
• The monitoring system controls the pace, not the applications. A hundred badly written applications cannot flood it.
• Automatic discovery. Prometheus queries the Kubernetes API to find out which pods exist. A new pod shows up as a target automatically, with nothing configured in the application.
• Trivial debugging. You can curl the /metrics endpoint from your laptop and see exactly what Prometheus sees.

The exception is very short-lived processes, such as our occupancy-reports CronJob: it may finish before Prometheus scrapes it. For those cases there is the Pushgateway, an intermediary the job pushes its metrics to and that Prometheus scrapes afterwards. It is the only legitimate exception and it should be used sparingly.

It is plain text. Each line is name{labels} value. The # HELP and # TYPE lines are metadata: description and type. That deliberately simple format is the entire contract between an application and Prometheus.

A time series in Prometheus is uniquely identified by its metric name plus the exact set of its labels. This is fundamental:

Changing the value of a single label creates a completely new series. Each series takes up memory and disk, and that is where the most frequently cited danger of Prometheus comes from: cardinality.

The golden rule of cardinality: never use as a label a value with many possible values. No user, booking or session identifiers, no client IP addresses, no timestamps and no full error messages.

A real example of the danger. If bookings-api labelled every request with the customer's ID number:

With 200,000 customers and 15 routes, you would have 3 million series from that metric alone. Prometheus would run out of memory within minutes. And, on top of that, you would be putting personal data into the monitoring system, with the implications we will see in 07-05.

Labels Prometheus adds by itself to everything it collects in Kubernetes (and that we will use constantly):

counter. A value that only grows: requests served, errors, bytes sent, bookings confirmed. It is never queried directly. The absolute value 184203 says nothing: is that a lot? since when? It is always queried with rate() or increase(), which calculate how much it has grown per unit of time. By convention, its name ends in _total.

Prometheus automatically detects counter resets (when a pod is recreated, the counter goes back to 0) and rate() compensates for them. You do not have to worry about it.

gauge. A value that goes up and down: active connections, temperature, memory in use, the size of a queue. It is queried directly. It is the type of almost everything cAdvisor exposes about memory.

histogram. The most powerful type and the worst understood. Instead of storing each individual value, the application classifies each observation into predefined cumulative buckets. A histogram metric actually exposes three things:

Reading it: 8140 requests took less than 50 ms; 9210 took less than 100 ms (including the 8140 above: they are cumulative); 9821 in total, so 9 requests took more than 1 second.

With that data, the histogram_quantile() function can estimate any percentile. The huge advantage over summary: because the buckets are ordinary counters, they can be summed across pods. You can calculate the 95th percentile of latency for the whole of bookings-api by aggregating its six replicas. With a summary that is mathematically impossible.

The cost: each bucket is a series. A histogram with 10 buckets and 15 routes is 150 series per pod. Choose the buckets with judgement, matched to your service's real latency.

summary. The application computes the percentiles itself and exposes them. Cheap to query, but not aggregatable across instances: the 95th percentile of pod A and that of pod B cannot be combined to get the service's. In Kubernetes, where everything has several replicas, this rules it out almost always. Use it only if you need an exact percentile for a single instance.

Prometheus has conventions worth respecting because tools and dashboards assume them:

• Prefix with the name of the system: rutasnorte_.
• The base unit in the name and always in SI base units: _seconds not _ms, _bytes not _mb.
• Counters end in _total.
• Names in snake_case, no capitals.

This is the section that resolves the doubt everybody has in their first week: "why do I need four different things to monitor Kubernetes?".

The short answer: because they are four different layers of reality and none of them can see the others.

Let us go through them one by one, because the distinction matters enormously when writing queries.

It is the same source that feeds metrics-server (07-02), but Prometheus reads it directly and in far more detail. Key metrics:

Look at the third one: the throttled periods counter that in 07-02 we said kubectl top could not give us. Here it is, and it is the one that will confirm the bookings-api throttling diagnosis.

A practical warning: these metrics also appear with container="" (the pod-level aggregate, including the infrastructure pause container). You will almost always want to filter with container!="" so as not to count twice.

It is a DaemonSet (one pod per node, as we studied in 06-02) that reads the host system's /proc and /sys. It sees things no container can see:

It is the one that tells you the node's disk is filling up, that the load average is through the roof or that the network card is saturated. No container can tell you that about itself.

This is the hardest one to grasp and the most necessary. It measures nobody's consumption. It connects to the Kubernetes API and turns the state of the objects into metrics:

With these metrics you can alert on things none of the other sources know about: a Deployment with fewer available replicas than desired, a pod restarting in a loop, a Job that has failed, a PVC that has been Pending for twenty minutes.

And something very useful: kube_pod_container_resource_requests exposes the declared requests. Cross-referencing it with cAdvisor's container_memory_working_set_bytes gives you, in a single PromQL query, exactly the recalibration analysis that in 07-02 we had to do with a bash script.

PostgreSQL does not speak the Prometheus format. An exporter is a process that connects to the software, queries its internal statistics and translates them into /metrics.

And here we connect back to 06-04: we already have it deployed. When we added the metrics exporter sidecar to the bookings-postgres pod, we said explicitly that Prometheus would consume it in this lesson. The moment has arrived.

What it exposes, and that we are going to need:

There are exporters for practically everything: redis_exporter for redis-cache, nginx-prometheus-exporter for web-store, blackbox_exporter to check from the outside that https://www.rutasnorte.example responds.

None of the four previous sources can tell you how many tickets you have sold. Only your code knows that. It is the next section.

Your own instrumentation is what separates an infrastructure dashboard from one the business cares about. "The pod is using 300 Mi of RAM" is of no interest to the Rutas Norte director. "We are confirming 12 bookings a minute, 40 % fewer than yesterday at this time" is.

bookings-api is in Node.js, so we use the official client library prom-client. There are equivalent libraries for Java, Go, Python, .NET and practically any language.

And the middleware that wires it into Express:

Two decisions worth highlighting:

1. Normalised route, not the real URL. req.route.path gives /api/bookings/:id, so every booking shares a series. Using req.originalUrl would give /api/bookings/48213, /api/bookings/48214... one series per booking. It is the most frequent cardinality mistake in the world.
2. /metrics on the same port as the application, or on a separate one. Here we leave it on 8080 for simplicity. In production it is preferable to expose it on a different port (9090, say) that is not published on the Ingress, so that the metrics are not reachable from the internet. The rutas-norte-pro NetworkPolicies (04-06) will have to allow traffic from the monitoring namespace to that port: remember that every new conversation needs its policy.

Installing Prometheus by hand in Kubernetes means managing about twenty objects across Deployments, ConfigMaps, Services, RBAC and storage. The community has packaged all that into a Helm chart called kube-prometheus-stack.

Note: Helm is the Kubernetes package manager and we will study it thoroughly in 10-03. Here we use it as an installation tool; for now it is enough to understand that a chart is a parameterisable template of manifests and that helm install generates and applies them.

A values file adapted to Rutas Norte:

What we have just installed:

Access to the Prometheus interface:

Here is where what we announced in 06-07 materialises. The Prometheus Operator is the canonical example of the operator pattern: a controller that watches custom resources and reconciles the real state until it matches.

Without an operator, adding a new target to Prometheus means editing a prometheus.yml file, putting it into a ConfigMap, reloading the configuration and waiting. A manual, centralised and error-prone process.

With an operator, the team that deploys bookings-api creates a ServiceMonitor object alongside their own application, in their own namespace, and the operator detects it and regenerates the Prometheus configuration automatically. Monitoring becomes a decentralised responsibility, versioned in Git alongside the rest of the manifests.

And AlertmanagerConfig, to configure alert routing per namespace. Also in 07-04.

As we saw in 06-06, all of these are CRDs. And like cert-manager's Certificate or VolumeSnapshot, they are handled with kubectl exactly like a Deployment.

An essential requirement: the ServiceMonitor references the port by name, so the Service must have it named.

Since bookings-postgres is a StatefulSet with a headless Service (06-01), we need an additional Service specifically for metrics. A headless Service (clusterIP: None) works just as well as the source of a ServiceMonitor, because the operator uses the Endpoints, not the ClusterIP.

With this, the sidecar we added in 06-04 starts feeding Prometheus. The promise is kept.

Our convention is clear: in rutas-norte-pro there is a deny-all and each conversation is authorised one at a time. Prometheus lives in monitoring and wants to talk to the metrics ports in rutas-norte-pro. Without a policy, every target will show up as down.

In the Prometheus interface, Status → Targets. Or from the command line:

This is the question of this lesson. A checklist in order of frequency:

Point 3 deserves detail because it is treacherous. The Prometheus resource has a serviceMonitorSelector. If the chart left it as release: monitoring, it will only collect ServiceMonitors with that label, and yours will be ignored silently: no event, no error, it simply does not show up.

Two fixes: set serviceMonitorSelectorNilUsesHelmValues: false in the values (which is what we did), or add the label release: monitoring to all your ServiceMonitors.

Checking directly whether the target is being scraped:

An empty array means the target is not configured (problems 1–4). A result with value: ["...", "0"] means it is configured but does not respond (problems 5–7).

PromQL is frightening at first and is simpler than it looks if you learn it in layers.

The simplest query is a metric name:

It returns every series with that name: one for each label combination on each pod. To filter, you use braces:

The four label comparison operators:

They can be combined, with an implicit AND:

Reading it: requests to /api/bookings in production that returned a 5xx code.

By adding [5m] you get, instead of one value per series, every value from the last 5 minutes:

This is called a range vector and cannot be plotted directly: it is raw material for the functions of the next level.

rate() — the most important function in PromQL.

It calculates the increment per second of a counter within a window. It is what turns a useless counter (184203) into a meaningful figure (12.4 requests per second).

Three things you need to know about rate():

• It only works with counters. Applying it to a gauge gives meaningless results.
• It automatically compensates for counter resets when a pod is recreated.
• The window must contain at least 4 samples. With interval: 30s, the minimum reasonable window is [2m]; [5m] is the usual choice. With [1m] and 30 s scrapes you would have two samples and erratic results.

increase() — the same as rate() but giving the total increment over the window instead of per second. It is literally rate() * window_seconds. More readable for human questions:

sum by and sum without — aggregation.

rate() returns one series per pod and per label combination. You almost never want that: you want the service total.

sum by (a, b) keeps only the a and b labels. sum without (pod, instance) keeps all but those. The second form tends to be more robust against change.

Other aggregations: avg, min, max, count, stddev, topk, bottomk.

topk() — the top N. Perfect for "who is eating the cluster?":

histogram_quantile() — percentiles from histograms.

Breaking it down, from the inside out:

1. rate(..._bucket[5m]) → the rate of increase of each bucket.
2. sum by (le, route) → sums the buckets of all the pods, keeping le (the bucket boundary) and route. The le label is mandatory: without it, histogram_quantile cannot work. It is the commonest error with this function.
3. histogram_quantile(0.95, ...) → estimates the 95th percentile.

The result is in seconds: 0.412 means that 95 % of requests are resolved in under 412 ms.

Accuracy: the result is an estimate by linear interpolation within the bucket. If your buckets are [0.1, 0.5, 1] and the real p95 is at 0.42 s, the estimate can be quite far off. That is why we chose buckets matched to our real latency in section 5.

Arithmetic operators and comparison between metrics.

Prometheus matches series on both sides by their common labels. If the labels do not match, the result is empty: it is the cause of half the queries that "return nothing". on() and ignoring() let you control the matching.

Requests per second by route:

Error rate (the proportion of 5xx over the total):

It returns a number between 0 and 1. Multiplied by 100 it is the percentage. A warning: if the denominator is 0 (no traffic), the result is NaN and disappears from the graph. That is correct behaviour: with no traffic there is no error rate to measure.

95th percentile of latency by route:

Bookings confirmed per minute (the business metric):

CPU saturation against the configured limit:

A value of 0.95 means the pod is using 95 % of its limit: an immediate candidate for throttling.

Confirming the throttling we could not see in 07-02:

It is the fraction of scheduling periods in which the container was throttled. Above 0.25 (25 %) there is a real performance problem. This is exactly the query that would have diagnosed the bookings-api problem in 07-02 in ten seconds.

Comparing real consumption with the requests: the 07-02 recalibration, now automated:

With Prometheus, this query over 30 days of history completely replaces the bash script from 07-02, and it also lets you ask for the real percentile using quantile_over_time.

Unavailable replicas (uses kube-state-metrics):

A target that is down:

Pods restarting in a loop:

PostgreSQL connections against the maximum (from the 06-04 sidecar):

External payment gateway failures:

This query lets you tell "our platform is failing" from "the external provider is failing", which is a very valuable distinction at three in the morning.

The four golden signals, popularised by Google's SRE book, are the framework that avoids the mistake of monitoring everything and understanding nothing.

One essential subtlety about latency: the latency of failing requests must be measured separately. A 500 error returned in 3 ms artificially lowers the average latency and can make a service that is failing massively look lightning fast.

The specific queries for bookings-api, which will be the basis of the 07-04 dashboard:

That last query closes a circle in the module: it is exactly the phenomenon that in 07-01 left bookings-api alive but useless. Back then we only had a readiness probe that took the pod out of rotation; now we also have the figure recorded and queryable after the fact.

Queries such as the p95 are expensive: they traverse many series. If a dashboard runs one every 15 seconds with 8 panels open, Prometheus suffers.

Recording rules precompute a query periodically and store the result as a new metric:

Now the dashboards and the alerts query bookingsapi:latency_p95:5m, which is a single already-computed series. In 07-04 we will use them heavily.

Prometheus writes to its own time-series database (TSDB):

1. New samples go into an in-memory block and into a write-ahead log (WAL) on disk, which protects against crashes.
2. Every 2 hours, that block is persisted as an immutable directory on disk.
3. A compaction process merges small blocks into larger ones.
4. Blocks beyond the retention are deleted whole.

An important consequence: retention does not delete individual samples, it deletes complete blocks. With retention: 30d you may occasionally have a little more than 30 days.

The accepted rule of thumb: between 1 and 2 bytes per sample after compression.

With a 50 GiB PVC we have plenty, even with room for compaction (which needs temporary space). That is also why we set retentionSize: 45GiB: it is the emergency brake if cardinality grows more than expected.

Seeing the real cardinality:

This second query is the first one to run when Prometheus starts consuming too much memory.

Three design limitations, all deliberate:

1. Local storage, not distributed. The data lives on a pod's disk. There is no replication between instances.
2. It does not scale horizontally by itself. Two Prometheus instances do not share data: each has its own.
3. Long retention is expensive. Keeping two years locally means hundreds of GB on a fast disk and considerable memory consumption when querying.

For long retention and a multi-cluster view there are two projects that extend Prometheus:

Both allow years of cheap retention and querying several clusters at once. For Rutas Norte with a single cluster, 30 days is enough today; when module 11-05 tackles multi-cluster management, Thanos will be the natural evolution.

Configuring remote_write from the operator is straightforward:

1. Cardinality explosion. The most serious and most expensive mistake. Labelling with booking identifiers, ID numbers, client IPs or unnormalised URLs multiplies the series by thousands and brings Prometheus down for lack of memory. Before adding a label, ask yourself how many distinct values it can have. If the answer is "I don't know", do not add it.

2. Querying a counter without rate(). Plain rutasnorte_requests_total returns a value accumulated since the pod started. It is a graph that only goes up and means nothing. Always rate() or increase().

3. A rate() window that is too short. With interval: 30s, a rate(...[1m]) has two samples and produces noise or gaps. Rule: the window must be at least 4 times the scrape interval.

4. Forgetting le in the sum by before histogram_quantile. If you aggregate without keeping the le label, the function cannot reconstruct the histogram and returns NaN or nothing. It is failure number one with percentiles.

5. The ServiceMonitor selecting pod labels instead of Service labels. A ServiceMonitor's selector is compared with the labels of the Service object. Because in Rutas Norte the Services carry the same labels as the pods, it is easy not to notice until a Service has different labels.

6. The operator's release label. If the Prometheus resource has serviceMonitorSelector: {matchLabels: {release: monitoring}}, your ServiceMonitors without that label are ignored with no error message at all. Checking it is the first thing to do when a target does not show up.

7. Prometheus without persistent storage. Without storageSpec, the chart uses an emptyDir and every pod restart wipes the whole history. It is exactly the problem we came here to solve.

8. No retentionSize. With only retention: 90d, if cardinality grows the disk fills up, Prometheus goes into CrashLoopBackOff and you lose your monitoring exactly when you need it most. Always set a size limit, at 80–85 % of the PVC.

9. Alerting on summary metrics. They are not aggregatable across pods. If your alert uses the p99 of a summary with six replicas, the number means nothing useful.

10. Monitoring the node with container metrics. cAdvisor cannot see the node's filesystem or its load average. That is what node-exporter is for. Confusing them leads to alerts that never fire.

11. Forgetting the NetworkPolicies. In rutas-norte-pro there is a deny-all. If Prometheus cannot reach the metrics ports, every target shows up as down and the error (context deadline exceeded) is not remotely obvious.

12. Instrumenting too late. Business instrumentation is added in the code, so it requires a development cycle. The sooner it is built into "ready for production", the better. Improvising it during an incident is impossible.

The team has deployed redis-cache with an exporter and has created these manifests. Nothing shows up on the Prometheus targets page.

1. Identify three distinct errors.
2. Write the corrected manifests.
3. Write the PromQL or curl command that confirms the problem is solved.

On the Saturday of the bank-holiday weekend, between 12:00 and 13:30, customers complained that the website was extremely slow and that some purchases failed. It is over, nobody took notes, and the director wants explanations on Monday.

Write the PromQL queries that answer each question. State which source (cAdvisor, kube-state-metrics, node-exporter, an exporter or your own instrumentation) each metric comes from.

1. How many requests per second was bookings-api serving in that interval, compared with the previous Saturday?
2. What was the 99th percentile of /api/bookings latency during the incident?
3. Was bookings-api suffering CPU throttling?
4. Did bookings-postgres run out of available connections?
5. Did the external payment gateway fail, or was the failure ours?
6. How many bookings were not confirmed compared with what was expected?

notifications-worker consumes from a Redis queue and sends confirmation emails. At present it exposes no metrics. Design its instrumentation:

1. List the metrics it should expose, with their full name, their type (counter, gauge, histogram) and their labels, justifying the cardinality of each label.
2. Write the corresponding ServiceMonitor (the worker has no Service because it receives no traffic: solve that problem).
3. Write the PromQL queries for the four golden signals for this component.

Solution 1

1. The three errors.

Error A — The ServiceMonitor selector does not match the Service labels. The ServiceMonitor looks for Services with app: redis-cache and environment: pro, but the Service has app.kubernetes.io/name: redis-cache and app.kubernetes.io/part-of: rutas-norte. Not a single one matches. The Service's spec.selector (which does use app/environment) selects pods; it is not what the ServiceMonitor looks at.

Error B — The Service port has no name, and the endpoint references metrics. The ServiceMonitor looks for a port called metrics in the Service; the Service defines port 9121 with no name field. The operator cannot find the port and discards the endpoint.

Error C — The ServiceMonitor is in monitoring with no namespaceSelector. By default, a ServiceMonitor looks for Services in its own namespace. With it in monitoring and the Service in rutas-norte-pro, they will never find each other. Two valid fixes: move the ServiceMonitor to the application's namespace (preferable, it keeps monitoring next to the code) or add a namespaceSelector.

2. Corrected manifests.

And the NetworkPolicy, because in rutas-norte-pro there is a deny-all:

3. Verification.

"1" is the confirmation. If it gave "0", the target is configured but does not respond: check the NetworkPolicy and that the exporter is listening on 9121.

An additional check that the operator has picked it up:

Solution 2

A general note: every query uses the offset modifier or an explicit time range, since the incident is over. Here is the decisive advantage over kubectl top from 07-02: this data exists.

1. Traffic during the incident, compared with the previous Saturday.

Source: your own instrumentation in bookings-api.

A result of 5.8 means that on the bank-holiday Saturday there was almost six times more traffic than on a normal Saturday: consistent with what we know about bank-holiday peaks.

2. 99th percentile of /api/bookings latency.

Source: your own instrumentation (histogram).

Compare it with the p95 and with the mean (_sum / _count) to see whether the problem affected every request or only the tail:

If the mean is fine and the p99 is through the roof, the problem affects a subset (for example, the requests that touch the payment gateway). If both go up, it is systemic.

3. CPU throttling.

Source: cAdvisor / kubelet.

A value sustained above 0.30 during the incident confirms that the CPU limit was holding the application back. A complementary one, to see whether it was pinned against the ceiling:

(The second part comes from kube-state-metrics: it is the perfect example of why both sources are needed.)

4. PostgreSQL connections.

Source: the exporter sidecar from 06-04, plus the pool's own metrics.

A numbackends / max_connections close to 1 confirms that the database was saturated with connections.

5. Did the external gateway fail or did we?

Source: your own instrumentation.

Interpretation: if the gateway failure rate goes up to 40 % while our general error_ratio also rises, the failure is the provider's and they should be held to account. If the gateway is at 0.1 % failures and we are returning 5xx, the problem is ours. That distinction, impossible without instrumentation, is what you take to Monday's meeting.

6. Bookings that were not confirmed.

Source: your own instrumentation (business metric).

And a particularly revealing cross-check: if traffic multiplied by 5.8 but confirmed bookings only by 1.3, the difference between those two factors is a direct estimate of the business lost. That number, in euros, is the one that justifies the infrastructure budget.

Solution 3

1. The component's metrics.

Total cardinality: about 43 series per pod. Perfectly manageable.

Labels deliberately dropped, and why:

• recipient (the customer's email address): unbounded cardinality and, above all, personal data in the monitoring system. Forbidden. We will come back to this in 07-05.
• booking_id: unbounded cardinality. That data belongs in the logs, not in the metrics. Metrics aggregate; logs give detail.
• The full error_message: unpredictable cardinality. It is replaced by reason, a closed set of categories.

2. A ServiceMonitor for a component with no Service.

notifications-worker receives no traffic, so it has no Service. Two valid options:

Option A (recommended): PodMonitor. It is exactly the resource designed for this case.

With the port declared in the Deployment:

Option B: create a headless Service just for metrics and use a normal ServiceMonitor. It works, but it creates an object that adds nothing else. The PodMonitor is cleaner.

And the corresponding NetworkPolicy:

3. The four golden signals.

An additional coverage query, very valuable for the business: it relates two different components to detect lost emails.

This kind of query — one that cross-references metrics from two components to verify a business invariant — is among the most useful you can write, and in 07-04 we will turn it into an alert.

Rutas Norte has gone from remembering nothing to having memory, a language and the ability to ask. In this lesson we have:

• Understood the pull model and why it is superior to push in an environment as dynamic as Kubernetes.
• Learned the data model: series identified by name and labels, the permanent threat of cardinality, and the four metric types, with histogram as the only one that allows percentiles to be aggregated across replicas.
• Cleared up the classic confusion about where the metrics come from: cAdvisor measures the containers' real consumption, node-exporter the node's operating system, kube-state-metrics the state of the API objects, exporters translate third-party software — including the bookings-postgres sidecar we added in 06-04, now finally connected — and your own instrumentation is the only thing that knows about the business.
• Instrumented bookings-api with requests by route and code, latency as a histogram, confirmed bookings and the state of the connection pool, minding cardinality in every decision.
• Deployed the kube-prometheus-stack and worked with the Prometheus Operator we announced in 06-07, creating ServiceMonitors that live alongside each application and knowing how to diagnose why a target does not show up.
• Written real PromQL: rate, increase, sum by, topk, histogram_quantile, and the specific queries for each component's four golden signals. Among them, the one that confirms the throttling we could only suspect in 07-02.
• Sized the storage and retention, and understood why Prometheus is not an eternal archive and when it will be time to look towards Thanos or Mimir.

But we still have a practical problem: all this lives in a spartan web interface where you have to type queries by hand, and nobody is looking at it at three in the morning. Having the data is worth nothing if nobody sees it and nobody gets the warning.

In 07-04 we will take that leap: with Grafana we will build the Rutas Norte dashboard, panel by panel, with variables that work for all three environments; we will write the platform's alert catalogue with PrometheusRule — including the one that predicts when the bookings-postgres disk will fill up before it happens; and we will configure Alertmanager so that the right alert reaches the right person, grouped, without noise and with a link to the procedure to follow.