The previous lesson ended with a problem that cannot be postponed: all of CicloUrbana's metrics live in memory and die with the process. Every automatic deployment from 08-05 wipes them; /actuator/metrics returns the total as of right now and has no history; there is no graph, no comparison with last week and, above all, not a single alert. The SLOs we defined with RED and USE are written down and nobody is watching them.

This lesson closes that gap with the two tools that have become the de facto standard: Prometheus, which collects and stores time series, and Grafana, which draws them. We will look at why Prometheus goes and fetches the data instead of waiting for it, how to expose Micrometer's metrics in its format, how to read that text format —including the histogram buckets we configured in 09-03—, real PromQL up to the point of computing the true p95 of the Ribalta network by aggregating all three instances, CicloUrbana's dashboard panel by panel and, the thing that really changes the team's life, the alerting rules that turn a number into a phone call when —and only when— it is needed.

1. Why an external system is needed
2. The pull model versus push
3. CicloUrbana's monitoring architecture
4. Exposing /actuator/prometheus
5. Reading the text format
6. Prometheus's data model
7. prometheus.yml for CicloUrbana
8. Target discovery in Kubernetes
9. Retention and storage
10. PromQL: what you need to know
11. CicloUrbana's queries, one by one
12. Bringing the stack up with docker-compose
13. Grafana: data source and imported dashboards
14. CicloUrbana's dashboard, panel by panel
15. Alerting rules
16. Alertmanager: routes, grouping and silences
17. Good alerting practice
18. Managed alternatives
19. Common Mistakes and Tips
20. Exercises

The MeterRegistry from 09-03 is a set of in-memory counters. That implies four shortcomings no Actuator configuration can resolve:

An external monitoring system solves all four: it collects periodically, stores with retention, aggregates across instances and evaluates rules to alert. The application only has to expose its numbers in a format that system understands, and with Micrometer that costs one dependency.

Prometheus goes and fetches the data: every 15 seconds it makes an HTTP request to each instance and stores what comes back. It is the design decision that most distinguishes it from systems like Datadog or StatsD, where it is the application that pushes its metrics to a collector.

The decisive advantage in practice is the third: with pull, Prometheus automatically generates an up metric per target, so "the application is down" is a query, not an inference. And for the case the model does not cover —ephemeral tasks such as the nightly import from 07-03, which may finish between two scrapes— there is the Pushgateway, an intermediary where the process deposits its metrics and from which Prometheus pulls afterwards. It is the exception, and it should be treated as such: using it for normal services defeats outage detection.

Five pieces and one responsibility each. The application only exposes a text endpoint. Prometheus collects, stores and evaluates the rules: note that alerts fire in Prometheus, not in Grafana. Alertmanager receives the fired alerts and decides who to notify, grouping, silencing and avoiding repetition. Grafana only queries and draws: it stores nothing. And the exporters —such as postgres_exporter— publish, in the same format, the metrics of systems that do not speak Prometheus, which is of interest because 09-01 made it clear that a good part of CicloUrbana's problems are in the database.

With that dependency alone, Micrometer registers a PrometheusMeterRegistry and Actuator adds the endpoint. It still has to be exposed:

Security: the EndpointRequest.toAnyEndpoint() chain from 07-01 already covers this endpoint and requires ADMIN, so Prometheus will need credentials (section 7). That is as it should be: /actuator/prometheus reveals endpoint names, versions and Ribalta's business volume. And it still lives on port 8081, which never leaves the internal network.

How it is read, element by element. # HELP is the description, which comes literally from the .description(...) we wrote in CicloUrbanaMetrics. # TYPE declares the type —counter, gauge, histogram, summary— and from it Grafana and PromQL know which operations make sense. Each following line is one time series: the name, braces with the labels and the current value; the name + labels combination is its identity, and here you can see why the cardinality of 09-03 matters so much —each line is a series Prometheus will index and store for ever—.

The suffixes deserve special attention, because they are the translation of what we configured in 09-03. A counter becomes _total by Prometheus convention. A histogram produces three families: _count (how many observations), _sum (the sum of all of them, in seconds) and _bucket with the le label (less or equal), one series per threshold. Notice that the buckets are cumulative: le="0.3" is 11,903 and it includes the 8,241 from le="0.1". The last one, le="+Inf", always matches _count. And those thresholds of 0.1 / 0.3 / 0.8 come exactly from the slo: we configured in 09-03.

With _count and _sum you get the average (_sum / _count = 1284.31 / 12801 = 100 ms), and with the buckets you get the real percentile: that is section 10.

A sample is a float64 value with a timestamp, and it belongs to a time series identified by a metric name and a set of labels. Nothing else: there are no tables, no schemas, no complex types.

The four types: counter (monotonically increasing; queried with rate, never by its absolute value), gauge (goes up and down; its value does mean something), histogram (cumulative buckets, aggregatable) and summary (pre-computed percentiles; what percentiles from 09-03 produces, not aggregatable).

Prometheus also adds some labels of its own to each sample: job (the name of the target group), instance (host:port) and any defined in the configuration. That is why, even though all three replicas publish the same counter, their series are distinguishable.

The decisions that matter. metrics_path is mandatory because Prometheus looks for /metrics by default and Actuator publishes at /actuator/prometheus: it is the number one configuration mistake. The 15-second interval is a compromise —more frequent gives more resolution and more storage; below 10 s it rarely pays off, and above 60 s you lose short spikes—. scrape_timeout lower than scrape_interval, or the scrapes overlap. Authentication is the price of having secured Actuator, with the password in a file and not in the YAML. And port 8081, the management one: Prometheus never touches the port carrying citizen traffic.

The static list stops working as soon as the 08-04 HPA scales from three to seven replicas: the new pods would not appear. The solution is dynamic discovery:

And in the 08-04 Deployment, the pod template is annotated:

The mechanism is elegant: Prometheus asks the Kubernetes API which pods exist and the relabel_configs filters and transforms that list. Every new pod born with those annotations starts being scraped on the next cycle, and every pod that dies stops being scraped, with no configuration change. That is the real reason Prometheus fits Kubernetes so well.

Prometheus stores the data in its own time-series database, on local disk, in compressed two-hour blocks. The startup parameters:

A useful estimate: each sample takes roughly 1-2 bytes once compressed. With 15,000 series and a scrape every 15 seconds, that comes to some 86 million samples a day, that is, of the order of 100-170 MB daily and around 3-5 GB a month. The operational conclusion: consumption depends on the number of series, not on the traffic, which takes us back once again to the cardinality of 09-03 —doubling the labels doubles the disk and the memory, even if the number of requests does not change—.

Thirty days is enough to operate on. For annual reports to the council you need long-term storage —Thanos, Cortex or Mimir—, which sit behind Prometheus and keep the blocks in object storage. And an important warning: the Prometheus volume must be persistent (a PersistentVolumeClaim in Kubernetes); if it lives on the container's filesystem, every restart wipes the history and we are back to the problem in section 1.

Selectors and matchers. The basic form is the metric name with a label filter in braces:

The four matchers are =, !=, =~ and !~; the last two are regular expressions anchored to the whole string, so status=~"5.." captures exactly the 5xxs.

rate(): the most important function. A counter only grows, so its absolute value says nothing useful —"we have served 18,422 rentals since the process started" cannot be compared with another instance or with yesterday—. What matters is the rate at which it grows:

rate computes the increase per second within the window and, crucially, handles restarts: if the counter resets to zero because the application restarted, rate detects it and does not produce an absurd negative value. It is precisely what makes the monotonic-counter model usable.

irate() uses only the last two points: it reacts very fast and is very noisy. The practical rule: rate for graphs and alerts, irate only to debug a spike live. And a condition people forget: the window must contain at least four samples, so with a 15 s scrape, never less than [1m].

Aggregation. The operators (sum, avg, max, min, count, topk) collapse series:

sum by (...) keeps only the named labels; sum without (...) keeps all but those. The second is often more practical: without (instance) aggregates the replicas while keeping the rest of the context.

histogram_quantile(): the real p95. This is where the 09-03 decision to use histograms pays off:

It is read from the inside out: rate over the buckets gives the rate of observations per bucket; sum by (le, uri) adds the three instances while keeping the threshold and the endpoint; histogram_quantile interpolates the percentile. The le in the by is mandatory: without it the histogram's structure is destroyed and the result means nothing. And this is the aggregated p95 that was impossible with percentiles computed in the JVM.

Other useful functions. increase(x[1h]) gives the total increase over the window —increase is rate × seconds, and it serves for "how many rentals in the last hour"—. avg_over_time(x[10m]) averages a gauge over time. changes(x[1h]) counts value changes, useful for detecting restarts. And absent(x) is 1 when the series does not exist, which is how you alert that something stopped being published.

Operations between series. Two queries can be divided if their labels match, which is how a ratio is computed:

With sum(...) on both sides the result is a scalar and there is no matching problem; without aggregating, you have to use on(...) or ignoring(...) to say which labels the series match on.

Five of them deserve comment because they contain an idea:

The error rate is expressed as a ratio, not as an absolute number: 20 errors a minute is a catastrophe with 100 requests and noise with 100,000. A practical detail: in the numerator it is better to sum outcome="SERVER_ERROR" and not status=~"5..", because outcome already distinguishes our own errors from the client's 4xxs —a 409 for attempting a second rental is the business rule working, not a failure—.

The p95 per endpoint is the query that justifies the whole module: it aggregates the three instances, keeps the per-endpoint breakdown and gives the number that was promised as an SLO in 09-03.

Rentals per minute and fare are the pulse of the business, and the * 60 is only so they can be read in human units: rate always returns per second.

Pool usage is a utilisation (USE), and it is enough to compare it with 0.8. hikaricp_connections_pending is the saturation, and its threshold is zero.

The cache hit rate closes 09-02: the by (cache) rule in numerator and denominator makes the division match series with series and give one value per cache.

Alongside the 07-04 docker-compose.yml, an observability file:

It is started alongside the main one with docker compose -f docker-compose.yml -f docker-compose.observability.yml up -d. Four details: the named volumes preserve the history and the dashboards between restarts; --web.enable-lifecycle allows rules to be reloaded without a restart; the Grafana password is mandatory —leaving admin/admin exposed is a classic—; and the postgres_exporter brings in the database metrics that in 09-01 had to be queried by hand with pg_stat_statements.

Grafana is configured through the interface, but the configuration that survives is the one provisioned from a file, versioned in the repository:

Import before you build. Before drawing a single panel, it is worth importing two community dashboards, giving their identifier in Dashboards → Import:

The honest comparison: importing gives 80 % of the value in five minutes, covers everything generic —JVM, GC, pool, HTTP— and is far better built than anything you would do by hand. What it cannot give you is the Ribalta-specific part: rentals by fare, bikes per station, cache hits. The sensible strategy is to import the generic and build a single dashboard of your own for the business side, instead of reinventing the JVM memory graph.

One warning: the imported dashboards assume specific label names, almost always application and instance. If your common tags from 09-03 do not include application, the panels will come out empty and it will look as though something is broken.

A good dashboard answers the question "is the service healthy?" in five seconds and allows you to drill down afterwards. It is organised in rows, from what the citizen sees to what the machine consumes:

Row 1 — Service state (RED). Four panels: Rate (sum(rate(http_server_requests_seconds_count[5m])), a line graph), Errors (the ratio from section 11, as a stat with colour thresholds: green < 0.5 %, amber < 1 %, red above), p50/p95/p99 latency (three queries overlaid on one graph, with the 800 ms SLO drawn as a threshold line) and Availability (avg(up{job="ciclourbana"}), as a stat).

Row 2 — Per endpoint. A table with topk(10, sum by (uri) (rate(...))) and a graph of p95 by uri. This row answers "which endpoint is to blame?" immediately after row 1 says something is wrong.

Row 3 — JVM and resources (USE). Heap memory after GC, rate(jvm_gc_pause_seconds_sum[5m]), live threads, hikaricp_connections_active against max on the same graph, and hikaricp_connections_pending as a stat with a threshold at 1. This is the causes row.

Row 4 — Business. Rentals per minute and fare (stacked bars), available bikes per station (a heatmap or a table sorted ascending, so the empty stations show first), median journey duration and hit rate per cache. This is the row the council understands without translation, and the one that detects a deployment that broke renting.

Template variables. Instead of duplicating the dashboard per environment, you define variables: environment as a query label_values(up, environment), and instance as label_values(up{environment="$environment"}, instance) with the All option enabled. Then every query uses {environment="$environment", instance=~"$instance"}. A single dashboard serves pre and prod, and lets you isolate a suspect replica with a dropdown.

Good visualisation practice, which separates a useful dashboard from a decorative one: always units (seconds, requests/s, bytes, per cent) because a number with no unit misleads; the percentiles on the same graph as the median, so you can see the gap between the typical experience and the bad one; thresholds drawn as a line, so the SLO is visible and not a fact buried in documentation; a logarithmic scale for latencies, which usually span orders of magnitude; no pie charts for time series; and about 12 panels at most, because a 40-panel dashboard never gets looked at.

The rules are evaluated in Prometheus, every evaluation_interval, and they live in the rule_files:

The four fields of a rule and their role. expr is a PromQL query: the alert is active while it returns any series. for is the most underrated element: it demands that the condition hold for that length of time before firing, and it is what removes 90 % of the noise —a two-second spike wakes nobody—. labels classifies the alert and is what Alertmanager will use to route it. And annotations is the message for the human, with Go templates ({{ $value }}, {{ $labels.instance }}) and, very much recommended, a runbook_url: at four in the morning, a link to the diagnostic steps is worth more than any graph.

The last rule deserves pausing on, because it is the most valuable and the hardest to write: it alerts that something that should be happening is not. It requires bounding the time of day —hour() returns the UTC hour, hence the offset— so that it does not fire at four in the morning, when zero rentals is normal in Ribalta.

Prometheus fires; Alertmanager decides what to do:

Three mechanisms that prevent notification hell. Grouping turns three identical alerts from the three replicas into a single message. Inhibition rules mute derived alerts: if the application is down, there is no point in also being told its latency looks odd. And silences, created from the Alertmanager interface with a duration and a reason, serve a known maintenance window —a heavy Flyway migration, for instance—; they should always be temporary, because a permanent silence is an alert deleted without saying so.

Alert on symptoms, not on causes. "The p99 latency is above 1 s" is a symptom the citizen notices; "the CPU is at 85 %" is a cause that may be perfectly normal. If you alert on causes, the team gets notifications about things that affect nobody and stops looking at them, which is exactly how an important alert gets missed.

Every alert must demand a human, immediate action. The deciding question is: if this arrives at four in the morning, is there anything to do now? If the answer is "look at it tomorrow", it is not an alert: it is a dashboard panel or, at most, a notice during working hours. The severity: critical versus severity: warning distinction from section 15 is exactly that, with different destinations.

Fight alert fatigue. A team receiving thirty notifications a day stops reading them within two weeks, and from then on the alerting system does not exist even though it keeps running. The tools: a generous for, grouping, inhibition and ruthlessly deleting alerts that have never led to an action.

Always add a runbook_url. Whoever receives the alert at four in the morning may not be the person who wrote the code. Three lines of "check this, look at that, if it is this then do the other" change the outcome completely.

Alert on absence too. Broken systems sometimes stop emitting instead of emitting errors. absent() and the "no rentals at peak hour" rule cover that gap.

Test the alerts. Prometheus ships promtool test rules, which lets you write unit tests for the rules with simulated series. An alert that has never been seen to fire is a hypothesis, not an alert.

The criterion. The question is not which is better, but who is going to operate this and on what budget. Self-hosted Prometheus is free in licences and expensive in attention; the managed ones invert the terms. For CicloUrbana —three instances, some 15,000 series, a team that already operates Kubernetes— the self-hosted stack is the right choice, with a natural exit towards Grafana Cloud if the maintenance starts to weigh, because the query language and the dashboards are preserved. That is, in fact, the strongest strategic argument in favour of the Prometheus ecosystem: it does not lock you in.

And a note that anticipates the end of the module: OpenTelemetry is becoming the common standard for emitting metrics, logs and traces, with a protocol (OTLP) that almost every destination accepts. Prometheus can already ingest OTLP, Micrometer has a bridge to it and the commercial backends support it. It is the direction everything is converging on, and the one we will take in 09-06 for distributed tracing.

Forgetting metrics_path: /actuator/prometheus. Prometheus looks for /metrics, the target shows up red and everything else looks broken. It is the number one failure: check it in Status → Targets.

Using a counter's absolute value. ciclourbana_rentals_started_total without rate() gives a number that only grows and resets to zero on every deployment. On a counter, always rate or increase.

Forgetting le in histogram_quantile's sum by. The result is a meaningless number that also looks plausible. Always sum by (le, ...).

Too short a rate window. With a 15 s scrape, rate(x[30s]) does not have enough samples and produces gaps. Rule: the window should be at least four times the scrape interval.

Alerting with no for. A two-second spike wakes somebody up. Almost every alert needs a for of between 2 and 15 minutes.

Alerting on causes. CPU at 85 %, memory at 70 %, frequent GC: these are diagnoses, not problems. Alert on what the user suffers and use those other metrics to investigate.

Not persisting the Prometheus or Grafana volume. A restart wipes the history and the hand-built dashboards. Named volumes and, better still, dashboards provisioned from files in the repository.

Leaving Grafana on admin/admin and reachable. The dashboards reveal business volumes, endpoints and versions. Password mandatory and sign-up disabled.

Tip: provision from files the data sources, dashboards and rules, and version them in the repository alongside the code. A dashboard built by hand in the interface is infrastructure with no version control.

Tip: start by importing 4701 and 12900 and build a single dashboard of your own for the Ribalta business side. Reinventing the JVM memory graph adds nothing.

Tip: write the alert at the same time as the metric. A business metric with no alert becomes a panel nobody looks at.

Write the PromQL query for each of the council's questions and explain every function you use: (a) how many rentals have started today at the "University" station; (b) what percentage of requests to /api/v1/stations are answered in under 300 ms —the SLO from 09-03—; (c) which instance consumes the most heap memory; (d) the hit rate of the stations cache over the last hour; (e) whether any replica has had the payments circuit breaker open for more than five minutes.

The team defined this rule and, during a real 20-minute outage in which the API returned 500 to everything, no notification arrived. Find the three possible reasons and explain how you would verify each one.

The council asks for a screen to be projected in the operations room that lets somebody with no technical background tell whether the Ribalta network is working. Design that dashboard: choose six panels at most, give the PromQL query for each, the visualisation type, the units and the colour thresholds, and justify what you have left out and why.

Solution 1.

(a) Today's rentals at "University":

increase gives the counter's total increase over the window —unlike rate, which gives the per-second rate— and sum collapses the three replicas and the three fare types into a single number. An honest caveat: [24h] is "the last 24 hours", not "since midnight"; for that you have to adjust the panel's range or use a daily recording rule.

(b) Percentage under 300 ms:

Here the slo: 300ms configuration from 09-03 pays off: since the buckets are cumulative, the le="0.3" one contains exactly the requests that met the target, and dividing it by the total gives the compliance ratio. It is the correct way to measure an SLO, better than histogram_quantile, because it answers the contractual question: "what percentage complied", not "what is the percentile". An indispensable requirement: that a bucket exists at exactly 0.3.

(c) Instance with the most heap:

sum by (instance) aggregates the different memory pools (Eden, Survivor, Old) of each JVM, and topk(1, ...) returns the largest while keeping the instance label, which is the answer being sought.

(d) Cache hit rate:

The denominator includes hits and misses because it does not filter by result. If the result is below 0.2, it is time to review the key or the self-invocation from 09-02; on a panel it is best shown as a percentage with thresholds.

(e) Circuit breaker open for more than five minutes:

min_over_time over the window is 1 only if the series has been 1 for the whole window, which expresses exactly "it has been open for five minutes" and not "it opened at some point". The idiomatic alternative is expr: ... == 1 with for: 5m in an alerting rule, which is how it would be written in production.

Solution 2.

Reason 1 — the expression uses a counter with no rate. http_server_requests_seconds_count is cumulative since startup, so > 100 is always true in any application with traffic... and that is why, paradoxically, it does not distinguish an outage from an ordinary day. Besides, if the outage caused restarts, the counter reset to zero and dropped below 100. The correct approach is a ratio with rate. How to verify it: run the expression in the Prometheus interface and see that it returns a large value even at a healthy moment.

Reason 2 — for: 30m is longer than the incident. The outage lasted 20 minutes, so the alert was in pending state that whole time and resolved before moving to firing: it never reached Alertmanager. How to verify it: look at the historical state in Alerts or query the ALERTS{alertstate="pending"} metric. Practical rule: the for must be considerably shorter than the duration of the incident you want to detect; 5 minutes is a reasonable value.

Reason 3 — severity: info is probably not routed to any receiver. The routes in section 16 send critical to the on-call rota and team=product to its channel; a label matching no specific route falls through to the default receiver, and if the team has that pointed at a channel nobody reads, the practical effect is the same as not alerting. How to verify it: use amtool config routes test severity=info or the routing tool in the Alertmanager interface.

In addition, the annotation "Errors in the API" does not say how many, or where, or what to do, and there is no runbook_url. Corrected version:

Solution 3.

Design principle: the audience is not technical, so each panel must answer one question in the language of the service, with colour as the primary information and the number as the detail. Six panels:

What is left out and why. Heap memory, GC pauses, Tomcat threads, pool usage, cache hit rate and per-endpoint latency: all of those are causes, not symptoms, and their audience is the technical team. A council operator can do nothing with "GC pause 180 ms", and its presence only means that the panels that do matter get lost in the noise. They live on the technical dashboard from section 14 and on the imported 4701, one click away.

Two additional decisions. The fourth panel's second row draws the SLO as a threshold line, so that "it is slow" is visible without interpreting numbers. And the rentals-per-minute panel is deliberately the largest: it is the only one that detects a functionally broken service with every technical indicator green, the case we identified in 09-03 as the one technical metrics never see. It is also worth setting the default time range to 6 hours and enabling auto-refresh every 30 seconds, consistent with the scrape interval.

CicloUrbana's metrics have left the process. You understand why an external system was needed —persistence, history, aggregation across instances and continuous evaluation— and why Prometheus goes and fetches the data instead of waiting for it, with the decisive advantage that "the instance is down" becomes a query (up == 0) instead of an inference, and with the Pushgateway as a bounded exception for ephemeral processes. You know how to expose the metrics with micrometer-registry-prometheus on the 07-01 management port, and to read the text format: # HELP, # TYPE, the series with their labels and the _count, _sum and _bucket suffixes with the cumulative le label, which are the literal translation of the histograms you configured in 09-03.

You have a complete prometheus.yml for Ribalta —with the metrics_path almost everybody forgets, the authentication that Actuator's security chain demands and intervals chosen sensibly— and you know how to replace the static list with discovery in Kubernetes using annotations and relabel_configs, so that every replica the 08-04 HPA creates starts measuring itself. You know the real cost of the storage and its dependence on the number of series, not on the traffic, which takes you back once again to cardinality.

And you know real PromQL: selectors with their four matchers, rate always on a counter and why irate is only good for debugging, sum by and sum without to aggregate the three instances, histogram_quantile with the mandatory le to get the real p95 that was impossible with percentiles computed in the JVM, plus increase, avg_over_time, absent and the operations between series that produce ratios. With those pieces you have written CicloUrbana's concrete queries: traffic and latency per endpoint, error rate as a ratio, rentals per minute and fare, bikes per station, the HikariCP pool, GC pauses and cache hits.

On the operational side, you have the stack up with docker-compose alongside the 07-04 one, with persistent volumes and the postgres_exporter; Grafana provisioned from files, with the strategy of importing the generic (4701, 12900) and building only the business side; the Ribalta dashboard organised in four rows from what the citizen sees to what the machine consumes, with template variables for environment and instance; and the alerting rules with expr, for, labels and annotations, including the most valuable and hardest to write —the one that warns that something that should be happening is not—. Alertmanager groups, inhibits and silences, and you have the golden rules of alerting: symptoms and not causes, every alert with an action attached, a runbook_url always, and ruthlessly deleting anything that only produces fatigue.

With this, two of the three pillars are standing. The metrics say that something is wrong and since when; the dashboards show it and the alerts announce it. But when the alert arrives at four in the morning saying the rental-start error rate is 3 %, the next question —what exactly failed, on which request, with which user, with which message— is answered by no graph. It is answered by the pillar we have been using since the very first lesson without ever treating it seriously: the event log. The next lesson, Logging and Log Management, turns it into a real tool: levels chosen with judgement, logback-spring.xml per profile, structured JSON logs, the traceId from the 03-06 TraceFilter propagated and searchable, what must never be logged about the citizens of Ribalta, and the centralised aggregation that lets you search the logs of all three instances as if they were one.