In 07-01 we opened CicloUrbana up from the inside with Actuator: health probes, version information, log levels at runtime, an endpoint of our own and a security chain requiring ADMIN on port 8081. We closed that lesson deliberately leaving one endpoint unopened, /actuator/metrics, with a promise: the instrumentation was in place and it would be read here.

That moment has arrived, and with a concrete need behind it. In 09-01 we tuned performance with load tests and EXPLAIN ANALYZE, that is, looking once and in a controlled environment. In 09-02 we set up caches whose usefulness depends entirely on a hit rate nobody is watching yet. Both lessons left the same debt: CicloUrbana does not measure itself, continuously and in production.

This lesson settles it. We will not repeat what Actuator is or how it is exposed: we will use the infrastructure already in place to instrument for real. We will look at Micrometer as the metrics facade, the five meter types and when to use each, the catalogue of metrics that exist without writing a line, the golden rule of cardinality —the one that separates a healthy monitoring system from one that collapses under its own weight—, the business metrics of the Ribalta network, the crucial difference between client-side percentiles and aggregatable histograms, and Spring Boot 3's unified observation model that produces a metric and a trace at the same time.

1. The three pillars of observability
2. Micrometer: the SLF4J of metrics
3. The meter types
4. The metrics that already exist without writing code
5. Querying metrics from /actuator/metrics
6. Tags and the golden rule of cardinality
7. Instrumenting CicloUrbana: CicloUrbanaMetrics
8. The Gauge, the weak reference and its classic mistake
9. @Timed and @Counted
10. Percentiles versus histograms
11. MeterFilter and MeterRegistryCustomizer
12. The pool, the database and the cache: which numbers to watch
13. The Observation API: one instrumentation, two signals
14. From metrics to SLOs: RED and USE
15. Testing that a metric is recorded
16. Common Mistakes and Tips
17. Exercises

Monitoring is watching a known set of indicators; observing is being able to answer questions you had not anticipated. The difference shows at three in the morning, when the problem is never the one you expected. Three signals are needed for that, and none of them substitutes for the others:

The real workflow of an incident in Ribalta uses all three in this order: a metrics-based alert warns that the rentals p99 has spiked; a trace shows that 80 % of the time is in the payment gateway span; the logs of that specific trace, filtered by its identifier, say what error it returned. Metrics to detect, traces to locate, logs to understand.

This lesson builds the first pillar inside the application. In 09-04 those metrics will leave the process for Prometheus and Grafana, because —and it is worth saying now— everything we instrument here lives in memory and dies with the process.

The comparison is literal and explains the whole design. Just as SLF4J lets you write log.info(...) without knowing whether Logback or Log4j2 is behind it, Micrometer lets you register a counter without knowing whether Prometheus, Datadog, New Relic or CloudWatch is behind it. Your code depends on a facade; the destination system is an interchangeable dependency.

MeterRegistry is the central piece: the factory and the registry of every meter. Spring Boot creates one automatically when it detects Actuator, and it is injected like any other bean. With no concrete implementation on the classpath an in-memory SimpleMeterRegistry is used, which is exactly what makes /actuator/metrics useful in development and in tests (section 15).

An important point of vocabulary: in Micrometer a meter is the generic abstraction, and Counter, Gauge, Timer, DistributionSummary and LongTaskTimer are its types. Each meter is identified by its name plus its set of tags, and each distinct combination is an independent time series. That sentence is the key to section 6.

Four selection criteria that settle almost every doubt:

Counter versus Gauge. If the question is "how many times has it happened?", it is a counter; if it is "how many are there right now?", it is a gauge. A counter is never decremented: for "rentals in progress" you do not use a counter that goes up on start and down on finish —that is a Gauge—, because a counter's value is its rate of growth, which is what rate() will exploit in 09-04.

Timer versus DistributionSummary. The Timer measures time and carries units; the DistributionSummary measures any quantity. The duration of a bike journey in minutes looks like time, but it is not the duration of a program operation: it is business data, and it goes in a DistributionSummary.

Timer versus LongTaskTimer. A Timer records the operation only when it finishes: if an import takes forty minutes, during those forty minutes the Timer says nothing. The LongTaskTimer reports on active tasks and how long they have been running, which is exactly what you want to watch on a @Scheduled task from 07-03.

Every Timer is also a counter. It publishes _count, _sum and _max, so timing an operation gives you the number of times it happened for free: no separate Counter is needed.

Before instrumenting anything, it is worth knowing what is there. With Actuator and the autoconfigurations active, CicloUrbana already publishes:

Two observations. The first: http.server.requests alone answers almost everything, because it has a count (traffic), a status tag (errors) and a distribution (latency), which are the three signals of the RED method. The second: spring.data.repository.invocations has to be enabled (management.metrics.data.repository.autotime.enabled: true) and it is the natural complement to the 09-01 query counter, now in production and continuously.

Recalling the 07-01 configuration —management port 8081 and ADMIN via httpBasic—, it is enough to add the endpoint to the exposed list:

Those three common tags are the difference between "latency is high" and "latency is high on production instance 3". They apply to every meter, including the automatic ones.

What matters is how it is read: TOTAL_TIME / COUNT is the average latency —4,021.83 s over 128,431 requests = 31 ms—, and we already know from 09-01 how little an average is worth. To get closer to what matters you filter with ?tag=, chaining several:

Notice the uri tag: it says /api/v1/rentals/{id}/finish, with the template and not the real identifier. That decision by Spring is what makes the metric usable, and it is exactly the subject of the next section.

And a limitation worth internalising: this endpoint gives you the accumulated value right now. There is no history, no graph, no "two hours ago". It serves to check that a metric exists and for an urgent one-off query; for everything else you need 09-04.

A tag (a label in Prometheus) is a dimension: it lets you break rentals.started down by fare type or http.server.requests down by endpoint. Without tags, a metric only says "things are happening"; with them, it says which things.

But each distinct combination of name and tag values is an independent time series, with its own memory in the application, its own series in Prometheus and its own cost in every query. Hence the golden rule:

A tag's cardinality must be low and bounded. If the number of possible values grows with the number of users, requests or rows, it is not a tag.

Applied to CicloUrbana:

What exactly happens when the rule is broken. Suppose Counter.builder("rentals.started").tag("user", userId). With 50,000 citizens, 50,000 series appear: the application retains 50,000 meter objects in memory and its MeterRegistry grows without stopping; /actuator/prometheus goes from returning 200 KB to returning tens of megabytes on every scrape; Prometheus multiplies its memory and index consumption; and Grafana queries become slow or fail outright. It is the number one cause of monitoring-system outages, it has a name of its own —cardinality explosion— and the worst part is that it does not fail the day it is deployed, but three weeks later, when nobody connects one thing with the other any more.

The practical rule that avoids the mistake: if a piece of data identifies an individual or a specific request, it goes in a log (09-05) or a trace (09-06), never in a metric tag. Metrics answer "how many" and "how much"; the "which one" is the job of the other two signals.

One intermediate case worth knowing: in Ribalta, 40 stations make a perfectly healthy tag. If CicloUrbana expanded to a metropolitan network of 4,000 stations the decision would have to be revisited —4,000 series per metric multiplied by every station metric starts to be a lot— and probably moved to tagging by district, leaving the specific station to the traces.

Automatic metrics tell you whether the system is healthy; business metrics tell you whether the service is healthy. They are different questions: an API answering 200 in 30 ms with zero rentals started at eight in the morning is technically perfect and functionally broken.

Everything is concentrated in a single component, which is the practice that ages best:

And its use from the domain service, which knows nothing about Micrometer:

Three design decisions deserve comment. A single component concentrates the metric names: without it, the strings "ciclourbana.rentals.started" scatter through the code and the variants appear (rentals_started, rental.started) that break dashboards. The names follow the Micrometer convention: lower case separated by dots and a hierarchy from the general to the particular (ciclourbana.rentals.started), which each system translates into its own style —Prometheus will turn it into ciclourbana_rentals_started_total—. And the instrumentation does not change behaviour: if registry were a test SimpleMeterRegistry, everything would carry on working just the same.

A warning about the Counter inside the method: Counter.builder(...).register(registry) on every invocation does not create a new counter, because register returns the existing one if the name and tags match. That is correct, though it has a small lookup cost; on a very hot path it is worth caching the counters in a Map keyed by tag.

A Gauge does not store values: it holds a reference to the object that has them and a function to read it, and that reference is weak. Micrometer does this on purpose, so as not to prevent the garbage collector from freeing objects the application no longer uses: a metric must never cause a memory leak.

The consequence is the most bewildering mistake in all of Micrometer:

This works perfectly for minutes and then the metric starts returning NaN. When the GC runs, the AtomicInteger —which nothing points to except the gauge's weak reference— disappears, and the meter is left with no source. And the failure is intermittent and depends on memory pressure, which makes it the classic "it worked in development".

The solution is the one in section 7: the field Map holds the strong reference for as long as the component lives. Two other correct ways: use registry.gauge("name", tags, anAlreadyLongLivedObject, function) on a long-lived object, or Gauge.builder("queue.pending", theQueue, Collection::size) on a collection that is already a field of the bean.

Three more rules about gauges. You register once and then only update the value: registering the same name with the same tags again returns the existing one, but registering it in a loop is a design smell. The function is invoked at scrape time, not when you call it: that is why it must be cheap and must not throw exceptions —never a Gauge that runs a database query, because Prometheus would fire it every 15 seconds—. And a gauge that never changes contributes nothing: it is a constant dressed up as a metric.

For Ribalta's availability, the gauge is refreshed from the 07-03 scheduled task, which already walks the stations every fifteen seconds, rather than being computed on every scrape.

To instrument without writing code, Micrometer offers two aspect-based annotations. They require the AOP dependency and the aspects to be registered:

@Timed generates a Timer with class, method and exception tags —the last of which is none when there was no error, which lets you separate the latency of successes from that of failures—. @Counted counts invocations with the same tags.

Their limits, worth being clear about before using them everywhere. They suffer the proxy trap for the fourth time in the course: an internal call is not measured. They cannot tag by business data: extraTags only accepts constants, so there is no way to tag by a fareType received as an argument —that needs the explicit code from section 7—. And they add an aspect per annotated method, with its invocation cost.

CicloUrbana's criterion: @Timed for cross-cutting, quick instrumentation —timing a whole service while investigating something— and explicit code for the business metrics that carry tags and are going to feed permanent dashboards.

This section is the most technical of the lesson and the one with the most consequences when scaling. In 09-01 it became clear that the SLO is written in p95 and p99. The question now is where those percentiles are computed, and there are two incompatible answers.

Why client-side percentiles cannot be aggregated. It is the consequence of that rule from 09-01: percentiles do not average. If instance 1 reports p95 = 200 ms and instance 2 reports p95 = 400 ms, the service's real p95 is not 300 ms, and there is no way to compute it from those two numbers: the information needed was lost when they were computed. With three replicas on Kubernetes (08-04), that "p95" graph is simply false.

How the histogram solves it. Instead of a percentile, each instance exports how many observations fell below each threshold: 1,200 below 100 ms, 1,380 below 300 ms, and so on. Those counts do add up across instances, and from the sum any percentile can be interpolated with histogram_quantile (09-04). You decide the percentile when you query, not when you instrument.

The price is cardinality: a histogram generates one series per bucket. That is why the three settings above exist: slo adds buckets exactly at the thresholds you care about —very useful, because it then lets you compute directly "what percentage of requests came in under 300 ms", which is the literal formulation of a service objective— and minimum-expected-value / maximum-expected-value bound the range, removing absurd buckets below 10 ms or above 10 s and greatly reducing the number of series.

CicloUrbana's decision: histograms on the few metrics that underpin the SLOs (http.server.requests and rental duration) and no distribution at all on the rest. Enabling percentiles-histogram for everything is one of the quickest ways to cause the problem from section 6.

When the instrumentation has to be corrected —yours or that of a library you do not control—, MeterFilter intervenes in the registration of every meter:

The five uses, in order of real usefulness. Denying metrics that are not used reduces the size of every scrape —jvm.buffer and some of the tomcat ones are rarely looked at—. maximumAllowableTags is the safety net against cardinality explosion: past 100 distinct stations, it stops registering new series instead of bringing Prometheus down; it is a lifebelt, not an excuse for tagging badly. maximumAllowableMetrics sets a global ceiling. renameTag and MeterFilter.commonTags let you adapt third-party metrics to your convention. And configuring the distribution in code applies rules to whole families of metrics by prefix, something YAML does not allow.

It is worth distinguishing the two beans: MeterRegistryCustomizer runs once over the registry, whereas the MeterFilter it installs runs for each meter that gets registered. And there is an ordering that surprises people: the filters are applied in the order they are added, and a later deny does not revoke an already-registered meter, so the filters have to be installed before the application starts instrumenting —hence they go in a @Bean and not in some @PostConstruct—.

With what was learned in 09-01 and 09-02, these are the metrics that turn those one-off diagnoses into continuous surveillance:

The most profitable of them all is the second to last combined with jvm.gc.pause: memory that goes up and does not come down after the pauses is the signature of a leak, and seeing it on a month-long graph is infinitely easier than in a heap dump. And the first, pending, is the one that turns the manual diagnosis from exercise 1 of 09-01 into an automatic alert.

So far we have recorded metrics. When it comes to recording traces in 09-06, the problem of instrumenting the same thing twice would appear. Spring Boot 3 solves it with the Micrometer Observation API: you declare one observation and the infrastructure produces a metric, a trace span and, if configured, a log entry, all at once.

The distinction between lowCardinalityKeyValue and highCardinalityKeyValue is the rule from section 6 turned into an API, and it is one of Micrometer's best ideas: low-cardinality keys become metric tags and span attributes; high-cardinality ones go only to the span, where they do no harm. The model forces you to think about cardinality as you write the code, which is exactly where it should be thought about.

The declarative version needs the corresponding aspect:

Today, with only Micrometer Metrics on the classpath, this produces a Timer identical to @Timed's. In 09-06, when Micrometer Tracing is added, the same code will start producing spans without touching a line. That is the reason to know it now: whatever is instrumented with the Observation API will be ready for distributed tracing; whatever is instrumented with a hand-written Timer will not.

As a complement, a custom ObservationHandler lets you react to every observation —for example, writing a log line with its duration— and ObservationPredicate lets you filter which ones are recorded, typically to ignore /actuator/** and avoid measuring yourself.

Instrumenting with no method produces hundreds of graphs nobody looks at. The two reference methods say what to look at and they complement each other:

RED, for services (what the user sees): Rate (requests per second), Errors (the proportion that fails) and Duration (the latency distribution).

USE, for resources (what the system consumes): Utilization (percentage in use), Saturation (work queued and waiting) and Errors (resource failures).

The last row is the most valuable and the one almost nobody adds. Every technical indicator can be green while the service is broken: a deployment that broke the rent button in the app leaves the API answering 200 to station queries and zero rentals started. An alert on that business metric detects in five minutes what the technical ones never detect at all.

Implementing these alerts is 09-04's job, with rules in Prometheus and Alertmanager. What is decided here is what gets measured and what threshold is considered acceptable, and that decision is as much a product one as an engineering one.

A business metric is observable behaviour: if a refactor stops incrementing the rentals counter, the dashboard lies and nobody notices. It is tested with SimpleMeterRegistry, Micrometer's in-memory implementation.

The second test is the gem: it verifies that the strong reference exists, which is precisely the failure that only shows up in production and after hours of running. System.gc() is a suggestion and not a guarantee, but in practice it is enough to make the test fail with the incorrect version from section 8.

In integration, @SpringBootTest with @AutoConfigureObservability brings up the real registry and lets you check that the automatic HTTP metrics appear —by default, tests disable the export—:

Tagging by user identifier, by URI with identifiers or by trace identifier. It is the serious mistake of the lesson: a cardinality explosion that brings down Prometheus first and the application afterwards, weeks after being introduced. Whatever identifies an individual goes in the log or the trace.

Losing the Gauge to the weak reference. It works for a while and then returns NaN. Always keep a strong reference in a field of the component, and write the test with System.gc().

Using a Counter for something that goes down. "Rentals in progress" is not a counter: it is a Gauge. A counter that gets decremented breaks rate() and produces absurd graphs.

Believing percentiles works with several instances. Percentiles computed in the JVM cannot be aggregated; with three replicas, that graph is false. You use histograms.

Enabling percentiles-histogram for every metric. It multiplies the series tenfold. Only on the ones that underpin an SLO, and bounded with minimum/maximum-expected-value.

Doing work inside a Gauge's function. It is invoked on every scrape, every 15 seconds, for ever. A database query in there is a permanent load nobody connects with anything.

Measuring only the technical side. With every indicator green the service can still be broken. The metric that warns you about a deployment that broke renting is ciclourbana.rentals.started dropping to zero.

Tip: centralise metric names in a component or in constants. Names are a contract with the dashboards and alerts of 09-04; a name change silently breaks panels. And follow the Micrometer convention —lower case with dots, application prefix—, letting each exporter translate it.

Tip: use the Observation API for anything new. It costs the same as a Timer and in 09-06 it will produce spans without touching the code.

Tip: add the common tags (application, environment, instance) from day one. Adding them later forces you to rewrite every query and to lose the continuity of the graphs.

For each of Ribalta council's needs, state the meter type, the name, the tags —justifying their cardinality— and whether it needs a histogram: (a) how many bikes are in maintenance right now, by workshop (there are 2 workshops); (b) how many charges the gateway has rejected, distinguishing the reason (BALANCE, CARD_EXPIRED, TECHNICAL); (c) how long the council's nightly import takes, knowing it runs for between 20 and 50 minutes and that it needs watching while it runs; (d) the amount billed for each rental, so that "what is the median and the p95 amount?" can be answered; (e) how many times each citizen has rented this month.

A colleague has added this instrumentation. Three weeks later, Prometheus is consuming 12 GB of memory, /actuator/prometheus takes 8 seconds to respond and the available-bikes graph shows gaps. Find the three problems, explain the symptom of each and write the corrected version.

The council is asking for a service level agreement. Define CicloUrbana's SLIs and SLOs following RED and USE: choose four service indicators and three resource ones, say which Micrometer metric each is computed from, propose a threshold and an evaluation period, and state which of those thresholds should wake somebody up in the middle of the night and which should not. Justify the management.metrics.distribution configuration that makes measuring them possible.

Solution 1.

(a) Bikes in maintenance per workshop: a Gauge, ciclourbana.bikes.maintenance, tag workshop (2 values: minimal cardinality). It is a value that goes up and down, so it is not a counter. No histogram —gauges have no distribution—. Watch out for the strong reference from section 8: a Map<String, AtomicInteger> as a field, refreshed from the scheduled task, and never a database query inside the gauge's function.

(b) Rejected charges: a Counter, ciclourbana.charges.rejected, tag reason with the enum's 3 fixed values. It only goes up and what matters is its rate, which rate() will extract in 09-04. The tag must come from an enum, never from the gateway's error message: free text returned by a third party is unlimited cardinality in disguise.

(c) Nightly import: a LongTaskTimer, ciclourbana.import.duration. It is the case that justifies its existence: a normal Timer would publish nothing during the 40 minutes it runs and would only record the result at the end, making it impossible to know whether it is running or hung. The LongTaskTimer publishes active_tasks and duration for tasks in progress, allowing an alert on "it has been going for more than 60 minutes". A useful complement: a Gauge with the timestamp of the last successful run, to detect that it has not run.

(d) Amount billed: a DistributionSummary, ciclourbana.rental.amount, baseUnit("euros"), with publishPercentileHistogram() because the question explicitly asks for the median and p95 and there are several instances. It is not a Timer because it does not measure time. A reasonable tag: fare (3 values). It is worth bounding it with minimum-expected-value: 0.5 and maximum-expected-value: 50 so as not to generate useless buckets.

(e) Rentals per citizen per month: no metric at all. It is unlimited cardinality —one series per citizen— and besides, it is not a monitoring question but a business one, with the rental row already stored in PostgreSQL: it is answered with SELECT user_id, count(*) ... GROUP BY user_id. The rule from section 6 in its purest form: metrics answer "how many" in aggregate, not "who".

Solution 2.

Problem 1 — a user tag with unlimited cardinality. One series per citizen; with 50,000 users, 50,000 series from a single counter. Symptom: Prometheus memory through the roof and slow queries. It is the main cause of the 12 GB.

Problem 2 — a uri tag with the rental identifier. Infinite cardinality: a new series for every rental created, for ever, and none of them ever reused. Symptom: /actuator/prometheus returning megabytes and taking 8 seconds, because its size grows linearly with the number of historical rentals. It is even worse than the previous one, because it has no ceiling.

Problem 3 — a Gauge registered on every request over a temporary object. Two failures in one line: it is registered inside the method, so a meter is attempted per request, and the AtomicInteger has no strong reference, so the GC frees it and the gauge returns NaN. Symptom: the gaps in the graph. On top of that, countFree() on the request path adds a query to every rental.

Corrected version:

The decisions: the instrumentation moves to the service, with metrics.rentalStarted(fareType, stationId) and its two low-cardinality tags; the URI and the status are already supplied by http.server.requests with the template, so the manual request counter was entirely surplus; and the free-bikes gauge is registered once in CicloUrbanaMetrics, with its strong reference and updated from the 07-03 scheduled task. As a safety net, the MeterFilter.maximumAllowableTags(...) from section 11 is added, which would have turned this outage into a truncated metric.

Solution 3.

Service indicators (RED), measured over http.server.requests:

Resource indicators (USE):

What wakes you and what does not. The criterion is twofold: the user is suffering it now and there is something to be done. That is why the error rate, sustained latency on the critical endpoint, the absence of rentals at peak hour and a saturated pool —which is the early warning of a complete outage— all wake you. GC pauses and thread utilisation do not: they are causes, not symptoms, and their place is a notice during working hours. It is the principle that will be developed in 09-04: alert on symptoms, not on causes, because every alert that does not demand immediate action erodes the credibility of all the rest.

Required configuration:

A histogram and not percentiles, because with three replicas the percentiles computed in each JVM cannot be aggregated and would give a false figure (section 10). The slo buckets at 300 ms and 800 ms let you answer directly "what percentage of requests met the target", which is the literal formulation of the SLO and the one that will go into the council's monthly report. And bounding between 10 ms and 5 s avoids dozens of useless buckets, keeping cardinality under control.

CicloUrbana no longer merely lets itself be asked: it measures itself, continuously. You are clear about the three pillars of observability and what each one answers —metrics to detect, traces to locate, logs to understand— and you understand Micrometer as the facade that lets your code register counters without knowing who will collect them, with an injectable MeterRegistry like any other bean. You know how to choose between the six meter types with criteria that settle the real doubts: a counter for "how many times" and a gauge for "how many there are now", a Timer for program operations and a DistributionSummary for business quantities, a LongTaskTimer for what has to be watched while it happens.

You know the catalogue of metrics that exist without writing code —with http.server.requests as the most valuable in the system— and how to query them through /actuator/metrics with ?tag=, with the limitation that there is no history there. And you have the golden rule of this lesson committed to memory: a tag's cardinality must be low and bounded; fare and station yes, userId, the URI with identifiers and the traceId never, because whatever identifies an individual belongs in a log or a trace and not in a metric.

You have instrumented the Ribalta network with CicloUrbanaMetrics: counters of rentals started and finished by fare, a gauge of available bikes per station —with the strong reference that avoids the weak reference's NaN, and its test with System.gc()—, a Timer for the fare calculation and a DistributionSummary for journey duration, all behind a facade that leaves the domain knowing nothing about Micrometer. You know when @Timed and @Counted are enough and where their three limits lie. You understand the difference that matters most when scaling: percentiles computed in the JVM do not aggregate across instances and histograms do, with slo, minimum-expected-value and maximum-expected-value so that this power is not paid for with an explosion of series. And you know how to correct and protect the instrumentation with MeterFilter, including the maximumAllowableTags safety net.

Finally, you have the numbers to watch in the pool, the database and the 09-02 caches; the Observation API as the unified model that turns cardinality into API (lowCardinalityKeyValue versus highCardinalityKeyValue) and that in 09-06 will produce spans without touching the code; and the RED and USE methods turned into concrete SLIs and SLOs, including the row almost nobody adds and which detects a broken service with everything green: rentals started = 0 at peak hour.

One problem has been present the whole time: all of this lives in memory and dies with the process. Every 08-05 deployment wipes the metrics; /actuator/metrics has no history; there are no graphs, no comparison with last week and, above all, not a single alert. The SLOs from section 14 are defined but watched by nobody. The next lesson, Using Prometheus and Grafana, closes that gap: the pull-based collection model, the format of /actuator/prometheus, real PromQL —rate, sum by, histogram_quantile over the buckets we have just configured—, CicloUrbana's dashboard panel by panel and the alerting rules that turn these numbers into a phone call when it genuinely matters.