Module 6 ended with a claim and a gap. The claim: CicloUrbana is correct, and ./mvnw verify proves it. The gap: it is not operable. Nobody can ask it whether it is healthy before sending traffic its way, nor whether the database is responding, nor which version is deployed on the Ribalta council's server. When the application hangs at three in the morning, the only tool available is to restart it and hope.

This lesson closes that gap with Spring Boot Actuator, the module that opens the application from the inside and exposes its state over HTTP and JMX. We will look at which endpoints it ships, which ones are worth exposing and which are not, how to secure them with the security chain we already wrote in 05-02, how /actuator/health really works — including the availability probes Kubernetes will need in 08-04 — how to write your own health indicators and endpoints for the Ribalta network, and how to find out, with a single request, exactly which commit is running in production.

1. What it means for an application to be operable
2. Adding Actuator and what shows up straight away
3. The full endpoint catalogue
4. Exposure and discovery
5. A separate management port
6. Securing Actuator with Spring Security
7. /actuator/health in depth
8. Health groups and availability probes
9. A custom HealthIndicator: StationNetworkHealth
10. Changing availability with AvailabilityChangeEvent
11. /actuator/info: which version is deployed
12. /actuator/loggers: changing the log level on the fly
13. mappings, configprops, env and secrets
14. A custom endpoint with @Endpoint
15. Actuator over JMX
16. Common Mistakes and Tips
17. Exercises

A correct application does what it promises. An operable application can also be managed by somebody who is not its author, in the middle of the night, without reading the code. These are two independent properties: a system can be impeccable on the inside and a black box on the outside. The difference shows up in concrete questions that the Ribalta council's operations team will ask sooner or later:

All of those answers already exist inside the process: Spring knows its beans, its configuration, its routes and the state of its connection pool. Actuator does not compute them, it publishes them. That is its entire job, and that is why adding it costs one dependency.

No version, because spring-boot-starter-parent governs it (01-04). On startup, the log shows a new line:

One endpoint, not fifteen. Actuator registers all of them internally, but only exposes health over HTTP by default. The reason is security, and it is a lesson learned the hard way: in Spring Boot 1.x almost everything was exposed, and there was a time when searching for /env on the internet returned database credentials belonging to real companies. Since Boot 2 the policy is the opposite: nothing goes out unless you ask for it by name.

The /actuator root returns a HAL index with links to whatever is exposed, handy for discovering what is available without knowing the names by heart. And health answers tersely on purpose:

The aggregate status is tolerably public information; the breakdown is not, which is why it is hidden by default. We open it up in section 7.

This is the lesson's reference table. The risk column is the one that decides what gets exposed in Ribalta.

Three observations about the table. First: shutdown is the only endpoint disabled by default, not merely unexposed — you have to enable it with management.endpoint.shutdown.enabled: true — because its effect is irreversible. Second: heapdump is the most dangerous of the lot and the one that looks least so; a heap dump of CicloUrbana contains the JWTs of every live session, the citizens' personal data and the signing secret. Third: httpexchanges does not work merely by exposing it; it also needs an HttpExchangeRepository bean, precisely so that nobody switches it on without realising what it is storing.

Three properties govern what is visible and where:

Telling enabled and exposure apart matters. A disabled endpoint does not exist: not over HTTP, not over JMX, not even as a bean. An enabled but unexposed one exists and works, but has no HTTP route. The usual combination in production is enabled + exposed + protected, not disabled, because the health indicators are still needed internally.

About include: "*": the quotes are mandatory in YAML — a bare asterisk is an alias — and, above all, it is an acceptable decision only in development. Writing out the explicit list takes thirty seconds and removes the class of accident where a Spring Boot upgrade exposes an endpoint that did not exist when you wrote the configuration. In 07-02 this list will differ per profile: generous in dev, minimal in prod.

With that, the Ribalta API stays on 8080 and all management moves to http://localhost:8081/actuator/.... It is a recommended practice for a reason that is not cryptographic but topological: it lets the firewall, rather than the application, decide who talks to management.

That last point is more useful than it looks: if the thread pool serving 8080 is exhausted, the health probe travelling over 8080 is left waiting too and the orchestrator concludes the application is dead. Warning: when you change management.server.port, the API's security chain stops applying to Actuator, because they are different ports. Authentication is still required — the next section — and you have to use EndpointRequest, which knows how to resolve the management port, instead of writing the path by hand.

In 05-02 we wrote a separate chain with @Order(1) and securityMatcher("/actuator/**"). That version worked, but it has two flaws: the path is written by hand — if somebody changes base-path, security silently stops applying — and it does not account for the management port. Actuator ships its own RequestMatcher that solves both:

The decisions, one by one:

• EndpointRequest.toAnyEndpoint() matches every exposed endpoint, whatever the base-path and whichever port it sits on. That is the difference between a rule that maintains itself and one that silently expires.
• EndpointRequest.to(HealthEndpoint.class, InfoEndpoint.class) identifies endpoints by their class, not by their path. There is an id-based variant, EndpointRequest.to("health", "info"), useful for custom endpoints such as the network one in section 14.
• anyRequest().hasRole("ADMIN") closes off the rest. Inside this chain anyRequest() means "any Actuator endpoint", because the securityMatcher has already narrowed the scope.
• httpBasic instead of JWT, deliberately: operations tooling — Prometheus in 09-04, an on-call curl, the alerting system — has no idea how to request a token from /api/v1/auth/login or renew it. A technical user with ADMIN and a long password, on a port that never leaves the internal network and over TLS, is more operable and no less secure. The mature alternative is mTLS between the monitoring system and the application.
• STATELESS and CSRF disabled, for consistency with the rest of the project (05-02): there is no session and there are no forms.

The RoleHierarchy from 05-03 still applies, so an ADMIN also covers whatever OPERATOR is allowed. And the permitAll on health is open to review: if the network allows it, closing health too and leaving only the per-group probes public (section 8) is better still.

health is not a check, it is an aggregation: Spring collects every bean implementing HealthIndicator, asks each one and combines the results.

With when-authorized and ADMIN credentials, the response changes:

The automatic indicators appear on their own depending on what is on the classpath: db with any DataSource (it asks for a connection and runs isValid()), diskSpace always (free space above a 10 MB threshold), ping always (answers UP while the application is serving), ssl from Boot 3.4 onwards (certificates about to expire) and one per infrastructure starter present: mail, redis, rabbit, kafka, elasticsearch. Each is switched off with management.health.<name>.enabled: false.

The aggregation rules are the part that produces the most surprises: UP yields 200, DOWN and OUT_OF_SERVICE yield 503, and UNKNOWN yields 200. The global status is the worst of them all according to a configurable order, and any DOWN brings the whole thing down. Hence this lesson's most common trap: an indicator for a non-critical system can pull the application out of production. If CicloUrbana declares an indicator for the Ribalta payment gateway and the gateway goes down, /actuator/health returns 503, the load balancer withdraws the instance and the entire network stops renting bikes over a problem that only affected billing. The fix is not to drop the indicator, but to take it out of the group the load balancer looks at.

A group is a subset of indicators with its own URL and its own policy:

Now /actuator/health/readiness and /actuator/health/liveness exist, each aggregating only its own members. The payment gateway from the previous example can keep appearing in the full /actuator/health — where the team sees it — without being in readiness, which is what the load balancer looks at.

Availability probes are the piece we will need in 07-04 and 08-04. With probes.enabled: true — and automatically when Spring detects it is running on Kubernetes — the framework manages two states of its own:

The distinction is critical and it is got wrong constantly. If you put the database in liveness, a thirty-second PostgreSQL outage will make the orchestrator kill every instance of CicloUrbana, which on restart will not find the database either, entering a restart loop that turns a database incident into a total outage. The rule is simple: liveness holds only what a restart fixes; readiness holds everything that prevents serving properly. A downed external dependency is never fixed by restarting.

CicloUrbana's health is not purely technical. If all four Ribalta stations have no bikes available, the application responds perfectly and the service, in practice, does not exist. That is a business indicator:

Four details that separate a useful indicator from a harmful one:

• The bean name is the component name. @Component("stationNetwork") produces that key in the JSON. Without it, Spring derives it from the class by stripping the HealthIndicator suffix; naming it explicitly stops a rename from breaking the readiness group configuration.
• Never let an exception escape. An indicator that throws produces a DOWN with the exception message inside the JSON, and that message may contain the database URL along with the username.
• It has to be fast. It runs on every probe, every three or five seconds, on every instance. A query that walks the rental history turns the probe into a performance problem; if the calculation is expensive, cache it (09-02) or read it from the in-memory ActiveRentalsRegistry.
• Think about which group it belongs to. stationNetwork in readiness means: if Ribalta runs out of bikes, the load balancer withdraws the instance. That is almost certainly not what you want — a citizen must still be able to browse stations even when they are empty — so it fits better in the general /actuator/health and in an alert. It is the perfect illustration of why the question "which group does it go in?" matters more than the code itself.

Variants: AbstractHealthIndicator saves you the try/catch by overriding doHealthCheck(Health.Builder); CompositeHealthContributor groups several sub-indicators under a common key — handy for per-station health — and in a WebFlux application you implement ReactiveHealthIndicator, whose health() returns a Mono<Health> so as not to block the event loop. CicloUrbana is servlet-based, so we use the blocking variant.

Sometimes the state is not inferred, it is decided. When the Ribalta team is about to apply a heavy migration, they want the instance to stay alive but stop receiving traffic. Spring models this with two enums, LivenessState and ReadinessState, changed by publishing an event:

From the publish onwards, /actuator/health/readiness returns 503 and the load balancer withdraws the instance without killing it, while liveness stays UP; in-flight requests finish thanks to the graceful shutdown from 01-05. Publishing LivenessState.BROKEN has the opposite and far more drastic effect: it tells the orchestrator "I am beyond recovery, kill me". The @EventListener records in the log who stopped accepting traffic and when, which is exactly what you go looking for afterwards during an investigation.

Of all the operations questions, "which version is deployed?" is the one that wastes the most time when it has no automatic answer. info answers it, but it comes empty by default and has to be filled from three sources: static properties, build data and the Git commit.

The two plugins that generate META-INF/build-info.properties and git.properties during packaging:

The combined result:

With that response, "is the fare calculation fix deployed?" is answered by comparing a seven-character hash, without logging into the machine. In 08-05 this becomes the basis of post-deployment verification. A custom InfoContributor additionally adds data computed on the fly:

Warning: info usually stays public, so do not put anything in it that you would not put on a billboard. The database URL, the server name or the list of dependencies with their versions — a perfect inventory of known vulnerabilities — must not appear.

This is the endpoint that rescues an on-call shift most often. It lets you raise the log detail of one specific package without restarting and without deploying, and lower it again when you are done.

Three operational tips. Be surgical: setting org.hibernate.SQL to DEBUG on an instance with real traffic can generate gigabytes in minutes and fill the disk, which really is an outage. Remember to undo it, because the change lives in memory and survives until the next restart, which may be weeks away. And remember it is a write endpoint: that is exactly why it demands ADMIN. The systematic treatment of logging — formats, structured JSON, aggregation — belongs to 09-05; here we only care about the lever.

/actuator/mappings returns the complete route inventory: the pattern, the HTTP methods, the produces/consumes and the Java method that serves it. It answers in seconds "does this route exist?", "why is this request landing in the wrong controller?" and "is there any debug route still published?".

/actuator/configprops shows the @ConfigurationProperties from 02-05 with their effective values — NetworkProperties, FareProperties, GatewayProperties — and, together with /actuator/env, answers the most frustrating question in Spring configuration: where does this value come from? env lists the sources in precedence order, so you can see that the password comes from an environment variable and not from the YAML.

And here is the risk: both endpoints see everything, including ciclourbana.jwt.secret and spring.datasource.password. That is why Spring sanitises:

With never, every value appears as "******". With when-authorized they are shown only to whoever holds the configured roles; and even then Spring masks keys whose name matches sensitive patterns (password, secret, key, token, credentials), masking that can be tuned with a SanitizingFunction bean.

The Ribalta rule: env and configprops are not exposed in production. The diagnostics they provide are real, but you can get them in pre with the same configuration, and no amount of sanitisation makes up for the day somebody adds a property called ciclourbana.gateway.master-credential — which matches no pattern — and publishes it in JSON. Masking is a second line of defence, not the first.

When the information operations needs fits neither health nor info, you write an endpoint. In CicloUrbana, the summary of the state of the Ribalta network:

Four points to bear in mind. The id must be lowercase alphanumeric and must not clash with another endpoint. You have to expose it explicitly in exposure.include, just like the built-in ones. An endpoint is not a controller: it does not go through the GlobalExceptionHandler from 03-06 nor the validation from 03-04, so validate the parameters yourself and return simple structures. And @Endpoint publishes over HTTP and JMX at once; to restrict it to one there are @WebEndpoint and @JmxEndpoint. The fact that this endpoint can switch maintenance on with a POST explains why the chain in section 6 demands ADMIN for everything that is not health or info.

As well as over HTTP, Actuator publishes its endpoints as MBeans under the org.springframework.boot domain, although it is disabled by default since Spring Boot 2.2:

With that, JConsole, VisualVM or JMC connected to the process see the endpoints as invokable operations. It is still useful in traditional deployments on machines reachable over SSH, but in containers the HTTP route is the usual one: it does not require opening the RMI port nor wrestling with JMX's random port negotiation, notoriously hostile to NAT and to Docker.

Actuator is, quite literally, a service door into the application. With env you read secrets, with heapdump you download the entire heap with the Ribalta citizens' JWTs inside, with loggers you fill the disk and with shutdown you switch the service off. An Actuator exposed unauthenticated to the internet is not a bad practice: it is a breach. Search engines specialising in exposed devices index /actuator/env routes systematically, and they find results every single day.

The minimum checklist before every deployment to Ribalta:

1. exposure.include is an explicit list, never *, in the prod profile.
2. env, heapdump, threaddump, beans and configprops are excluded or restricted to ADMIN, and shutdown remains disabled.
3. There is a chain with EndpointRequest.toAnyEndpoint() and anyRequest() is not left without a rule.
4. management.server.port is a port closed at the firewall to external traffic, and show-details and show-values are not set to always.
5. Everything travels over TLS, because basic authentication sends the password encoded, not encrypted.

Exposing everything with include: "*" and forgetting about it. It works on day one, gets copied into the production profile and survives for years. Write the explicit list from the start.

Putting the database in liveness. It turns a PostgreSQL outage into a restart loop across every instance. liveness holds only what a restart fixes.

A slow HealthIndicator. It runs on every probe of every instance: a two-second query turns the health check into a performance problem and causes false DOWNs through timeouts.

A non-critical indicator that brings down the global health. The aggregate status is the worst of them all: if the payment gateway should not pull the instance out of the load balancer, take it out of the readiness group.

Confusing enabled with exposure. Disabling health "so it cannot be seen" also breaks the internal probes. What you want is to leave it enabled and not expose it, or expose it protected.

Tip: put Actuator on its own port from day one, because changing it later means touching load balancers, firewalls, alerts and manifests. And use EndpointRequest, never the literal path: "/actuator/**" stops matching the day somebody changes base-path, and security disappears without a single error in the log.

Tip: document your health groups — what the load balancer looks at, what the orchestrator looks at, what fires an alert — and write a test (06-04) that requests /actuator/env without credentials and expects 401: that way the exposure policy is defended by the build.

Configure Actuator for CicloUrbana to these requirements: management port 8081; exposure of health, info, metrics, loggers, flyway and the custom network endpoint; health details only for authorised users with the ADMIN role; availability probes enabled; a readiness group containing only the database; the disk space indicator disabled; and info with build and Git data. Write the YAML and the security chain, and explain what each URL answers to an anonymous client.

Write a HealthIndicator named connectionPool that queries the HikariDataSource from 04-02 through its HikariPoolMXBean and returns DOWN if more than three threads are waiting for a connection or if active connections exceed 90 % of the maximum, and UP otherwise, with the breakdown as details. Decide, with reasons, which health group it should go in.

A colleague has pushed this production configuration. Find every problem and propose the fix.

Solution 1

With the security chain from section 6, what an anonymous client sees on port 8081 is:

The difference between 401 and 404 is informative: 404 means "not exposed" and 401 means "exposed and protected". And it is worth remembering that these URLs are only reachable from the internal network, because port 8081 is not published externally.

Solution 2

Decisions. We inject HikariDataSource rather than DataSource, because the MXBean is Hikari-specific; the initial null is handled because the pool is created lazily and a very early probe may arrive first. In that case we return UNKNOWN and not DOWN, because UNKNOWN maps to 200 and a freshly started instance should not declare itself sick. The thresholds are constants for brevity; in the real project they would be @ConfigurationProperties (02-05) tunable per environment. The indicator is cheap: it only reads in-memory counters, without touching the database.

Which group it goes in. This one really is a good candidate for readiness, and it is the exact counterpoint to stationNetwork. A saturated pool means the instance cannot serve more requests properly, so withdrawing it from the load balancer is the right call: traffic goes to healthy instances and this one recovers by draining its queue. Putting it in liveness would be a serious mistake — a load spike would kill the very instances that are holding on. A final caution: if every instance saturates its pool at the same time, they all declare themselves REFUSING_TRAFFIC and the Ribalta network is left with none available; that is why the threshold must be high (90 %, not 60 %) and why an alert must fire long before the probe acts.

Solution 3

Seven problems, in order of severity:

There is an eighth problem, of form: the chain uses the literal path "/actuator/**", so if somebody changes base-path or enables management.server.port, the rule stops matching without any warning.

The corrected configuration is the one from solution 1 plus the chain from section 6. And there is an underlying lesson: problems 1 and 2 together let anyone switch off a city's bike network with a one-line curl. It is not a Spring Boot flaw — all of its defaults are safe — it is the result of having changed them one by one in search of convenience during development and having promoted the file to production. Hence the next module.

CicloUrbana now lets itself be questioned. You know what it means for an application to be operable and why that is a different property from being correct. You know the full catalogue of Actuator endpoints along with the risk of each one, you understand why only health goes out over HTTP out of the box and why that policy was born of a lesson learned the hard way. You know how to expose them with include/exclude, move them to a different base-path and, above all, separate them onto their own port so that the firewall — and not a rule in code — decides who talks to management. You have closed the security chain with EndpointRequest.toAnyEndpoint(), which does not expire when the path changes, leaving only health and info public and demanding ADMIN for the rest.

You have mastered /actuator/health from the inside: aggregation to the worst status, show-details, the automatic indicators and how to switch them off, and the piece that will be needed in Docker and in Kubernetes — the health groups and the liveness and readiness probes — with the rule that avoids the most expensive mistake: liveness holds only what a restart fixes. You have written StationNetworkHealth, which measures the health of the service and not just of the process, with the four cautions that separate a useful indicator from a harmful one, and you know how to change availability at will with AvailabilityChangeEvent to drain traffic from an instance without killing it. /actuator/info finally answers which version and which commit are deployed in Ribalta, /actuator/loggers raises the log detail without restarting, mappings and configprops serve as diagnostic tools, and you have the warning about env and secrets committed to memory. And you have written your own @Endpoint(id = "network") with read, write and delete operations.

One endpoint has been deliberately left unopened: /actuator/metrics. Actuator publishes it, but exploiting it — business counters and timers, @Timed, percentiles, MeterRegistry — is the job of 09-03, and its collection by Prometheus and its display in Grafana, the job of 09-04. Here we have built the instrumentation; there it will be read.

And now the problem that exercise 3 laid bare comes into view. All the configuration in this lesson — what is exposed, how many details are shown, whether shutdown is enabled — must be different on a developer's laptop and on the council's server. The same goes for the database (H2 versus PostgreSQL), for the log level, for Swagger, for ddl-auto, for CORS and for JWT expiry. Until now we have been changing them by hand in a single application.yml, which is exactly how a development configuration ends up deployed in production. The next lesson, Spring Boot Profiles, solves that with the principle of build once, deploy many: one artefact, several environments, no recompilation.