The previous lesson left a promise unkept: we said that one service calls another, that the JWT and the trace identifier have to be forwarded, that a circuit breaker is needed and that one service's outage must not cascade. None of that have we written yet. And it is needed even if CicloUrbana is never split up: as soon as the application calls the Ribalta payment gateway — an external system, with its latency, its outages and its 500 errors — exactly the same problems appear.

This lesson solves them with code. First the client: what options Spring offers, how RestClient is configured to talk to the gateway, why timeouts are the most expensive setting to forget, how the trace and the token are forwarded, and how a remote error is translated into a domain exception. Then resilience: the five Resilience4j patterns one by one, the order in which they are applied, and the decision — a business one, not a technical one — about what CicloUrbana does when the gateway does not answer. And finally, how all of this is tested by simulating a service that is slow, down and broken.

1. Spring's HTTP clients
2. The payment gateway client
3. Timeouts: the setting you cannot forget
4. Interceptors: forwarding the trace and the token
5. Translating remote errors into the domain
6. The declarative interface with @HttpExchange
7. Resilience4j: installation and configuration
8. Retries
9. Circuit breakers
10. Rate limiter, bulkhead and time limiter
11. The order of the decorators
12. Graceful degradation
13. Observability of resilience
14. Testing failures with WireMock
15. Asynchronous communication: the bare minimum
16. Common Mistakes and Tips
17. Exercises

Two useful clarifications. RestTemplate is neither deprecated nor going away, but it receives no new functionality; migrating to RestClient is almost mechanical because they share the infrastructure (ClientHttpRequestFactory, interceptors, converters). And using WebClient in a servlet application just to call a service and then .block() is a common anti-pattern: it drags in the whole reactive stack to obtain RestClient's behaviour with more complexity. The choice for CicloUrbana: RestClient as the foundation and @HttpExchange to expose it as a domain interface.

We start from the GatewayProperties of 02-05, extended with the timeouts:

Four decisions. It is a bean, not a RestClient created on every call: creating one per invocation throws away the connection pool and adds a full TLS handshake to every request. The API key goes as a default header, so no method can forget it — and never in the URL, where it would end up in access logs and in intermediate proxies. The User-Agent identifies CicloUrbana in the gateway's logs, which saves arguments when an incident has to be investigated with the provider. And there is one client per destination: if a mapping service appears tomorrow, it will have its own bean with its own timeouts and its own circuit breaker.

Not configuring the timeouts is the most expensive mistake in the whole lesson, and it is easy to make because it produces no symptom until the day the remote system degrades.

Without them, a request to a gateway that accepts the connection and never answers waits indefinitely, and the thread serving the citizen is blocked. With two hundred citizens trying to pay, all two hundred Tomcat threads are held, and from that point the whole of CicloUrbana stops responding: no station queries, no rental starts, not even /actuator/health if it shares the port. A third party's problem has turned into a total outage of the municipal service.

The rule for setting them: start from your own endpoint's latency budget. If POST /api/v1/rentals/{id}/finish must answer in under a second, it cannot afford a thirty-second read timeout; it is more useful to fail fast and degrade (section 12) than to wait for an answer that is no longer any use to anyone. And a warning about retries: the effective wait multiplies — with a 5 s read and three attempts, the worst case is 15 s plus the pauses — which is why section 11 insists on the order of the decorators and on an overall limit.

An interceptor runs before every outgoing request and is the natural place for cross-cutting headers:

With this header, the trace identifier from the TraceFilter of 03-06 travels to the remote system and appears in its logs: when the gateway provider asks about a specific transaction, the reference is the same on both sides. In 09-06, Micrometer Tracing will do this in a standard way with the W3C traceparent headers. To forward the JWT to another CicloUrbana service — the token relay from 07-05 — the interceptor reads the security context:

Never towards a third party: this interceptor must only be registered on clients pointing at our own services; sending a Ribalta citizen's token to the external payment gateway would be a credential leak, and that is why the gateway uses its own apiKey. And SecurityContextHolder is a ThreadLocal: if the call goes out from an asynchronous thread (07-03), there will be nothing here unless the executor is wrapped in DelegatingSecurityContextAsyncTaskExecutor.

Untreated, a 402 Payment Required from the gateway turns into an HttpClientErrorException that rises to the GlobalExceptionHandler of 03-06, which does not recognise it and returns a generic 500. The citizen is told "internal error" when what is really happening is that their card has no funds.

And in the global handler, three new rules consistent with the ProblemDetail of 03-06:

Two principles lie behind that table. The transport exception must not escape: whether the client uses RestClient, Feign or a raw socket is an implementation detail, and HttpClientErrorException in a domain service's signature couples the application to the library. And the status code towards the citizen is not the remote one: the gateway returning 400 because we sent a bad field does not mean the citizen's request was incorrect.

Be careful, too, about what gets logged: the body of a payment gateway's response may contain card data, so it must never be dumped in full to the log, still less into the message of the exception that reaches the client.

With more than two or three operations, a bare RestClient scatters URIs and types through the code. The declarative interface gathers them into a readable contract:

Spring generates the implementation, which underneath uses the RestClient from section 2 with its timeouts, its interceptors and its error translator. The advantages over the bare client: the interface is the contract and reads at a glance, the domain service depends on an abstraction of our own rather than on an HTTP library, and in tests it is replaced by a Mockito mock (06-03) without starting anything.

@HttpExchange is the standard equivalent of OpenFeign without depending on Spring Cloud, with an almost one-to-one correspondence (@FeignClient → @HttpExchange, @GetMapping → @GetExchange).

spring-boot-starter-aop is not optional: Resilience4j's annotations work through proxies, with the very same self-invocation traps as @Transactional (04-07) and @Async (07-03) — an annotated method called with this. is protected by nothing. All the configuration is YAML, with an inheritable configs.default plus per-instance settings:

ignoreExceptions with PaymentDeclinedException is the most important line in the whole block. A card with no funds is a correct response from the gateway: it must not be retried nor counted as a failure for opening the circuit breaker. If it were counted, a thousand citizens with expired cards would be enough to leave the entire Ribalta network unable to charge.

A retry solves transient failures: some packet loss, a 503 during the provider's deployment, a momentary spike.

Two concepts you have to understand.

Exponential backoff with jitter. Retrying every 500 ms on a fixed schedule has a perverse effect: if a thousand requests fail at once because the gateway restarted, all thousand retry at once and knock it over again — the retry storm. Exponential backoff separates the attempts (500 ms, 1 s, 2 s) and jitter (enableRandomizedWait) adds random variation: with randomizedWaitFactor: 0.5, the second wait falls between 250 and 750 ms.

Idempotency: what is safe to retry. This is the rule that decides whether the retry helps or duplicates charges, and it links back to the HTTP method semantics of 03-03:

The dangerous case is an expired timeout: you do not know whether the operation ran or not — the response was lost, but the charge may have been applied. The solution is the idempotency key: CicloUrbana sends with each charge a key derived from the rental (charge-rental-42) and the gateway, if it has seen it before, returns the original result without charging again. Without that mechanism on the server side, retrying a charge POST is unacceptable.

Retries are for passing failures. When the failure is sustained, retrying makes things worse: it consumes threads, lengthens responses and hammers a system that is already down. The circuit breaker cuts that feedback loop.

The OPEN state is where the value lies: CallNotPermittedException is thrown immediately, so you stop waiting five seconds for every request and fail in microseconds, without touching the downed system.

Four rules about fallbackMethod, all of them a source of frequent mistakes:

1. The same signature plus a final Throwable parameter. If it does not match, Resilience4j does not find it and the original exception escapes, silently, at run time.
2. It must be in the same class and it may be private.
3. There can be several, specialised by exception type; the most specific one wins.
4. It must not fail or be slow. A fallback that calls another remote service reintroduces the very problem it was there to solve.

The two thresholds. failureRateThreshold: 50 opens the circuit at half failures, but slowCallRateThreshold: 80 with slowCallDurationThreshold: 3s is just as important — a slow service does more damage than a downed one, because it holds threads without giving anything back — and minimumNumberOfCalls: 10 stops two failures during a quiet moment opening the circuit.

@RateLimiter protects against exceeding a quota: the Ribalta gateway allows 50 requests a second and going over produces 429 and, in some contracts, a surcharge. With timeoutDuration: 200ms, a call that does not get permission within that window throws RequestNotPermitted. It is different from Bucket4j in 05-05: that one limited what comes into CicloUrbana; this one limits what goes out to the third party.

@Bulkhead — named after a ship's watertight compartments — limits the concurrent calls to a resource: with @Bulkhead(name = "paymentGateway", type = Bulkhead.Type.SEMAPHORE) and maxConcurrentCalls: 10, at most ten threads wait on the gateway at once; the rest fail fast instead of piling up. It is what stops a slow third party exhausting Tomcat's pool: without a bulkhead, two hundred concurrent requests to a five-second service hold all two hundred threads and bring the whole of CicloUrbana down. It is the same principle we applied in 07-03 by giving e-mail its own executor.

@TimeLimiter imposes an overall cap, and it only works on methods that return CompletableFuture, because it needs to be able to cancel:

In a blocking application such as CicloUrbana, the RestClient timeouts (section 3) cover the usual case; @TimeLimiter adds the cap for the complete operation, retries and their pauses included, which is precisely what the individual timeouts do not bound.

When several annotations apply to the same method, the order matters a great deal. The one Resilience4j applies by default, from the outside in:

It reads like this: first you ask for a place in the bulkhead; inside, the time limiter bounds the whole operation; then the rate limiter grants permission; then the circuit breaker decides whether it is attempted at all; and the retry sits innermost, wrapping only the actual call. That Retry sits inside CircuitBreaker is the key decision:

The default order is the right one in almost every case, and it can be changed with the ...retryAspectOrder and ...circuitBreakerAspectOrder properties if needed.

Here comes the central question of the lesson: what does CicloUrbana do when the payment gateway does not answer?

CicloUrbana's decision is the second: finish the rental and defer the charge, which is what deferCharge in section 9 does. The reasoning is a business one, not a technical one: the citizen has already returned the bike — that is a physical fact that happened — and blocking the operation does not undo it, it only leaves a dock out of use and a citizen stranded. The amount is small, the citizen's identity is known and the charge can be retried later with a scheduled task (07-03) over the queued PendingCharge entries.

And that decision is not the developer's to take: it would change completely if the amount were a thousand euros, if the citizen were anonymous or if regulation required payment up front. The general rule is that the fallback is a product decision and the code merely implements it, and the questions to take into that conversation are always the same: can we serve slightly stale data? accept the operation and complete it later? offer reduced functionality? or is this operation genuinely indispensable?

A catalogue of typical degradations: serving the last cached response (09-02) when the data tolerates being stale; returning a safe default value; queue and confirm, as here; or disabling the feature while keeping the rest.

A circuit breaker that opens without anyone noticing is as bad as not having one: the system degrades in silence. With registerHealthIndicator: true, each circuit appears as a health component in /actuator/health (07-01), and Resilience4j adds endpoints of its own:

Careful with the health indicator. If the gateway's circuit goes into the readiness group, opening would pull the instance out of the load balancer — precisely the scenario we warned about in 07-01. An open circuit breaker means "the third party is down and I am degrading correctly", not "I am sick": it should be visible in /actuator/health and fire an alert, but not be in readiness.

The metrics Resilience4j publishes to Micrometer — resilience4j_circuitbreaker_state, ..._calls, resilience4j_retry_calls — are exploited in 09-03 and displayed in Grafana in 09-04, with two minimum alerts: circuit open for more than a minute and retry rate above normal, which is the early warning of a degradation before the circuit gets as far as opening.

All of the above is only worth anything if it is tested, and you cannot test it by asking the provider to go down. WireMock starts an HTTP server that pretends to be the gateway and behaves however it is told to (dependency org.wiremock:wiremock-standalone, scope test).

The assertions that give these tests their value are the ones that do not look at the happy path: verify(0, ...) with the circuit open proves that not a single request went out, which is the essence of the pattern, and the timing check proves that a third party taking thirty seconds does not drag CicloUrbana down with it. For the retry, WireMock offers stateful scenarios (inScenario(...).whenScenarioStateIs(STARTED)...willSetStateTo(...)), which let you simulate "fail once and then work" and check it with verify(2, ...); exercise 2 develops this.

Two essential details: reset the circuit breaker between tests, because its state is global to the context and one test would leave the circuit open for the next; and lower the thresholds in application-test.yml (minimumNumberOfCalls: 5, waitDurationInOpenState: 1s) so that the suite takes seconds rather than minutes. The alternative to WireMock is MockServer on Testcontainers, going back to 06-05, useful when you want the same container-based approach as the rest of the suite.

When the answer is not needed in order to carry on (07-05), messaging turns an outage into a delay. With spring-kafka — or spring-boot-starter-amqp for RabbitMQ — publishing and consuming are a handful of lines:

Publishing is symmetrical — kafka.send("ciclourbana.rentals", String.valueOf(event.rentalId()), event) — with the rentalId as the partition key so that the events of a single rental keep their order.

The dead-letter queue deserves a paragraph. Without it, a message that always fails — an event with a corrupt field — is retried indefinitely and blocks the whole partition: no later message is processed. With it, the problematic message is set aside in a .DLT topic for manual review and the flow carries on. And that topic has to be watched: a dead-letter queue nobody looks at is a mail folder nobody opens.

Not configuring the timeouts. The most expensive mistake: a slow third party exhausts the threads and brings the whole of CicloUrbana down.

Creating an HTTP client on every call. The connection pool is lost and every request pays a full TLS handshake.

Retrying a non-idempotent POST. An expired timeout does not mean the operation did not run: it charges twice.

Counting business errors as circuit failures. A thousand cards with no funds would open the circuit breaker and leave the whole network unable to charge. ignoreExceptions.

A wrong fallbackMethod signature. Resilience4j does not find it and the original exception escapes, with no compilation error.

A fallback that calls another remote service. It reintroduces exactly the problem it came to solve.

Letting the transport exception escape. HttpClientErrorException in a domain service couples the application to the HTTP library.

Forwarding the citizen's JWT to a third party. It is a credential leak: outwards goes the system's key, not the user's.

Putting the circuit breaker in the readiness group. Opening means the third party is down, not that the instance is sick.

Tip: one client, one circuit breaker and one bulkhead per destination, because sharing them makes one slow third party affect calls that have nothing to do with it; and set the timeouts from your endpoint's latency budget, not from how long the remote system usually takes.

Tip: test the failures, not just the happy path. A circuit breaker never seen to open in a test is a hypothesis, not a protection.

rentals-service needs to ask stations-service whether bike RB-0142 is available before starting a rental. Design the complete client: the @HttpExchange interface, a RestClient with timeouts and JWT forwarding, the Resilience4j configuration and the fallback. The query is read-only and the rental endpoint must answer in under a second. Justify each value and decide what should happen if stations does not answer.

Write the WireMock tests that prove, for the client from the previous exercise: that a one-off 503 is retried and eventually works; that after sustained failures the circuit opens and no traffic goes out; that a response taking 20 seconds does not block the citizen; and that a 404 (a non-existent bike) does not count as a circuit failure. State the application-test.yml configuration required.

Find every problem in this code and rewrite it.

Solution 1

Justification of each value. The budget is one second for the whole endpoint, and this query is only part of it: hence 300 ms to connect and 600 ms to read, with two attempts at most and a 100 ms pause, which bounds the worst case at around 1.6 s before the fallback kicks in — already over budget, which is why the circuit is aggressive. slowCallDurationThreshold: 500ms is deliberately low: on the critical path, slow is equivalent to broken. waitDurationInOpenState: 15s is short because it is one of our own services, which recovers quickly after a deployment. ignoreExceptions with the non-existent bike stops a legitimate 404 counting as a failure. And the bulkhead of 30 concurrent calls protects Tomcat's pool.

What happens if stations does not answer, and why. Here the decision is the opposite of the payments one: the rental is refused with a 503. The difference is that the deferred charge had a possible compensation — it is charged later — whereas starting a rental without knowing whether the bike is available has none: two citizens could take the same bike, or somebody could rent one that is in the workshop. When the missing information is what makes the operation correct, the correct degradation is to refuse. It is the same question as section 12 with a different answer, and that is why it is decided case by case.

Solution 2

Comments. The first uses WireMock's stateful scenarios, the only way to simulate "fail once and then work", and the verify(2, ...) is what proves the retry; without it, the test would pass just the same with no retry configured at all. The second holds the most valuable assertion in the exercise: verify(0, ...) after resetRequests() proves that with the circuit open not a single request goes out, which is precisely the point of the pattern. The third checks that 20 seconds of delay are cut short in under 3, which proves the read timeout is configured. And the fourth is the subtlest: it verifies that a business error does not degrade the system; without ignoreExceptions, six queries for a non-existent plate would take the whole Ribalta network out of service.

Every test starts from stations.resetAll() and registry.circuitBreaker("stations").reset() in a @BeforeEach; without that, the execution order would determine the result and the suite would be flaky.

Solution 3

Nine problems:

The corrected version is the one in the body of the lesson: RestClient as a bean with the timeouts from GatewayProperties, the key in a default header, the @HttpExchange interface, the defaultStatusHandler that translates into PaymentDeclinedException / GatewayUnavailableException, @CircuitBreaker + @Retry + @Bulkhead with ignoreExceptions over the business errors, the idempotency key charge-rental-{id} on every request, and a deferCharge(Long, BigDecimal, Throwable) that queues the charge and returns ChargeResult.deferred() — never null.

Module 7 ends and CicloUrbana has changed in nature. It started out as a correct application: well built, well tested and completely incapable of living outside a developer's laptop. It ends as an operable application. Actuator answers whether it is healthy, which version is running and what is failing, with availability probes that an orchestrator understands. Profiles make the same artefact — the one that passed ./mvnw verify — serve both the laptop and the council's server without recompiling, with the secrets outside the repository. Scheduled tasks expire Ribalta's forgotten rentals and recalculate occupancy, coordinated across instances; asynchronous execution sends the confirmation e-mail without making the citizen wait, with the trace and the identity travelling to the other thread. The container image packages the application, its JRE and its time zone into a reproducible artefact, with layers that make rebuilding a matter of seconds, an unprivileged user and a docker-compose.yml where PostgreSQL and the application wait for each other through their health checks. And this last lesson has added the missing piece: CicloUrbana now knows how to talk to the outside world without dying in the attempt — timeouts that bound the wait, retries with backoff and jitter only where retrying is safe, a circuit breaker that stops hammering what is down, bulkheads that stop a slow third party exhausting the threads, and a graceful degradation that is a business decision, written in code and demonstrated with tests that fake the outage. It also knows when it makes sense to split into microservices and, more valuable still, when it does not. What remains is no longer to build, but to deliver: the Ribalta network works, is observable, holds up and is packaged, but it still lives on development machines. Module 8, Deploying Spring Boot Applications, finally puts it into the citizens' hands: what deploying means and what has to be decided beforehand, a first simple platform with Heroku, real infrastructure on AWS, orchestration with Kubernetes — where the probes from 07-01 and the image from 07-04 finally fall into place — and a continuous integration and delivery pipeline that takes every commit from the repository to the city without anybody touching a server by hand.