In 06-02 we compared the design of Aroma Store with that of CafeSocial and closed with an uncomfortable thesis: there is no universal REST design. Now comes the last matter, the one that separates a well-designed API from an API that is still alive three years later: what happens after you publish it. Because on launch day the API is yours; from the first integration onwards it belongs to your consumers, and every field you return becomes a promise. This lesson walks through the first three years of the Aroma Store API as a timeline with concrete milestones: the first incident, the post-mortem of the Christmas outage, the contract debt that keeps piling up, the migration to v2 with its twelve-month schedule and the silent maintenance work nobody sees but which holds up everything else.

1. The timeline of an API in production
2. Day one: what you watch and what is normal
3. Incident management: severities, mitigation and communication
4. The blameless post-mortem (complete document)
5. Listening to your consumers
6. Contract debt
7. Adding without breaking: a catalogue of safe changes
8. When a v2 is due and how to migrate in 12 months
9. Retiring features and the cost of what almost nobody uses
10. Continuous maintenance: dependencies, secrets, SLOs and cost
11. API inventory and zombie APIs
12. Governance with several teams
13. Common mistakes, exercises and conclusion

It helps to see the full cycle before going into each milestone. These are the moments that marked the first three years of https://api.aromastore.example/v1.

None of these milestones is a new project: they are all maintenance. And in an API with several external consumers, maintenance consumes considerably more cumulative effort than the initial build.

In month 0 the API is published with two in-house consumers: the SPA https://aromastore.example and the internal panel. All the instrumentation we set up in 04-07 (pino for structured logs, prom-client for metrics, traces with OpenTelemetry, /health/live and /health/ready) stops being an exercise and becomes the only place you can look.

The four signals you watch from minute one are the classic ones for a request-response service:

And there is a fifth signal almost nobody looks at on day one which turns out to be the most informative: the distribution of the 4xx.

The real output from the second day:

What is normal and what is not. The 401s are normal: the SPA retries with an expired token and renews it. The 404s on /coffees/{id} are too: there are old indexed links out there. What is not normal is 412 invalid_parameter on GET /v1/coffees: that is not a clumsy client, it is a badly explained contract. Inspecting the error details reveals the pattern: clients are sending ?roast=Medium with an initial capital, and our enum only accepts light|medium|dark. The documentation said so; the error message did not. The message was fixed —not the validation— and the 400s dropped to 20 a day.

Day one rule: a repeated 4xx error does not accuse the client, it describes a design or documentation defect in your API.

The first incident (month 2). With the launch of Aroma Mobile the first 429s appear. The app polls the cart every 5 seconds and the rate limiting from 04-04 starts returning rate_limit_exceeded with Retry-After. Diagnosis took twenty minutes because the Aroma-RateLimit-Remaining headers were not being written to the logs. The solution was not to raise the limit: it was to add an ETag to the cart (we already had one on /coffees, see 04-05) so that the polling would answer 304 Not Modified at almost no cost, and to publish in the integration guide that the recommended interval was 30 seconds.

In 04-07 we set an availability SLO of 99.9 % monthly for the read routes. That 0.1 % is 43 minutes of error budget per month. The budget is what turns a discussion of opinions ("is this serious?") into an arithmetic decision: if 60 % of the monthly budget has been consumed in three days, feature deployments are frozen and the team works on reliability until the rate recovers.

Alerts do not fire on instantaneous thresholds but on the burn rate of the budget: consuming the budget 14 times faster than is sustainable for 5 minutes is an immediate page; 6 times faster for an hour is a normal one.

It is the hardest rule for a technical team to internalise, because curiosity pushes towards the why. During an incident, the right order is:

1. Restore the service with whatever it takes: roll back the last deployment, turn off the feature flag (05-04), send traffic back to the previous colour in the blue-green, degrade a feature.
2. Preserve the evidence: capture traces, the Aroma-Trace-Id of failed requests, EXPLAIN output, metrics for the interval.
3. Afterwards, work out the root cause calmly.

A deployment rolled back in 4 minutes costs you a post-mortem; a deployment debugged live for 40 minutes costs you the quarter's error budget.

With external consumers —SwiftShip and CataBox— saying nothing is worse than being wrong. The pattern we adopted: an entry on the status page within 15 minutes even if you know nothing yet ("we are investigating errors when creating orders"), updates every 30 minutes, and a final note with a link to the post-mortem once it is published. No internal infrastructure details; yes to the observable impact and advice on retrying.

Month 7, 18 December. The Christmas campaign multiplies traffic by six. Two weeks earlier the ?origin= filter had been added to GET /v1/coffees —a textbook backwards-compatible change, three lines of code— but with no index on the corresponding column. With the catalogue grown and campaign traffic, the database saturates and the API becomes unreachable for 34 minutes.

This is the document as it ended up in docs/incidents/. Notice what it does not contain: any name attached to the cause.

The verification of action 1, so that it is clear what gets checked:

Blameless does not mean ownerless. The actions have an owner and a deadline; the cause has no owner. The person who added the filter without an index did what the system allowed them to do: there was no checklist, no load test, no alert. The system failed, and the system is what gets fixed.

From month 9 onwards, the question stops being "does it work?" and becomes "what are they actually doing with it?". Three sources:

Usage analytics per endpoint and per client. Every JWT carries a client_id (04-02). The request metric is labelled with that identifier, but carefully, because of cardinality: labelling by route with parameters (/v1/coffees/cof_001) creates one time series per coffee and blows up Prometheus's memory. You label by route template and by client, which are small, bounded sets.

Which fields nobody uses. A REST API returns the full representation, so you do not know what the client reads… unless you ask. Two cheap techniques: (a) measure the use of the projection parameter if you have one (?fields=), and (b) ask directly in the annual integrators' survey. At Aroma Store that is how we discovered that tastingNotes was only consumed by the SPA and CataBox, and that the _links.self field on each collection element was used by absolutely nobody: the clients built the URLs by concatenation. An uncomfortable data point that went into the HATEOAS self-critique of 06-01.

Recurring 4xx errors per client. It is the best documentation to-do list in existence:

And the support channel: an [email protected] address that reaches a human, with a public commitment to reply within 2 working days. Every ticket is tagged as a defect, documentation or feature request; documentation ones are closed by editing the openapi.yaml, never by replying only over email.

Month 11. The team makes a list of "things we would fix if we were starting today" and realises that none of them can be fixed. That is contract debt: published decisions that are no longer the best ones, but which hold up real consumers.

It accumulates for three reasons, and none of them is negligence:

• The domain changes. When roast was designed with three values, filter roast did not exist in the catalogue.
• The standard changes, or you get to know it better. application/problem+json (RFC 9457) existed, but we opted for a format of our own; today it would be the obvious choice.
• You get it 80 % right. The {"data": [...], "total": n} envelope worked well, but it makes every response more expensive and confuses clients that expect a bare array.

In 06-01 we already criticised four decisions. Now they have a price:

The artefact that stops this being forgotten is the contract debt register, versioned alongside the code in docs/contract-debt.yaml. It is a living document: it is reviewed every quarter and it is the raw material for the decision about v2.

Notice the note on CD-007: part of the debt can be paid off without breaking anything if you think in terms of content negotiation (02-05). Not all debt demands a new version.

Before contemplating a v2, you have to exhaust what can be done inside v1. We saw the general rule in 02-07: adding is safe, removing and changing meaning are not. The detail matters far more than it looks.

* The asterisk on the new field. "Adding a field breaks nothing" is only true if the clients are tolerant readers: if they ignore what they do not know. A client that validates the response against a strict schema with additionalProperties: false, or that uses a language that fails when deserialising unknown fields, breaks on a new field. That is why the Aroma Store integration guide says this on its first page:

The dangerous case: tightening a validation. In month 14 it was discovered that tastingNotes accepted text of any length and somebody had stored 40 KB in it. The temptation is to add .max(500) to the Zod schema (03-02) and deploy. That turns previously valid requests into 422 invalid_data: it is a breaking change even though it does not touch a single field name. The correct procedure has two stages.

If after six weeks the counter is still climbing, phase 2 is not deployed: you call the client. The date moves; the contract is not broken by surprise.

A v2 is not an achievement, it is an invoice: two codebases or two translation layers, two test suites, two sets of documentation, two versions of the openapi.yaml, and consumers who will take months to move. Honest criteria:

Signs that a v2 is NOT due:

• The change can be made by adding (section 7).
• It annoys the team but not the consumers (tastingNotes on its own justifies nothing).
• It is a documentation problem dressed up as a design problem.
• There are fewer than three high-impact debts accumulated.

Signs that it IS due:

• Several high-impact debts that can only be resolved by breaking, and which cause incidents or recurring support.
• The domain model has genuinely changed (Aroma Store started selling subscriptions, and a recurring order does not fit the current /orders resource).
• Non-negotiable security changes.
• The cost of maintaining the workarounds exceeds the cost of migrating.

Month 18. Aroma Store decides on the v2 with four high-impact debts and a new domain (subscriptions). An ADR is written in docs/decisions/0031-launch-v2.md with the rejected alternative (keep patching v1 with negotiated problem+json and a /subscriptions resource hanging off v1) and why it is rejected.

Recall the published policy: versioning only in the path, and v1 and v2 coexist for at least 6 months. In practice, for an API with external partners, six months is the legal minimum and twelve is the reasonable minimum.

It is the document that decides whether the migration takes the consumer two days or two months. No prose: literal equivalences.

Picking up 02-07, this is how v1 responds from month 1 onwards:

With the version label on the counter from section 5, the question answers itself:

That second result is the reason you never switch off a version by calendar without looking at the metrics: a forgotten consumer always turns up. And the third is a reminder that with mobile apps you do not control when the user updates: the old app will stay alive for months.

A brownout is a brief, announced switch-off of v1: during the window, every request receives a 410. Its function is not technical, it is psychological: it turns a distant date into a real incident in the consumer's environment, which is the only thing that shifts priorities.

And the final switch-off, now without Retry-After:

410 Gone and not 404: the difference communicates that the resource existed and was removed on purpose, and it stops a client thinking they have got the route wrong.

Year 2. The endpoint GET /v1/coffees/{id}/pairings, added in month 4 at marketing's request, receives 40 requests a month from a single client. Its cost is not zero:

• It appears in openapi.yaml, so it has to be documented and linted with Spectral.
• It has tests that run on every CI build and that sometimes fail because of test data.
• It has a table with its migration, which has to be dragged along on every schema change.
• It blocks decisions: any refactor of the Coffee resource has to take it into account.
• It takes up mental space in every design review.

Retiring one specific feature follows the same protocol as a version, in miniature: announcement, Deprecation/Sunset on that route only, contact with the single consumer, and a 410. It took three months. What you must never do is delete it because "almost nobody uses it": that "almost" is a company that depends on it.

Recurring work that produces no features and without which the API degrades on its own.

Dependencies and CVEs. An automated weekly audit in the ci.yml from 05-04:

The agreed policy: a critical, exploitable vulnerability in a dependency, patched within 48 h; high, within 7 days; the rest, in the monthly maintenance window.

Node version. Node 20 reaches end of support and you have to jump to the next LTS. It is an invisible change for the consumer and a dangerous one for you: it is done by canary (05-04), with 5 % of the traffic for 48 h, comparing p95 and error rate between the two groups. It goes into the calendar before support expires, not after.

Rotating secrets and signing keys. Three different clocks:

SLO review. Every six months. If the 99.9 % SLO has never been anywhere near breached in a year, either it is badly measured or it is too lax. If it is missed every month, either it is not achievable with the current architecture, or the error budget is being used as an excuse. At Aroma Store, the latency SLO for /coffees was tightened from 400 ms to 300 ms in year 2 after the indexing work.

Security review. An annual pass over the OWASP API Top 10 (04-01) against the real state of things: object-level authorisation on every new endpoint, consumption limits, data exposure. It is documented like any design review.

Infrastructure cost as a design signal. The API's bill is a design metric dressed up as a financial one. If an endpoint costs disproportionately much, there is almost always a contract decision behind it: GET /v1/orders with no mandatory pagination returning thousands of elements, no ETag on a heavily queried resource, or a client polling where there should be a webhook. Before scaling the infrastructure, review the contract: it comes out cheaper.

In 04-02 we defined ownership for every API and in 05-06 we set up the developer portal with its inventory. Their real usefulness shows up right now: during the migration to v2 a consumer called erp-integration appeared that nobody remembered authorising. It existed, it worked and it was in production.

A zombie API is an API (or a version, or an endpoint) that still responds, consumes resources and presents attack surface, but has no identifiable owner. They are detected by cross-referencing three sources: the declared inventory, the real traffic metrics and the register of issued credentials. Any row that appears in one source and is missing from another is an alarm.

Every inventory entry must answer, as a minimum: who maintains it, who consumes it, which version is current, what retirement date it has if it is deprecated, and where its openapi.yaml lives. No owner, no deployment: it is the only way to stop the inventory ageing.

Year 3. There is no longer one team, there are three: catalogue, orders and subscriptions. Without governance, in six months you have three APIs that look as if they came from three different companies: one with snake_case, another with plain-text errors, another paginating with page/size.

Aroma Store's governance rests on four pieces, all of them already familiar:

• The style guide (02-01) as a written standard, not as a recommendation. It is the reference that settles arguments in a review.
• The design review: before writing any code, the team presents the proposed fragment of openapi.yaml. Ten minutes with two people from other teams save months of debt. The contract is reviewed, not the implementation.
• Automated gates in CI (05-04/05-05): Spectral validates the style and oasdiff detects breaking changes. What is automated is not up for discussion, and that is precisely its virtue.
• ADRs in docs/decisions/: every structural decision with its context, alternatives and consequences. It exists so that in two years' time nobody "fixes" something that was that way on purpose.

And the complete human flow of a contract change:

• Debugging live instead of mitigating. Every minute of diagnosis during an S1 is paid for out of the error budget. Roll back, turn off the flag, and investigate afterwards.
• Post-mortems that end in "lack of attention". If the action cannot be closed with a verifiable link, it is not an action. And if the document names a person as the cause, next time nobody will tell you what happened.
• Confusing "blameless" with "consequence-free". The actions have an owner and a deadline, and they are reviewed in the following month's retrospective.
• Believing that adding a field never breaks anything. It is only true with tolerant-reader clients. Publish that requirement in the integration guide from day one.
• Tightening validations with no warning phase. It is the breaking change that slips through most often because it does not touch the visible schema. Measure first, reject afterwards.
• Labelling metrics by concrete path or by id. Cardinality explodes and you lose your observability exactly when you need it. Route templates and closed sets.
• Switching off v1 by calendar without looking at per-version metrics. A forgotten consumer always turns up. Metrics override the calendar; an announced Sunset is never brought forward, but it can be pushed back.
• Launching a v2 out of internal discomfort. If the pain is felt only by your team, solve it internally. A new version is an invoice paid by every one of your consumers.
• Treating 4xx as the client's fault. They are your documentation to-do list, sorted by impact.
• A final tip: publish the deprecation calendar somewhere stable and honour it even when it hurts. A Sunset brought forward destroys more trust than a 34-minute incident.

For each proposed change to Aroma Store's v1, state whether it is backwards compatible, whether it is breaking, and what the correct strategy would be:

1. Adding the certifiedOrigin field (boolean) to the Coffee representation.
2. Adding the value filter to the roast enum in responses.
3. Changing the default limit of GET /v1/coffees from 20 to 50.
4. Starting to reject a negative priceMin with 422 invalid_data.
5. Adding the optional filter ?certified=true.

An internal invoicing API has accumulated these debts: (a) the total field is in euros as a decimal number and causes rounding errors; (b) GET /invoices does not paginate and returns up to 4,000 elements; (c) the name invoice_id uses snake_case while the rest of the API uses camelCase. There are only two consumers, both internal. Would you launch a v2? Justify it with criteria, not with taste.

A Friday afternoon deployment introduced a bug in token renewal: for 18 minutes, every request from Aroma Mobile received 401 not_authenticated. It was detected because a user posted on social media; nobody on the team saw any alert. Write five actions for the post-mortem, each with a type (corrective, detection, preventive, organisational, communication) and a verifiable closing criterion.

Exercise 1

Exercise 2

No, not yet. Debt by debt:

• (a) total as a decimal in euros. It is real and serious (rounding on money), but it is solved by adding: publish totalCents as a new field, document it as preferred, deprecate total in the documentation and measure its use. It does not require a version.
• (b) GET /invoices with no pagination. It can be paginated in a backwards-compatible way: accept limit/offset, and as long as they are not sent, return the current behaviour. With two internal consumers, you can also negotiate a maximum limit with prior notice. It does not require a version.
• (c) invoice_id in snake_case. It is aesthetic discomfort. Solution: also emit invoiceId (both fields coexist), document the new one, and remove the old one when the usage metrics allow it or when a v2 motivated by something else comes along.

Besides, with two internal consumers the coordination cost is tiny compared with that of maintaining two versions. The applicable criterion: none of the three debts forces a break, therefore there is no v2. What is appropriate is to open three entries in the contract debt register with v2_candidate: true and review them every quarter.

Exercise 3

Notice that action 2 is the most valuable: the bug lasted 18 minutes, but the real problem is that detection came from outside. An incident discovered by a user on social media is, above all, an observability failure.

Designing an API is a bounded exercise; maintaining it is an open-ended commitment. In this lesson we have walked through three years of the Aroma Store API and seen that almost all the work after launch consists of protecting the people who already trust you: watching the right signals from day one, mitigating before diagnosing when something breaks, writing post-mortems that fix the system instead of hunting for culprits, listening to what the 4xx and the usage analytics are telling you, recording contract debt instead of pretending it does not exist, exhausting everything that can be added without breaking, and —only when there is no alternative left— planning a v2 with its schedule, its Deprecation and Sunset headers, its brownouts and its final 410. On top of that comes the maintenance nobody applauds: CVEs, Node versions, key rotation, SLO review, an inventory with no zombies and a governance built on Spectral, oasdiff and ADRs that keeps things coherent once there is no longer a single team.

The underlying conclusion is simple and demanding at the same time: an API is a long-term commitment to whoever consumes it. Every published field is a promise, every announced Sunset is a contract, and the trust you build over three years can be lost with one "harmless" change deployed on a Friday afternoon. The quality of an API is not measured on launch day, but by how easily an integrator can keep working with it when nobody from the original team is still at the company.

That closes our run of case studies and analysis. Now it is your turn: in 06-04, Final Project: Design and Build Your Own RESTful API, you will apply everything you have learned in phases —contract design, implementation, security, documentation and deployment— to a domain of your own, with a clear rubric for assessing yourself. Everything we have done so far has been preparing you for that brief.