If you ask AlpinaShop's management what level of service they expect from the shop, the answer will be "that it never goes down". It is an understandable answer and a completely useless one, for three reasons: it cannot be met, it cannot be measured, and it does not let you decide anything.

It cannot be met because 100 % availability does not exist: hardware fails, the customer's network fails, the DNS provider fails, a deployment fails, a person makes a mistake. It cannot be measured because nobody has defined what counts as "down". And it does not let you decide anything because if the objective is perfection, any investment in reliability is justified and none is ever enough, which in practice means decisions get taken on intuition.

The previous lesson ended by pointing at exactly that: AlpinaShop pays for Cloud SQL's high availability, has duplicated the VPN tunnels and keeps the global load balancer without any declared objective that justifies those three decisions in numbers. And since 07-04 it has been carrying a debt rated critical: nobody has ever checked that the backups can be restored.

This lesson turns reliability into engineering. By the end you will know how to define SLIs, SLOs and SLAs precisely, calculate and use an error budget to decide when you deploy and when you stop, create SLOs and burn rate alerts in Cloud Monitoring, choose the level of high availability for each component knowing what it costs and what it gives you, design against real failure modes, and have a disaster recovery plan with RTO and RPO — rehearsed, not written.

1. What reliability means and why perfection is not an objective
2. SLI, SLO and SLA: the three acronyms everybody mixes up
3. Choosing good SLIs for AlpinaShop
4. AlpinaShop's SLOs, with their real queries
5. Error budget: what it is and what it is really for
6. SLOs in Cloud Monitoring and burn rate alerts
7. Tiered high availability architectures
8. Failure modes and their mitigation
9. Disaster recovery: RTO, RPO and strategies
10. AlpinaShop's recovery plan
11. Backups that have not been tested do not exist
12. Operational practice: on-call, runbooks and post-mortems
13. Chaos testing at a sensible scale

Reliability is the probability that a system does what is expected of it over a given period. It is not a binary property: it is a continuous quantity that you choose, pay for and manage.

The curve to keep in your head:

Every additional nine multiplies the cost, and not linearly. Going from 99 % to 99.9 % is usually a matter of good practice. Going from 99.99 % to 99.999 % requires active-active multi-region, 24×7 on-call, dedicated engineering and accepting that you deploy much more slowly.

And the other half of the equation, the one that almost never gets calculated:

With 1,200 orders a month and an average basket of €65, going from 99.9 % to 99.99 % would save AlpinaShop around €70 a month in orders not lost — and it would cost active-active multi-region, that is, several hundred euros a month plus the engineering work.

The uncomfortable, correct conclusion: for AlpinaShop, 99.99 % would be a bad investment. And being able to say that with the numbers in hand, instead of chasing nines out of inertia, is one of the most valuable things an engineer can contribute.

With two caveats that need to be made explicit:

1. Not all time is worth the same. Forty minutes of downtime in the small hours of February costs almost nothing; forty minutes on the Friday of the autumn campaign can cost twenty times more. Availability matters weighted by traffic.
2. There are costs that are not lost orders: reputation, customers who do not come back, support, and — in a long incident — the wear on the team.

The relationship between the three, which is what has to sink in:

Three rules that avoid almost every mistake:

1. The SLO is always stricter than the SLA. If you promise 99.5 % and your internal objective is 99.5 %, you will breach the contract half the time. The usual practice is a generous margin: SLA 99.5 %, SLO 99.9 %.
2. The SLI is measured from where the user is, not from where it is convenient. A server responding perfectly while the load balancer returns 502 has an SLI of 100 % and a furious user.
3. An SLO with no consequence is an ornament. If nothing changes when you miss it, it is not an objective: it is a number on a dashboard.

And a precise definition of an SLI that resolves most of the doubts:

SLI = good events / valid events

Everything difficult lies in defining those two terms. Does a 404 count as "bad"? (No: the user asked for something that does not exist.) And a 429 from the rate limiter? (It depends: if it is an attack, no; if it is a legitimate customer, yes.) And a request from a crawler? (It probably should not be a valid event at all.) Those decisions are 80 % of the work of defining an SLI, and they have to be taken in writing.

A good SLI meets four conditions:

The commonest mistake is measuring what is easy instead of what is important. CPU is extremely easy to measure and no customer cares about it. The "four golden signals" — latency, traffic, errors and saturation — are a good starting point, but only the first two and the third are SLI candidates; saturation is a cause, not an experience.

AlpinaShop's four SLIs, chosen by walking the customer journey:

Valid events: HTTP requests to the load balancer on shop routes, excluding known crawlers. Good events: those returning a code < 500.

Decisions taken and written down:

Valid events: the same requests as in SLI 1. Good events: those served in less than 800 ms.

Watch out for the classic mistake: a latency SLI is not "the p95 is under X". It is the proportion of requests faster than X. The difference matters: a percentile cannot be combined with an error budget, and a proportion can.

The 800 ms threshold is not chosen on intuition: you measure the current distribution, check where the point is beyond which users abandon, and pick a value that is comfortably met today but that will detect a real degradation.

The most important of the four, because it is the one that touches money.

Valid events: payment attempts started. Good events: those ending in a confirmation or in a legitimate refusal from the gateway (insufficient funds, expired card).

The nuance that makes it correct: a gateway refusal for insufficient funds is not our failure. Counting it as an error would make the SLI depend on the solvency of the customers, not on the quality of the system. What does count as bad: timeouts, 5xx errors, exceptions and transactions left in an indeterminate state.

Different from the previous ones because it is not a request: it is a process.

Valid events: scheduled runs of the orders pipeline. Good events: those that leave the data in BigQuery with less than 2 hours of lag.

Note that there is no SLI for CPU, memory, instance count or queue length. All of that is monitored — you need it to diagnose — but it is not a service level objective, because no customer cares about it.

Why 28 days and not a calendar month: a rolling 28-day window contains exactly four weeks, so it always includes the same number of weekends and Mondays. With calendar months, February and March are not comparable, and the budget "resets" on the 1st, which creates the perverse incentive of waiting for the month to turn before deploying anything risky.

The real queries, over the metrics from 06-04, in PromQL — which is the language Cloud Monitoring supports today alongside MQL:

For checkout, the custom metric the application has been emitting since 06-04 is used:

Separating rechazado_legitimo from error_tecnico is an instrumentation decision that has to be taken in the code, and it is exactly where you can see whether somebody has thought the SLI through or copied it. If the application only records "success" and "failure", the checkout SLI is irreparably ambiguous.

Error budget = 100 % − SLO

With a 99.9 % SLO over 28 days, the budget is 0.1 %: 40 minutes and 19 seconds of unavailability, or its equivalent in failed requests.

The "nines" table translated into time, worth keeping to hand:

Here is the part that turns the concept from a curiosity into a management tool. The error budget resolves the permanent tension between development and operations:

Without an objective criterion, that tension is resolved by authority, by exhaustion or by who shouts loudest. With an error budget, it is resolved with a number:

While there is budget left, you deploy. When it runs out, you stop and stabilise.

And this turns reliability into something development wants too, because a fragile system consumes the budget and stops it shipping features. The incentives are no longer opposed.

Written, agreed and in the README of alpinashop-infra:

And the clause that makes the policy credible:

If the budget is exhausted because of a failure in Google's platform and not because of our changes, it is documented and there is no freeze. The objective is to fix what is within our reach, not to punish the team for what it does not control.

Without that clause, the first regional Google outage would turn the policy into something the team would stop respecting — and a policy that gets ignored is worse than not having one.

The error budget is not an objective to be minimised. If at the end of the 28 days you have used 5 %, you have not won: you were probably being too conservative and could have shipped faster, or your SLO is too lax and does not reflect what users notice.

A budget that is never touched is as bad a sign as one that is always exhausted. Healthy consumption is around 50-80 %.

Cloud Monitoring has native support for SLOs, and it is worth using instead of building it by hand.

And in Terraform, which is where it should live:

Two types of SLI to distinguish:

For AlpinaShop, request-based. With one honest caveat: in the small hours, with three requests a minute, a single error is 33 % of that minute. Request-based SLIs are noisy at low traffic, and that is why the alerts in the next section require long windows for the softer thresholds.

This is the most valuable part of the section, and it replaces the fixed-threshold alerts from 06-04.

The burn rate is how many times faster than sustainable the budget is being spent:

• A rate of 1 = at this pace, the budget runs out exactly at the end of the 28 days. It is the sustainable pace.
• A rate of 14.4 = it runs out in 2 days.
• A rate of 6 = it runs out in a little under 5 days.

The problem with alerting on a fixed error threshold is well known: if the threshold is sensitive, you get false alarms; if it is lax, it does not detect slow degradations. Burn rate alerts solve both with two complementary alerts.

Why two windows per alert. The long window (1 h) avoids false alarms from a thirty-second spike. The short window (5 min) makes the alert clear quickly when the problem is resolved — without it, an alert with a one-hour window would still be firing fifty minutes after the fault was fixed. It is a small detail with an enormous impact on the life of whoever is on call.

The documentation block is the most important part of the policy, and the one almost everybody leaves empty. An alert that arrives on the phone at 3 in the morning with the text "Threshold exceeded" forces the person to rebuild the context half asleep. One that arrives with the first three steps and the link to the runbook turns fifteen minutes of disorientation into two of action. It is developed in section 12.

High availability is built by removing single points of failure, and it is done in tiers, each with its own price.

The relative frequency of each event is what should guide the decision, and it is rarely looked at:

The last row is the one that changes the priorities: most outages are not caused by infrastructure but by a change. Investing in multi-region while you have neither canary deployments nor fast rollback is armour-plating the door and leaving the window open. AlpinaShop did the right thing: pipeline and rollback first (module 6 and 07-02), architecture afterwards.

An important observation: a large part of AlpinaShop's architecture is already multi-region with no effort, because Google's global services are multi-region by design. The load balancer, DNS, CDN, Armor, Firestore, BigQuery and Secret Manager need no decision at all. The two components that are regional and critical are Cloud Run and Cloud SQL.

It deserves detail because you pay double for it and many people do not know what they are buying.

The last row is the one to underline. Regional HA protects against infrastructure failures. It does not protect against a DELETE FROM pedidos with no WHERE, which is replicated instantly to the standby instance. That is what backups and point-in-time recovery are for — section 11.

Documented in alpinashop-infra as a design ready to switch on:

Three commands. The global load balancer sends each request to the nearest region with capacity, and if a region fails, it stops sending traffic there automatically.

What stops it being switched on today is not Cloud Run: it is Cloud SQL. With the database only in europe-west1, the Madrid instances would have to cross regions on every query — latency and traffic cost — and if europe-west1 goes down, the database goes down anyway. High availability is determined by the least available component, and at AlpinaShop that is Cloud SQL.

Distributed systems fail in specific, well-known ways. Designing against them pays off better than adding redundancy.

The first thing is to distinguish two types, because the treatment is the opposite:

The commonest design flaw is treating a soft dependency as if it were hard. A recommender that takes 5 seconds must not make the product page take 5 seconds: it must disappear from the page.

Every network call must have a timeout. No exceptions.

A missing or over-long timeout is the mechanism by which a small failure turns into a total outage: requests pile up waiting, threads run out, and the service stops responding to everything, including the things that did not depend on the slow system.

Golden rule: the timeout of a call must always be shorter than the timeout of the caller. If Cloud Run cuts off at 60 s and your query waits 90 s, that code never runs.

Retrying is fine. Retrying badly makes the failure worse:

The four elements, and why each one:

1. Exponential backoff: giving the other side time to recover.
2. Jitter: without it, a thousand clients that failed at the same time retry at the same time, creating a synchronised wave that knocks the service over again just as it was getting back up. It is a real and frequent failure, and jitter costs one line.
3. A retry limit: retrying indefinitely turns a failure into an attack on your own system.
4. Transient errors only: retrying a 400 or a 403 will never fix it; it only wastes time and resources.

When a service has been failing for a while, stop calling it:

The double benefit: the caller stops waiting (it fails in microseconds instead of seconds) and the failing service stops receiving load, which lets it recover.

The principle that makes a partial failure invisible:

The shop carries on selling even if the recommender fails.

The three levels are the key to the pattern: plan A the recommender; plan B something simple and local — the category's bestsellers, which almost nobody will tell apart; plan C nothing, and the page is served anyway. A customer can buy in all three cases, which is the only thing that matters.

And the detail that makes it measurable: the log carries "degradado": true. Without that marker, the service runs degraded for weeks and nobody notices, because the availability SLI stays green. An alert on the percentage of degraded requests is what stops graceful degradation becoming silent degradation.

When demand exceeds capacity, there are two options: degrade for everybody or reject some and serve the rest well. The second is almost always better.

The last one is the most sophisticated and the most useful: when there is saturation, not all requests are worth the same. A user at checkout is worth far more than a crawler indexing page 47 of the catalogue.

High availability and disaster recovery are not the same thing, and confusing them leads to expensive architectures that do not protect against what actually happens:

A fact that reorders priorities: the commonest cause of data loss is not a natural disaster or an attack. It is somebody running something in the wrong environment. And against that, high availability is no use at all: it replicates the mistake faithfully.

With daily backups at 03:00 and a disaster at 10:15, the real RPO is 7 hours and 15 minutes of lost data. For AlpinaShop, that would be a whole morning's orders: unacceptable.

How you choose, and it is not by the technology: you calculate the cost of an hour of downtime and compare it against the monthly cost of each strategy. For AlpinaShop, an hour of downtime on a normal day is about €100 of orders; during the campaign, perhaps €800. With those numbers, active-active — several hundred euros a month, every month — is not justified.

Written, with owners, and also kept outside Google Cloud (because if the disaster is about access to Google Cloud, the plan has to be reachable).

Scenario 2 is by far the most likely, and its 5-minute RTO has already been solved since 07-02. It is a good reminder that the most profitable reliability work was already done before this lesson.

The four lines that do the work:

• point_in_time_recovery_enabled = true is what turns a 24-hour RPO into an RPO of minutes. Without it, you can only restore to the moment of the last backup; with it, to any instant within the transaction log window.
• location = "eu" keeps the backups outside the region. Backups in the same region that dies with the region are not backups: they are an illusion.
• transaction_log_retention_days = 7 defines the point-in-time recovery window. Seven days covers the case of "somebody deleted something on Monday and we noticed on Thursday".
• deletion_protection prevents the accidental deletion of the whole instance.

Data is not the only thing you have to be able to rebuild:

This is where the debt 07-04 flagged as critical gets paid.

An untested backup is not a backup: it is a hope with a technical name.

The reasons a restore fails on the day you need it, all real and all frequent:

The complete procedure, run quarterly by Marta, and timed.

Step 1 — Prepare and note the start time.

Step 2 — Restore to a NEW instance.

Never over the production one. The restore creates a separate instance, which makes the drill completely safe:

Step 3 — Verify the integrity, which is what almost nobody does.

Restoring without verifying only proves that the command works:

Step 4 — Start the application against the restored copy.

This is the step that uncovers the real problems:

Step 5 — Time it, clean up and document.

And here is the value of the exercise, because the first drill never goes well:

Five problems that would have turned a 1-hour restore into a 3-hour one, with the shop down and everybody watching. And none of them was detectable without a drill.

The five fixes were written into a runbook, creating the user and resetting the sequences was automated in a post-restore script, and the RTO for scenario 3 was adjusted from 1 hour to 90 minutes, which is an honest number.

The drill is not there to confirm that the backups work. It is there to find the five problems that stop them working.

Reliability is held up by people, and a burnt-out team is the biggest reliability risk there is. With three technical people, a traditional 24×7 rota is unworkable: it would mean two weeks out of every three permanently on call.

AlpinaShop's realistic model:

And the rules that make it sustainable, which matter as much as the calendar:

1. Only wake someone for what has to be fixed now. If an alert arrives in the small hours and the action is "look at it tomorrow", that alert must not fire at night. It is the rule that most reduces burnout.
2. Real compensation. Hours or money, agreed in writing. Unpaid on-call lasts until somebody has had enough.
3. A maximum of two interrupted nights in a row. If it happens, they are relieved.
4. Every night-time alert is reviewed the next day: was it necessary? If not, it is tuned or removed.
5. Fewer alerts is better. Alert fatigue (06-04) is a reliability problem: a person who has learnt to ignore their phone will also ignore the one that mattered.

A runbook is the document that says what to do about a specific alert. It lives in alpinashop-infra/runbooks/ and is linked from the documentation field of the alert policy.

gcloud run revisions list --service=alpinashop-web --region=europe-west1 --limit=5

gcloud run services update-traffic alpinashop-web --region=europe-west1
--to-revisions=PREVIOUS_REVISION=100

The three properties of a useful runbook: it starts by confirming (not by diagnosing), it puts the most likely cause first — a recent deployment, in 70 % of cases — and it says explicitly when to escalate, so that the person on call is not left going round in circles alone at 4 in the morning.

And a fourth that you can see in the header: "Tested in a drill: yes". A runbook nobody has ever executed is a hypothesis.

After a significant incident, a document with a fixed structure:

Blameless does not mean there are no responsibilities. It means one very specific thing:

If the answer to "why did it happen" is a person's name, the analysis has stopped too soon.

An applied example:

The practical — not moral — reason for the absence of blame: in a culture where the culprit is hunted down, people hide mistakes, and hidden mistakes do not get fixed. In a blameless culture, people tell you what happened and the system improves. It is a decision about effectiveness.

All the post-mortems in one place, tagged. After a year it lets you see patterns that no isolated incident reveals: "60 % of our incidents are deployments" is a sentence that changes an entire team's priorities.

Chaos engineering is causing failures on purpose to verify that the system tolerates them. Netflix popularised the idea by killing instances in production; a small business does not need that, but it does need something.

Every system has disaster recovery tests. Some run them on purpose; others wait for them to happen.

AlpinaShop's experiments, ordered from least to most aggressive:

The rules that make this responsible rather than reckless:

1. A written hypothesis beforehand. If you do not know what you expect, it is not an experiment: it is breaking things.
2. Start in development.
3. A small blast radius, widened only when the previous one went well.
4. A stop button and the ability to abort at any moment.
5. Never during the campaign. Nor on a Friday afternoon.
6. Tell the team. Unannounced chaos with real on-call is a nasty practical joke.

Experiment 3 deserves a comment, because it is frightening and should not be: the Cloud SQL failover is going to happen one day, chosen by Google and at the worst possible moment. Provoking it on a Tuesday at 10 in the morning, with everybody watching and the ability to fix it, is infinitely better than discovering on a campaign Friday that the application does not reconnect because it lacks pool_pre_ping (07-02).

And experiment 7 is the most underrated: that only Marta knows how to roll back a deployment is a single point of human failure. That Dani and Lucía have each done it at least once is worth more than any infrastructure redundancy.

• Setting "it must never go down" as the objective. It cannot be met, measured or used to decide. Pick a number and pay for it.
• Confusing SLO and SLA. The SLO always stricter than the SLA, with a generous margin.
• Measuring the SLI from the server. It is measured from where the user is: the load balancer or an external check.
• Defining the latency SLI as "the p95 < X". It is "the proportion of requests < X". A percentile does not combine with an error budget.
• Counting legitimate gateway refusals as errors. The SLI would end up depending on the solvency of the customers.
• An SLO on CPU or memory. They are not user experience. You monitor them, you do not turn them into an objective.
• Calendar-month windows. They create the incentive to wait for the 1st before deploying anything risky. Use a rolling 28-day window.
• An SLO with no associated policy. If nothing changes when you miss it, it is an ornament.
• Freezing because the budget ran out when it was the platform's fault. The policy loses credibility and the team stops respecting it.
• Chasing a budget consumed at 5 %. A sign of excessive conservatism or of an SLO that is too lax.
• Fixed-threshold alerts on errors. Either they are noisy or they miss slow degradations. Use burn rate with two windows.
• Leaving the alert's documentation field empty. At 3 in the morning, that text is worth more than the alert itself.
• Network calls with no timeout. It is the mechanism by which a small failure turns into a total outage.
• Retries without jitter. A thousand clients retrying at once knock the service over just as it was recovering.
• Retrying permanent errors. A 403 does not improve with persistence.
• Treating a soft dependency as hard. The recommender must not be able to take down the product page.
• Degrading without marking it in the logs. The service runs half-broken for weeks and the SLI stays green.
• Backups in the same region. They are not backups: they are an illusion.
• Believing HA protects against an accidental deletion. It replicates it instantly to the standby instance.
• Not testing the restore. It is the most dangerous debt because it is invisible until the day it matters.
• On-call with no compensation. It lasts until somebody has had enough, and then you have neither on-call nor the person.
• Hunting for culprits in a post-mortem. Mistakes get hidden, and hidden mistakes do not get fixed.
• Tip: start with an availability SLO and nothing else. Four SLOs from day one is too many. One good one is worth more than four copied ones.
• Tip: fast rollback is worth more than redundancy. The commonest cause of an outage is a change, not a hardware failure.
• Tip: make sure the whole team knows how to roll back a deployment. It is the cheapest redundancy there is.
• Tip: time the restore drill. The real number always surprises you, and it is the one that gives you an honest RTO.

Dani is about to launch alpinashop-buscador (the service from the 07-02 exercise). Define its complete SLI and SLO:

1. Which events are valid and which are good, explicitly resolving at least six edge cases.
2. The SLO and the window, justified.
3. The resulting error budget in minutes and in requests.
4. The two burn rate alerts with their thresholds.
5. The policy for what is done in each budget state.

Bear in mind that the search service is a soft dependency of the catalogue: if it fails, a link to browsing by category can be shown.

AlpinaShop is launching AlpinaPro, a subscription service for mountaineering clubs: a monthly fee, exclusive content and equipment bookings. The facts:

• 400 subscribers, €25/month each: €10,000/month of recurring revenue.
• Its own database alpinapro-suscripciones (PostgreSQL).
• An integration with the gateway for recurring charges on the 1st of each month.
• The clubs book equipment 48 h in advance; a lost booking is a serious complaint.
• Management says it "must never go down", but the additional budget is €150/month.

Design: the service's SLIs and SLOs, the RTO and RPO justified with business numbers, the recovery strategy chosen within the budget, and the specific backup configuration. Explain what you answer management about "it must never go down".

The facts. Tuesday, 18:42. The fast burn alert fires. The reconstructed timeline:

• 18:30 — Cloud Build deploys revision alpinashop-web-00051 with a 10 % canary. The smoke tests pass.
• 18:38 — Marta approves the move to 100 % (she was in a hurry, she was leaving at 18:45).
• 18:42 — Alert: burn rate 22×. 500 errors on 35 % of requests.
• 18:44 — Marta sees the alert already in the car. She cannot act.
• 18:51 — Dani, alerted by Marta, connects and rolls back to revision 00050.
• 18:53 — The errors disappear.
• Impact: 11 minutes, ~230 failed requests, 4 lost orders, 27 % of the monthly budget consumed.
• Technical cause: the new revision read an environment variable TARIFA_ENVIO that was only defined in alpinashop-dev.

Write the complete post-mortem following the structure from section 12, with the five whys taken through to a systemic cause, and at least five specific actions with an owner. Also point out what the team did well.

Solution 1 — Defining the SLO for alpinashop-buscador

1. Defining the SLI.

Availability SLI = search requests with a useful response / valid search requests

Edge cases, resolved and written down:

The "0 results" case is the one that generates the most argument and it is important to get it right. Counting it as bad would make the SLI measure the quality of the catalogue instead of the reliability of the service, and it would drop every time somebody searched for something we do not sell. Search quality is measured separately, with product metrics — the rate of searches with no results, the click-through rate — which are not an SLO.

Latency SLI = searches served in < 500 ms / valid searches

500 ms and not 800 ms like the catalogue, because a search box is interactive: the user types and waits. The perception of slowness kicks in sooner.

2. SLO and window.

The justification for the 99.5 % is the interesting part of the exercise. The search service is a soft dependency: if it fails, the catalogue shows a link to browsing by category and the customer can carry on buying. A soft service does not need the same objective as a critical one, and setting it at 99.9 % would mean spending money and engineering on a level of reliability that does not change the business outcome.

This is the reasoning to take away: the SLO is derived from the business impact of the failure, not from how fond you are of the service.

3. Error budget.

At 99.5 % over 28 days: 0.5 % = 3 h 21 min 36 s.

In requests: if the search service receives around 150,000 valid searches over 28 days, the budget is 750 failed searches.

It is comfortable, and deliberately so: it lets search improvements be deployed frequently without a one-off failure blocking the team.

4. The two alerts.

Neither of them wakes anybody up, and that is the logical consequence of being a soft dependency with a 99.5 % SLO. Getting up at 4 in the morning because the search service is failing, while the shop carries on selling, is exactly the kind of alert that produces fatigue and gets the one that does matter ignored.

The exception to plan for: if the search service goes down and on top of that the catalogue starts failing — because the timeout was not cutting in and the threads ran out — the alert that will fire is the catalogue's, which is a paging one. The alert design has to reflect the dependency design.

5. Budget policy.

The red row is the key to the exercise. When a soft service exhausts its budget, the priority question is not "how do I make it more reliable" but "is it properly isolated?". A search service that fails 2 % of the time without affecting anybody is a minor problem; a search service that fails 0.5 % of the time and every failure degrades the product page is a serious one.

Solution 2 — Designing a recovery plan for AlpinaPro

Starting point: translating the business into numbers.

And here is the nuance that separates a good analysis from a superficial one: in a subscription service, revenue is not lost by being down. The 400 subscribers pay regardless. What is lost is something else, and it is worse:

Conclusion: a 4-hour outage that causes 5 cancellations costs €3,000, not €55. The pro rata revenue analysis underestimates the impact by more than an order of magnitude, and it is a very common mistake in subscription services.

Proposed SLIs and SLOs.

The third is the most interesting and the one usually forgotten. The recurring charge is not a continuous service: it is a process that runs one day a month. A monthly SLO over it makes no sense; what is needed is a different objective: "99.99 % of scheduled charges execute correctly within the following 24 hours", with retries. And a specific architectural treatment: a queue with retries and a DLQ, not a single execution that either works or is lost.

RTO and RPO justified.

A strategy within €150/month.

A very tight budget. The split:

Strategy chosen: pilot light. The database replicated in the second region is the only thing kept switched on; the Cloud Run compute is brought up with three commands (section 7) because the image already exists and Cloud Run scales from zero. Estimated RTO: 30-60 minutes, better than the 6 h objective.

What is not done and why: active-active would cost €400-600/month to cover a very low probability event, when the entire budget is €150.

Backup configuration.

Two decisions worth pointing out: the Wednesday maintenance window, deliberately far from the 1st — Google maintenance during the charging cycle would be the worst possible moment — and the DR replica in ZONAL mode, because paying for HA on a replica only used in a disaster is doubling the cost of an insurance policy.

What you answer management.

"We cannot guarantee it will never go down, and anybody who promises that is not telling the truth. What we can do is decide how much downtime we are willing to accept and pay for it.

With the €150 a month we have, the commitment is this: if a machine or a specific data centre fails, it recovers by itself in five minutes and without losing any data. If somebody deletes something by mistake, we recover it in two hours losing at most five minutes of bookings. And if a whole Google region went down — something very infrequent but possible — we would be running from Madrid in under an hour.

For the charging process on the 1st we have separate treatment, because it is the critical moment of the month: if something fails, it is retried automatically for 24 hours and no charge is lost.

What we do not cover with this budget is two regions going down simultaneously or a global Google problem. Covering it would cost three to four times more, every month, for a scenario that may never happen.

And there is something I want to ask for besides the budget: testing all of this every quarter. A recovery plan that is not rehearsed does not work on the day you need it, and we know that because it happened to us with the shop: the first time we tried restoring the database we found five problems that would have tripled the recovery time."

Solution 3 — Post-mortem of the 18:42 incident

Post-mortem: 500 errors in the catalogue after deployment 00051

Incident date: Tuesday 2026-08-04, 18:42-18:53 Author: Marta Ruiz · Reviewed by: Dani, Lucía Status: actions in progress · Classification: blameless

Summary. A deployment promoted to 100 % of traffic contained code that depended on an environment variable that did not exist in production, causing 500 errors on 35 % of requests for 11 minutes.

Impact.

Timeline.

Root cause — the five whys.

1. Why did the requests fail? The code read os.environ["TARIFA_ENVIO"], which did not exist in production, raising KeyError.
2. Why did it not exist? It was added to alpinashop-dev during development and was not propagated to production.
3. Why was it not propagated? Production environment variables are managed in Terraform, but development configures them by hand with gcloud. There is no mechanism guaranteeing parity.
4. Why did the canary not catch it? The smoke tests only check /salud and /producto/1, and neither of those routes uses TARIFA_ENVIO (only the basket one does). On top of that the canary ran for 8 minutes at low traffic.
5. Why was it promoted to 100 % with that coverage? Because the process allows manual approval with no automated check on the candidate revision, and the approval was given in a hurry at the end of the day.

Systemic root cause: there is no automated verification that a candidate revision has all its configuration present in the target environment, nor an objective check that has to pass before promotion to 100 %. Manual approval is the only control and it depends on the state and availability of one person.

What went well — and this section matters as much as the previous one:

• The alert worked: 4 minutes from the first error to the notification. Exactly what 06-04 and the burn rate alerts promised.
• The rollback was trivial: one command, 2 minutes, nothing to rebuild. The work from 07-02 paid for itself in this incident.
• The escalation worked: Marta, unable to act, alerted Dani within 3 minutes instead of trying to fix it while driving.
• The error budget did its job: it turned "11 minutes, it was not that bad" into "27 % of the monthly budget", which is a figure that forces action.
• Nobody looked for culprits during the incident. It was fixed first and analysed afterwards.

Actions.

A note on action 5. It might look like a bureaucratic rule that slows the team down, and it deserves justifying: deploying in the afternoon is not forbidden — the canary can run all night at 10 % — what is forbidden is promoting to 100 % when there is nobody left available to react. The restriction is not about the risk of the change, but about the capacity to respond. That distinction is what makes the team accept the rule instead of working around it.

What this incident was NOT. It was not a mistake by Dani for writing code that reads a variable, nor by Marta for approving in a hurry. It was a system that allowed a change whose configuration nobody had verified to be promoted to 100 % of traffic, at a moment when nobody could react. Any of the three would have done the same, and that is why all seven actions are about the system and none about the people.

AlpinaShop has stopped aspiring to "never going down" and started managing reliability with numbers.

You know that every nine multiplies the cost and you have done the sum almost nobody does: going from 99.9 % to 99.99 % would save AlpinaShop around €70 a month in orders and would cost several hundred. Saying "99.99 % would be a bad investment" with the table in front of you is one of the most valuable things an engineer contributes, and it requires having calculated the two caveats too: that not all downtime is worth the same and that there are costs that are not lost orders.

You distinguish SLI, SLO and SLA precisely, with the three rules that avoid almost every mistake — the SLO always stricter than the SLA, the SLI measured from where the user is, and an SLO with no consequence is not an objective — and with the operational definition that resolves the doubts: good events / valid events, where the real work is deciding what each of those is.

You have chosen four SLIs by walking the customer journey — availability, latency, checkout success and analytics freshness — resolving the edge cases in writing: a 404 is good, a 429 to an attacker is not even valid, and a refusal for insufficient funds is not our failure, a decision that has to be taken in the code by instrumenting rechazado_legitimo against error_tecnico. And you know why there is no CPU SLI: because no customer cares.

You have mastered the error budget: the nines table translated into minutes, the rolling 28-day window that avoids the perverse incentive of the calendar month, and above all what it is really for — resolving with a number the tension between deploying fast and not breaking anything, with a four-state policy and the credibility clause of not freezing when the failure was the platform's. Plus the use almost nobody makes: a budget consumed at 5 % is as bad a sign as one that is exhausted.

You know how to create SLOs in Cloud Monitoring from Terraform, to distinguish request-based from window-based SLIs, and to set up burn rate alerts with two thresholds and two windows each — the long one to avoid false alarms, the short one so the alert clears when the problem is fixed. And you know that the documentation field, which almost everybody leaves empty, is worth more at 3 in the morning than the alert itself.

You have the tiered high availability architectures with AlpinaShop's component-by-component table, the observation that much of the platform is already multi-region with no effort because Google's global services are, and the two pieces that really are regional and critical. You know exactly what you buy with Cloud SQL HA — synchronous disk replication, another zone, failover in 1-2 minutes — and what it does not cover: a DELETE with no WHERE is replicated instantly. And you know that multi-region Cloud Run is three commands, but that it is not switched on because availability is determined by the least available component.

You design against the failure modes: hard and soft dependencies — with the commonest design flaw, treating a soft one as hard — timeouts on every network call with the rule that the inner one must always be shorter than the outer, retries with exponential backoff and jitter so as not to create synchronised waves, circuit breakers, graceful degradation in three levels with the "degradado": true marker without which degradation becomes silent, and load shedding with priority-based rejection.

You know that high availability and disaster recovery are not the same thing, and the fact that reorders priorities: the commonest cause of data loss is somebody running something in the wrong environment, and against that HA is no use because it replicates the mistake. You can handle RTO and RPO and the four strategies with their costs, choosing by comparison against the real cost of an hour of downtime.

You have AlpinaShop's plan with seven scenarios — where the most likely one, a faulty deployment, was already solved back in 07-02 — the backup configuration with the four lines that do the work (point_in_time_recovery, location = "eu", log retention and deletion_protection) and the configuration export, including the DR plan kept outside Google Cloud.

And you have paid off 07-04's critical debt: the restore drill, timed, with integrity verification and the application started against the copy. With the result that matters: five problems found — 47 minutes instead of "a few minutes", the non-existent application user, missing permissions, out-of-sync sequences and no written procedure — that would have tripled the real recovery time. The drill does not confirm that the backups work: it finds the five problems that stop them working.

Finally, you have the operational practice: sustainable on-call for three people with the rule that most avoids burnout — if the action is "look at it tomorrow", that alert does not fire at night — runbooks linked from the alerts that start with the most likely cause and say when to escalate, blameless post-mortems with the practical and not moral reason for that culture, and chaos testing at a sensible scale with its six rules, where the most underrated experiment is everybody on the team knowing how to roll back a deployment.

Everything in this lesson works because there are three people holding it up. And there is the limit: the SLOs live on a dashboard somebody maintains, the error budget policy is respected because the team agreed it, the runbooks get updated because Marta remembers, and nothing technically stops somebody creating a VM with a public IP in a United States region tomorrow, or disabling an audit log, or deleting a whole project by mistake.

That is no longer reliability: it is governance. And it is the only thing left to build. AlpinaShop's organization today has five projects, three folders, dozens of service accounts, four repositories, a half-finished perimeter and no policy governing the whole — plus a list of debts from 07-04 that starts with "organization policies not applied" and "audit logs without immutable retention".

The last lesson of the module closes the circle: organization structure, policies that are inherited and enforce themselves, an inventory of everything that exists, real auditing with immutable retention, and landing zones deployed with Terraform — so that what has been built following AlpinaShop does not depend on three people remembering.