Everything we have built so far has a go-live date. What comes afterwards does not: it is the day-to-day of whoever keeps the platform alive, and it is exactly what no tutorial covers. Tutorials end when the pod reaches Running. Reality starts there.

Reality is three in the morning on a Saturday with an alert on your phone. It is the meeting where somebody from finance asks why the cloud bill has gone up 60 % in four months. It is the developer who wants to deploy at six on a Friday afternoon. It is the awkward conversation about whether you can promise 99.99 % availability or only 99.9 %. It is the cluster upgrade that has to happen before the version goes out of support, and that nobody wants to touch because it works.

This lesson closes the module with that part: incident management, runbooks, the error budget as a decision-making tool, change management, capacity planning, costs and a maintenance calendar. And it also closes the whole Rutas Norte journey.

Compliance warning. The incident management procedures described here affect service continuity and, when an incident involves improper access to, loss of, or exposure of personal data, they trigger notification obligations with strict legal deadlines. The severity classification, the communication channels and the postmortem templates must be reviewed by the organisation's compliance officer, who must also determine in which cases an incident constitutes a notifiable security breach.

1. Incident management
2. Runbooks
3. SLIs, SLOs and the error budget
4. Change management
5. Capacity planning for the May bank-holiday weekend
6. Costs
7. Maintenance calendar
8. Team maturity

Severity is not determined by how alarming the problem sounds, but by the impact on the user and on the business. Defining it in advance avoids the "is this serious?" debate exactly when there is no time to have it.

Two rules that prevent a lot of damage:

• When in doubt, raise the severity. Lowering it once you understand the problem is cheap. Discovering two hours in that a sev 1 was being treated as a sev 3 is expensive.
• Anyone can declare an incident. No permission and no hierarchy needed. A false positive costs twenty minutes; a real incident left undeclared costs hours.

In severity 1 and 2 incidents, three roles are explicitly separated. In small teams one person can hold two, but command and hands-on work are never combined.

Why separating them is not bureaucracy but a lesson learned the hard way. The person debugging cannot at the same time answer on three channels, talk to the board and keep the timeline: if they try, they do all four badly. With nobody deciding, you end up with three people applying contradictory mitigations to the same system — one restarts pods, another does a rollout undo, a third scales by hand — which is a real and frequent pattern. With no dedicated communications, either nobody gets informed, or the responders spend the incident answering "is it fixed yet?" instead of fixing it. And the commander, with no hands on the keyboard, keeps perspective: they are the one who notices that everybody has spent forty minutes chasing a hypothesis that leads nowhere.

Minutes 0-2. Scope. What is failing, for whom and since when. The answers get written in the channel:

Minutes 2-4. What has changed? Between 70 and 80 % of incidents are caused by a recent change. It is the highest-yield question in the whole process.

Minutes 4-7. Mitigate, do not diagnose. This is the point where most teams go wrong. The natural impulse is to understand what is happening. The correct goal is to make it stop hurting, and to understand afterwards.

Minutes 7-10. Communicate. A short, honest message, with the next update committed to:

What makes this message useful: it states the impact in user terms, not infrastructure terms; it says what is being done; and it commits to a specific time for the next update, which stops interruptions at the source.

Rules: communicate before they ask; say what you know and explicitly what you do not; never promise a resolution time you cannot guarantee (promise the next update, not the resolution); and give bad news early.

It is held within 48 hours of every severity 1 and 2 incident, and of any incident with something to learn from.

Blameless means something very specific: you start from the position that everybody acted reasonably with the information and the tools they had at the time. If somebody ran a destructive command, the question is not why they ran it, but why the system made such a command easy to run by mistake. If somebody did not see the alert, the question is not why they missed it, but why the alert was not visible.

The reason is not kindness, it is effectiveness: as soon as blame is on the table, people stop telling you what actually happened, and without that there is no analysis to be had.

The template:

Four criteria for actions to be worth anything: a named owner (not "the team"), a specific date, prioritised (three that get done beat twelve that do not), and reviewed in the weekly meeting until they are closed.

Action number 2 in this example is the most valuable and the most expensive: the underlying problem was not the query, it was that pre did not resemble production. That kind of action is what prevents whole families of future incidents.

A runbook is the written procedure for a specific symptom. Its real value is that it lets somebody who is not an expert in that component respond correctly at three in the morning.

Rules for a useful runbook: commands that can be copied and run as they stand, not descriptions; mitigation before diagnosis; linked from the annotation of the alert that fires it; and reviewed after every use, because an out-of-date runbook is worse than none.

A runbook is only useful if it turns up at the moment of the alert. The runbook annotation on the rules from 07-04 is what achieves that:

And one simple policy that keeps the quality up: every alert that wakes somebody up must have a runbook. If it does not, either you write one or the alert should not be waking anybody.

In 07-04 we saw what they are. Here they are going to be used to make decisions, which is the only thing that distinguishes them from decoration on a dashboard.

An SLI measures what the user experiences, not what the infrastructure does. "CPU at 80 %" is not an SLI; "99.7 % of searches respond in under 400 ms" is.

The classic mistake is choosing the number for aesthetic reasons: "99.99 % sounds serious". Each extra nine multiplies the cost roughly tenfold.

The three questions that set the number: what does each minute of downtime cost? (at Rutas Norte, about €90/min on a normal day and about €900/min during the May bank-holiday weekend); what does the user perceive? (nobody notices three minutes a month, everybody notices seven hours); and what does the competition do and what has been promised contractually?

The Rutas Norte SLOs:

Note that none of them is 100 %. A 100 % SLO means any change is unacceptable, and that paralyses the product.

Error budget = 100 % − SLO. With a 99.9 % SLO over 30 days, the budget is 43 minutes and 12 seconds of unavailability per month.

And here is the idea that changes everything: that budget is a resource you can spend deliberately. Having part of it consumed is not a failure; it is what it exists for.

The Rutas Norte policy, approved by the board and by the team:

What is valuable about this policy is that it turns an argument about opinions into a decision based on a number. When product asks for one more feature and platform says the system is fragile, the conversation stops being "I reckon that…" and becomes "the budget is at 130 %, and this is what we agreed to do in that case".

And it works in both directions, which is what makes it fair: if the budget is at 20 % halfway through the month, platform cannot block a deployment on grounds of risk. The system is being more reliable than promised, and that means you can go faster.

Alerting on the budget: you alert on the burn rate, not on an absolute threshold. The expression combines a long window and a short one to avoid false positives, and it fires when the pace would exhaust thirty days' budget in two.

Scheduled freezes:

A freeze is not "no work happens": it is "production does not change". During the May bank-holiday freeze the team does what it cannot do the rest of the year: write runbooks, improve tests, review noisy alerts, pay down technical debt that does not touch production.

It is not superstition, it is the arithmetic of detection time and response capacity. Problems do not appear instantly: memory leaks, table bloat, connection exhaustion and disks filling up take hours or days, and a deployment at 18:00 on Friday can fail at 04:00 on Saturday. At the weekend there are fewer people and they are less available: the 12-minute time to restore from 11-03 is measured in working hours, and on a Saturday morning it is easily triple that. Weekend traffic is different: at Rutas Norte more leisure tickets are sold on a Saturday than on any weekday, so that is when a failure hurts most. Nobody is watching, because whoever deployed has gone home. And there is a perverse effect: knowing all of this, whoever deploys at 18:00 on a Friday does it in a hurry and skips checks, so they can leave.

The Rutas Norte rule: the last deployment to production is Thursday at 15:00, and Friday is spent watching what was deployed during the week. The exception is incident fixes, which have no window because the cost of not making them is higher.

The peak is not evenly spread: it concentrates between 10:00 and 13:00 on the Wednesday and Thursday before the long weekend, when people buy their return ticket.

bookings-api. From the load test in 09-06: one replica with 200m of CPU sustains around 70 requests/s with a p95 of 180 ms. For 2,400 requests/s:

And you have to verify that the arithmetic adds up along the whole chain, including the PostgreSQL connections, which were the bottleneck in 11-02:

The namespace ResourceQuota (03-04): with 41 API replicas plus everything else, the current 20 vCPU quota falls short. It has to go up to 32 before the bank holiday, or the HPA will ask for pods the quota will reject: a silent and baffling failure.

PostgreSQL: 3,400 queries/s against the usual 320. The load test showed that the primary holds up to 4,200 queries/s before degrading, with the report reads already diverted to replicas. Enough, but with no margin for a new badly indexed query, which is exactly what happened in incident INC-2026-041.

Results of the test on 15 April 2026:

The note: Karpenter took three minutes to provision the peak's nodes. During that time there were pods in Pending and the p95 rose to 1.1 s. Action taken: over-provision with low-priority pods (a negative PriorityClass, from 06-05) that Karpenter evicts instantly when real load arrives, leaving nodes already warm.

Point 11 is systematically forgotten and it is the reason next year's bill starts 20 % higher.

And one underlying cause that encompasses them all: nobody sees the bill for what they deploy. A developer who puts requests: 2 instead of 200m gets no signal at all.

The fact that monitoring costs more than rutas-norte-dev always comes as a surprise, and it is normal: Prometheus with long retention and many series is expensive.

This is where the labels from 02-07 stop being a convention and become a management instrument.

__unallocated__ is the quality indicator for your labelling: €341 that nobody can explain. The Kyverno policy from 11-05 that requires the rutasnorte.example/team label exists precisely to drive that number to zero.

Informational allocation works better than real internal billing. A monthly report per team, compared with the previous month, changes behaviour without the bureaucracy of actually charging. It is enough for somebody to see that their service went from 300 to 700 euros.

Lever 1: right-sizing. The VPA in recommendation mode, applying no changes, tells you what is being wasted:

With requests: 200m and target: 186m, bookings-api is well sized. The interesting case was notifications-worker, with requests: 1 of CPU and a target of 120m: 88 % waste multiplied by its replicas.

Lever 2: switching off at night. The best return on effort of them all.

With start-up at 07:30, dev and pre run 57 hours a week instead of 168: a 66 % saving in those environments.

Lever 3: spot instances. Continuing what was decided in 10-06:

Lever 4: orphaned resources. The sweep worth doing quarterly:

Rutas Norte's first sweep turned up 14 orphaned volumes (1.8 TB, about €180/month) from the ephemeral environments in 11-03 deleted before the cleanup CronJob existed.

Lever 5: retention. Rutas Norte's revised policy:

That last row matters: retention of backups containing personal data is not a cost lever, it is a compliance decision. Reducing it to save money is a serious mistake.

Nearly 27,000 euros a year, for about three person-weeks of work and without degrading any SLO. The data for the period confirms that purchase availability stayed at 99.94 %, above the target.

The obligatory warning: cost optimisation has a point beyond which it starts costing reliability. Cutting requests below what the load needs causes evictions; moving everything to spot instances causes simultaneous interruptions; trimming metric retention leaves an incident analysis with no data. The bill is not the SLO. When they conflict, the SLO wins, and it is worth having that decision written down before it happens.

None of this is optional; the only optional part is whether it is done as planned work or as an emergency.

About the certificates: the cert-manager ones renew themselves, but there is always one outside the automation (client certificates towards the payment gateway, internal mesh certificates, signing certificates). An inventory with alerts at 30, 14 and 7 days is worth having.

The maintenance budget. Adding up the tasks, maintenance consumes roughly 20-25 % of the platform team's capacity. It is a number that has to be said out loud and defended to whoever does the planning: a team 100 % on new features is a team accumulating operational debt on credit, and that debt is collected in the form of incidents.

A platform's maturity is not measured by the tools it uses. It is measured by what happens when something goes wrong.

Rutas Norte went from reactive to managed in about fourteen months, and it was not the technology that made it possible but four changes of habit: blameless postmortems on every incident, which by the sixth revealed that half of them shared an underlying cause; runbooks written by whoever suffered the incident, immediately afterwards; the error budget as a shared argument, which turned the tension between product and platform into a conversation with data; and the maintenance calendar defended as real work, with a slot in the plan rather than in the spare moments.

Three signs that a team is genuinely maturing: incidents get shorter before they stop happening (you do not eliminate failures, you respond better); the number of indispensable people falls, because only one person knowing how to fix something is a risk and not a virtue; and the time spent on prevention rises, which is the definitive indicator of having left firefighting mode behind.

• Diagnosing before mitigating. Understanding the root cause is important, but not while users are suffering. First make it stop hurting.
• The incident commander with their hands on the keyboard. They lose perspective and stop coordinating. In sev 1 and 2, the commander touches nothing.
• Postmortems that look for someone to blame. People stop telling you what happened and the analysis loses all its value.
• Follow-up actions with no owner and no date. They do not get done. Three actions with a name and a date beat twelve on a list.
• Alerts with no runbook that wake somebody up. Either you write the runbook, or the alert should not be waking anybody.
• A 99.99 % SLO without having worked out what it costs. Each nine multiplies the cost by roughly ten. Choose the number on business criteria.
• The error budget as a decorative dashboard. If it does not change what the team does when it runs out, it is worth nothing.
• Optimising costs without watching the SLOs. Cutting requests too far causes evictions; trimming retention blinds the next investigation.
• Forgetting to undo the peak adjustments. Raising maxReplicas and the quota for the bank holiday and not lowering them afterwards is the most common way for the bill to rise permanently.
• Tip: time your runbooks. A runbook that in practice takes forty minutes when it says ten needs revising.
• Tip: make the on-call rota genuinely rotate. If the same person always answers, the knowledge does not spread and that person burns out.
• Tip: celebrate well-handled incidents. A team that only gets attention when something goes wrong ends up hiding problems.

Saturday, 09:14. The BookingsApiHighErrorRate alert arrives (12 % of 5xx errors). It is the first weekend of July, with a lot of leisure traffic. The last deployment to pro was Thursday at 14:30. You are on call and there is one other person reachable. Describe: (a) the severity you declare and why; (b) the roles you assign; (c) the exact commands for your first five minutes; (d) the communication message you send at ten minutes.

It is 18 September. The error budget for purchase availability (SLO 99.9 %, 30-day window) is 118 % consumed: two sev 2 incidents took 51 minutes. Product wants to deploy the new price comparator, which has been ready for three weeks and which the board wants for the autumn campaign. Platform has four outstanding postmortem actions. Apply the policy from section 3.3: what do you decide, who decides, what do you propose as an alternative and what conditions would you set for lifting the freeze?

A cluster's monthly bill is: pro compute €3,100, non-production compute €1,900, observability €1,200, storage €800, network €400, load balancers €300. Additional data: the VPA recommends an average of 45 % of the current requests; dev and pre run 24×7; Prometheus retains 45 days at full resolution; there are 22 volumes in Released state; no workload uses spot instances. Propose a plan ordered by return, with the estimated saving of each lever, the associated risk and what you would measure to check that the service does not degrade.

Solution 1.

(a) Severity 2. A 12 % error rate is a serious degradation of critical functionality (purchasing), but not a total outage: 88 % of attempts work. It would be sev 1 if it went above 50 % or if buying were impossible altogether. The context of a July weekend with heavy traffic reinforces the urgency and justifies calling the second person.

(b) With two of us: I take command and communications; the reachable person takes the responder role. If it escalates to sev 1, a third person is called and command and communications are separated. What is not done is the commander starting to debug.

(c) The first five minutes:

The key reasoning: since the deployment was 43 hours ago and the problem starts now, the most likely hypothesis is not a code change but saturation from weekend traffic (HPA at the ceiling, connections exhausted, disk full) or a cumulative problem (memory leak, table bloat). You go to the latency runbook and the disk one.

(d) Communication at ten minutes:

Solution 2. Decision: the price comparator is not deployed. The budget is exhausted (118 %), which triggers the policy's feature freeze. The decision belongs to the board, not the team, precisely because an exception has a business cost and must be owned by whoever holds that responsibility. Platform's role is to present the number, the agreed policy and the alternatives, not to block unilaterally.

The alternative proposed: over the next two weeks the team closes the four outstanding actions, and the comparator is deployed behind a switched-off feature flag (11-04). That way the code reaches production without changing behaviour, it is validated technically, and activation is ready for the moment the budget recovers. This partly satisfies product without consuming budget.

Conditions for lifting the freeze: (1) the four actions closed and verified; (2) the budget in the 30-day rolling window below 100 %, which will happen automatically as the incidents leave the window provided there are no new ones; (3) the comparator must be deployed with a full canary, with no promote --full, with analysis of the conversion metric; (4) a joint review of whether the 99.9 % SLO is still the right one, because two sev 2 incidents in a month may indicate the target is too demanding for the current investment, or that there is an unresolved underlying problem.

Solution 3. Current total: €7,700/month.

The order is justified: first the zero-risk, low-effort items (1, 5), then the biggest absolute saving (2), and last the ones requiring most care (3, 4).

What to measure to check the service does not degrade: (a) the error budget of every SLO before and after each lever, which is the definitive indicator; (b) evictions from memory pressure and OOMKilled restarts after applying lever 2; (c) p95 latency and HPA reaction time after lever 4, because interruptions increase pod churn; (d) incident resolution time after lever 3, which is where the lack of history would show. Rule of application: one lever at a time, with two weeks of observation between them, so that any degradation can be attributed to its cause.

We have walked through the day-to-day of whoever keeps a platform alive. Incident management with severities defined in advance, separated roles — and the commander not touching the keyboard — the script for the first ten minutes that puts mitigating ahead of diagnosing, and the blameless postmortem whose actions have an owner and a date. Runbooks, with their template and four real Rutas Norte cases linked from the alert annotations. SLIs and SLOs chosen on business criteria, and the error budget turned into a policy that decides what the team does next month. Change management, with its windows, its May bank-holiday freeze and the arithmetical reasons why you do not deploy on Friday afternoons. Capacity planning, with the ×10 numbers and the load test that validates them. Costs, with the levers ordered by return and a real 35 % reduction without touching a single SLO. And the maintenance calendar, which consumes a quarter of the team and has to be defended as real work.

That closes module 11 and, with it, the Rutas Norte journey. We started with a company selling coach tickets and a handful of loose manifests. We gave it Pods, Deployments and Services; configuration and secrets; networking, Ingress and TLS; persistent storage with backups; StatefulSets, operators and advanced scheduling; observability with metrics, logs and alerts; security from the image through to RBAC; autoscaling for the May bank-holiday weekend; Helm, Kustomize, GitOps and a managed cluster. And in this last module we have seen it all working together: a complete web application in production with its thirteen objects, a stateful database with its measured failover and its timed restore, a pipeline that goes from git push to production without ever touching kubectl apply, canary deployments that abort by themselves when Prometheus says something is wrong, a fleet of clusters governed from one repository, and the daily operation that holds all of the above up.

If you have got this far, there is something worth saying plainly: you already have the knowledge the Kubernetes certifications measure. The CKA asks about clusters, nodes, networking, storage and troubleshooting: that is what we did in modules 1 to 7 and in this module's incidents. The CKAD asks about Pods, Deployments, configuration, probes and jobs: that is modules 2 to 6 and the whole of lesson 11-01. The CKS asks about hardening, policies, image security and auditing: that is the whole of module 8.

What is missing is not knowledge, it is format. The exams are practical, timed, with a terminal and the official documentation as your only help, and they punish slowness with kubectl mercilessly. It is a different kind of skill from the one needed to operate a real platform: there, what matters is deciding well; here, you also have to do it fast and without second-guessing the syntax.

That is what module 12, Preparing for Kubernetes Certification, is about: what exactly each exam measures, how the time is distributed, which kubectl shortcuts make the difference between finishing and not finishing, what to look for in the permitted documentation and how to prepare with timed mock exams. We will start with the CKA.