Fifty-five lessons. srv-tramontana is installed, hardened, monitorable, backed up, automated, encrypted in transit and running a database that no longer uses the factory configuration. You know how to rebuild it in fifty minutes, recover the database to a specific second, diagnose a latency problem with eBPF and decide with numbers whether high availability is needed.

And even so something is missing, because "it works" and "it is in production" are not the same thing. A production system is one that other people depend on without knowing it exists: it has an owner, an agreed service objective, somebody who looks at it every morning, alerts that get attended to, a procedure for when it breaks and a record of why it is the way it is. None of that is software; all of it is operations.

This lesson closes the course by putting that layer in place. The complete checklist with its evidence, the monitoring with Prometheus and Grafana that has been postponed since 05-07, alerts you can attend to without hating them, the daily routine, the blameless post-incident review, and the error budget that turns the 99.8 % from 07-07 into an executable decision. And the three debts still open since Module 5, which are closed today.

1. What being in production means
2. The going-to-production checklist
3. Monitoring and alerting: they are not the same thing
4. Prometheus and node_exporter
5. The essential PromQL
6. Metrics of your own: the ones you already have without knowing it
7. Grafana and the minimum dashboard
8. Alerts you can act on
9. Centralised logs: the debt from 05-06
10. Daily, weekly and monthly operations
11. The duty log and the change log
12. The blameless post-incident review
13. Change management and deployment windows
14. SLIs, SLOs and the error budget
15. Closing the outstanding debts
16. Compliance: GDPR and data retention
17. Closing the course

The last row is the one that sums the lesson up. The difference between an amateur system and a professional one is not the technology: it is that when somebody asks "is it all right?" there is a figure, not an impression.

And there is a practical test, more useful than any definition:

Could you go away for two weeks' holiday without a computer?

To answer yes you need six things: that the system heals itself for the anticipated failures, that somebody receives the alerts, that that somebody has written procedures they can follow without being you, that the backups verify themselves, that there is an escalation path, and that a record exists of what has been changed and why. This lesson builds all six.

Every row carries verifiable evidence: a command that returns a result, not a box ticked in good conscience. Without evidence, a checklist is a list of good intentions.

Summary: 30 rows of 42 satisfied. Nine are closed in this lesson; three remain as planned work with a date. That figure is in itself evidence number 43: knowing exactly what is missing.

It is the distinction that avoids the most expensive mistake in this area.

Monitor everything you can; alert on almost nothing. It is counter-intuitive and it is the only way for alerts to be of any use. A system that sends forty notifications a day is a system without alerts, because nobody reads them.

From 05-07, now with real instrumentation:

Two nuances that make the difference between measuring and measuring well:

Latency is measured in percentiles, never as an average. An average of 80 ms can hide the fact that 1 % of users wait 4 seconds. And you have to separate the latency of successful requests from that of failed ones: a 500 error returned in 2 ms improves the average and makes the service worse.

The latency of errors is excluded from the SLI. Otherwise a total outage — where everything fails quickly — would show up as a performance improvement.

And the USE method, complementary, for each resource: Utilisation, Saturation and Errors.

Prometheus is a time-series database that scrapes metrics: instead of services sending it data, it asks HTTP endpoints for them periodically.

That second row is an enormous advantage: with scraping, a service that is down immediately produces up == 0, which is an explicit signal.

The package's units are reasonable but they do not meet the course's standard. Drop-ins, as always:

promtool check config is Prometheus' nginx -t, and it is used the same way: always before reloading.

About retention and disk, which is the question that always comes up:

Thirty days is enough to investigate incidents and see monthly trends. For years of history there are Thanos or Mimir, and for Tramontana they are unnecessary.

PromQL is intimidating at first and it boils down to five patterns that cover 90 % of the cases.

1. up: the most important and the simplest.

2. rate() over counters. A counter only goes up. Its absolute value says nothing; what matters is the rate at which it grows.

rate() handles restarts correctly: if the counter goes back to zero because the process restarted, it detects it and does not produce an absurd negative value.

Rule: rate() only over counters (_total). For gauges — memory, temperature, connections — the value is used directly.

3. Aggregations and ratios.

4. histogram_quantile(): the latency percentiles.

The le label (less or equal) defines the histogram's buckets and must be kept in the by. It is the most common PromQL mistake: aggregating without by (le) and getting meaningless results.

5. Prediction, which is what makes an alert useful.

Alerting on "the disk is 90 % full" is alerting too late or alerting for nothing — a disk sitting stable at 91 % is not a problem. Alerting on "at this rate it fills up in four hours" is actionable, which is the property that defines a good alert.

node_exporter and friends give you technical metrics. The ones that really matter are those that answer business questions, and most of them you are already calculating in the course's scripts — they just have to be exposed.

The mechanism is the textfile collector: any script leaves a .prom file in a directory and node_exporter publishes it.

Four design decisions in that script:

1. The atomic write with mktemp and mv. node_exporter can read the file at any moment; a truncated one makes it discard the whole contents.
2. umask 022, against the course's general convention: node_exporter runs as a different user and needs to read it. It is a justified exception, and that is why it is commented.
3. Metric 8, the freshness mark, is what makes the other seven trustworthy. Without it, if this script stops running, Prometheus would carry on publishing the last known values indefinitely and everything would look fine for ever. With it, you can alert that the metrics have gone stale.
4. The business metric, tramontana_bookings_total, is the one Marta cares about and the one that detects the most dangerous failure: the one that breaks nothing. If the system answers 200 to everything but nobody manages to book, no technical metric will reveal it.

And for the application, what Luis has to expose on /metrics:

And the access, making use of what is already built: over the VPN from 08-04, without publishing anything new.

The minimum dashboard, six panels and not one more:

A piece of advice that goes against instinct: a dashboard with forty charts does not get looked at. The main dashboard has to fit on one screen and answer "is it all right?" in three seconds. The detail lives on secondary dashboards you go to when you need them.

Every alert must require immediate human action. If the recipient cannot do anything, or it can wait until tomorrow, it is not an alert: it is a dashboard or a report.

Alert fatigue is the characteristic failure mode of monitoring, and its progression is always the same:

1. Alerts are configured for everything, "just in case".
2. Ten a day arrive, almost all of them irrelevant.
3. People start ignoring them.
4. A mail filter is created that archives them.
5. The important alert arrives and nobody sees it.

The system ends up worse than with no alerts at all, because there is a false sense of vigilance. And the remedy is counter-intuitive: delete alerts.

Rule: alert on symptoms, diagnose with causes. A for: 5m on the error rate detects any failure that affects the user, including the ones nobody anticipated. A 90 % CPU alert fires when the nightly backup runs and the service is working perfectly.

With two legitimate exceptions, which are cause alerts and must exist because their symptom arrives too late:

• The disk is going to fill up. By the time the symptom appears, the service is already down.
• The certificate is going to expire. By the time the symptom appears, the site no longer loads.

Both share the key property: they warn hours or days in advance and the action is obvious.

The inhibit_rules deserve attention: when the service goes down, the latency, error and database-connection alerts would fire as well. Receiving five notifications about the same incident is alert fatigue in its purest form. Inhibition sends one: the cause.

And the scheduled silence during maintenance, without which the deployment window generates an avalanche of notifications:

Silences are always set with an expiry. An indefinite silence set on a Tuesday is an alert that will never sound again, and nobody will remember.

In 05-06 one piece was left pending: the journal is persistent and local. With a single server that works; as soon as there are two — the application nodes from 07-07, the load balancer, the replica — diagnosing means logging into each machine and correlating by hand.

Why centralising is needed:

That second row is pure security and it links back to 06-04: if the logs only live on the attacked machine, the attacker edits them. It is the same reasoning by which the AIDE database lives outside the server.

Loki as the lightweight option:

Loki is the choice for Tramontana for two reasons: it does not index the full text — which is what makes Elasticsearch expensive — and it shares labels and syntax with Prometheus, so that you can jump from a latency chart to the logs of that same instant on the same dashboard. That correlation is what makes a logging system useful.

And the obligatory warning: logs contain personal data — IP addresses, user identifiers, sometimes email addresses. Centralising them multiplies the reach of a leak. Bounded retention, restricted access and the same treatment as the database.

The duty log is a text file where you note down what happens. It sounds trivial and it is the tool that saves the most time during an incident, because it answers "has this happened to us before?".

Four rules to make it work:

1. It is written at the time, not at the end of the day. What gets postponed does not get written.
2. Timestamps always, so that it can be correlated with the charts.
3. The observations are worth as much as the incidents. The 16:10 entry is the one that will explain an alert two months from now.
4. It lives outside the server — in the Ansible repository — because it will be needed precisely when the server is not there.

The change log already exists: it is the git log of ~/tramontana-infra. The discipline is that every configuration change goes through it, so that "what changed before it started failing?" can be answered.

An incident is a failure of the system, not of a person. If somebody was able to delete 40,000 rows with one command, the problem is that the system allowed it without friction, not that that person made a mistake.

This is not corporate kindness: it is the only way to get truthful information. In a culture where culprits are sought, people hide their mistakes, and then incidents repeat themselves because nobody knows they happened. The blameless review is an engineering decision.

Post-incident review: accidental deletion of bookings

Incident: INC-2026-003 · Date: 2026-08-18 · Severity: High Duration of impact: 34 min · Data lost: none Written by: Operations · Reviewed with: Marta, Luis

1. What happened

During a routine cleanup of old data, a DELETE whose WHERE clause was not the intended one was run against the production database, removing 40,218 historical bookings. It was detected 19 minutes later and the rows were recovered by point-in-time recovery. There was no data loss and no interruption of service.

2. Impact

Dimension	Impact
Service availability	None: the site carried on working
Data permanently lost	None
Data temporarily inaccessible	40,218 historical bookings, 34 min
Customers affected	None (data for internal consultation)
Error budget consumed	0 %

3. Timeline

Time	Event
14:30	The cleanup of data prior to 2024 begins
14:32	The DELETE is run with the wrong condition
14:51	A report returns anomalous figures: it is detected
14:53	All further writing is stopped. The incident is opened
14:58	The exact timestamp is located in pg_stat_statements
15:02	The PITR restore begins on srv-tramontana-test
15:19	Recovery completed and verified on the test machine
15:24	Data reimported into production with psql -1
15:26	Final verification. Incident closed

4. Root cause: the five whys

#	Question	Answer
1	Why were 40,218 bookings deleted?	A DELETE with the wrong WHERE
2	Why was it run with the wrong condition?	It was typed by hand, with no prior SELECT
3	Why was it typed by hand?	There is no procedure for data cleanups
4	Why is there none?	A bulk cleanup had never been done before
5	Why was one needed now?	The audit table grew with no retention policy

Root cause: the absence of a data retention policy forced an improvised manual cleanup, with no procedure and no safeguards.

And an observation about the method: the five whys cut through the easy answer — "somebody mistyped something" — until they reach a systemic and actionable cause. Stopping at why number 2 would have produced the useless conclusion that "we have to be more careful".

5. What went well

This section is compulsory and it is always forgotten:

• Point-in-time recovery worked exactly as rehearsed (RB-DB-02, 08-02). The quarterly rehearsal proved its worth three days later.
• Restoring on a separate machine preserved the later transactions. Restoring over production would have lost 19 minutes of real bookings.
• Detection in 19 minutes was reasonable, though it can be improved.
• It was reported immediately, with no concealment. That made it possible to act quickly.

6. What changes

#	Action	Owner	Date	Status
1	A retention policy for audit agreed with Marta	Marta + Ops	25/08	Pending
2	A cleanup runbook: a mandatory SELECT count(*) before any DELETE	Ops	22/08	In progress
3	An svc_cleanup role with statement_timeout and no bulk DELETE permission	Ops	25/08	Pending
4	Purging scheduled as an idempotent, tested timer, not manual	Ops	31/08	Pending
5	An alert if a table loses more than 10 % of its rows in 5 min	Ops	25/08	Pending
6	Production psql with \set ON_ERROR_ROLLBACK off and a warning in the prompt	Ops	22/08	Done

No action is "be more careful". A corrective action that depends on human attention is not a corrective action: it is a wish.

7. Lessons

1. Recovery rehearsals pay for themselves. The one in 08-02 was done on the 15th; on the 18th it was genuinely needed, and nobody had to improvise.
2. A table that grows with no retention policy ends up forcing a risky operation. Silent growth is a debt that gets collected all at once.
3. Detecting in 19 minutes is a long time for a bulk deletion. Action 5 will bring it down to under 5.

Most interruptions do not come from hardware failures: they come from changes. And yet change is necessary. Managing it means putting friction on it in proportion to the risk, not preventing it.

Tramontana's window: Tuesdays and Wednesdays, from 10:00 to 12:00. And the reasons why you do not deploy on a Friday afternoon, which is the industry's most famous rule:

1. Problems appear under real load, hours later. On a Friday at 18:00, that load arrives on Monday.
2. People go home. If something fails at 20:00 on a Friday, whoever deployed it is no longer around.
3. The weekend is when most bookings are made at Tramontana: the worst possible moment for a failure.
4. A Friday incident drags through the weekend, with the person on call working on a change they did not make.

The rule, correctly formulated: you do not deploy if there is not enough time ahead to detect and revert the problem with the available team. Tuesday morning satisfies that; Friday afternoon does not.

The pre-deployment checklist:

Three concepts that get confused and are different:

The SLO is always set stricter than the SLA, so that there is headroom before the contract is breached.

Taking the 99.8 % proposed in 07-07 and turning it into concrete numbers:

SLI: the proportion of HTTP requests to bookings.tramontana.example answered with a code below 500 and in under 2 seconds.

SLO: ≥ 99.8 % of those requests, measured over a rolling 30-day window.

Here is the idea that changes the way you work:

Error budget = 100 % − SLO. It is the amount of failure you can afford. It is not a leftover: it is a resource that gets spent.

Tramontana can be down for 1 h 26 min every 30 days and still meet its objective. That time is a budget that can be spent deliberately: on risky deployments, on upgrades, on experiments.

And out of that comes a management rule that replaces the endless arguments between "we have to move faster" and "we have to be more stable":

31.4 % consumed: there is headroom. That number goes on the main dashboard, and it is what turns "shall we deploy?" from an argument of opinions into a query against data.

The most valuable thing about the error budget, and it usually comes as a surprise: exhausting it is as bad as not spending it. A team that ends the month with 5 % consumed is being too conservative — it could have deployed more, delivered more value and taken more risk within what was agreed. Excessive reliability has a cost too: the features that never got built.

The problem. restic uploads the backups with a token that can also delete. An attacker with access to srv-tramontana can run restic forget --prune and destroy the whole history. It is exactly how modern ransomware operates: first it deletes the backups, then it encrypts the data.

The solution: an append-only token, plus a restic in --append-only mode on the repository side.

The general principle, applicable far beyond this: the backed-up system must not be able to destroy its own backups. If it can, they are not backups: they are a replica with a delay.

Checklist row 23: closed.

What it is. authorized_keys2 is an obsolete file from OpenSSH 2.x, when there were separate files for SSH-1 and SSH-2 keys. Modern versions of OpenSSH ignore it entirely, unless AuthorizedKeysFile mentions it explicitly.

And there is the problem: Ubuntu's default configuration does include it. That 2048-bit RSA key, belonging to a provider you no longer work with, gives access to the server today.

It is a perfect example of a real risk: it is not a software vulnerability, it is a file nobody looked at for eighteen months.

And in Ansible, so that it is permanent and verified:

The lesson of method: in an access inventory, a file that cannot be explained is a file that gets withdrawn. And the underlying mistake was never having a periodic access review — which is now the monthly row in the table in section 10.

Checklist row 40: closed.

It is closed by formalising what was proposed in 07-07:

Internal service level agreement · Tramontana Bookings Agreed on 18/08/2026 between Operations and Management (Marta Vidal). Review: August 2027.

Item	Value
SLI	Requests with a code below 500 and latency under 2 s
SLO	99.8 % over a rolling 30-day window
Error budget	1 h 26 min every 30 days
Maintenance window	Tuesdays and Wednesdays, 10:00-12:00
RPO	15 minutes
RTO	2 hours
On-call hours	Working days 8:00-20:00; critical 24×7
Compliance review	Monthly, on the dashboard

Checklist row 33: closed.

Tramontana holds customers' personal data: name, contact details and dates of stay. That triggers specific obligations under the General Data Protection Regulation, and some of them are technical and they are yours.

And the point that connects directly with the incident in section 12: the audit table takes up 1,204 MB, more than all the booking data put together, and nobody has decided how long it is kept for. That is not only a space problem: keeping personal data with no defined time limit is a breach of the storage limitation principle.

Anonymising instead of deleting preserves the statistical usefulness — how many bookings there were, for which cottages — without keeping the personal data. It is the technically correct solution and the one most often overlooked.

The final warning, and it is important: GDPR is not a technical matter, it is a legal matter with technical implications. Nothing in this section replaces professional advice. What is your responsibility as an administrator is: that the technical measures exist, that the retention is enforceable with one command, that there is traceability of access, and that whoever decides — Marta — has the information to decide. An administrator who implements a purge before anybody has decided the time limits is taking a legal decision without the authority to do so.

• Confusing monitoring with alerting. Monitor everything; alert on almost nothing.
• Alerting on causes instead of symptoms. "CPU at 90 %" fires when the backup runs. "More than 5 % errors" detects any real failure.
• Not pruning alerts. Ten notifications a day is equivalent to none. The monthly alert review is as important as creating them.
• Alerts with no action. If the recipient cannot do anything, it is a dashboard, not an alert. Every alert carries its action and its link to the runbook.
• Silences with no expiry. An indefinite silence is an alert switched off for ever.
• Not monitoring the monitoring. A dead metrics script makes everything look fine indefinitely. The freshness mark solves it.
• Average latency instead of percentiles. An average of 80 ms hides the fact that 1 % waits 4 seconds.
• Including errors in the latency. A total outage, where everything fails quickly, would show up as an improvement in performance.
• rate() over a gauge. It only makes sense over counters (_total).
• Aggregating histograms without by (le). It is the classic PromQL mistake: the percentiles come out meaningless.
• High-cardinality labels. A label with the booking ID creates one series per booking and blows up Prometheus' memory.
• A dashboard with forty charts. Nobody looks at it. The main one has to fit on one screen.
• Post-incident reviews that look for culprits. They guarantee that the next incident gets hidden.
• Corrective actions of the "be more careful" kind. They are not actions: they are wishes. They have to change the system.
• Deploying on a Friday afternoon. The problems appear under real load and by then there is nobody around.
• An SLO with no error budget. An objective without its operational consequence changes no decisions.
• Spending 5 % of the error budget. You are being too conservative: you could have delivered more value.
• Backups the server itself can delete. They are not backups: they are a replica with a delay, and ransomware knows it.
• Not reviewing access periodically. An authorized_keys2 from a provider of eighteen months ago gives access today.
• Keeping personal data with no defined time limit. It is a GDPR breach, not just a space problem.
• A tip on method. When somebody asks "is the system all right?", if the answer is not a number, you are not in production.

At 03:47 HighErrorRate fires. Document the complete incident response — from the notification to the closure — and write the resulting post-incident review.

Design the monthly alert review: what is measured about each alert, the criteria for deciding whether to delete it, adjust it or keep it, and apply the method to the alerts configured in this lesson.

Marta asks for a system status report for the annual meeting with the partners: what has been done this year, where things stand and what is needed. Write it.

Solution 1

03:47 — The notification.

03:49 — Confirm the problem is real, before touching anything.

Confirmed: one user in three is getting an error. It is not a false alarm.

03:51 — Narrow the scope down with the dashboard, not by guessing.

The latency rose three minutes before the errors, and there are 47 clients waiting for a connection. The bottleneck is in the database, not in the application and not in Nginx.

03:53 — The change log, which is the obligatory question.

No recent changes. It is not a deployment: it is something that has happened on its own.

03:55 — Diagnosis in the database.

Root cause found in eight minutes. A VACUUM FULL on app.availability takes an ACCESS EXCLUSIVE lock that prevents all reading and writing of that table. The requests pile up waiting, they exhaust the PgBouncer pool, and the application returns 502s.

application_name = psql and started at 03:30: somebody launched it by hand. Checking the duty log confirms it — a note from the previous afternoon: "availability at 31.7 % dead tuples, look into it".

03:58 — Mitigation.

04:00 — Verify and close.

04:05 — Note it in the duty log while it is fresh.

Post-incident review: 502 errors caused by a VACUUM FULL lock

Incident: INC-2026-004 · Date: 2026-08-19 · Severity: Critical Duration of impact: 17 min (03:41-03:58) · Detection: 6 min

1. What happened

A VACUUM FULL run manually on app.availability acquired an exclusive lock that prevented all reading and writing of the table for 28 minutes. The requests piled up, the connection pool was exhausted and the application returned 502 errors to 34 % of requests for 17 minutes.

2. Impact

Dimension	Value
Duration of impact	17 min
Failed requests	~2,100 (34 % of 6,200)
Estimated bookings lost	1-2 (early morning, low traffic)
Error budget consumed	19.7 % of the month (17 of 86 min)
Data lost	None

3. Timeline

Time	Event
03:30	VACUUM FULL app.availability is launched manually
03:38	p95 latency rises to 8 s (an early symptom, with no alert)
03:41	The first 502s: the PgBouncer pool is exhausted
03:47	HighErrorRate fires and the notification arrives
03:49	The problem is confirmed
03:51	It is narrowed down with the dashboard: the database
03:55	The lock and the VACUUM FULL are identified
03:58	pg_cancel_backend. Service restored
04:00	Verified. Incident closed

4. Root cause: the five whys

#	Question	Answer
1	Why were there 502s?	The application could not query the database
2	Why could it not?	An ACCESS EXCLUSIVE lock on app.availability
3	Why was that lock there?	A manual VACUUM FULL
4	Why was it run by hand in production?	31.7 % dead tuples were spotted and somebody wanted to fix it quickly
5	Why was there so much bloat?	autovacuum is not tuned for that table

Root cause: autovacuum with the default values (a 20 % threshold) does not keep up with the write rate of app.availability. The resulting bloat prompted a manual intervention with a tool whose locking was not anticipated.

Contributing cause: there was no control preventing a blocking operation being run in production, and no alert to warn before users were affected.

5. What went well

• The alert was a symptom alert and it worked: it detected a failure nobody had anticipated. A cause alert on VACUUM did not exist and would not have been needed.
• The dashboard narrowed the problem down in two minutes. Without it, the diagnosis would have been blind.
• The change log ruled out a deployment in thirty seconds.
• The mitigation was reversible: cancelling rather than terminating the connection.
• The duty log gave the context — the previous day's note — that explained who and why.

6. What changes

#	Action	Owner	Date	Status
1	Tune autovacuum for app.availability to 2 % (08-02)	Ops	19/08	Done
2	lock_timeout = 5s by default in the production psql session	Ops	20/08	Done
3	Install pg_repack and document it as the tool for bloat	Ops	26/08	Pending
4	A LongLockWait alert (> 30 s) — it warns before the symptom	Ops	22/08	Pending
5	Runbook RB-DB-03: database maintenance, with the list of blocking operations	Ops	26/08	Pending
6	A dashboard with dead tuples per table, so it is not discovered by chance	Ops	22/08	Pending
7	No manual maintenance operations in production outside the window	The team	20/08	Agreed

# Action 2: /etc/postgresql/16/main/conf.d/50-protection.conf
# If a query does not get a lock within 5 s, it fails instead of
# waiting. A query that fails fast is infinitely preferable to one
# that blocks the entire service.
lock_timeout = 5s

7. Lessons

1. The latency rose 9 minutes before the errors. That is the margin we lost by not having an early alert. Action 4 recovers it.
2. The intention was good and the result was an outage. It is not a failure of the person: it is a failure of the system, which allowed a blocking operation to be run in production with no friction at all. Actions 2, 5 and 7 add that friction.
3. 19.7 % of the error budget in a 17-minute incident demonstrates why the SLO changes decisions: with two incidents like that, the month enters the caution zone and deployments are moderated.
4. VACUUM FULL is not "a bigger vacuum". It locks the whole table. It deserves to be in a runbook with that warning in capitals.

Solution 2

Why this review exists. Alerts are created after every incident and they are almost never removed. In a year, a system accumulates thirty alerts of which five are useful. The monthly review is the counterweight.

The four things measured about each alert:

The decision tree, applied to each alert:

Applied to this lesson's alerts:

Analysis and correction of the two problematic ones:

BackupTooOld: 412 firings, 0.2 % precision. It is the textbook case of alert fatigue: it fired every 30 minutes, every night, between the backup running and it being verified. Four hundred and twelve notifications nobody read, and which on top of that trained the team to ignore the channel.

LatencyDegraded: 38 firings, 21 % precision, 45 minutes until a reaction. The 45 minutes are the revealing figure: the team was already ignoring it. It fires every afternoon during Marta's billing report, which is slow and expected.

And for the reports, which are worth watching but with a different threshold:

The result of the review:

Four principles this review sums up:

1. An alert with low precision is worse than none at all, because it trains the team to ignore the channel the good ones also arrive on.
2. Reaction time measures whether the alert is attended to. Forty-five minutes means it is being ignored, and that figure is more honest than any opinion.
3. An alert that never fires is not useless, if it covers a serious risk and its cost is zero. ServiceDown has not fired in a year and it must stay there.
4. Adjust before deleting. Both problematic ones covered real risks; the problem was in the threshold, not in the idea.

And a final note on method: this review is written into the duty log, with the before and after of each modified alert. Six months from now, when somebody wonders why the BackupTooOld threshold is 26 hours and not 8, the answer will be written down.

Solution 3

Annual systems report · Tramontana Bookings

For: Marta Vidal and partners · From: Systems Operations Year: 2026 · Date: 18 August 2026

---

Executive summary

Over the last year, the system that holds up Tramontana Bookings has gone from being a hand-configured machine that nobody but me knew how to rebuild to a documented, automated, monitored infrastructure with written procedures that somebody else could follow.

The three most important changes, one line each:

• Our customers' data now travels encrypted. Until June it did not.
• We can recover the database to any specific instant, and we have rehearsed it: 24 minutes.
• Rebuilding the whole server went from 8 hours to 50 minutes, and that is measured, not estimated.

Availability measured over the last 30 days: 99.94 %, above the 99.8 % objective we are proposing to formalise.

---

1. Where we were and where we are

	A year ago	Today
The server's configuration	In my head	In code, versioned
Rebuilding after a disaster	8 h estimated	50 min measured
Data we could lose	Up to 24 h	15 minutes
Recovering an accidental deletion	Impossible without losing a day	To the second before
Have the backups been tested?	Never	A documented quarterly rehearsal
Encryption of web traffic	No	Yes
Secure remote access	Only from the office	From anywhere, over an encrypted tunnel
Do we know whether it is all right?	"I think so"	99.94 %, on a dashboard
If something fails at 3 a.m.	We find out in the morning	An alert to the phone
Operational documentation	None	6 written and rehearsed procedures

---

2. What has been done, grouped

Reliability. Complete automation of the configuration: the server is rebuilt from scratch with one command, and that has cut disaster recovery time from eight hours to fifty minutes. It is the improvement that has increased availability the most, and it has not cost a euro in equipment.

Data protection. Automatic verified backups, encrypted, stored off the server and — since this month — impossible to delete from the server itself for 30 days, which is the concrete defence against a ransomware attack. And the ability to rewind the database to any instant, rehearsed.

Security. A restrictive firewall, intrusion detection, access auditing, passwords and certificates stored encrypted, and the service running with the fewest possible permissions. An independent automated assessment scores our hardening level at 82 out of 100, where a default installation is around 60.

Performance. Tuning the database to the actual machine, which has reduced disk access by 93 %. And a report query that used to take 3.8 seconds now takes 0.18.

Visibility. This is the most recent change and the one that most changes the day to day: a dashboard that answers "is it all right?" in three seconds, with 30 days of history, and alerts that warn before the problem affects customers.

---

3. This year's incidents

Date	What happened	Impact	Status
17/08	The backup disk filled up and blocked the database log	None: detected beforehand	Resolved + a new alert
18/08	Accidental deletion of 40,218 historical bookings	None: recovered in 34 min	Resolved + 6 measures
19/08	Errors for 17 min caused by a maintenance operation	~2,100 failed requests	Resolved + 7 measures

No incident caused data loss. All three were analysed with the same method — what happened, why, what changes — and generated fifteen concrete improvements, all of them applied or with a date.

I want to highlight the one on 18 August: 40,000 bookings were recovered without losing anything, using a procedure we had rehearsed three days earlier. That rehearsal, which looked like bureaucracy, paid for itself in 72 hours.

---

4. What we propose agreeing

A service objective. Until now we had none, and without one it is impossible to decide how much to invest. We propose:

Item	Proposal	What it means in practice
Availability	99.8 % monthly	At most 1 h 26 min of failure every 30 days
Recoverable data	15 minutes	In the worst case we would lose 15 min of bookings
Recovery time	2 hours	From a total disaster to the service restored
Change window	Tuesday and Wednesday mornings	Never Friday afternoon or the weekend

We are above that objective today (99.94 %). We are proposing it that way deliberately: an objective that is already met comfortably allows improvements to be deployed calmly, and one that is too demanding would force us to slow the product's development.

And one idea I want to explain because it changes how we decide. That 0.2 % of margin is a budget: we can "spend" it on changes and improvements. If in a given month we use less than half of it, we carry on deploying normally. If we get close to the limit, new features are frozen and the effort goes into stabilising. It turns the argument "are we going too fast?" into a query against a number. This month we have used 31 % of it.

---

5. What is needed

High priority, a management decision:

1. A data retention policy. We keep the application's activity history with no defined time limit, and it already takes up more space than all the bookings put together. As well as being a space problem, keeping personal data with no defined period is not defensible under the Data Protection Regulation. I need a decision on how long we keep each thing; I can apply it in a week. I recommend legal advice to set the periods.
2. A record of processing activities. It is a document GDPR requires and that we do not have. It is not technical work, but it needs my input.

Medium priority, moderate investment:

3. A second application server with load balancing. It is the recommendation I analysed in detail: it would eliminate interruptions caused by deployments and by a machine failing, which are the majority. Cost: two machines and a week of work, plus three or four weeks of prior development with Luis. It would take us from the current 99.8 % to a much more comfortable margin.
4. A database replica. Justified above all because it would let us run the heavy reports without affecting the website, and incidentally upgrade without cutting the service.

What I do NOT recommend, and I want it in writing:

• A full high-availability architecture (five machines, automatic server failover). Analysed with numbers: it costs several times more than option 3 and delivers barely 670 € of additional annual saving. And it would add a complexity that with the current staffing we cannot watch properly — a sophisticated system badly maintained is less reliable than a simple one well looked after.
• Kubernetes, which is the fashionable technology. I have set it up and evaluated it in the test environment: at our size it would consume between two and four weeks of work a year just to keep it running, in order to solve problems we do not have.

---

6. Risks we are consciously accepting

I would rather they were written down than discovered on the bad day:

Risk	Consequence	Why we accept it
A single application machine	A hardware failure = the service down for up to 2 h	Proposal 3 would resolve it
Manual database failover	5-15 additional minutes in that case	Automating it requires 5 machines and would create a greater risk
A single person with operational knowledge	Holidays or illness = a slow response	Partly mitigated: written procedures
Dependence on one provider	An outage of theirs affects all of us	Normal at our scale

The third is the one that worries me most in the medium term, and it is the reason I have put so much effort this year into writing procedures: today, somebody with general technical knowledge could follow most of them without ever having worked on our system. A year ago, they could not.

---

7. Conclusion

The system is in a considerably better position than a year ago and — what matters most — it is in a position I can demonstrate with data rather than assert with confidence. We have gone from "I think it is all right" to "99.94 %, here is the dashboard".

What remains outstanding is mostly decisions, not technical work: how long we keep the data, what level of service we commit to providing, and whether we invest in the second server. With those three decisions taken, we have a system that can be depended on.

The course ends where it had to end: not with one more tool, but with the layer that turns a server that works into a service that can be depended on. You have a checklist of forty-two rows with verifiable evidence on each one, and you know exactly which ones are missing, which is a way of knowing that is worth more than believing they are all there. You have monitoring with thirty days of history, alerts that can be acted on because they are few and they are symptom alerts, and a dashboard that answers the only question that matters in three seconds.

Above all you have operational judgement. You know that monitoring and alerting are not the same thing and that you have to do a great deal of the first and very little of the second. You know that an alert with 2 % precision is worse than none, because it trains the team to ignore the channel the good ones also arrive on. You know that a post-incident review that looks for culprits guarantees that the next one gets hidden, and that a corrective action of the "be more careful" kind is not an action but a wish. And you know that the error budget turns "shall we deploy?" into a query against a number, with the conclusion that comes as a surprise: spending only 5 % of the budget means having been too conservative.

The three debts are closed. The off-site copy can no longer be deleted by the server it backs up, because a system that can destroy its own backups does not have backups. The authorized_keys2 turned out to be what these things usually are: not an exotic vulnerability, but a file from a provider of eighteen months ago that still gave access because nobody had looked at it — and what has been corrected is not the file, it is the absence of periodic review that let it sit there. And the availability objective finally exists, with its number, its window and its budget.

And with that the whole course closes. Eight modules.

You began in Module 1 not knowing what a kernel was, and you built the system from an installation image. In 2 you stopped fearing the terminal: permissions, inodes, links and man as the first port of call. In 3 the shell stopped being an interpreter of isolated orders and became a language: pipes, find, awk, processes, signals. In 4 you wrote scripts that do not break — set -euo pipefail, trap, flock, idempotence — and deploy.sh with its atomic rollback is still in production fifty lessons later. 5 was real administration: users, packages with pinning, LVM, hardened systemd, a persistent journal, and backups with an agreed RPO and RTO instead of improvised ones. 6 made security a discipline and not a list of tricks: a threat model, an allowlist firewall, AIDE with its database elsewhere, encrypted secrets and that TLS certificate that waited two modules. 7 looked under the bonnet — boot, strace, perf, eBPF, the kernel, KVM, containers, Ansible — and finished with the honest analysis of whether high availability was needed. And 8 applied all of it to complete builds, from beginning to end.

What you take away from here is not a list of commands. It is a method: measure before and after, because a tuning change without measurement is superstition. Test on the test machine before production. --dry-run before acting. A copy and a diff -u before editing. Validate before reloading, whether that is nginx -t, promtool check, testparm -s or pg_hba_file_rules. Never close the door you are coming in through. Silence if all is well. And always say what a measure protects against and what it does not, which is what separates an honest report from a sales pitch.

And an idea that has appeared in every module wearing different clothes: the professional answer is not always yes. No to the full high-availability architecture, no to Kubernetes, no to raising max_connections, no to copying six terabytes to the cloud. Knowing how to say no, with numbers and with an alternative, is worth as much as knowing how to build it.

On how to carry on. The LPIC-1 and LPIC-2 certifications or the RHCSA organise and accredit what you already know; the RHCSA is practical and demanding, and preparing for it genuinely teaches you something. The natural specialisations from here are four: reliability engineering, which is this last lesson taken to scale; platform and cloud, where Kubernetes and infrastructure as code are the day job; cybersecurity, extending Module 6; and databases, extending 08-02. Take part in the community — mailing lists, forums, a well-written bug report, a documentation fix — because explaining something is the best way to discover whether you understand it. And read the release notes of the kernel and of your distribution: it is the cheapest way of not falling behind.

But if you can keep only one thing, let it be this: keep the lab. That test VM, or the media server from 08-03, or the k3s cluster you decided not to take to production. A place of your own where you can break things on purpose, cause failures during working hours instead of waiting for them, and try out what you do not know yet. Everything you have learned on this course you have learned by doing it, and that is the only way there is. The difference between somebody who reads about systems and somebody who administers them is not what they have studied: it is what they have broken and put back together. Switch the machine on and carry on.