You can rebuild srv-tramontana in fifty minutes. You can test any change before applying it, detect an intrusion, encrypt the secrets, diagnose an anomalous latency and recover a broken boot. And even so, if that machine goes down, Tramontana Bookings is down.
A disk that fails, a power supply, a network outage at the provider, a kernel that will not boot after an update, or simply scheduled maintenance on the infrastructure it lives on. In the best case, fifty minutes of interruption — and that assumes somebody is awake to launch the playbook. All the work of seven modules rests on a single machine, which is the exact definition of a single point of failure.
This lesson attacks that problem, and it does so with a warning up front: high availability is expensive, complex and, done badly, it worsens reliability rather than improving it. A system with more pieces has more ways of failing. The lesson ends with an honest analysis of whether Tramontana needs this, because the professional answer is not always yes.
Contents
- Vocabulary: availability, nines, MTBF and MTTR
- Vertical and horizontal scaling
- Redundancy: active-passive and active-active
- State is the hard problem
- Load balancing: layers, algorithms and health checks
- HAProxy in practice
- High availability for the load balancer: keepalived and VRRP
- Split-brain and quorum
- Data replication
- Clusters, Pacemaker and fencing
- A design for Tramontana and the cost analysis
- Testing failures by causing them
Vocabulary: availability, nines, MTBF and MTTR
The field is full of terms that get used as synonyms and are not. Pinning them down saves expensive misunderstandings.
Availability is the fraction of time in which the service responds correctly. It is expressed as a percentage, and in "nines":
| Availability | Name | Maximum downtime per year | Per month | Per week |
|---|---|---|---|---|
| 99% | "two nines" | 3 days 15 h | 7 h 18 min | 1 h 41 min |
| 99.9% | "three nines" | 8 h 46 min | 43 min 50 s | 10 min 5 s |
| 99.95% | 4 h 23 min | 21 min 55 s | 5 min 2 s | |
| 99.99% | "four nines" | 52 min 34 s | 4 min 23 s | 1 min |
| 99.999% | "five nines" | 5 min 15 s | 26 s | 6 s |
Reading that table slowly corrects almost everybody's intuition. 99.99% means less than a minute of downtime a week, updates, reboots for a new kernel and provider failures included. With a single server, one monthly three-minute reboot already puts you below three and a half nines. And each additional nine multiplies the cost by roughly three.
Three distinctions worth having clear:
- Reliability is the probability of running without failure over an interval. A system can be highly available and not very reliable: it fails often, but it recovers in seconds.
- Durability refers to the data: the probability of not losing it. It is independent of availability — a system can be down and have its data perfectly safe.
- MTBF (mean time between failures) and MTTR (mean time to repair) break availability down:
And that formula contains the whole lesson's strategic decision: there are two ways of raising availability. Increasing MTBF — failing less: better hardware, more testing, fewer changes — or reducing MTTR — recovering sooner. In practice, reducing MTTR is almost always cheaper and more effective, because MTBF has a physical ceiling and MTTR does not.
It is exactly what you did in 07-06 without calling it that: taking MTTR from 8 hours to 50 minutes multiplied availability without buying a single component.
And the relationship with what you already have agreed with Marta:
| Metric | What it measures | Current value |
|---|---|---|
| RPO | How much data can be lost | 4 hours |
| RTO | How long the interruption can last | 2 hours (revised in 07-06) |
| Availability | Fraction of time running | No formal target |
There is a gap there: nobody has set an availability target. And without one, you cannot decide how much to invest.
Vertical and horizontal scaling
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| What you do | A bigger machine | More machines |
| Limit | The biggest hardware there is | Practically none |
| Cost | Grows more than linearly | Roughly linear |
| Complexity | None | High: distribution, state, consistency |
| Effect on availability | None: there is still a single point of failure | It is what enables it |
| Requires application changes | No | Almost always yes |
The row that matters here is the second to last. Doubling srv-tramontana's memory would make it faster and exactly as fragile. Horizontal scaling is what allows the failure of one machine not to be the failure of the service, and that is why this lesson is about horizontal scaling even though the problem is not one of capacity.
Redundancy: active-passive and active-active
Having two machines is not the same as having redundancy: you have to decide what each one does.
| Active-passive | Active-active | |
|---|---|---|
| The second node | Waits, serving no traffic | Serves traffic as well |
| Utilisation | 50% of the hardware | ~100% |
| Failover time | Seconds to minutes | Immediate |
| Complexity | Medium | High |
| It requires the application to… | Start on the other node | Be capable of running in several instances at once |
| Characteristic risk | The passive node not working when it is needed | Split-brain, data consistency |
The active-passive risk is always underestimated and deserves a name of its own: a passive node that is never used is a node you do not know works. It was installed eight months ago, it has not had the latest updates, it has an expired certificate or a full disk. On the day of the failure, you discover that the backup fails too. The only defence is to use it periodically: fail over deliberately every few weeks.
State is the hard problem
Here is the conceptual core of the lesson, and what separates a design that works from one that looks as though it works.
Replicating processes is easy: you start the same application on two machines and you have two. Replicating state is hard, and state lives in more places than it seems.
An inventory of Tramontana Bookings' state:
| State | Where it lives | The problem when duplicating |
|---|---|---|
| Booking data | PostgreSQL | The central problem: two databases diverge |
| Uploaded files | /opt/tramontana/shared/uploads |
A photo uploaded to node A is not on B |
| User sessions | In the process's memory | A request to node B does not recognise the session started on A |
| Logs | /var/log/tramontana/ |
They are spread across nodes: they have to be centralised to diagnose anything |
| Scheduled tasks | systemd timers | They would run TWICE: two backups, two purges |
| TLS certificate | /etc/letsencrypt/ |
It has to be synchronised or terminated at the load balancer |
That scheduled-tasks row is the one that causes the most trouble in practice, because nobody thinks about it when duplicating a server. With two identical nodes, tramontana-backup.timer fires at 02:30 on both: two simultaneous backups to the same destination, competing for the flock — which is local to each machine and therefore protects nothing. The solution requires a distributed lock or running the tasks only on one designated node.
A stateless application is one whose instances keep nothing between requests: all the state is in the database or in a shared store. It is the requirement for active-active, and it is almost never met without development work. In Tramontana:
| Component | Stateless? | What it would take |
|---|---|---|
| Serving pages and querying bookings | Yes | Nothing |
| User sessions | No | Move them to Redis or to the database |
| Photo uploads | No | Shared storage or an object store |
| Database | No, by definition | Replication |
Honest conclusion: Tramontana is not a stateless application today. Before setting anything up, two changes to the application are needed: externalised sessions and files in shared storage. That is development work, and telling Marta before buying servers is part of the job.
Load balancing: layers, algorithms and health checks
A load balancer distributes requests among several servers. It operates at one of two layers:
| Layer 4 (transport) | Layer 7 (application) | |
|---|---|---|
| What it inspects | IP and port | HTTP headers, paths, cookies |
| Performance | Very high | High, but lower |
| TLS termination | It cannot | Yes |
| Routing by path or domain | No | Yes |
| Retrying a failed request | No | Yes |
| Examples | IPVS, nftables, NLB |
HAProxy, Nginx, Traefik |
For HTTP, layer 7 is almost always the right one: it lets you terminate TLS at a single point, route by path, retry a request that fails and make meaningful health checks.
Algorithms
| Algorithm | How it distributes | When |
|---|---|---|
roundrobin |
In turns | Homogeneous requests, identical servers |
leastconn |
To whichever has fewest connections | Requests of variable duration |
source |
A hash of the source IP | When affinity is needed (with reservations) |
uri |
A hash of the path | Caches: the same path always to the same node |
random |
At random, with two probes | Many load balancers in parallel |
leastconn is the most suitable for Tramontana: an availability query takes 40 ms and a billing report takes several seconds, so distributing in turns would pile the slow ones onto the same node.
Health checks
This is the piece that makes the load balancer useful: without it, it would carry on sending traffic to a server that is down.
| Type | How it works | Advantage |
|---|---|---|
| Active | The load balancer probes every N seconds | It detects the failure even when there is no traffic |
| Passive | It watches the errors in real traffic | No added cost; it detects partial failures |
The right thing is to use both. And this is where something you have had since Module 4 fits in: health_check.sh returns 0, 1 or 2 — OK, warning, critical. That gradation is exactly what a load balancer needs, provided the application exposes an equivalent endpoint:
| State | HTTP | What the load balancer should do |
|---|---|---|
| 0 OK | 200 | Send traffic normally |
| 1 warning | 200 with a warning header | Send, but with less weight |
| 2 critical | 503 | Send no traffic |
And the distinction almost nobody gets right: a shallow health check is worse than none. If the health endpoint only says "I am alive" without checking the database, the load balancer will carry on sending traffic to a node that cannot serve a single request. If it checks too much — a heavy query, for instance — a database problem marks every node as down and the service disappears entirely, when it could at least have served the static pages.
The balance: check the critical dependencies with a cheap operation. A SELECT 1 with a time limit, not a report.
Sticky sessions: an anti-pattern
A sticky session ties each client to one node, by cookie or by IP. It solves the problem of in-memory sessions... and creates others:
- The distribution becomes unbalanced: one node can end up with three times the load.
- When a node goes down, all of its users lose their session at once.
- It prevents draining a node for maintenance without affecting anybody.
- It masks the real problem instead of solving it.
It is an acceptable stopgap while sessions are being externalised. As a permanent solution, no.
Connection draining
To deploy without cutting the service, a node has to be able to leave the rotation in an orderly way: stop receiving new requests and finish the ones it already has in flight.
# 1. Take the node out of the rotation (it stops receiving new requests)
$ echo "set server tramontana/app1 state drain" | \
sudo socat stdio /run/haproxy/admin.sock
# 2. Wait for the in-flight connections to finish
$ while [[ "$(echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock \
| awk -F, '$1=="tramontana" && $2=="app1" {print $5}')" != "0" ]]; do
sleep 1
done
# 3. Deploy in peace
$ ssh [email protected] 'sudo ~/scripts/deploy.sh 3.3.0'
# 4. Put it back in the rotation
$ echo "set server tramontana/app1 state ready" | \
sudo socat stdio /run/haproxy/admin.sockRepeated node by node, that is a zero-downtime deployment. And it is the answer to something left outstanding since 04-07: deploy.sh restarts the service, and that restart is a few seconds of rejected requests. With two nodes and draining, zero.
HAProxy in practice
# /etc/haproxy/haproxy.cfg
global
log /dev/log local0
chroot /var/lib/haproxy
# Admin socket: allows draining nodes on the fly
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4000
# Modern TLS profile (06-05). Never invent the cipher suite list.
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
# Retry on ANOTHER server if the first one fails. Only safe for
# idempotent requests: hence 'if-none', which excludes POST.
retry-on all-retryable-errors
option redispatch
retries 3
timeout connect 5s
timeout client 30s
timeout server 30s
timeout http-request 10s
# Draining: wait for connections to finish when taking a node out
default-server init-addr last,libc,none
# ---------- Entry point ----------
frontend tramontana_https
bind *:443 ssl crt /etc/haproxy/certs/bookings.tramontana.example.pem alpn h2,http/1.1
bind *:80
# Redirect HTTP to HTTPS, except for the Let's Encrypt challenge
acl acme_challenge path_beg /.well-known/acme-challenge/
http-request redirect scheme https code 301 unless { ssl_fc } || acme_challenge
# HSTS (06-05). Start with a short max-age and raise it later.
http-response set-header Strict-Transport-Security "max-age=63072000"
# Per-IP rate limiting: it complements fail2ban at layer 7
stick-table type ip size 100k expire 60s store http_req_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
default_backend tramontana_app
# ---------- Application servers ----------
backend tramontana_app
# leastconn: the requests vary greatly in duration
balance leastconn
# ACTIVE health check at layer 7.
# The /health endpoint checks the critical dependencies cheaply.
option httpchk
http-check send meth GET uri /health ver HTTP/1.1 hdr Host bookings.tramontana.example
http-check expect status 200
# Headers so the application knows who the real client is
http-request set-header X-Forwarded-Proto https if { ssl_fc }
option forwardfor
# inter: how often to probe. rise/fall: how many times in a row
# before considering the node healthy or down. slowstart: on its
# return, raise the weight gradually instead of taking the whole load at once.
server app1 10.0.2.21:8080 check inter 3s rise 2 fall 3 slowstart 30s maxconn 200
server app2 10.0.2.22:8080 check inter 3s rise 2 fall 3 slowstart 30s maxconn 200
# PASSIVE check: if a server returns errors on real traffic, it is
# taken out temporarily even if its /health responds.
option redispatch
# ---------- Statistics, internal network only ----------
listen stats
bind 10.0.2.20:8404
stats enable
stats uri /
stats refresh 5s
stats admin if TRUE
acl internal_network src 10.0.2.0/24
http-request deny unless internal_network$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid
$ sudo systemctl reload haproxy
$ echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | \
awk -F, 'NR==1 || $1=="tramontana_app" {print $1","$2","$18","$5}'
# pxname,svname,status,scur
tramontana_app,app1,UP,12
tramontana_app,app2,UP,9
tramontana_app,BACKEND,UP,21Four details in that configuration deserve an explanation:
slowstart 30s. When a node comes back after a failure, its caches are cold and its connection pools empty. Sending it a third of the traffic all at once can bring it down again, and start a cycle of outages. slowstart raises its weight gradually over 30 seconds.
rise 2 fall 3. Asymmetric on purpose: three consecutive failures are needed to take a node out (avoiding false positives from a spike) but only two successes to bring it back. Taking a node out on a single transient failure is how a load balancer causes an outage.
option redispatch with retry-on. If the chosen server fails, it retries on another. With one important nuance: retrying a POST can duplicate a booking. HAProxy only retries when the request has not been sent or the method is idempotent.
The stick-table with rate limiting. It is 06-03's fail2ban at layer 7: it counts requests per IP in 10-second windows and returns 429 when the threshold is exceeded. Complementary, not a substitute: fail2ban blocks at the network level and this one at the application level.
Nginx can do the same thing with upstream, and it is the natural option if you already have it as a reverse proxy — which is what you will set up in 08-01:
upstream tramontana {
least_conn;
server 10.0.2.21:8080 max_fails=3 fail_timeout=30s;
server 10.0.2.22:8080 max_fails=3 fail_timeout=30s;
}The practical difference: Nginx only does passive checks in its free version; the active ones belong to the commercial version. HAProxy has them as standard, and that is why it is preferable when balancing is the main job.
High availability for the load balancer: keepalived and VRRP
And here is the most common design error in the field, so frequent that it deserves stating on its own:
Putting a load balancer in front of two servers does not remove the single point of failure: it moves it to the load balancer.
Before, you had one machine that could go down. Now you have three, and if the one that distributes goes down, the service disappears just the same — with the aggravating factor that there are now more pieces that can fail.
The solution is VRRP (Virtual Router Redundancy Protocol): two load balancers share a floating virtual IP, the one DNS points at. One holds it; if it stops advertising, the other takes it over within seconds.
# /etc/keepalived/keepalived.conf — PRIMARY NODE (lb1, 10.0.2.20)
global_defs {
router_id lb1
enable_script_security
script_user keepalived_script
}
# Check on the LOCAL service. If HAProxy dies on this node, the priority
# drops and the other one takes the IP. Without this, a node with
# keepalived alive and HAProxy dead keeps the IP and the service goes down.
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight -40 # drops the priority by 40 points on failure
fall 2
rise 2
}
vrrp_instance TRAMONTANA_VIP {
state MASTER
interface enp0s3
virtual_router_id 51 # IDENTICAL on both nodes
priority 100 # higher than the backup
advert_int 1 # advertise every second
authentication {
auth_type PASS
auth_pass {{ vault_vrrp_pass }}
}
virtual_ipaddress {
10.0.2.30/24 dev enp0s3
}
track_script {
check_haproxy
}
notify_master "/usr/local/sbin/vrrp-notify master"
notify_backup "/usr/local/sbin/vrrp-notify backup"
notify_fault "/usr/local/sbin/vrrp-notify fault"
}# BACKUP NODE (lb2, 10.0.2.21): identical except for three lines
state BACKUP
priority 90
router_id lb2$ sudo systemctl enable --now keepalived
$ ip -brief addr show enp0s3
enp0s3 UP 10.0.2.20/24 10.0.2.30/24 # <- the VIP is on lb1
# On lb2 the VIP does NOT appear: it is on standby
$ ssh [email protected] 'ip -brief addr show enp0s3'
enp0s3 UP 10.0.2.21/24The mechanism: the node with the higher priority advertises the VIP every second. If the backup stops hearing advertisements for three intervals, it assumes the primary has gone down, assigns itself the IP and sends a gratuitous ARP so that the network switches update their tables. The failover takes between 3 and 4 seconds.
The track_script with weight -40 is what avoids the silliest failure in this architecture: without it, a node where keepalived works but HAProxy is dead cheerfully keeps the VIP, and the service is down even though the other load balancer is perfectly fine. With it, lb1's priority drops from 100 to 60, below lb2's 90, and the VIP moves.
And enable_script_security with script_user is a real security precaution: keepalived runs as root and executes those scripts; without those directives, a script with loose permissions is a privilege escalation.
Split-brain and quorum
The characteristic failure of any redundant system, and the one that causes the most serious data losses.
Split-brain happens when the nodes stop seeing each other but both keep running. Each concludes that the other has gone down, and both declare themselves primary:
- Two load balancers advertise the same virtual IP → unpredictable traffic, broken connections.
- Two databases accept writes → the data diverges, and reconciling it afterwards can be impossible.
The scenario that causes it is not a node going down — that is handled well — but the network between them being cut while both are still alive and serving clients. It is more frequent than it sounds: a switch that fails, a badly placed firewall rule, a momentary saturation.
The three defences:
| Defence | How it works | Limitation |
|---|---|---|
| Quorum | Only the group with a majority of nodes acts | Requires 3 or more nodes, an odd number |
| Fencing (STONITH) | The survivor powers off the other one at the hardware level | Requires remote power or hypervisor control |
| Witness | A third resource arbitrates (a disk, an IP) | Less robust than real quorum |
And the conclusion to hold on to:
With two nodes there is no possible quorum. Neither can have a majority of two. That is why a serious cluster has three nodes, or two plus a witness.
This has a direct consequence for Tramontana's design: a pair of load balancers with VRRP is acceptable, because the worst case — two nodes advertising the same IP for a few seconds — is annoying but recoverable. A pair of databases with automatic failover is not: the worst case is data divergence, and it does not recover. Hence the recommendation in the next section.
Data replication
PostgreSQL replicates through the write-ahead log (WAL): the primary sends its changes to one or more replicas.
| Mode | The primary commits when… | Risk of loss | Cost |
|---|---|---|---|
| Asynchronous | It has written locally | The last few transactions | None |
| Synchronous | The replica has confirmed | None | Latency on every write |
And the trade-off, which has to be understood before choosing: with synchronous replication, every write waits for the round trip to the replica. On the same local network, one or two milliseconds. Between data centres, tens. And if the replica goes down, the primary blocks waiting for confirmation — unless synchronous_standby_names is configured with the right syntax, which opens the door to losing data exactly when it matters most.
For Tramontana, with an agreed RPO of 4 hours, asynchronous replication is more than enough: losing the last few seconds of transactions is well within what is tolerated.
# postgresql.conf on the PRIMARY
wal_level = replica
max_wal_senders = 3
wal_keep_size = 1GB
archive_mode = on
archive_command = 'test ! -f /srv/wal/%f && cp %p /srv/wal/%f'
hot_standby = on# Create the replica from scratch
$ sudo -u postgres pg_basebackup -h 10.0.2.15 -U replicator \
-D /var/lib/postgresql/16/main -R -P -X stream -c fast
$ sudo -u postgres psql -c "SELECT client_addr, state, sync_state,
pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes FROM pg_stat_replication;"
client_addr | state | sync_state | lag_bytes
-------------+-----------+------------+-----------
10.0.2.22 | streaming | async | 0That lag_bytes is the metric to watch: it is how much data would be lost if the primary went down now. It goes straight into the monitoring, alongside the age of the last backup from 05-08.
Failover: manual versus automatic
| Manual | Automatic | |
|---|---|---|
| Recovery time | Minutes | Seconds |
| Split-brain risk | None | High without quorum |
| Requires somebody available | Yes | No |
| Tools | pg_ctl promote |
Patroni, repmgr, pg_auto_failover |
The professional recommendation, and it goes against intuition:
With two nodes, database failover must be manual. Without quorum, a network failure between primary and replica can promote the replica while the primary is still accepting writes, and then there are two diverging databases. Recovering from that can mean losing transactions or reconciling by hand.
Tools such as Patroni do automatic failover safely, but they require a distributed configuration store with quorum of its own (etcd or Consul), which in turn needs three nodes. In other words: automating failover safely requires at least five machines. It is an architectural decision, not a box to tick.
Shared storage
For /opt/tramontana/shared/uploads:
| Option | Advantage | Drawback |
|---|---|---|
| NFS | Simple, works without touching the application | It is a new single point of failure |
| GlusterFS / Ceph | Distributed and replicated | High complexity |
| Object store (S3, MinIO) | Scalable, managed, replicated | Requires changing the application |
Periodic rsync |
Trivial | Not real time: files get lost |
NFS has the characteristic irony: it is mounted to provide high availability and, if it is not made redundant too, it worsens reliability, because now two servers depend on a third machine.
The modern answer is the object store: it is what applications designed to scale horizontally use. It demands development work, and it is the same conclusion as the section on state.
Clusters, Pacemaker and fencing
For more complex scenarios — databases, shared filesystems, resources with dependencies — there is the Pacemaker + Corosync stack:
- Corosync: communication between nodes, cluster membership and quorum.
- Pacemaker: resource management, with dependencies and constraints.
$ sudo pcs status
Cluster name: tramontana
Cluster Summary:
* Stack: corosync
* Current DC: node1 (version 2.1.6) - partition with quorum
* 3 nodes configured
* 4 resource instances configured
Full List of Resources:
* vip_tramontana (ocf:heartbeat:IPaddr2): Started node1
* postgresql (ocf:heartbeat:pgsql): Master node1A resource is anything the cluster manages (an IP, a service, a filesystem), and the constraints express rules: "the IP must be where the PostgreSQL primary is", "this service must start after that one".
And the concept without which none of this really works:
Fencing (or STONITH, Shoot The Other Node In The Head): before taking over the resources of a node that appears to be down, the survivor makes sure it is powered off, by cutting its power or destroying its virtual machine.
Why it is indispensable: if node A appears to be down but is in fact merely frozen or cut off, and B takes over its resources — mounts the filesystem, promotes the database, takes the IP — then when A recovers both will write to the same data. The resulting corruption can be unrecoverable.
# Fencing over libvirt: destroys the other node's virtual machine
$ sudo pcs stonith create fence_node2 fence_virsh \
ip=10.0.2.10 login=fence identity_file=/etc/pacemaker/id_rsa \
pcmk_host_list=node2 action=offOn physical hardware it is done with IPMI, iLO or manageable power distribution units. And the rule in the field is blunt: a cluster with no fencing configured is not a reliable cluster. Many projects disable it "temporarily" because it complicates testing, and that is exactly the configuration that causes data loss on the day of the failure.
A design for Tramontana and the cost analysis
graph TD
I["Internet"] --> DNS["DNS: bookings.tramontana.example<br/>→ 10.0.2.30 (VIP)"]
DNS --> VIP{"Floating virtual IP<br/>10.0.2.30"}
VIP -.->|VRRP| LB1["lb1 · HAProxy<br/>MASTER prio 100"]
VIP -.->|VRRP| LB2["lb2 · HAProxy<br/>BACKUP prio 90"]
LB1 --> APP1["app1 · Tramontana<br/>10.0.2.21:8080"]
LB1 --> APP2["app2 · Tramontana<br/>10.0.2.22:8080"]
LB2 -.-> APP1
LB2 -.-> APP2
APP1 --> DB1["PostgreSQL primary<br/>10.0.2.15"]
APP2 --> DB1
DB1 -->|asynchronous WAL| DB2["PostgreSQL replica<br/>10.0.2.16"]
APP1 --> OBJ["Object store<br/>uploads"]
APP2 --> OBJ
What each layer protects against, and what it does not:
| Failure | Covered? | Recovery time |
|---|---|---|
app1 or app2 goes down |
Yes, automatically | 3-9 s (fall 3 × inter 3s) |
lb1 goes down |
Yes, automatically | 3-4 s (VRRP) |
| The PostgreSQL primary goes down | No: promotion is deliberately manual | 5-15 min |
| The network between lb1 and lb2 fails | Partially: a brief split-brain, recoverable | Seconds |
| The host running everything goes down | No | The full RTO |
| A logical failure (deletion, corruption) | No: it replicates instantly | Restore a backup |
| A faulty deployment | No: it is deployed to both | Rollback |
The last three rows are the ones to underline for management. Redundancy does not protect against logical errors: a mistaken DELETE replicates to the replica in milliseconds. That is what the backups from 05-08 are for, and this architecture does not replace them in the slightest.
The cost analysis
What the lesson has been promising, and what Marta has been asking for over three replies:
Cost of the complete architecture (5 machines against 1):
| Item | Annual estimate |
|---|---|
| 4 additional machines | The current VPS cost × 4 |
| Development work (sessions + object store) | 3-4 weeks, once |
| Setup and automation with Ansible | 2 weeks, once |
| Additional maintenance | ~4 h/month indefinitely |
| Object store | Low, by volume |
| Complexity: more failure modes | Hard to quantify, real |
The cost of unavailability. The question that has to be answered first:
Bookings per month: ~500 Average amount: €313.70 Average revenue per hour (24×30): 500 × 313.70 / 720 ≈ €218/h
But that raw calculation is misleading for three reasons, and saying so is part of the analysis:
- Bookings are not uniform. They cluster in the afternoon and at weekends. A two-hour outage at four in the morning on a Tuesday may cost nothing; the same outage on a Friday afternoon costs much more than the average.
- A lost booking is not always lost. Many customers try again. The real cost is between 20% and 50% of the raw figure.
- There is a reputational cost, non-linear and hard to quantify, especially if the outage coincides with a campaign.
A reasonable estimate of the annual cost of unavailability, at different targets:
| Availability | Annual downtime | Estimated cost (at €218/h × 35%) |
|---|---|---|
| 99.5% (estimated current situation) | 43.8 h | ~€3,340 |
| 99.9% | 8.8 h | ~€670 |
| 99.95% | 4.4 h | ~€335 |
The saving from going from 99.5% to 99.9% is about €2,700 a year. That has to be compared against the cost of the complete architecture — four more machines, six weeks of initial work and four hours a month indefinitely — which very probably exceeds it.
And here is the analysis almost nobody does, and which changes the recommendation:
| Measure | Cost | Estimated availability |
|---|---|---|
| The current situation | — | ~99.5% |
| Just two application nodes + one load balancer | 2 machines, 1 week | ~99.8% |
| The complete architecture with a replicated DB | 4 machines, 6 weeks | ~99.9% |
The first step captures most of the benefit for a fraction of the cost, and this is the usual pattern in availability: the first nines are cheap and the following ones get exponentially more expensive. The reason is that most interruptions do not come from a hardware failure but from deployments, reboots and maintenance — and all of that is covered by two application nodes and connection draining, without touching the database.
Recommendation
Phase 1 (now): set a formal availability target with Marta. Without a target you cannot decide how much to invest. And measure it, because the current 99.5% is an estimate, not a figure.
Phase 2 (1-2 months): two application nodes and one load balancer. Externalised sessions and files in an object store, which is development work. It eliminates interruptions caused by deployment and by the failure of one node, which are the majority.
Phase 3 (assess later): a second load balancer with VRRP, and a PostgreSQL replica with manual promotion.
Phase 4 (probably never): automatic database failover. It requires quorum, which is to say five machines and a distributed store. For a company of this size, the complexity outweighs the benefit.
Testing failures by causing them
An untested high-availability system is not a high-availability system: it is an architecture you believe works. And the history of the field is full of cases where the backup failed on the day it was needed.
# --- Test 1: an application node goes down ---
$ ssh [email protected] 'sudo systemctl stop tramontana'
# Expected: HAProxy takes it out in ~9 s (fall 3 x inter 3s), with no visible errors
$ for i in {1..60}; do
curl -s -o /dev/null -w '%{http_code} ' https://bookings.tramontana.example/houses
sleep 1
done; echo
# Criterion: zero responses != 200
# --- Test 2: the primary load balancer goes down ---
$ ssh [email protected] 'sudo systemctl stop keepalived'
# Expected: the VIP moves to lb2 in 3-4 s
$ ssh [email protected] 'ip -brief addr show enp0s3 | grep 10.0.2.30'
# --- Test 3: HAProxy dies but keepalived lives ---
# This is the test that validates the track_script. Without it, the VIP
# does NOT move and the service goes down with both load balancers "alive".
$ ssh [email protected] 'sudo systemctl stop haproxy'
$ sleep 6 && ssh [email protected] 'ip -brief addr show enp0s3 | grep 10.0.2.30'
# --- Test 4: a network partition between the load balancers (split-brain) ---
$ ssh [email protected] 'sudo nft add rule inet filter input ip saddr 10.0.2.21 drop'
# Expected: BOTH take the VIP. Document the observed behaviour.
$ ssh [email protected] 'sudo nft flush chain inet filter input'
# --- Test 5: a zero-downtime deployment ---
$ echo "set server tramontana_app/app1 state drain" | sudo socat stdio /run/haproxy/admin.sock
$ ssh [email protected] 'sudo ~/scripts/deploy.sh 3.3.0'
$ echo "set server tramontana_app/app1 state ready" | sudo socat stdio /run/haproxy/admin.sock
# Criterion: zero errors throughout the processTest 4 is the most uncomfortable and the most important: it cannot be "passed" with two nodes, because without quorum split-brain is unavoidable. What you do is document the observed behaviour and the recovery procedure, so that on the day it happens nobody has to improvise.
Chaos engineering takes this further: injecting failures continuously and automatically in production, on the logic that if failures happen every day, the system is forced to tolerate them and the team is forced to know how to respond. For Tramontana it is disproportionate; the principle — cause the failures yourself, during working hours, instead of waiting for them — applies at any scale, and it fits the six-monthly rehearsal you proposed in 07-06.
Common Mistakes and Tips
- Putting in a load balancer and thinking you now have high availability. All you have done is move the single point of failure. The load balancer needs redundancy of its own.
- Configuring VRRP without
track_script. A node withkeepalivedalive and HAProxy dead keeps the virtual IP, and the service is down with both load balancers "working". - Automatic database failover with two nodes. Without quorum, a network partition produces two primaries and diverging data. Manual, until there are three nodes.
- Disabling fencing "temporarily". It is the configuration that causes data corruption on the day of the failure. A cluster with no fencing is not reliable.
- Shallow health checks. A
/healththat only says "I am alive" makes the load balancer send traffic to a node that cannot serve anything. - Health checks that are too deep. A database failure marks every node as down, when they could have served something.
- Taking a node out on the first failure. A transient spike takes the node out, the load concentrates on the rest, and they fall in a cascade.
fall 3andslowstart. - Forgetting that the timers run on every node. Two simultaneous backups, two purges. The
flockis local and protects nothing. - Sticky sessions as a permanent solution. They mask the real problem, unbalance the load, and when a node goes down it loses all of its sessions.
- Believing that replication replaces backups. An accidental deletion replicates in milliseconds. Redundancy protects against hardware failure, not against human error.
- NFS with no redundancy of its own to provide high availability. You add a single point of failure that two servers now depend on.
- Not testing the failures. An untested high-availability system is an architecture you believe works.
- A tip on method. Before designing anything, set the availability target and measure the current one. Without those two numbers, any investment is a hunch dressed up as architecture.
Exercises
Exercise 1
Design the /health endpoint the application should expose so that HAProxy makes good decisions. Specify what it checks, what it does not, what codes it returns and with what latency, and explain how it relates to health_check.sh.
Exercise 2
Write the Ansible role that deploys the HAProxy and keepalived configuration on both load balancers, bearing in mind that the configuration differs between them and that applying it badly leaves the service with no IP.
Exercise 3
Marta asks directly: "Do we need high availability?". Write the answer with the complete analysis and a clear recommendation.
Solutions
Solution 1
Design of the /health endpoint:
| It checks | How | Why |
|---|---|---|
| That the process responds | That the request arrives | Trivial but necessary |
| The PostgreSQL connection | SELECT 1 with a 1 s timeout |
Without the database it cannot serve anything |
| The connection pool | That at least one is free | With the pool exhausted, requests queue up |
| The file store | Writing one byte every 30 s (cached) | Without it uploads fail, but queries do not |
Space in /var/log |
A 95% threshold (cached) | With no room for logs, traceability is lost |
| It does NOT check | Why |
|---|---|
| Real business queries | Expensive; a data problem would bring down every node |
| External services (mail, payments) | A third-party failure would take the whole service out of the rotation |
| The state of other nodes | Each node reports only on itself |
| Performance metrics | That is monitoring, not health |
The three states and their responses:
// 200 OK — state 0, OK
{
"status": "ok",
"version": "3.2.1",
"checks": {
"db": {"ok": true, "latency_ms": 2},
"pool": {"ok": true, "free": 68, "total": 80},
"storage": {"ok": true},
"log_disk": {"ok": true, "usage_pct": 34}
}
}// 200 OK with the header X-Health-Warning: degraded — state 1
// It STILL receives traffic: it can serve queries
{
"status": "degraded",
"warnings": ["storage_unavailable"],
"checks": {
"db": {"ok": true, "latency_ms": 3},
"pool": {"ok": true, "free": 12, "total": 80},
"storage": {"ok": false, "error": "timeout"},
"log_disk": {"ok": true, "usage_pct": 34}
}
}// 503 Service Unavailable — state 2, critical
// It receives NO traffic
{
"status": "critical",
"errors": ["db_unreachable"],
"checks": {
"db": {"ok": false, "error": "connection refused"}
}
}The relationship with health_check.sh is the conceptual part of the exercise. Both answer the same question from different sides, and that complementarity should be exploited rather than duplicating work:
health_check.sh |
/health |
|
|---|---|---|
| Who asks | The administrator or a timer | HAProxy, every 3 s |
| From where | Outside the process, on the machine | From inside the process |
| Sees the internal state | No | Yes: connection pool, caches |
| Frequency | Minutes | Seconds |
| Acceptable cost | High | Very low |
| Codes | 0 / 1 / 2 | 200 / 200+header / 503 |
The right approach is for health_check.sh to consume /health rather than reimplementing the checks:
# A fragment of health_check.sh, adapted
check_application() {
local response code status
response="$(curl -s -m 5 -w '\n%{http_code}' \
"http://127.0.0.1:${TRAMONTANA_PORT}/health")" || {
error "the application is not responding"
return 2
}
code="$(tail -1 <<<"$response")"
status="$(sed '$d' <<<"$response" | jq -r '.status')"
case "$status" in
ok) log "application OK"; return 0 ;;
degraded) error "degraded: $(sed '$d' <<<"$response" | jq -r '.warnings|join(", ")')"
return 1 ;;
critical) error "critical: $(sed '$d' <<<"$response" | jq -r '.errors|join(", ")')"
return 2 ;;
*) error "unknown status (HTTP $code)"; return 2 ;;
esac
}Four design decisions that have to be justified:
- The degraded state returns 200, not 503. It is counterintuitive and it is correct: if the file store fails, the node can carry on serving availability queries, which are most of the traffic. Taking it out of the rotation would reduce capacity and gain nothing. The header lets the monitoring detect it without the load balancer acting on it.
- Latency bounded by design. With a 3-second interval and two nodes, that is 40 requests a minute. The target is under 50 ms, and hence the expensive checks are cached for 30 seconds.
- No authentication, but restricted by network. It must be reachable without credentials so that HAProxy can query it, and it must not be exposed to the Internet: the detail in the JSON is useful information for an attacker. It is blocked in the frontend:
http-request deny if { path_beg /health } !{ src 10.0.2.0/24 } - No side effects and no blocking. A
/healththat writes to the database or takes a lock can cause the failure it is meant to detect. The store's write check is cached and runs in the background.
And the final warning, which is the most common mistake: /health must not check dependencies that do not belong to this node. If it checked the state of the database with a heavy query, a PostgreSQL problem would take every node out at once and the service would disappear entirely, when it could at least have returned a decent error page.
Solution 2
# roles/loadbalancer/defaults/main.yml
haproxy_vip: 10.0.2.30
haproxy_vip_cidr: 24
haproxy_interface: enp0s3
haproxy_vrrp_id: 51
haproxy_backends:
- { name: app1, address: 10.0.2.21, port: 8080 }
- { name: app2, address: 10.0.2.22, port: 8080 }
haproxy_check_inter: 3s
haproxy_check_rise: 2
haproxy_check_fall: 3
haproxy_slowstart: 30s# inventory/production.yml (fragment)
loadbalancers:
hosts:
lb1:
ansible_host: 10.0.2.20
vrrp_state: MASTER
vrrp_priority: 100
lb2:
ansible_host: 10.0.2.21
vrrp_state: BACKUP
vrrp_priority: 90# roles/loadbalancer/tasks/main.yml
---
# =====================================================================
# WARNING: an incorrect keepalived configuration can leave the service
# WITH NO VIRTUAL IP, that is, completely down. This role deploys the
# nodes ONE AT A TIME and verifies between them.
# =====================================================================
- name: "Check that it is not running against both load balancers in parallel"
ansible.builtin.assert:
that: ansible_play_hosts | length == 1
fail_msg: >-
This role must be run with --limit against ONE load balancer at a time,
or with serial: 1 in the play. Configuring both at once can leave the
service with no virtual IP.
- name: Install HAProxy and keepalived
ansible.builtin.apt:
name: [haproxy, keepalived, socat]
state: present
tags: [packages]
# ---------- HAProxy ----------
- name: Create the certificates directory
ansible.builtin.file:
path: /etc/haproxy/certs
state: directory
owner: root
group: haproxy
mode: '0750'
tags: [tls]
- name: Install the combined certificate for HAProxy
ansible.builtin.copy:
content: "{{ vault_certificate_pem }}"
dest: /etc/haproxy/certs/bookings.tramontana.example.pem
owner: root
group: haproxy
mode: '0640'
no_log: true # it contains the private key
notify: Reload haproxy
tags: [tls]
- name: Deploy the HAProxy configuration
ansible.builtin.template:
src: haproxy.cfg.j2
dest: /etc/haproxy/haproxy.cfg
owner: root
group: root
mode: '0644'
backup: true
# VALIDATES before installing: a broken config stops it starting
validate: 'haproxy -c -f %s'
notify: Reload haproxy
tags: [haproxy]
- name: Enable HAProxy
ansible.builtin.systemd_service:
name: haproxy
enabled: true
state: started
tags: [haproxy]
# ---------- keepalived ----------
# The sysctl that lets HAProxy listen on an IP this node does not have
# yet (the VIP while it is on the other one). Without this, HAProxy
# does not start on the BACKUP node.
- name: Allow binding to non-local IP addresses
ansible.posix.sysctl:
name: net.ipv4.ip_nonlocal_bind
value: '1'
sysctl_file: /etc/sysctl.d/71-loadbalancer.conf
sysctl_set: true
reload: true
tags: [keepalived]
- name: Create the user for the keepalived scripts
ansible.builtin.user:
name: keepalived_script
system: true
shell: /usr/sbin/nologin
create_home: false
tags: [keepalived]
- name: Install the VRRP notification script
ansible.builtin.template:
src: vrrp-notify.j2
dest: /usr/local/sbin/vrrp-notify
owner: root
group: root
mode: '0755' # NOT writable by the script's user
tags: [keepalived]
- name: Deploy the keepalived configuration
ansible.builtin.template:
src: keepalived.conf.j2
dest: /etc/keepalived/keepalived.conf
owner: root
group: root
mode: '0600' # it contains the VRRP password
backup: true
no_log: true
notify: Restart keepalived
tags: [keepalived]
- name: Enable keepalived
ansible.builtin.systemd_service:
name: keepalived
enabled: true
state: started
tags: [keepalived]
# ---------- Verification ----------
- name: Flush the handlers before verifying
ansible.builtin.meta: flush_handlers
- name: "Verify | HAProxy responds on the admin socket"
ansible.builtin.shell:
cmd: >-
set -o pipefail;
echo "show stat" | socat stdio /run/haproxy/admin.sock
| awk -F, '$1=="tramontana_app" && $2=="BACKEND" {print $18}'
register: backend_status
changed_when: false
failed_when: "'UP' not in backend_status.stdout"
tags: [verification]
- name: "Verify | The backends are healthy"
ansible.builtin.shell:
cmd: >-
set -o pipefail;
echo "show stat" | socat stdio /run/haproxy/admin.sock
| awk -F, '$1=="tramontana_app" && $2 ~ /^app/ && $18=="UP"' | wc -l
register: backends_up
changed_when: false
failed_when: backends_up.stdout | int < 1
tags: [verification]
- name: "Verify | The VIP is on the MASTER and only there"
ansible.builtin.shell:
cmd: "ip -brief addr show {{ haproxy_interface }} | grep -c {{ haproxy_vip }} || true"
register: has_vip
changed_when: false
tags: [verification]
- name: "Verify | The VIP state is consistent"
ansible.builtin.assert:
that:
- (vrrp_state == 'MASTER' and has_vip.stdout | int == 1) or
(vrrp_state == 'BACKUP' and has_vip.stdout | int == 0)
fail_msg: >-
Inconsistent VIP state on {{ inventory_hostname }}
({{ vrrp_state }}, VIP present: {{ has_vip.stdout }}).
Check before continuing with the other load balancer.
tags: [verification]
- name: "Verify | The service responds through the VIP"
ansible.builtin.uri:
url: "https://bookings.tramontana.example/health"
validate_certs: true
status_code: 200
timeout: 10
delegate_to: localhost
become: false
run_once: true
tags: [verification]# roles/loadbalancer/handlers/main.yml
- name: Reload haproxy
ansible.builtin.systemd_service:
name: haproxy
state: reloaded # reload, not restart: it does not cut the connections
- name: Restart keepalived
ansible.builtin.systemd_service:
name: keepalived
state: restarted{# roles/loadbalancer/templates/keepalived.conf.j2 #}
# GENERATED BY ANSIBLE — DO NOT EDIT BY HAND
global_defs {
router_id {{ inventory_hostname }}
enable_script_security
script_user keepalived_script
}
vrrp_script check_haproxy {
script "/usr/bin/killall -0 haproxy"
interval 2
weight -40
fall 2
rise 2
}
vrrp_instance TRAMONTANA_VIP {
state {{ vrrp_state }}
interface {{ haproxy_interface }}
virtual_router_id {{ haproxy_vrrp_id }}
priority {{ vrrp_priority }}
advert_int 1
authentication {
auth_type PASS
auth_pass {{ vault_vrrp_pass }}
}
virtual_ipaddress {
{{ haproxy_vip }}/{{ haproxy_vip_cidr }} dev {{ haproxy_interface }}
}
track_script {
check_haproxy
}
notify_master "/usr/local/sbin/vrrp-notify master"
notify_backup "/usr/local/sbin/vrrp-notify backup"
notify_fault "/usr/local/sbin/vrrp-notify fault"
}And the play that guarantees deployment one node at a time:
# loadbalancers.yml
- name: Configure the load balancers
hosts: loadbalancers
become: true
# serial: 1 -> one node at a time. If the first fails, the second is NOT touched.
serial: 1
# max_fail_percentage: 0 -> abort on the first failure
max_fail_percentage: 0
# DELIBERATE ORDER: the BACKUP first. If something goes wrong, the MASTER
# keeps the VIP and the service is not interrupted.
order: reverse_inventory
roles:
- loadbalancerThe decisions that stop the service being left with no IP:
| Decision | What it protects against |
|---|---|
serial: 1 + a single-host assert |
Configuring both at once and ending up with no node holding the VIP |
order: reverse_inventory (BACKUP first) |
If the role fails, the MASTER keeps the VIP and the service carries on |
max_fail_percentage: 0 |
A failure on lb2 going undetected before lb1 is touched |
validate: haproxy -c -f %s |
Installing a configuration that stops HAProxy starting |
meta: flush_handlers before verifying |
Verifying the state as it was before the changes |
state: reloaded on HAProxy |
restart would cut every in-flight connection |
The VIP consistency assert |
Detecting that both or neither hold the VIP |
uri against the VIP with run_once |
Checking the service end to end, not just the pieces |
net.ipv4.ip_nonlocal_bind |
HAProxy does not start on the BACKUP if it cannot listen on the absent VIP |
no_log: true on the certificate and VRRP |
The private key and the password would appear in the output |
A 0755 script owned by root |
enable_script_security rejects scripts writable by the user that runs them |
And what has to be added to the runbook: this role is tested first in the test environment with two virtual load balancers, and in production it is run with --check --diff before running for real. The virtual_router_id must be unique on the network segment: two VRRP clusters with the same ID on the same network interfere with each other, and it is a baffling failure to diagnose.
Solution 3
Does Tramontana Bookings need high availability? To: Marta Vidal · From: Systems Operations · 18 August 2026
Short answer: full high availability, no. A first step, yes, and with a very favourable cost-benefit ratio. I recommend two application servers with a load balancer, which captures most of the benefit for a fraction of the cost.
First, a figure we are missing. We have no agreed availability target and we do not measure it. My estimate is that we are around 99.5%, some 44 hours of downtime a year, counting reboots for updates, deployments and the occasional incident. It is an estimate, not a figure, and the first thing I propose is that we start measuring it — it costs very little and without that number any investment is a hunch.
What we have already achieved without spending anything. Availability depends on two things: how often the system fails and how long it takes to recover. In recent weeks we have drastically reduced the second: rebuilding the server completely has gone from eight hours to fifty minutes, by automating its configuration. That has already improved availability without buying a single machine, and it is the kind of improvement to exhaust before duplicating infrastructure.
What being down would cost. With around 500 bookings a month at an average of €313.70, average revenue is about €218 an hour. That number has to be qualified in three ways:
- Bookings cluster in the afternoon and at weekends. An outage in the small hours of a Tuesday may cost nothing; the same one on a Friday afternoon costs considerably more than the average.
- Many customers who find the site down try again. The real cost is probably between 20% and 50% of the raw figure.
- There is a cost to our image, non-linear and hard to quantify, especially if it coincides with a campaign.
With a prudent estimate (35%):
Availability Downtime per year Estimated cost 99.5% (current situation) 44 h ~€3,340 99.8% 17.5 h ~€1,330 99.9% 8.8 h ~€670 The three options, with their real cost.
What it includes Investment Availability Annual saving A. Do nothing The current setup 0 ~99.5% — B. Two servers + load balancer 2 machines, 1 week of work Low ~99.8% ~€2,000 C. The complete architecture 4 machines, 6 weeks, +4 h/month High ~99.9% ~€2,670 The important part is the comparison between B and C: option C costs several times more than B and adds barely €670 of additional annual saving. The reason is that the first nines are cheap and the following ones get much more expensive — and, above all, that most of our interruptions do not come from hardware failures but from deployments, reboots and maintenance, and that is resolved entirely by option B.
What option B would give us exactly:
- Zero-downtime deployments. Today every deployment means a few seconds of service cut off. With two servers one is updated while the other serves, and the customer notices nothing.
- Maintenance with no window. Rebooting for a new kernel stops being a scheduled interruption.
- Tolerance of one application server failing. The system detects it in under ten seconds and stops sending it traffic on its own.
- Capacity for peaks, as a secondary benefit.
What it would NOT give us, and this has to be said clearly:
- If the database goes down, the service goes down. Promoting the replica would be manual, 5 to 15 minutes.
- If the load balancer goes down, the service goes down. That is option C.
- If the provider's infrastructure goes down, everything goes down.
- And none of this protects against an accidental deletion or data corruption: that would replicate instantly to every node. That is what the backups are for, and they would remain just as necessary.
There is a prerequisite that is not a systems matter. Tramontana Bookings currently keeps two things inside each server: the sessions of the people browsing, and the photos that get uploaded. With two servers, anybody who logged in on one would appear logged out when served by the other, and a photo uploaded to one would not be visible from the other. Before setting anything up, development work is needed — externalising the sessions and storing the files in a shared store — some three or four weeks. It is the part of the project to plan with Luis, and without it option B does not work.
And a professional warning. Redundancy adds pieces, and each piece can fail in new ways. A badly built system is less reliable than a simple one that is well maintained. That is why I do not recommend option C: the complexity it adds — replication, promotion, the risk of two servers writing incompatible data at the same time — demands a level of vigilance and testing we cannot sustain with the current headcount. A simple architecture we understand and test is worth more than a sophisticated one nobody has rehearsed.
Recommendation, in three steps.
- This month, at zero cost: set a formal availability target — I propose 99.8% — and start measuring it. With that we will be able to decide on data rather than on estimates.
- In two or three months: develop the two prerequisites, and deploy option B. I would start with a single load balancer, accepting that it is a single point of failure but one far less prone to going down than the complete application.
- Review in a year, or sooner if something changes: if the booking volume grows significantly, if an outage costs us an important customer, or if we measure availability worse than estimated.
And one last thing. Quite apart from all of the above, I want to schedule a six-monthly rehearsal: causing the failures deliberately, during working hours, and checking that the recovery works. A backup system that has never been tested is a system we know nothing about, and this applies as much to the future architecture as to the rebuild procedure we already have. The first one in September.
Conclusion
You know what people are talking about when they talk about high availability, and with the precision the subject demands. The nines have concrete numbers — 99.99% is less than a minute of downtime a week — availability breaks down into MTBF and MTTR, and reducing MTTR is usually far cheaper than increasing MTBF: which is exactly what you did in 07-06 by taking the rebuild from eight hours to fifty minutes, without buying anything. You know that vertical scaling adds no availability, that a passive node that is never used is a node you know nothing about, and above all that state is the hard problem: sessions, uploaded files and scheduled tasks are what stops you simply duplicating a server.
You have set up a load balancer with HAProxy that distributes by leastconn, probes health every three seconds with asymmetric fall 3 and rise 2, brings nodes back with slowstart so as not to knock them over again, and allows connections to be drained so you can deploy without cutting anybody off — resolving what was left outstanding since 04-07. You have given it redundancy with keepalived and a floating virtual IP, with the track_script that avoids the silliest failure in this architecture: a load balancer with keepalived alive and HAProxy dead holding on to the IP. And you know the two concepts without which all of this is theatre: split-brain, which with two nodes cannot be avoided because no quorum is possible, and fencing, without which a cluster is not reliable however well it starts.
And you have done the analysis Marta had been asking for over three lessons, with the conclusion that is not always welcome: the complete architecture is not worth it. Two application servers with a load balancer capture most of the benefit for a fraction of the cost, because most interruptions do not come from hardware failures but from deployments and maintenance. That the professional answer to "do we need high availability?" can be "part of it, and in this order" is as important as knowing how to build it. Along with the warning that closes the matter: redundancy done badly worsens reliability, and a simple system that is understood and tested is worth more than a sophisticated one nobody has rehearsed.
That closes Module 7, and with it the most advanced part of the course. You have learned to look inside the system: the complete boot chain and how to intervene when it breaks, diagnosis with strace, perf and eBPF when the counters are not enough, and kernel tuning with the rule of not touching what you have not measured. And you have learned to multiply it: virtualisation with KVM and the test environment the course had been missing since Module 4, containers understood as what they are — isolated processes, not small machines — automation with Ansible that turned the configuration into code and the RTO into fifty minutes, and finally the redundancy that takes away srv-tramontana's status as a single point of failure.
In Module 8: Practical Projects all of that is applied to complete builds, end to end. You will set up a web server with Nginx as a reverse proxy and TLS in front of the application — and there the encryption in transit that was prepared in 06-05 and has been waiting ever since is finally closed off. You will configure a proper PostgreSQL database server, with its tuning, its backups and its replication. You will build a home media server, which is the project where you can confirm that everything you have learned applies just as well outside work. You will bring up a VPN server with WireGuard to reach the internal network without exposing anything. You will deploy a Kubernetes cluster, which is where the containers from 07-05 and orchestration come together. And you will finish with going into production: the complete checklist, the monitoring with Prometheus and Grafana that was mentioned in 05-07, and the daily operating procedure for a system that is no longer a laboratory. Update your VM's snapshot and I will see you there.
Linux Course: From Beginner to System Administrator
Module 1: Introduction to Linux
- What Is Linux?
- History of Linux
- Linux Distributions
- Installing Linux
- First Contact with the System
- The Linux File System Structure
Module 2: Basic Linux Commands
- Introduction to the Command Line
- Getting Help and System Documentation
- Navigating the File System
- File and Directory Operations
- Viewing and Editing Files
- Hard and Symbolic Links
- File Permissions and Ownership
Module 3: Advanced Command-Line Skills
- The Shell Environment: Variables, Aliases and History
- Using Wildcards and Regular Expressions
- Searching Files and Content: find, locate and grep
- Pipes and Redirection
- Text Processing: cut, sort, uniq, sed and awk
- Process Management
- Scheduling Tasks with Cron
- Networking Commands
Module 4: Shell Scripting
- Introduction to Shell Scripting
- Variables and Data Types
- Script Input, Output and Arguments
- Control Structures
- Functions and Libraries
- Debugging and Error Handling
- Production Scripts: Best Practices
Module 5: System Administration
- User and Group Management
- sudo and Special Permissions
- Package Management
- Disk Management
- systemd and Service Management
- System Logs: journald and syslog
- System Monitoring and Performance Tuning
- Backup and Restore
Module 6: Networking and Security
- Network Configuration
- SSH and Remote Access
- Firewalls and Perimeter Security
- Intrusion Detection Systems
- Secrets Management and TLS Certificates
- Securing Linux Systems
Module 7: Advanced Topics
- The Boot Process and System Recovery
- Advanced Diagnostics: strace, perf and eBPF
- Linux Kernel Tuning
- Virtualization with Linux
- Linux Containers and Docker
- Automation with Ansible
- High Availability and Load Balancing
