You can rebuild srv-tramontana in fifty minutes. You can test any change before applying it, detect an intrusion, encrypt the secrets, diagnose an anomalous latency and recover a broken boot. And even so, if that machine goes down, Tramontana Bookings is down.

A disk that fails, a power supply, a network outage at the provider, a kernel that will not boot after an update, or simply scheduled maintenance on the infrastructure it lives on. In the best case, fifty minutes of interruption — and that assumes somebody is awake to launch the playbook. All the work of seven modules rests on a single machine, which is the exact definition of a single point of failure.

This lesson attacks that problem, and it does so with a warning up front: high availability is expensive, complex and, done badly, it worsens reliability rather than improving it. A system with more pieces has more ways of failing. The lesson ends with an honest analysis of whether Tramontana needs this, because the professional answer is not always yes.

Contents

  1. Vocabulary: availability, nines, MTBF and MTTR
  2. Vertical and horizontal scaling
  3. Redundancy: active-passive and active-active
  4. State is the hard problem
  5. Load balancing: layers, algorithms and health checks
  6. HAProxy in practice
  7. High availability for the load balancer: keepalived and VRRP
  8. Split-brain and quorum
  9. Data replication
  10. Clusters, Pacemaker and fencing
  11. A design for Tramontana and the cost analysis
  12. Testing failures by causing them

Vocabulary: availability, nines, MTBF and MTTR

The field is full of terms that get used as synonyms and are not. Pinning them down saves expensive misunderstandings.

Availability is the fraction of time in which the service responds correctly. It is expressed as a percentage, and in "nines":

Availability Name Maximum downtime per year Per month Per week
99% "two nines" 3 days 15 h 7 h 18 min 1 h 41 min
99.9% "three nines" 8 h 46 min 43 min 50 s 10 min 5 s
99.95% 4 h 23 min 21 min 55 s 5 min 2 s
99.99% "four nines" 52 min 34 s 4 min 23 s 1 min
99.999% "five nines" 5 min 15 s 26 s 6 s

Reading that table slowly corrects almost everybody's intuition. 99.99% means less than a minute of downtime a week, updates, reboots for a new kernel and provider failures included. With a single server, one monthly three-minute reboot already puts you below three and a half nines. And each additional nine multiplies the cost by roughly three.

Three distinctions worth having clear:

  • Reliability is the probability of running without failure over an interval. A system can be highly available and not very reliable: it fails often, but it recovers in seconds.
  • Durability refers to the data: the probability of not losing it. It is independent of availability — a system can be down and have its data perfectly safe.
  • MTBF (mean time between failures) and MTTR (mean time to repair) break availability down:
Availability = MTBF / (MTBF + MTTR)

And that formula contains the whole lesson's strategic decision: there are two ways of raising availability. Increasing MTBF — failing less: better hardware, more testing, fewer changes — or reducing MTTR — recovering sooner. In practice, reducing MTTR is almost always cheaper and more effective, because MTBF has a physical ceiling and MTTR does not.

It is exactly what you did in 07-06 without calling it that: taking MTTR from 8 hours to 50 minutes multiplied availability without buying a single component.

And the relationship with what you already have agreed with Marta:

Metric What it measures Current value
RPO How much data can be lost 4 hours
RTO How long the interruption can last 2 hours (revised in 07-06)
Availability Fraction of time running No formal target

There is a gap there: nobody has set an availability target. And without one, you cannot decide how much to invest.

Vertical and horizontal scaling

Vertical (scale up) Horizontal (scale out)
What you do A bigger machine More machines
Limit The biggest hardware there is Practically none
Cost Grows more than linearly Roughly linear
Complexity None High: distribution, state, consistency
Effect on availability None: there is still a single point of failure It is what enables it
Requires application changes No Almost always yes

The row that matters here is the second to last. Doubling srv-tramontana's memory would make it faster and exactly as fragile. Horizontal scaling is what allows the failure of one machine not to be the failure of the service, and that is why this lesson is about horizontal scaling even though the problem is not one of capacity.

Redundancy: active-passive and active-active

Having two machines is not the same as having redundancy: you have to decide what each one does.

Active-passive Active-active
The second node Waits, serving no traffic Serves traffic as well
Utilisation 50% of the hardware ~100%
Failover time Seconds to minutes Immediate
Complexity Medium High
It requires the application to… Start on the other node Be capable of running in several instances at once
Characteristic risk The passive node not working when it is needed Split-brain, data consistency

The active-passive risk is always underestimated and deserves a name of its own: a passive node that is never used is a node you do not know works. It was installed eight months ago, it has not had the latest updates, it has an expired certificate or a full disk. On the day of the failure, you discover that the backup fails too. The only defence is to use it periodically: fail over deliberately every few weeks.

State is the hard problem

Here is the conceptual core of the lesson, and what separates a design that works from one that looks as though it works.

Replicating processes is easy: you start the same application on two machines and you have two. Replicating state is hard, and state lives in more places than it seems.

An inventory of Tramontana Bookings' state:

State Where it lives The problem when duplicating
Booking data PostgreSQL The central problem: two databases diverge
Uploaded files /opt/tramontana/shared/uploads A photo uploaded to node A is not on B
User sessions In the process's memory A request to node B does not recognise the session started on A
Logs /var/log/tramontana/ They are spread across nodes: they have to be centralised to diagnose anything
Scheduled tasks systemd timers They would run TWICE: two backups, two purges
TLS certificate /etc/letsencrypt/ It has to be synchronised or terminated at the load balancer

That scheduled-tasks row is the one that causes the most trouble in practice, because nobody thinks about it when duplicating a server. With two identical nodes, tramontana-backup.timer fires at 02:30 on both: two simultaneous backups to the same destination, competing for the flock — which is local to each machine and therefore protects nothing. The solution requires a distributed lock or running the tasks only on one designated node.

A stateless application is one whose instances keep nothing between requests: all the state is in the database or in a shared store. It is the requirement for active-active, and it is almost never met without development work. In Tramontana:

Component Stateless? What it would take
Serving pages and querying bookings Yes Nothing
User sessions No Move them to Redis or to the database
Photo uploads No Shared storage or an object store
Database No, by definition Replication

Honest conclusion: Tramontana is not a stateless application today. Before setting anything up, two changes to the application are needed: externalised sessions and files in shared storage. That is development work, and telling Marta before buying servers is part of the job.

Load balancing: layers, algorithms and health checks

A load balancer distributes requests among several servers. It operates at one of two layers:

Layer 4 (transport) Layer 7 (application)
What it inspects IP and port HTTP headers, paths, cookies
Performance Very high High, but lower
TLS termination It cannot Yes
Routing by path or domain No Yes
Retrying a failed request No Yes
Examples IPVS, nftables, NLB HAProxy, Nginx, Traefik

For HTTP, layer 7 is almost always the right one: it lets you terminate TLS at a single point, route by path, retry a request that fails and make meaningful health checks.

Algorithms

Algorithm How it distributes When
roundrobin In turns Homogeneous requests, identical servers
leastconn To whichever has fewest connections Requests of variable duration
source A hash of the source IP When affinity is needed (with reservations)
uri A hash of the path Caches: the same path always to the same node
random At random, with two probes Many load balancers in parallel

leastconn is the most suitable for Tramontana: an availability query takes 40 ms and a billing report takes several seconds, so distributing in turns would pile the slow ones onto the same node.

Health checks

This is the piece that makes the load balancer useful: without it, it would carry on sending traffic to a server that is down.

Type How it works Advantage
Active The load balancer probes every N seconds It detects the failure even when there is no traffic
Passive It watches the errors in real traffic No added cost; it detects partial failures

The right thing is to use both. And this is where something you have had since Module 4 fits in: health_check.sh returns 0, 1 or 2 — OK, warning, critical. That gradation is exactly what a load balancer needs, provided the application exposes an equivalent endpoint:

State HTTP What the load balancer should do
0 OK 200 Send traffic normally
1 warning 200 with a warning header Send, but with less weight
2 critical 503 Send no traffic

And the distinction almost nobody gets right: a shallow health check is worse than none. If the health endpoint only says "I am alive" without checking the database, the load balancer will carry on sending traffic to a node that cannot serve a single request. If it checks too much — a heavy query, for instance — a database problem marks every node as down and the service disappears entirely, when it could at least have served the static pages.

The balance: check the critical dependencies with a cheap operation. A SELECT 1 with a time limit, not a report.

Sticky sessions: an anti-pattern

A sticky session ties each client to one node, by cookie or by IP. It solves the problem of in-memory sessions... and creates others:

  • The distribution becomes unbalanced: one node can end up with three times the load.
  • When a node goes down, all of its users lose their session at once.
  • It prevents draining a node for maintenance without affecting anybody.
  • It masks the real problem instead of solving it.

It is an acceptable stopgap while sessions are being externalised. As a permanent solution, no.

Connection draining

To deploy without cutting the service, a node has to be able to leave the rotation in an orderly way: stop receiving new requests and finish the ones it already has in flight.

# 1. Take the node out of the rotation (it stops receiving new requests)
$ echo "set server tramontana/app1 state drain" | \
    sudo socat stdio /run/haproxy/admin.sock

# 2. Wait for the in-flight connections to finish
$ while [[ "$(echo 'show stat' | sudo socat stdio /run/haproxy/admin.sock \
      | awk -F, '$1=="tramontana" && $2=="app1" {print $5}')" != "0" ]]; do
      sleep 1
  done

# 3. Deploy in peace
$ ssh [email protected] 'sudo ~/scripts/deploy.sh 3.3.0'

# 4. Put it back in the rotation
$ echo "set server tramontana/app1 state ready" | \
    sudo socat stdio /run/haproxy/admin.sock

Repeated node by node, that is a zero-downtime deployment. And it is the answer to something left outstanding since 04-07: deploy.sh restarts the service, and that restart is a few seconds of rejected requests. With two nodes and draining, zero.

HAProxy in practice

$ sudo apt install haproxy socat
$ haproxy -v
HAProxy version 2.8.5-1ubuntu3 2024/01/31
# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    chroot /var/lib/haproxy
    # Admin socket: allows draining nodes on the fly
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon
    maxconn 4000

    # Modern TLS profile (06-05). Never invent the cipher suite list.
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    # Retry on ANOTHER server if the first one fails. Only safe for
    # idempotent requests: hence 'if-none', which excludes POST.
    retry-on all-retryable-errors
    option  redispatch
    retries 3
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    timeout http-request 10s
    # Draining: wait for connections to finish when taking a node out
    default-server init-addr last,libc,none

# ---------- Entry point ----------
frontend tramontana_https
    bind *:443 ssl crt /etc/haproxy/certs/bookings.tramontana.example.pem alpn h2,http/1.1
    bind *:80
    # Redirect HTTP to HTTPS, except for the Let's Encrypt challenge
    acl acme_challenge path_beg /.well-known/acme-challenge/
    http-request redirect scheme https code 301 unless { ssl_fc } || acme_challenge

    # HSTS (06-05). Start with a short max-age and raise it later.
    http-response set-header Strict-Transport-Security "max-age=63072000"

    # Per-IP rate limiting: it complements fail2ban at layer 7
    stick-table type ip size 100k expire 60s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

    default_backend tramontana_app

# ---------- Application servers ----------
backend tramontana_app
    # leastconn: the requests vary greatly in duration
    balance leastconn

    # ACTIVE health check at layer 7.
    # The /health endpoint checks the critical dependencies cheaply.
    option httpchk
    http-check send meth GET uri /health ver HTTP/1.1 hdr Host bookings.tramontana.example
    http-check expect status 200

    # Headers so the application knows who the real client is
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    option forwardfor

    # inter: how often to probe. rise/fall: how many times in a row
    # before considering the node healthy or down. slowstart: on its
    # return, raise the weight gradually instead of taking the whole load at once.
    server app1 10.0.2.21:8080 check inter 3s rise 2 fall 3 slowstart 30s maxconn 200
    server app2 10.0.2.22:8080 check inter 3s rise 2 fall 3 slowstart 30s maxconn 200

    # PASSIVE check: if a server returns errors on real traffic, it is
    # taken out temporarily even if its /health responds.
    option redispatch

# ---------- Statistics, internal network only ----------
listen stats
    bind 10.0.2.20:8404
    stats enable
    stats uri /
    stats refresh 5s
    stats admin if TRUE
    acl internal_network src 10.0.2.0/24
    http-request deny unless internal_network
$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

$ sudo systemctl reload haproxy
$ echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | \
      awk -F, 'NR==1 || $1=="tramontana_app" {print $1","$2","$18","$5}'
# pxname,svname,status,scur
tramontana_app,app1,UP,12
tramontana_app,app2,UP,9
tramontana_app,BACKEND,UP,21

Four details in that configuration deserve an explanation:

slowstart 30s. When a node comes back after a failure, its caches are cold and its connection pools empty. Sending it a third of the traffic all at once can bring it down again, and start a cycle of outages. slowstart raises its weight gradually over 30 seconds.

rise 2 fall 3. Asymmetric on purpose: three consecutive failures are needed to take a node out (avoiding false positives from a spike) but only two successes to bring it back. Taking a node out on a single transient failure is how a load balancer causes an outage.

option redispatch with retry-on. If the chosen server fails, it retries on another. With one important nuance: retrying a POST can duplicate a booking. HAProxy only retries when the request has not been sent or the method is idempotent.

The stick-table with rate limiting. It is 06-03's fail2ban at layer 7: it counts requests per IP in 10-second windows and returns 429 when the threshold is exceeded. Complementary, not a substitute: fail2ban blocks at the network level and this one at the application level.

Nginx can do the same thing with upstream, and it is the natural option if you already have it as a reverse proxy — which is what you will set up in 08-01:

upstream tramontana {
    least_conn;
    server 10.0.2.21:8080 max_fails=3 fail_timeout=30s;
    server 10.0.2.22:8080 max_fails=3 fail_timeout=30s;
}

The practical difference: Nginx only does passive checks in its free version; the active ones belong to the commercial version. HAProxy has them as standard, and that is why it is preferable when balancing is the main job.

High availability for the load balancer: keepalived and VRRP

And here is the most common design error in the field, so frequent that it deserves stating on its own:

Putting a load balancer in front of two servers does not remove the single point of failure: it moves it to the load balancer.

Before, you had one machine that could go down. Now you have three, and if the one that distributes goes down, the service disappears just the same — with the aggravating factor that there are now more pieces that can fail.

The solution is VRRP (Virtual Router Redundancy Protocol): two load balancers share a floating virtual IP, the one DNS points at. One holds it; if it stops advertising, the other takes it over within seconds.

$ sudo apt install keepalived
# /etc/keepalived/keepalived.conf — PRIMARY NODE (lb1, 10.0.2.20)
global_defs {
    router_id lb1
    enable_script_security
    script_user keepalived_script
}

# Check on the LOCAL service. If HAProxy dies on this node, the priority
# drops and the other one takes the IP. Without this, a node with
# keepalived alive and HAProxy dead keeps the IP and the service goes down.
vrrp_script check_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight -40        # drops the priority by 40 points on failure
    fall 2
    rise 2
}

vrrp_instance TRAMONTANA_VIP {
    state MASTER
    interface enp0s3
    virtual_router_id 51        # IDENTICAL on both nodes
    priority 100                # higher than the backup
    advert_int 1                # advertise every second

    authentication {
        auth_type PASS
        auth_pass {{ vault_vrrp_pass }}
    }

    virtual_ipaddress {
        10.0.2.30/24 dev enp0s3
    }

    track_script {
        check_haproxy
    }

    notify_master "/usr/local/sbin/vrrp-notify master"
    notify_backup "/usr/local/sbin/vrrp-notify backup"
    notify_fault  "/usr/local/sbin/vrrp-notify fault"
}
# BACKUP NODE (lb2, 10.0.2.21): identical except for three lines
    state BACKUP
    priority 90
    router_id lb2
$ sudo systemctl enable --now keepalived
$ ip -brief addr show enp0s3
enp0s3  UP  10.0.2.20/24 10.0.2.30/24     # <- the VIP is on lb1

# On lb2 the VIP does NOT appear: it is on standby
$ ssh [email protected] 'ip -brief addr show enp0s3'
enp0s3  UP  10.0.2.21/24

The mechanism: the node with the higher priority advertises the VIP every second. If the backup stops hearing advertisements for three intervals, it assumes the primary has gone down, assigns itself the IP and sends a gratuitous ARP so that the network switches update their tables. The failover takes between 3 and 4 seconds.

The track_script with weight -40 is what avoids the silliest failure in this architecture: without it, a node where keepalived works but HAProxy is dead cheerfully keeps the VIP, and the service is down even though the other load balancer is perfectly fine. With it, lb1's priority drops from 100 to 60, below lb2's 90, and the VIP moves.

And enable_script_security with script_user is a real security precaution: keepalived runs as root and executes those scripts; without those directives, a script with loose permissions is a privilege escalation.

Split-brain and quorum

The characteristic failure of any redundant system, and the one that causes the most serious data losses.

Split-brain happens when the nodes stop seeing each other but both keep running. Each concludes that the other has gone down, and both declare themselves primary:

  • Two load balancers advertise the same virtual IP → unpredictable traffic, broken connections.
  • Two databases accept writes → the data diverges, and reconciling it afterwards can be impossible.

The scenario that causes it is not a node going down — that is handled well — but the network between them being cut while both are still alive and serving clients. It is more frequent than it sounds: a switch that fails, a badly placed firewall rule, a momentary saturation.

The three defences:

Defence How it works Limitation
Quorum Only the group with a majority of nodes acts Requires 3 or more nodes, an odd number
Fencing (STONITH) The survivor powers off the other one at the hardware level Requires remote power or hypervisor control
Witness A third resource arbitrates (a disk, an IP) Less robust than real quorum

And the conclusion to hold on to:

With two nodes there is no possible quorum. Neither can have a majority of two. That is why a serious cluster has three nodes, or two plus a witness.

This has a direct consequence for Tramontana's design: a pair of load balancers with VRRP is acceptable, because the worst case — two nodes advertising the same IP for a few seconds — is annoying but recoverable. A pair of databases with automatic failover is not: the worst case is data divergence, and it does not recover. Hence the recommendation in the next section.

Data replication

PostgreSQL replicates through the write-ahead log (WAL): the primary sends its changes to one or more replicas.

Mode The primary commits when… Risk of loss Cost
Asynchronous It has written locally The last few transactions None
Synchronous The replica has confirmed None Latency on every write

And the trade-off, which has to be understood before choosing: with synchronous replication, every write waits for the round trip to the replica. On the same local network, one or two milliseconds. Between data centres, tens. And if the replica goes down, the primary blocks waiting for confirmation — unless synchronous_standby_names is configured with the right syntax, which opens the door to losing data exactly when it matters most.

For Tramontana, with an agreed RPO of 4 hours, asynchronous replication is more than enough: losing the last few seconds of transactions is well within what is tolerated.

# postgresql.conf on the PRIMARY
wal_level = replica
max_wal_senders = 3
wal_keep_size = 1GB
archive_mode = on
archive_command = 'test ! -f /srv/wal/%f && cp %p /srv/wal/%f'
hot_standby = on
# Create the replica from scratch
$ sudo -u postgres pg_basebackup -h 10.0.2.15 -U replicator \
      -D /var/lib/postgresql/16/main -R -P -X stream -c fast

$ sudo -u postgres psql -c "SELECT client_addr, state, sync_state,
    pg_wal_lsn_diff(sent_lsn, replay_lsn) AS lag_bytes FROM pg_stat_replication;"
 client_addr | state     | sync_state | lag_bytes
-------------+-----------+------------+-----------
 10.0.2.22   | streaming | async      |         0

That lag_bytes is the metric to watch: it is how much data would be lost if the primary went down now. It goes straight into the monitoring, alongside the age of the last backup from 05-08.

Failover: manual versus automatic

Manual Automatic
Recovery time Minutes Seconds
Split-brain risk None High without quorum
Requires somebody available Yes No
Tools pg_ctl promote Patroni, repmgr, pg_auto_failover

The professional recommendation, and it goes against intuition:

With two nodes, database failover must be manual. Without quorum, a network failure between primary and replica can promote the replica while the primary is still accepting writes, and then there are two diverging databases. Recovering from that can mean losing transactions or reconciling by hand.

Tools such as Patroni do automatic failover safely, but they require a distributed configuration store with quorum of its own (etcd or Consul), which in turn needs three nodes. In other words: automating failover safely requires at least five machines. It is an architectural decision, not a box to tick.

Shared storage

For /opt/tramontana/shared/uploads:

Option Advantage Drawback
NFS Simple, works without touching the application It is a new single point of failure
GlusterFS / Ceph Distributed and replicated High complexity
Object store (S3, MinIO) Scalable, managed, replicated Requires changing the application
Periodic rsync Trivial Not real time: files get lost

NFS has the characteristic irony: it is mounted to provide high availability and, if it is not made redundant too, it worsens reliability, because now two servers depend on a third machine.

The modern answer is the object store: it is what applications designed to scale horizontally use. It demands development work, and it is the same conclusion as the section on state.

Clusters, Pacemaker and fencing

For more complex scenarios — databases, shared filesystems, resources with dependencies — there is the Pacemaker + Corosync stack:

  • Corosync: communication between nodes, cluster membership and quorum.
  • Pacemaker: resource management, with dependencies and constraints.
$ sudo pcs status
Cluster name: tramontana
Cluster Summary:
  * Stack: corosync
  * Current DC: node1 (version 2.1.6) - partition with quorum
  * 3 nodes configured
  * 4 resource instances configured

Full List of Resources:
  * vip_tramontana  (ocf:heartbeat:IPaddr2):  Started node1
  * postgresql      (ocf:heartbeat:pgsql):    Master node1

A resource is anything the cluster manages (an IP, a service, a filesystem), and the constraints express rules: "the IP must be where the PostgreSQL primary is", "this service must start after that one".

And the concept without which none of this really works:

Fencing (or STONITH, Shoot The Other Node In The Head): before taking over the resources of a node that appears to be down, the survivor makes sure it is powered off, by cutting its power or destroying its virtual machine.

Why it is indispensable: if node A appears to be down but is in fact merely frozen or cut off, and B takes over its resources — mounts the filesystem, promotes the database, takes the IP — then when A recovers both will write to the same data. The resulting corruption can be unrecoverable.

# Fencing over libvirt: destroys the other node's virtual machine
$ sudo pcs stonith create fence_node2 fence_virsh \
    ip=10.0.2.10 login=fence identity_file=/etc/pacemaker/id_rsa \
    pcmk_host_list=node2 action=off

On physical hardware it is done with IPMI, iLO or manageable power distribution units. And the rule in the field is blunt: a cluster with no fencing configured is not a reliable cluster. Many projects disable it "temporarily" because it complicates testing, and that is exactly the configuration that causes data loss on the day of the failure.

A design for Tramontana and the cost analysis

graph TD
    I["Internet"] --> DNS["DNS: bookings.tramontana.example<br/>→ 10.0.2.30 (VIP)"]
    DNS --> VIP{"Floating virtual IP<br/>10.0.2.30"}
    VIP -.->|VRRP| LB1["lb1 · HAProxy<br/>MASTER prio 100"]
    VIP -.->|VRRP| LB2["lb2 · HAProxy<br/>BACKUP prio 90"]
    LB1 --> APP1["app1 · Tramontana<br/>10.0.2.21:8080"]
    LB1 --> APP2["app2 · Tramontana<br/>10.0.2.22:8080"]
    LB2 -.-> APP1
    LB2 -.-> APP2
    APP1 --> DB1["PostgreSQL primary<br/>10.0.2.15"]
    APP2 --> DB1
    DB1 -->|asynchronous WAL| DB2["PostgreSQL replica<br/>10.0.2.16"]
    APP1 --> OBJ["Object store<br/>uploads"]
    APP2 --> OBJ

What each layer protects against, and what it does not:

Failure Covered? Recovery time
app1 or app2 goes down Yes, automatically 3-9 s (fall 3 × inter 3s)
lb1 goes down Yes, automatically 3-4 s (VRRP)
The PostgreSQL primary goes down No: promotion is deliberately manual 5-15 min
The network between lb1 and lb2 fails Partially: a brief split-brain, recoverable Seconds
The host running everything goes down No The full RTO
A logical failure (deletion, corruption) No: it replicates instantly Restore a backup
A faulty deployment No: it is deployed to both Rollback

The last three rows are the ones to underline for management. Redundancy does not protect against logical errors: a mistaken DELETE replicates to the replica in milliseconds. That is what the backups from 05-08 are for, and this architecture does not replace them in the slightest.

The cost analysis

What the lesson has been promising, and what Marta has been asking for over three replies:

Cost of the complete architecture (5 machines against 1):

Item Annual estimate
4 additional machines The current VPS cost × 4
Development work (sessions + object store) 3-4 weeks, once
Setup and automation with Ansible 2 weeks, once
Additional maintenance ~4 h/month indefinitely
Object store Low, by volume
Complexity: more failure modes Hard to quantify, real

The cost of unavailability. The question that has to be answered first:

Bookings per month: ~500
Average amount:     €313.70
Average revenue per hour (24×30):  500 × 313.70 / 720 ≈ €218/h

But that raw calculation is misleading for three reasons, and saying so is part of the analysis:

  1. Bookings are not uniform. They cluster in the afternoon and at weekends. A two-hour outage at four in the morning on a Tuesday may cost nothing; the same outage on a Friday afternoon costs much more than the average.
  2. A lost booking is not always lost. Many customers try again. The real cost is between 20% and 50% of the raw figure.
  3. There is a reputational cost, non-linear and hard to quantify, especially if the outage coincides with a campaign.

A reasonable estimate of the annual cost of unavailability, at different targets:

Availability Annual downtime Estimated cost (at €218/h × 35%)
99.5% (estimated current situation) 43.8 h ~€3,340
99.9% 8.8 h ~€670
99.95% 4.4 h ~€335

The saving from going from 99.5% to 99.9% is about €2,700 a year. That has to be compared against the cost of the complete architecture — four more machines, six weeks of initial work and four hours a month indefinitely — which very probably exceeds it.

And here is the analysis almost nobody does, and which changes the recommendation:

Measure Cost Estimated availability
The current situation — ~99.5%
Just two application nodes + one load balancer 2 machines, 1 week ~99.8%
The complete architecture with a replicated DB 4 machines, 6 weeks ~99.9%

The first step captures most of the benefit for a fraction of the cost, and this is the usual pattern in availability: the first nines are cheap and the following ones get exponentially more expensive. The reason is that most interruptions do not come from a hardware failure but from deployments, reboots and maintenance — and all of that is covered by two application nodes and connection draining, without touching the database.

Recommendation

Phase 1 (now): set a formal availability target with Marta. Without a target you cannot decide how much to invest. And measure it, because the current 99.5% is an estimate, not a figure.

Phase 2 (1-2 months): two application nodes and one load balancer. Externalised sessions and files in an object store, which is development work. It eliminates interruptions caused by deployment and by the failure of one node, which are the majority.

Phase 3 (assess later): a second load balancer with VRRP, and a PostgreSQL replica with manual promotion.

Phase 4 (probably never): automatic database failover. It requires quorum, which is to say five machines and a distributed store. For a company of this size, the complexity outweighs the benefit.

Testing failures by causing them

An untested high-availability system is not a high-availability system: it is an architecture you believe works. And the history of the field is full of cases where the backup failed on the day it was needed.

# --- Test 1: an application node goes down ---
$ ssh [email protected] 'sudo systemctl stop tramontana'
# Expected: HAProxy takes it out in ~9 s (fall 3 x inter 3s), with no visible errors
$ for i in {1..60}; do
      curl -s -o /dev/null -w '%{http_code} ' https://bookings.tramontana.example/houses
      sleep 1
  done; echo
# Criterion: zero responses != 200

# --- Test 2: the primary load balancer goes down ---
$ ssh [email protected] 'sudo systemctl stop keepalived'
# Expected: the VIP moves to lb2 in 3-4 s
$ ssh [email protected] 'ip -brief addr show enp0s3 | grep 10.0.2.30'

# --- Test 3: HAProxy dies but keepalived lives ---
# This is the test that validates the track_script. Without it, the VIP
# does NOT move and the service goes down with both load balancers "alive".
$ ssh [email protected] 'sudo systemctl stop haproxy'
$ sleep 6 && ssh [email protected] 'ip -brief addr show enp0s3 | grep 10.0.2.30'

# --- Test 4: a network partition between the load balancers (split-brain) ---
$ ssh [email protected] 'sudo nft add rule inet filter input ip saddr 10.0.2.21 drop'
# Expected: BOTH take the VIP. Document the observed behaviour.
$ ssh [email protected] 'sudo nft flush chain inet filter input'

# --- Test 5: a zero-downtime deployment ---
$ echo "set server tramontana_app/app1 state drain" | sudo socat stdio /run/haproxy/admin.sock
$ ssh [email protected] 'sudo ~/scripts/deploy.sh 3.3.0'
$ echo "set server tramontana_app/app1 state ready" | sudo socat stdio /run/haproxy/admin.sock
# Criterion: zero errors throughout the process

Test 4 is the most uncomfortable and the most important: it cannot be "passed" with two nodes, because without quorum split-brain is unavoidable. What you do is document the observed behaviour and the recovery procedure, so that on the day it happens nobody has to improvise.

Chaos engineering takes this further: injecting failures continuously and automatically in production, on the logic that if failures happen every day, the system is forced to tolerate them and the team is forced to know how to respond. For Tramontana it is disproportionate; the principle — cause the failures yourself, during working hours, instead of waiting for them — applies at any scale, and it fits the six-monthly rehearsal you proposed in 07-06.

Common Mistakes and Tips

  • Putting in a load balancer and thinking you now have high availability. All you have done is move the single point of failure. The load balancer needs redundancy of its own.
  • Configuring VRRP without track_script. A node with keepalived alive and HAProxy dead keeps the virtual IP, and the service is down with both load balancers "working".
  • Automatic database failover with two nodes. Without quorum, a network partition produces two primaries and diverging data. Manual, until there are three nodes.
  • Disabling fencing "temporarily". It is the configuration that causes data corruption on the day of the failure. A cluster with no fencing is not reliable.
  • Shallow health checks. A /health that only says "I am alive" makes the load balancer send traffic to a node that cannot serve anything.
  • Health checks that are too deep. A database failure marks every node as down, when they could have served something.
  • Taking a node out on the first failure. A transient spike takes the node out, the load concentrates on the rest, and they fall in a cascade. fall 3 and slowstart.
  • Forgetting that the timers run on every node. Two simultaneous backups, two purges. The flock is local and protects nothing.
  • Sticky sessions as a permanent solution. They mask the real problem, unbalance the load, and when a node goes down it loses all of its sessions.
  • Believing that replication replaces backups. An accidental deletion replicates in milliseconds. Redundancy protects against hardware failure, not against human error.
  • NFS with no redundancy of its own to provide high availability. You add a single point of failure that two servers now depend on.
  • Not testing the failures. An untested high-availability system is an architecture you believe works.
  • A tip on method. Before designing anything, set the availability target and measure the current one. Without those two numbers, any investment is a hunch dressed up as architecture.

Exercises

Exercise 1

Design the /health endpoint the application should expose so that HAProxy makes good decisions. Specify what it checks, what it does not, what codes it returns and with what latency, and explain how it relates to health_check.sh.

Exercise 2

Write the Ansible role that deploys the HAProxy and keepalived configuration on both load balancers, bearing in mind that the configuration differs between them and that applying it badly leaves the service with no IP.

Exercise 3

Marta asks directly: "Do we need high availability?". Write the answer with the complete analysis and a clear recommendation.

Solutions

Solution 1

Design of the /health endpoint:

GET /health
It checks How Why
That the process responds That the request arrives Trivial but necessary
The PostgreSQL connection SELECT 1 with a 1 s timeout Without the database it cannot serve anything
The connection pool That at least one is free With the pool exhausted, requests queue up
The file store Writing one byte every 30 s (cached) Without it uploads fail, but queries do not
Space in /var/log A 95% threshold (cached) With no room for logs, traceability is lost
It does NOT check Why
Real business queries Expensive; a data problem would bring down every node
External services (mail, payments) A third-party failure would take the whole service out of the rotation
The state of other nodes Each node reports only on itself
Performance metrics That is monitoring, not health

The three states and their responses:

// 200 OK — state 0, OK
{
  "status": "ok",
  "version": "3.2.1",
  "checks": {
    "db": {"ok": true, "latency_ms": 2},
    "pool": {"ok": true, "free": 68, "total": 80},
    "storage": {"ok": true},
    "log_disk": {"ok": true, "usage_pct": 34}
  }
}
// 200 OK with the header X-Health-Warning: degraded — state 1
// It STILL receives traffic: it can serve queries
{
  "status": "degraded",
  "warnings": ["storage_unavailable"],
  "checks": {
    "db": {"ok": true, "latency_ms": 3},
    "pool": {"ok": true, "free": 12, "total": 80},
    "storage": {"ok": false, "error": "timeout"},
    "log_disk": {"ok": true, "usage_pct": 34}
  }
}
// 503 Service Unavailable — state 2, critical
// It receives NO traffic
{
  "status": "critical",
  "errors": ["db_unreachable"],
  "checks": {
    "db": {"ok": false, "error": "connection refused"}
  }
}

The relationship with health_check.sh is the conceptual part of the exercise. Both answer the same question from different sides, and that complementarity should be exploited rather than duplicating work:

health_check.sh /health
Who asks The administrator or a timer HAProxy, every 3 s
From where Outside the process, on the machine From inside the process
Sees the internal state No Yes: connection pool, caches
Frequency Minutes Seconds
Acceptable cost High Very low
Codes 0 / 1 / 2 200 / 200+header / 503

The right approach is for health_check.sh to consume /health rather than reimplementing the checks:

# A fragment of health_check.sh, adapted
check_application() {
    local response code status
    response="$(curl -s -m 5 -w '\n%{http_code}' \
        "http://127.0.0.1:${TRAMONTANA_PORT}/health")" || {
        error "the application is not responding"
        return 2
    }
    code="$(tail -1 <<<"$response")"
    status="$(sed '$d' <<<"$response" | jq -r '.status')"

    case "$status" in
        ok)       log "application OK"; return 0 ;;
        degraded) error "degraded: $(sed '$d' <<<"$response" | jq -r '.warnings|join(", ")')"
                  return 1 ;;
        critical) error "critical: $(sed '$d' <<<"$response" | jq -r '.errors|join(", ")')"
                  return 2 ;;
        *)        error "unknown status (HTTP $code)"; return 2 ;;
    esac
}

Four design decisions that have to be justified:

  1. The degraded state returns 200, not 503. It is counterintuitive and it is correct: if the file store fails, the node can carry on serving availability queries, which are most of the traffic. Taking it out of the rotation would reduce capacity and gain nothing. The header lets the monitoring detect it without the load balancer acting on it.
  2. Latency bounded by design. With a 3-second interval and two nodes, that is 40 requests a minute. The target is under 50 ms, and hence the expensive checks are cached for 30 seconds.
  3. No authentication, but restricted by network. It must be reachable without credentials so that HAProxy can query it, and it must not be exposed to the Internet: the detail in the JSON is useful information for an attacker. It is blocked in the frontend:
    http-request deny if { path_beg /health } !{ src 10.0.2.0/24 }
    
  4. No side effects and no blocking. A /health that writes to the database or takes a lock can cause the failure it is meant to detect. The store's write check is cached and runs in the background.

And the final warning, which is the most common mistake: /health must not check dependencies that do not belong to this node. If it checked the state of the database with a heavy query, a PostgreSQL problem would take every node out at once and the service would disappear entirely, when it could at least have returned a decent error page.

Solution 2

# roles/loadbalancer/defaults/main.yml
haproxy_vip: 10.0.2.30
haproxy_vip_cidr: 24
haproxy_interface: enp0s3
haproxy_vrrp_id: 51
haproxy_backends:
  - { name: app1, address: 10.0.2.21, port: 8080 }
  - { name: app2, address: 10.0.2.22, port: 8080 }
haproxy_check_inter: 3s
haproxy_check_rise: 2
haproxy_check_fall: 3
haproxy_slowstart: 30s
# inventory/production.yml (fragment)
loadbalancers:
  hosts:
    lb1:
      ansible_host: 10.0.2.20
      vrrp_state: MASTER
      vrrp_priority: 100
    lb2:
      ansible_host: 10.0.2.21
      vrrp_state: BACKUP
      vrrp_priority: 90
# roles/loadbalancer/tasks/main.yml
---
# =====================================================================
# WARNING: an incorrect keepalived configuration can leave the service
# WITH NO VIRTUAL IP, that is, completely down. This role deploys the
# nodes ONE AT A TIME and verifies between them.
# =====================================================================

- name: "Check that it is not running against both load balancers in parallel"
  ansible.builtin.assert:
    that: ansible_play_hosts | length == 1
    fail_msg: >-
      This role must be run with --limit against ONE load balancer at a time,
      or with serial: 1 in the play. Configuring both at once can leave the
      service with no virtual IP.

- name: Install HAProxy and keepalived
  ansible.builtin.apt:
    name: [haproxy, keepalived, socat]
    state: present
  tags: [packages]

# ---------- HAProxy ----------
- name: Create the certificates directory
  ansible.builtin.file:
    path: /etc/haproxy/certs
    state: directory
    owner: root
    group: haproxy
    mode: '0750'
  tags: [tls]

- name: Install the combined certificate for HAProxy
  ansible.builtin.copy:
    content: "{{ vault_certificate_pem }}"
    dest: /etc/haproxy/certs/bookings.tramontana.example.pem
    owner: root
    group: haproxy
    mode: '0640'
  no_log: true                    # it contains the private key
  notify: Reload haproxy
  tags: [tls]

- name: Deploy the HAProxy configuration
  ansible.builtin.template:
    src: haproxy.cfg.j2
    dest: /etc/haproxy/haproxy.cfg
    owner: root
    group: root
    mode: '0644'
    backup: true
    # VALIDATES before installing: a broken config stops it starting
    validate: 'haproxy -c -f %s'
  notify: Reload haproxy
  tags: [haproxy]

- name: Enable HAProxy
  ansible.builtin.systemd_service:
    name: haproxy
    enabled: true
    state: started
  tags: [haproxy]

# ---------- keepalived ----------
# The sysctl that lets HAProxy listen on an IP this node does not have
# yet (the VIP while it is on the other one). Without this, HAProxy
# does not start on the BACKUP node.
- name: Allow binding to non-local IP addresses
  ansible.posix.sysctl:
    name: net.ipv4.ip_nonlocal_bind
    value: '1'
    sysctl_file: /etc/sysctl.d/71-loadbalancer.conf
    sysctl_set: true
    reload: true
  tags: [keepalived]

- name: Create the user for the keepalived scripts
  ansible.builtin.user:
    name: keepalived_script
    system: true
    shell: /usr/sbin/nologin
    create_home: false
  tags: [keepalived]

- name: Install the VRRP notification script
  ansible.builtin.template:
    src: vrrp-notify.j2
    dest: /usr/local/sbin/vrrp-notify
    owner: root
    group: root
    mode: '0755'      # NOT writable by the script's user
  tags: [keepalived]

- name: Deploy the keepalived configuration
  ansible.builtin.template:
    src: keepalived.conf.j2
    dest: /etc/keepalived/keepalived.conf
    owner: root
    group: root
    mode: '0600'      # it contains the VRRP password
    backup: true
  no_log: true
  notify: Restart keepalived
  tags: [keepalived]

- name: Enable keepalived
  ansible.builtin.systemd_service:
    name: keepalived
    enabled: true
    state: started
  tags: [keepalived]

# ---------- Verification ----------
- name: Flush the handlers before verifying
  ansible.builtin.meta: flush_handlers

- name: "Verify | HAProxy responds on the admin socket"
  ansible.builtin.shell:
    cmd: >-
      set -o pipefail;
      echo "show stat" | socat stdio /run/haproxy/admin.sock
      | awk -F, '$1=="tramontana_app" && $2=="BACKEND" {print $18}'
  register: backend_status
  changed_when: false
  failed_when: "'UP' not in backend_status.stdout"
  tags: [verification]

- name: "Verify | The backends are healthy"
  ansible.builtin.shell:
    cmd: >-
      set -o pipefail;
      echo "show stat" | socat stdio /run/haproxy/admin.sock
      | awk -F, '$1=="tramontana_app" && $2 ~ /^app/ && $18=="UP"' | wc -l
  register: backends_up
  changed_when: false
  failed_when: backends_up.stdout | int < 1
  tags: [verification]

- name: "Verify | The VIP is on the MASTER and only there"
  ansible.builtin.shell:
    cmd: "ip -brief addr show {{ haproxy_interface }} | grep -c {{ haproxy_vip }} || true"
  register: has_vip
  changed_when: false
  tags: [verification]

- name: "Verify | The VIP state is consistent"
  ansible.builtin.assert:
    that:
      - (vrrp_state == 'MASTER' and has_vip.stdout | int == 1) or
        (vrrp_state == 'BACKUP' and has_vip.stdout | int == 0)
    fail_msg: >-
      Inconsistent VIP state on {{ inventory_hostname }}
      ({{ vrrp_state }}, VIP present: {{ has_vip.stdout }}).
      Check before continuing with the other load balancer.
  tags: [verification]

- name: "Verify | The service responds through the VIP"
  ansible.builtin.uri:
    url: "https://bookings.tramontana.example/health"
    validate_certs: true
    status_code: 200
    timeout: 10
  delegate_to: localhost
  become: false
  run_once: true
  tags: [verification]
# roles/loadbalancer/handlers/main.yml
- name: Reload haproxy
  ansible.builtin.systemd_service:
    name: haproxy
    state: reloaded      # reload, not restart: it does not cut the connections

- name: Restart keepalived
  ansible.builtin.systemd_service:
    name: keepalived
    state: restarted
{# roles/loadbalancer/templates/keepalived.conf.j2 #}
# GENERATED BY ANSIBLE — DO NOT EDIT BY HAND
global_defs {
    router_id {{ inventory_hostname }}
    enable_script_security
    script_user keepalived_script
}

vrrp_script check_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight -40
    fall 2
    rise 2
}

vrrp_instance TRAMONTANA_VIP {
    state {{ vrrp_state }}
    interface {{ haproxy_interface }}
    virtual_router_id {{ haproxy_vrrp_id }}
    priority {{ vrrp_priority }}
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass {{ vault_vrrp_pass }}
    }

    virtual_ipaddress {
        {{ haproxy_vip }}/{{ haproxy_vip_cidr }} dev {{ haproxy_interface }}
    }

    track_script {
        check_haproxy
    }

    notify_master "/usr/local/sbin/vrrp-notify master"
    notify_backup "/usr/local/sbin/vrrp-notify backup"
    notify_fault  "/usr/local/sbin/vrrp-notify fault"
}

And the play that guarantees deployment one node at a time:

# loadbalancers.yml
- name: Configure the load balancers
  hosts: loadbalancers
  become: true
  # serial: 1 -> one node at a time. If the first fails, the second is NOT touched.
  serial: 1
  # max_fail_percentage: 0 -> abort on the first failure
  max_fail_percentage: 0
  # DELIBERATE ORDER: the BACKUP first. If something goes wrong, the MASTER
  # keeps the VIP and the service is not interrupted.
  order: reverse_inventory
  roles:
    - loadbalancer

The decisions that stop the service being left with no IP:

Decision What it protects against
serial: 1 + a single-host assert Configuring both at once and ending up with no node holding the VIP
order: reverse_inventory (BACKUP first) If the role fails, the MASTER keeps the VIP and the service carries on
max_fail_percentage: 0 A failure on lb2 going undetected before lb1 is touched
validate: haproxy -c -f %s Installing a configuration that stops HAProxy starting
meta: flush_handlers before verifying Verifying the state as it was before the changes
state: reloaded on HAProxy restart would cut every in-flight connection
The VIP consistency assert Detecting that both or neither hold the VIP
uri against the VIP with run_once Checking the service end to end, not just the pieces
net.ipv4.ip_nonlocal_bind HAProxy does not start on the BACKUP if it cannot listen on the absent VIP
no_log: true on the certificate and VRRP The private key and the password would appear in the output
A 0755 script owned by root enable_script_security rejects scripts writable by the user that runs them

And what has to be added to the runbook: this role is tested first in the test environment with two virtual load balancers, and in production it is run with --check --diff before running for real. The virtual_router_id must be unique on the network segment: two VRRP clusters with the same ID on the same network interfere with each other, and it is a baffling failure to diagnose.

Solution 3

Does Tramontana Bookings need high availability? To: Marta Vidal · From: Systems Operations · 18 August 2026

Short answer: full high availability, no. A first step, yes, and with a very favourable cost-benefit ratio. I recommend two application servers with a load balancer, which captures most of the benefit for a fraction of the cost.


First, a figure we are missing. We have no agreed availability target and we do not measure it. My estimate is that we are around 99.5%, some 44 hours of downtime a year, counting reboots for updates, deployments and the occasional incident. It is an estimate, not a figure, and the first thing I propose is that we start measuring it — it costs very little and without that number any investment is a hunch.

What we have already achieved without spending anything. Availability depends on two things: how often the system fails and how long it takes to recover. In recent weeks we have drastically reduced the second: rebuilding the server completely has gone from eight hours to fifty minutes, by automating its configuration. That has already improved availability without buying a single machine, and it is the kind of improvement to exhaust before duplicating infrastructure.

What being down would cost. With around 500 bookings a month at an average of €313.70, average revenue is about €218 an hour. That number has to be qualified in three ways:

  • Bookings cluster in the afternoon and at weekends. An outage in the small hours of a Tuesday may cost nothing; the same one on a Friday afternoon costs considerably more than the average.
  • Many customers who find the site down try again. The real cost is probably between 20% and 50% of the raw figure.
  • There is a cost to our image, non-linear and hard to quantify, especially if it coincides with a campaign.

With a prudent estimate (35%):

Availability Downtime per year Estimated cost
99.5% (current situation) 44 h ~€3,340
99.8% 17.5 h ~€1,330
99.9% 8.8 h ~€670

The three options, with their real cost.

What it includes Investment Availability Annual saving
A. Do nothing The current setup 0 ~99.5% —
B. Two servers + load balancer 2 machines, 1 week of work Low ~99.8% ~€2,000
C. The complete architecture 4 machines, 6 weeks, +4 h/month High ~99.9% ~€2,670

The important part is the comparison between B and C: option C costs several times more than B and adds barely €670 of additional annual saving. The reason is that the first nines are cheap and the following ones get much more expensive — and, above all, that most of our interruptions do not come from hardware failures but from deployments, reboots and maintenance, and that is resolved entirely by option B.

What option B would give us exactly:

  • Zero-downtime deployments. Today every deployment means a few seconds of service cut off. With two servers one is updated while the other serves, and the customer notices nothing.
  • Maintenance with no window. Rebooting for a new kernel stops being a scheduled interruption.
  • Tolerance of one application server failing. The system detects it in under ten seconds and stops sending it traffic on its own.
  • Capacity for peaks, as a secondary benefit.

What it would NOT give us, and this has to be said clearly:

  • If the database goes down, the service goes down. Promoting the replica would be manual, 5 to 15 minutes.
  • If the load balancer goes down, the service goes down. That is option C.
  • If the provider's infrastructure goes down, everything goes down.
  • And none of this protects against an accidental deletion or data corruption: that would replicate instantly to every node. That is what the backups are for, and they would remain just as necessary.

There is a prerequisite that is not a systems matter. Tramontana Bookings currently keeps two things inside each server: the sessions of the people browsing, and the photos that get uploaded. With two servers, anybody who logged in on one would appear logged out when served by the other, and a photo uploaded to one would not be visible from the other. Before setting anything up, development work is needed — externalising the sessions and storing the files in a shared store — some three or four weeks. It is the part of the project to plan with Luis, and without it option B does not work.

And a professional warning. Redundancy adds pieces, and each piece can fail in new ways. A badly built system is less reliable than a simple one that is well maintained. That is why I do not recommend option C: the complexity it adds — replication, promotion, the risk of two servers writing incompatible data at the same time — demands a level of vigilance and testing we cannot sustain with the current headcount. A simple architecture we understand and test is worth more than a sophisticated one nobody has rehearsed.

Recommendation, in three steps.

  1. This month, at zero cost: set a formal availability target — I propose 99.8% — and start measuring it. With that we will be able to decide on data rather than on estimates.
  2. In two or three months: develop the two prerequisites, and deploy option B. I would start with a single load balancer, accepting that it is a single point of failure but one far less prone to going down than the complete application.
  3. Review in a year, or sooner if something changes: if the booking volume grows significantly, if an outage costs us an important customer, or if we measure availability worse than estimated.

And one last thing. Quite apart from all of the above, I want to schedule a six-monthly rehearsal: causing the failures deliberately, during working hours, and checking that the recovery works. A backup system that has never been tested is a system we know nothing about, and this applies as much to the future architecture as to the rebuild procedure we already have. The first one in September.

Conclusion

You know what people are talking about when they talk about high availability, and with the precision the subject demands. The nines have concrete numbers — 99.99% is less than a minute of downtime a week — availability breaks down into MTBF and MTTR, and reducing MTTR is usually far cheaper than increasing MTBF: which is exactly what you did in 07-06 by taking the rebuild from eight hours to fifty minutes, without buying anything. You know that vertical scaling adds no availability, that a passive node that is never used is a node you know nothing about, and above all that state is the hard problem: sessions, uploaded files and scheduled tasks are what stops you simply duplicating a server.

You have set up a load balancer with HAProxy that distributes by leastconn, probes health every three seconds with asymmetric fall 3 and rise 2, brings nodes back with slowstart so as not to knock them over again, and allows connections to be drained so you can deploy without cutting anybody off — resolving what was left outstanding since 04-07. You have given it redundancy with keepalived and a floating virtual IP, with the track_script that avoids the silliest failure in this architecture: a load balancer with keepalived alive and HAProxy dead holding on to the IP. And you know the two concepts without which all of this is theatre: split-brain, which with two nodes cannot be avoided because no quorum is possible, and fencing, without which a cluster is not reliable however well it starts.

And you have done the analysis Marta had been asking for over three lessons, with the conclusion that is not always welcome: the complete architecture is not worth it. Two application servers with a load balancer capture most of the benefit for a fraction of the cost, because most interruptions do not come from hardware failures but from deployments and maintenance. That the professional answer to "do we need high availability?" can be "part of it, and in this order" is as important as knowing how to build it. Along with the warning that closes the matter: redundancy done badly worsens reliability, and a simple system that is understood and tested is worth more than a sophisticated one nobody has rehearsed.


That closes Module 7, and with it the most advanced part of the course. You have learned to look inside the system: the complete boot chain and how to intervene when it breaks, diagnosis with strace, perf and eBPF when the counters are not enough, and kernel tuning with the rule of not touching what you have not measured. And you have learned to multiply it: virtualisation with KVM and the test environment the course had been missing since Module 4, containers understood as what they are — isolated processes, not small machines — automation with Ansible that turned the configuration into code and the RTO into fifty minutes, and finally the redundancy that takes away srv-tramontana's status as a single point of failure.

In Module 8: Practical Projects all of that is applied to complete builds, end to end. You will set up a web server with Nginx as a reverse proxy and TLS in front of the application — and there the encryption in transit that was prepared in 06-05 and has been waiting ever since is finally closed off. You will configure a proper PostgreSQL database server, with its tuning, its backups and its replication. You will build a home media server, which is the project where you can confirm that everything you have learned applies just as well outside work. You will bring up a VPN server with WireGuard to reach the internal network without exposing anything. You will deploy a Kubernetes cluster, which is where the containers from 07-05 and orchestration come together. And you will finish with going into production: the complete checklist, the monitoring with Prometheus and Grafana that was mentioned in 05-07, and the daily operating procedure for a system that is no longer a laboratory. Update your VM's snapshot and I will see you there.

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

© Copyright 2026. All rights reserved