The Project | About Us | Contribute | Donations | License

HOME

There is a debt in this course that has been open since Module 6. In 06-05 you issued a Let's Encrypt certificate for bookings.tramontana.example, you verified it, you scheduled its renewal with certbot.timer and you wrote check_certificate.sh to warn you twenty days before it expires. All correct. And yet, today Tramontana Bookings is still being served unencrypted, because the certificate is sitting on a disk and there is nothing using it.

The application listens on 127.0.0.1:8080, in plain text, and that is the point where the design was left half finished. This lesson closes that debt by putting Nginx in front of the application as a reverse proxy: it terminates TLS with the certificate you already have, adds the security headers, serves the static content, rate-limits requests and leaves the application exactly where it is, on the loopback, invisible from outside.

It is also the first project of the module, and therefore the first time the work is organised the way a real project is organised: objective and requirements, justified design decisions, build, verification, automation with Ansible and operation. That will be the pattern for all six lessons.

Objective, prerequisites and starting state
What a reverse proxy is and why it goes in front
Choosing the web server: Nginx, Apache or Caddy
Installation and the structure of /etc/nginx
Anatomy of the configuration: contexts, server and location
Building Tramontana's virtual server
Static content, compression and rate limiting
Logging and rotation
Integrating the already-issued certificate with certbot
Verification and measurement, before and after
Automation with Ansible: the web role
Operation and maintenance

Objective, prerequisites and starting state

Objective. That https://bookings.tramontana.example responds encrypted, with an A grade on the usual TLS checkers, serving the application that runs on 127.0.0.1:8080 without exposing that port, with static content served directly and with basic protection against abuse.

Prerequisites, all of them already met in earlier modules:

Requirement	Where it comes from	Check
Certificate issued and renewing	06-05	`sudo certbot certificates`
DNS pointing at the server	06-01	`dig +short bookings.tramontana.example`
Ports 80 and 443 open, 8080 closed	06-03	`sudo ufw status numbered`
Application active on 127.0.0.1:8080	05-05	`curl -I http://127.0.0.1:8080/`
Test environment available	07-04	`srv-tramontana-test`
Ansible with the production inventory	07-06	`~/tramontana-infra`

The starting state, measured before touching anything — because you measure before and after:

$ curl -I http://127.0.0.1:8080/
HTTP/1.1 200 OK
Server: tramontana/3.2.1
Content-Type: text/html; charset=utf-8
Content-Length: 4812

$ curl -sI https://bookings.tramontana.example/ ; echo "code: $?"
curl: (7) Failed to connect to bookings.tramontana.example port 443: Connection refused
code: 7

There is the debt, in two commands: the application works and port 443 does not exist.

What a reverse proxy is and why it goes in front

A reverse proxy is a server that receives requests from clients and forwards them to one or more internal servers, returning their response as if it were its own. The client never talks to the application: it talks to the proxy.

The word "reverse" distinguishes this case from the classic proxy. A forward proxy works for the client (a company filtering its employees' browsing); a reverse proxy works for the server.

	Forward proxy	Reverse proxy
Who it serves	The client	The server
Who configures it	The client or their network	The owner of the service
What it hides	The client's identity	The internal topology of the service
Typical example	Corporate filtering, Squid	Nginx in front of an application
The end user sees it	Yes, they configure it	No, it is transparent

And the question that matters: if the application already speaks HTTP, why not let it listen on 443 directly? Six reasons, and none of them is optional in production:

TLS termination. The proxy manages the certificates, the protocols and the cipher suites. The application does not need to know anything about TLS, it does not need to read the private key and it does not need to restart when the certificate is renewed. A single point where the encryption policy is applied.
Privileged ports without privileges. Listening on 443 requires CAP_NET_BIND_SERVICE. Nginx does it by starting as root and dropping privileges immediately; the application carries on as svc-tramontana on a high port, with no special capability at all. It is exactly the principle of least privilege from 05-02.
Static content. Serving a 40 KB CSS file should not consume an application thread. Nginx does it with sendfile(), without copying the data into user space, and at two orders of magnitude less cost.
Headers and policy. HSTS, CSP, X-Frame-Options, compression, redirects: all of that is configured once, in one place, and applies to everything that goes out — without depending on the developer remembering it in every response.
Rate limiting and protection. Abuse is cut off at the edge, before it reaches the business logic and before it consumes a PostgreSQL connection.
A distribution point. It is what allows a second application node to be added without changing anything visible from outside — option B recommended in 07-07.

And an additional reason you appreciate on deployment day: with the proxy in front, deploy.sh can restart the application while Nginx returns a decent error page instead of a refused connection.

Choosing the web server: Nginx, Apache or Caddy

	Nginx	Apache httpd	Caddy
Concurrency model	Asynchronous, few processes	Processes or threads per connection (MPM)	Asynchronous (Go)
Consumption with many connections	Very low	High with `prefork`	Low
Automatic TLS	With certbot	With certbot	Built in, no configuration
Configuration	Declarative, its own	Directives + `.htaccess`	JSON/Caddyfile, very short
Per-directory `.htaccess`	No (and that is an advantage)	Yes	No
Hot-loadable modules	No, you have to recompile	Yes, dynamic	Compiled plugins
As a reverse proxy	Excellent	Fine	Excellent
Market share and documentation	Enormous	Enormous	Growing
In the Ubuntu 24.04 repositories	Yes (1.24)	Yes (2.4)	No (its own repository)

The decision for Tramontana is Nginx, and for these specific reasons:

It is in the official Ubuntu 24.04 repositories, which fits the package and pinning policy from 05-03: no third-party repositories to maintain.
Its asynchronous model is the right one for a reverse proxy: thousands of slow connections do not cost thousands of processes, which matters with 3.8 GB of RAM.
It is what you already partly know from 07-07, where it appeared as an alternative to HAProxy with upstream.
The absence of .htaccess is deliberately a good thing: all the configuration lives in /etc/nginx, versioned in Ansible, with no stray files changing behaviour without leaving a trace.

Caddy would be a very reasonable choice on a new project — its automatic certificate management eliminates a whole class of incidents — but here the certificate already exists and the renewal flow is set up and tested. Apache is still excellent, especially if you need mod_php or per-directory configuration; for a pure reverse proxy, Nginx is lighter.

Installation and the structure of /etc/nginx

$ sudo apt update && sudo apt install nginx
$ nginx -v
nginx version: nginx/1.24.0 (Ubuntu)

$ systemctl is-active nginx
active

Ubuntu starts Nginx with a welcome site. Before anything else, check that the installation has not opened a door you did not want:

$ sudo ss -tlnp | grep nginx
LISTEN 0 511 0.0.0.0:80  0.0.0.0:*  users:(("nginx",pid=2841,fd=6))
LISTEN 0 511    [::]:80     [::]:*  users:(("nginx",pid=2841,fd=7))

It listens on 80 on every interface, which is what is expected and what ufw has been allowing since 06-03.

The structure of /etc/nginx on Ubuntu:

Path	What it contains
`nginx.conf`	Global configuration; it includes the other files
`conf.d/*.conf`	Global fragments of the `http` context (Debian/Ubuntu)
`sites-available/`	Virtual servers defined, active or not
`sites-enabled/`	Symbolic links to the active ones
`snippets/`	Reusable fragments (`ssl-params.conf`, etc.)
`mime.types`	Extension → MIME type mapping
`modules-enabled/`	Enabled dynamic modules

The distinction between conf.d and sites-available is a Debian and Ubuntu convention worth understanding so as not to mix them up:

conf.d/ is included inside the http context and serves for cross-cutting configuration: log formats, rate-limit zones, shared proxy settings. It is what Red Hat-based distributions use for everything.
sites-available/ + sites-enabled/ is the Debian pattern: you define each virtual server in a file and enable it by creating a symbolic link. Disabling a site means deleting the link, not the file — reversible, and with the history intact.

$ grep -n 'include' /etc/nginx/nginx.conf | tail -3
 62:	include /etc/nginx/conf.d/*.conf;
 63:	include /etc/nginx/sites-enabled/*;

$ ls -l /etc/nginx/sites-enabled/
lrwxrwxrwx 1 root root 34 Aug 18 09:12 default -> /etc/nginx/sites-available/default

The first thing to do is remove the default site: it serves a welcome page that reveals the Nginx version and answers to any domain name.

$ sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak-$(date +%F)
$ sudo rm /etc/nginx/sites-enabled/default

And two global settings in nginx.conf, inside the http context:

    # Do not reveal the version in the Server header or on error pages
    server_tokens off;

    # Maximum request size: the photographs of the cottages
    client_max_body_size 20m;

server_tokens off changes Server: nginx/1.24.0 (Ubuntu) to Server: nginx. It is not a serious defence — the version can be inferred in other ways — but it removes the easy result for automated scanners, in line with the hardening from 06-06.

Anatomy of the configuration: contexts, server and location

Nginx configuration is a tree of nested contexts. Directives are only valid in certain contexts, and they inherit downwards: what you put in http applies to every server, unless one of them redefines it.

Context	What it configures	Typical directives
main (root)	The process	`user`, `worker_processes`, `pid`
`events`	The connection model	`worker_connections`, `multi_accept`
`http`	All HTTP traffic	`log_format`, `gzip`, `include`, `upstream`
`server`	One virtual server	`listen`, `server_name`, `ssl_certificate`
`location`	A set of paths	`proxy_pass`, `root`, `expires`

How Nginx picks the `server` block

This is the mechanism that causes the most confusion, and it is resolved in two steps:

By listen: the server blocks that are not listening on the request's address and port are discarded. The most specific match wins (listen 10.0.2.15:443 beats listen 443).
By server_name, compared against the request's Host header, in this order: exact name → wildcard at the start (*.tramontana.example) → wildcard at the end (www.*) → regular expression, in order of appearance.

If nothing matches, the default server wins: the one marked with default_server in its listen, or the first one defined for that port. Hence why leaving the default site enabled is a problem: any request with an unknown Host — including a scanner arriving by IP — ends up in it.

How Nginx picks the `location` block

Here the order of precedence is not the order in the file, and getting it wrong produces baffling behaviour:

Priority	Modifier	Example	Meaning
1	`=`	`location = /health`	Exact match. Always wins and stops the search
2	`^~`	`location ^~ /static/`	A prefix that, if it is the longest, prevents the regexes being evaluated
3	`~` / `~*`	`location ~* \.(jpg\|css)$`	Case-sensitive / insensitive regex, in file order
4	(prefix)	`location /`	The longest prefix, if no regex matched

The complete algorithm: Nginx looks for the longest prefix that matches; if it carries =, it finishes; if it carries ^~, it finishes too; otherwise it evaluates the regular expressions in order and the first one that matches wins; if none matches, the saved longest prefix is used.

The practical consequence to memorise: a regex beats a prefix even when the prefix is more specific. A location ~ \.php$ will capture /static/foo.php even though location /static/ exists. That is why static directories are declared with ^~.

Building Tramontana's virtual server

Here is the complete file, commented directive by directive. It is the core of the lesson.

# /etc/nginx/sites-available/bookings.tramontana.example
# Reverse proxy with TLS termination for Tramontana Bookings 3.2.1

# ---------------------------------------------------------------
# Rate-limit zones. They are declared in the http context (here via
# sites-enabled, which is included inside http). The memory is
# shared between the worker processes.
# 10m holds around 160,000 IP addresses.
# ---------------------------------------------------------------
limit_req_zone  $binary_remote_addr zone=general:10m rate=30r/s;
limit_req_zone  $binary_remote_addr zone=login:10m   rate=5r/m;
limit_conn_zone $binary_remote_addr zone=connections:10m;

# Group of application servers. One today; tomorrow the two from 07-07
# without touching anything other than these lines.
upstream tramontana_app {
    server 127.0.0.1:8080 max_fails=3 fail_timeout=30s;
    # Reuses TCP connections to the application instead of opening one
    # per request. Requires HTTP/1.1 and an empty Connection (below).
    keepalive 16;
}

# ---------------------------------------------------------------
# Server on 80: it only redirects, with the ACME challenge as the exception
# ---------------------------------------------------------------
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name bookings.tramontana.example;

    # Certbot in webroot mode needs to serve this directory WITHOUT TLS.
    location ^~ /.well-known/acme-challenge/ {
        root /var/www/html;
        allow all;
    }

    # Everything else, to HTTPS. Permanent 301: browsers cache it.
    location / {
        return 301 https://$host$request_uri;
    }
}

# ---------------------------------------------------------------
# Server on 443: the real one
# ---------------------------------------------------------------
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    http2 on;                      # syntax for Nginx >= 1.25.1
    server_name bookings.tramontana.example;

    # ---------- TLS (certificate issued in 06-05) ----------
    # fullchain.pem, NOT cert.pem: it includes the intermediates that many
    # clients (old Android, curl without a complete store) need in order
    # to build the chain. Serving cert.pem produces failures that only
    # show up on some clients, and they are a classic.
    ssl_certificate     /etc/letsencrypt/live/bookings.tramontana.example/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/bookings.tramontana.example/privkey.pem;

    # The "intermediate" profile from ssl-config.mozilla.org. Never invent
    # the list of suites: it is copied from a maintained source (06-05).
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;   # in TLS 1.3 the client decides

    # Session cache: avoids the full handshake on reconnections.
    # 10m of cache holds around 40,000 sessions. 'shared' = shared
    # between workers; 'off' for tickets because they break forward
    # secrecy if the keys are not rotated.
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;

    # OCSP stapling: Nginx asks the CA for the revocation status and
    # attaches it to the handshake. The client does not have to contact
    # the CA, which improves latency and privacy. chain.pem contains the
    # intermediate used to verify the response.
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/bookings.tramontana.example/chain.pem;
    resolver 10.0.2.2 valid=300s;
    resolver_timeout 5s;

    # ---------- Security headers ----------
    # HSTS: the browser refuses to use HTTP for this domain for max-age.
    # Start with 300 and raise it to 2 years once it is verified:
    # once sent, it CANNOT be revoked from the client's browser.
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
    # Stops the browser guessing the MIME type (upload attacks)
    add_header X-Content-Type-Options "nosniff" always;
    # Stops the site being loaded inside somebody else's iframe (clickjacking)
    add_header X-Frame-Options "SAMEORIGIN" always;
    # Do not leak the full URL when navigating to another domain
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    # Basic CSP: own resources only. Adjust with Luis if the
    # application loads external fonts or scripts.
    add_header Content-Security-Policy "default-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; frame-ancestors 'self'" always;

    # ---------- Logging ----------
    access_log /var/log/nginx/tramontana_access.log tramontana;
    error_log  /var/log/nginx/tramontana_error.log warn;

    # ---------- Site-wide limits ----------
    limit_conn connections 20;        # 20 simultaneous connections per IP
    limit_req  zone=general burst=60 nodelay;
    limit_req_status 429;
    limit_conn_status 429;

    # ---------- Static content served by Nginx ----------
    # ^~ so that no later regex captures it.
    location ^~ /static/ {
        alias /opt/tramontana/app/static/;
        expires 30d;
        add_header Cache-Control "public, immutable";
        access_log off;               # unnecessary noise
        try_files $uri =404;
    }

    location ^~ /uploads/ {
        alias /opt/tramontana/shared/uploads/;
        expires 7d;
        add_header Cache-Control "public";
        # Never execute anything from here: these are user-uploaded files
        add_header X-Content-Type-Options "nosniff" always;
        try_files $uri =404;
    }

    # ---------- Health endpoint (07-07) ----------
    # Exact match: it always wins and it is not logged.
    location = /health {
        access_log off;
        allow 127.0.0.1;
        allow 10.0.2.0/24;
        deny all;
        proxy_pass http://tramontana_app;
    }

    # ---------- Login form: a stricter limit ----------
    location = /login {
        limit_req zone=login burst=3 nodelay;
        proxy_pass http://tramontana_app;
        include /etc/nginx/snippets/proxy-tramontana.conf;
    }

    # ---------- Everything else: to the application ----------
    location / {
        proxy_pass http://tramontana_app;
        include /etc/nginx/snippets/proxy-tramontana.conf;
    }
}

And the reusable fragment with the proxy parameters, which is where the most important part lives:

# /etc/nginx/snippets/proxy-tramontana.conf

# HTTP/1.1 and an empty Connection: needed for the upstream's
# 'keepalive' to work. Without this Nginx opens a TCP connection per request.
proxy_http_version 1.1;
proxy_set_header Connection "";

# --- The four indispensable headers and why ---

# Host: without it, the application would see "tramontana_app" as the host
# and would generate broken absolute links and cookies with the wrong domain.
proxy_set_header Host $host;

# X-Real-IP: the client's IP. Without it, the application logs
# 127.0.0.1 on EVERY request and access.log stops being good for
# anything: no analytics, no fail2ban, no diagnosis.
proxy_set_header X-Real-IP $remote_addr;

# X-Forwarded-For: the chain of proxies. Nginx appends the client's IP
# to the existing list. It is the de facto standard header.
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# X-Forwarded-Proto: tells the application that the client used HTTPS
# even though it receives HTTP. Without it, an application that forces
# HTTPS goes into an infinite redirect loop, and "Secure" cookies are
# not issued.
proxy_set_header X-Forwarded-Proto $scheme;

# --- Timeouts ---
proxy_connect_timeout 5s;    # opening the connection to the app
proxy_send_timeout   30s;    # sending it the request
proxy_read_timeout   60s;    # waiting for its response (slow reports)

# --- Buffering ---
# With buffering enabled, Nginx reads the complete response from the
# application and then sends it to the client at the client's pace. That
# way a slow client (a phone with poor coverage) does NOT keep an
# application thread busy. It is one of the main reasons for the proxy.
proxy_buffering on;
proxy_buffers 8 16k;
proxy_buffer_size 16k;

# Exception: for streaming responses (events, long downloads) it has to
# be disabled per location with 'proxy_buffering off'.

# Do not pass the application's 5xx errors through as-is if there is an own page
proxy_intercept_errors off;

Four decisions that deserve an explicit justification:

X-Forwarded-Proto is not optional. It is the cause of the most frequent incident when setting up a reverse proxy for the first time: the application, which knows it is behind HTTPS, receives an HTTP request, redirects to HTTPS, the proxy forwards it again as HTTP, and the browser shows "too many redirects". Luis will have to make sure the application trusts that header only when the request comes from 127.0.0.1; trusting it from any source would let an attacker forge their own IP in the logs.

keepalive 16 on the upstream. Without this directive, every client request opens and closes a fresh TCP connection to 127.0.0.1:8080. With 500 bookings a month you do not notice it, but with real traffic sockets pile up in TIME_WAIT and you waste a handshake per request.

default_server on both blocks. Having removed the default site, somebody has to handle requests with an unknown Host. A stricter alternative — and a recommended one if there are ever several domains — is a default server that returns 444 (close with no response) and leaving Tramontana's without default_server.

HSTS with a long max-age, but not on day one. The header is irrevocable from the server side: if the certificate fails in two months' time, the browsers that received it will refuse to connect over HTTP. The correct practice is to deploy with max-age=300, verify over a few days that the renewal works, and only then raise it to two years.

Static content, compression and rate limiting

Compression

In the http context of nginx.conf:

    gzip on;
    gzip_vary on;              # adds "Vary: Accept-Encoding" (caches)
    gzip_proxied any;
    gzip_comp_level 5;         # 5 is the balance point; 9 burns CPU
    gzip_min_length 1024;      # compressing 200 bytes costs more than it saves
    gzip_types
        text/plain text/css text/xml application/json
        application/javascript application/xml+rss
        image/svg+xml application/atom+xml;

Three warnings:

text/html is always compressed and must not appear in gzip_types; putting it there produces a warning.
Do not compress what is already compressed. JPEG, PNG, WebP, MP4, ZIP: you burn CPU and the file grows by a few bytes.
Brotli compresses between 15% and 20% better than gzip for text, and every current browser supports it. It does not come in the Ubuntu package: it needs libnginx-mod-http-brotli from a PPA, or compiling the module. For Tramontana's volume it does not justify the maintenance debt it introduces into the package policy from 05-03; gzip is enough. The elegant alternative is to pre-compress the static files at deployment time and serve them with gzip_static on.

`sendfile` and company

    sendfile on;         # copies file -> socket inside the kernel
    tcp_nopush on;       # groups headers and data into fewer packets
    tcp_nodelay on;      # disables Nagle on keepalive connections

sendfile() is a system call that copies data from a file descriptor to a socket without going through user space. It is the reason Nginx serves static files at almost no CPU cost. You can see it with the tools from 07-02:

$ sudo strace -c -p $(pgrep -f 'nginx: worker' | head -1) -e trace=sendfile,read,write &
$ ab -n 200 -c 10 https://bookings.tramontana.example/static/style.css >/dev/null 2>&1
$ kill %1
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 61.20    0.004112          20       200           sendfile
 22.10    0.001485           7       200           read
 16.70    0.001122           5       201           write

Rate limiting: how it really works

limit_req_zone implements a leaky bucket. rate=30r/s does not mean "30 requests and then I block": it means one request is processed every 33 ms. Without burst, request number 2 arriving in the same millisecond is rejected with 429, and that breaks any real page, which loads twenty resources at once.

Configuration	Behaviour
`limit_req zone=general;`	Strictly one every 33 ms. Rejects legitimate bursts
`limit_req zone=general burst=60;`	Queues up to 60 and serves them at the rate. Adds latency
`limit_req zone=general burst=60 nodelay;`	Serves the burst instantly and rejects beyond it. The usual choice

The two separate zones answer different threats: general (30 r/s) protects against aggressive crawling and limits the damage from a runaway client; login (5 r/m with burst=3) protects against brute force on credentials. This second one complements fail2ban from 06-03, it does not replace it: fail2ban bans at the network level after several failures, whereas limit_req slows the pace down from the first second, before the application queries the database.

limit_conn connections 20 attacks something else: simultaneous connections per IP, which is the defence against slowloris-style attacks, where the attacker opens hundreds of connections and keeps them open by sending one byte every few seconds.

Logging and rotation

A log format of your own in the http context:

    log_format tramontana '$remote_addr - $remote_user [$time_local] '
                          '"$request" $status $body_bytes_sent '
                          'rt=$request_time urt=$upstream_response_time '
                          'ua="$http_user_agent" ref="$http_referer"';

The two variables that justify a custom format:

$request_time: the total time from the first line of the request to the last byte sent to the client. It includes the client's network.
$upstream_response_time: how long the application took.

The difference between the two separates two culprits that are constantly confused. If rt=3.500 and urt=0.045, the application was fast and the client has a slow connection: there is nothing to optimise on the server. If rt=3.500 and urt=3.480, the problem is in the application or in PostgreSQL, and that is where you apply latency_diagnostics.sh from 07-02.

$ sudo awk '{print $NF}' /dev/null; \
  sudo grep -oP 'urt=\K[0-9.]+' /var/log/nginx/tramontana_access.log | \
  sort -n | awk '{a[NR]=$1} END {print "p50:",a[int(NR*0.50)]," p95:",a[int(NR*0.95)]," p99:",a[int(NR*0.99)]}'
p50: 0.041  p95: 0.220  p99: 1.180

That p99 of 1.18 s is the kind of figure that goes into the monitoring in 08-06.

Rotation. The Ubuntu package ships /etc/logrotate.d/nginx, which rotates daily keeping 14 and sends USR1 to Nginx so it reopens the files. New logs are covered automatically by the /var/log/nginx/*.log pattern. All that is worth checking is that the retention matches the policy from 05-06.

The alternative is to send everything to the journal, which unifies querying with journalctl and takes advantage of the persistent journal you already configured:

    access_log syslog:server=unix:/dev/log,tag=nginx_access,severity=info tramontana;
    error_log  syslog:server=unix:/dev/log,tag=nginx_error warn;

$ journalctl -t nginx_access -f --since "10 min ago"

	Files + logrotate	To the journal
Unified querying with the rest of the system	No	Yes
Performance under heavy traffic	Better	Worse: every line goes through journald
Analysis tools (GoAccess)	Direct	Require exporting
Retention	`logrotate`	`SystemMaxUse`

For Tramontana, at its volume, the journal is the better option: a single query tool and automatic time correlation with the application's errors. On a high-traffic site, files.

Integrating the already-issued certificate with certbot

The certificate has existed since 06-05, issued in standalone or webroot mode. What is missing is for certbot to know that there is now an Nginx that must be reloaded after every renewal.

$ sudo certbot certificates
Found the following certs:
  Certificate Name: bookings.tramontana.example
    Domains: bookings.tramontana.example
    Expiry Date: 2026-10-29 08:14:00+00:00 (VALID: 71 days)
    Certificate Path: /etc/letsencrypt/live/bookings.tramontana.example/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/bookings.tramontana.example/privkey.pem

Seventy-one days: there is margin, and therefore this is done without rushing and tested first.

There are two routes:

# Option A: let certbot write the Nginx configuration
$ sudo certbot --nginx -d bookings.tramontana.example --dry-run

certbot --nginx modifies the virtual server file by inserting the TLS directives. It is convenient to start with and counterproductive here: the configuration is written by hand, commented and — in the next section — managed by Ansible. Letting certbot rewrite it would cause a divergence between the real file and the template, which is exactly the problem 07-06 came to solve.

# Option B (the chosen one): certbot only renews; Nginx serves the challenge
$ sudo cat /etc/letsencrypt/renewal/bookings.tramontana.example.conf
[renewalparams]
authenticator = webroot
webroot_path = /var/www/html,

With the location ^~ /.well-known/acme-challenge/ in the port 80 block pointing at /var/www/html, renewal works without stopping Nginx — which was a limitation of standalone mode. All that is missing is the reload hook:

$ printf '#!/bin/sh\nnginx -t && systemctl reload nginx\n' | \
      sudo tee /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh
$ sudo chmod 0755 /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh

$ sudo certbot renew --dry-run
Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/bookings.tramontana.example/fullchain.pem (success)
Running deploy-hook command: /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh

The hook lives in deploy/, not in post/: the ones in deploy run only if the certificate was actually renewed, whereas the ones in post run on every attempt. And the nginx -t && before the reload is an application of the rule that governs this whole lesson.

Verification and measurement, before and after

Rule number one, no exceptions: nginx -t before any reload.

$ sudo ln -s /etc/nginx/sites-available/bookings.tramontana.example \
             /etc/nginx/sites-enabled/
$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ sudo systemctl reload nginx

reload and not restart. The difference matters: reload sends SIGHUP, the master process starts new workers with the new configuration and lets the old ones finish the requests in flight before dying. Zero connections cut. restart kills everything and starts again: a few hundred milliseconds of refused connections, and if the configuration is invalid, the service does not come back. With reload, an invalid configuration simply is not applied and the service carries on with the previous one.

The battery of checks

# 1. Redirect from HTTP
$ curl -sI http://bookings.tramontana.example/houses | head -3
HTTP/1.1 301 Moved Permanently
Server: nginx
Location: https://bookings.tramontana.example/houses

# 2. HTTPS and security headers
$ curl -sI https://bookings.tramontana.example/
HTTP/2 200
server: nginx
strict-transport-security: max-age=63072000; includeSubDomains
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin-when-cross-origin
content-security-policy: default-src 'self'; img-src 'self' data: ...

# 3. Chain, protocol and OCSP stapling
$ echo | openssl s_client -connect bookings.tramontana.example:443 \
      -servername bookings.tramontana.example -status 2>/dev/null | \
      grep -E 'Protocol|Cipher|Verify return code|OCSP Response Status'
OCSP Response Status: successful (0x0)
Protocol  : TLSv1.3
Cipher    : TLS_AES_256_GCM_SHA384
Verify return code: 0 (ok)

# 4. TLS 1.0 and 1.1 rejected
$ openssl s_client -connect bookings.tramontana.example:443 -tls1_1 2>&1 | \
      grep -c 'no protocols available\|alert protocol version'
1

# 5. 8080 IS STILL CLOSED FROM OUTSIDE (checked from another machine)
$ ssh [email protected] 'nc -z -w3 10.0.2.15 8080; echo "rc=$?"'
rc=1

# 6. Compression active
$ curl -sI -H 'Accept-Encoding: gzip' \
      https://bookings.tramontana.example/static/style.css | grep -i encoding
content-encoding: gzip

# 7. Rate limiting working
$ for i in $(seq 1 12); do
      curl -s -o /dev/null -w '%{http_code} ' https://bookings.tramontana.example/login
  done; echo
200 200 200 200 429 429 429 429 429 429 429 429

Check 5 is the one that avoids the most dangerous mistake in this lesson. A reverse proxy in front of an application that is also still directly reachable provides no security at all: the attacker bypasses the TLS, the headers and the rate limiting by going to 8080. The application listens on listen=127.0.0.1 according to /etc/tramontana/app.conf and ufw blocks the port: two layers, and both verified.

Measuring before and after

# Reusable format file
$ cat > /tmp/curl-format.txt <<'EOF'
     dns:  %{time_namelookup}s
     tcp:  %{time_connect}s
     tls:  %{time_appconnect}s
   first:  %{time_starttransfer}s
   total:  %{time_total}s
    size:  %{size_download} bytes
EOF

# BEFORE (straight to the application, unencrypted)
$ curl -w "@/tmp/curl-format.txt" -o /dev/null -s http://127.0.0.1:8080/static/style.css
     dns:  0.000012s
     tcp:  0.000198s
     tls:  0.000000s
   first:  0.004120s
   total:  0.004230s
    size:  41208 bytes

# AFTER (through Nginx, with TLS and gzip)
$ curl -w "@/tmp/curl-format.txt" -o /dev/null -s --compressed \
      https://bookings.tramontana.example/static/style.css
     dns:  0.001840s
     tcp:  0.002210s
     tls:  0.021500s
   first:  0.023900s
   total:  0.024010s
    size:  8940 bytes

The honest reading of those numbers, which is what has to go into a report:

Metric	Before	After	Comment
Latency of the first request	4.2 ms	24.0 ms	The TLS handshake costs ~19 ms
Latency with a reused connection	4.2 ms	~5.1 ms	The real steady-state cost
Bytes transferred	41,208	8,940	78% less thanks to gzip
Load on the application	1 request	0 requests	Nginx serves it on its own
Encryption in transit	No	Yes	The 06-05 debt, closed

TLS costs around 19 ms on first contact and practically nothing afterwards, thanks to ssl_session_cache and to TLS 1.3 reducing the handshake to a single round trip. In exchange, 78% less data is transferred and the application stops serving static files entirely. The trade is clearly favourable, and it was not negotiable anyway: serving personal booking data unencrypted is not a defensible option under the GDPR.

Automation with Ansible: the web role

All of the above was done by hand once, in order to understand it. Now it becomes code, following the role structure from 07-06.

# ~/tramontana-infra/roles/web/defaults/main.yml
---
web_domain: bookings.tramontana.example
web_upstream_host: 127.0.0.1
web_upstream_port: 8080
web_upstream_keepalive: 16
web_hsts_max_age: 63072000          # lower to 300 on the first deployment
web_rate_general: 30r/s
web_rate_login: 5r/m
web_burst_general: 60
web_connections_per_ip: 20
web_client_max_body: 20m
web_static: /opt/tramontana/app/static/
web_uploads: /opt/tramontana/shared/uploads/
web_health_networks:
  - 127.0.0.1
  - 10.0.2.0/24
web_log_to_journal: true

# ~/tramontana-infra/roles/web/tasks/main.yml
---
- name: Install Nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true
    cache_valid_time: 3600
  tags: [packages]

- name: Check that the certificate exists before configuring TLS
  ansible.builtin.stat:
    path: "/etc/letsencrypt/live/{{ web_domain }}/fullchain.pem"
  register: tls_cert

- name: Abort if there is no certificate
  ansible.builtin.assert:
    that: tls_cert.stat.exists
    fail_msg: >-
      /etc/letsencrypt/live/{{ web_domain }}/fullchain.pem does not exist.
      Issue it with certbot before applying this role: a configuration
      with ssl_certificate pointing at a nonexistent file stops Nginx
      from starting.

- name: Remove Ubuntu's default site
  ansible.builtin.file:
    path: /etc/nginx/sites-enabled/default
    state: absent
  notify: Reload nginx

- name: Global settings in nginx.conf
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
    backup: true
    # Validates the COMPLETE configuration with the temporary file as root.
    # If it fails, Ansible does not install it and the task reports an
    # error: it is impossible to leave the server with a configuration
    # that will not start.
    validate: 'nginx -t -c %s'
  notify: Reload nginx

- name: Install the proxy parameters snippet
  ansible.builtin.copy:
    src: proxy-tramontana.conf
    dest: /etc/nginx/snippets/proxy-tramontana.conf
    owner: root
    group: root
    mode: '0644'
  notify: Reload nginx

- name: Deploy the virtual server
  ansible.builtin.template:
    src: site.conf.j2
    dest: "/etc/nginx/sites-available/{{ web_domain }}"
    owner: root
    group: root
    mode: '0644'
    backup: true
  notify: Reload nginx
  # Note: 'validate' is NOT used here because a sites-available file on
  # its own is not a valid configuration by itself (it lacks the http
  # context). The real validation is done by the handler.

- name: Enable the virtual server
  ansible.builtin.file:
    src: "/etc/nginx/sites-available/{{ web_domain }}"
    dest: "/etc/nginx/sites-enabled/{{ web_domain }}"
    state: link
  notify: Reload nginx

- name: Install the reload hook for after the certificate renews
  ansible.builtin.copy:
    content: |
      #!/bin/sh
      nginx -t && systemctl reload nginx
    dest: /etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh
    owner: root
    group: root
    mode: '0755'
  tags: [tls]

- name: Allow HTTP and HTTPS through the firewall
  community.general.ufw:
    rule: allow
    port: "{{ item }}"
    proto: tcp
  loop: ['80', '443']
  tags: [firewall]

- name: Make sure Nginx is active and enabled
  ansible.builtin.systemd:
    name: nginx
    state: started
    enabled: true

# --- End-to-end verification, inside the role itself ---
- name: Force the pending reload before verifying
  ansible.builtin.meta: flush_handlers

- name: Verify that HTTPS responds correctly
  ansible.builtin.uri:
    url: "https://{{ web_domain }}/health"
    return_content: false
    status_code: 200
  register: verify
  retries: 3
  delay: 2
  until: verify is succeeded
  tags: [verify]

- name: Verify that the HSTS header is present
  ansible.builtin.assert:
    that: "'strict-transport-security' in verify.msg | lower or true"
    success_msg: "HTTPS operational on {{ web_domain }}"

# ~/tramontana-infra/roles/web/handlers/main.yml
---
- name: Reload nginx
  # ALWAYS validate before reloading. If 'nginx -t' fails, the task
  # fails and the service carries on with the previous configuration,
  # which works. It is the Ansible version of the golden rule.
  ansible.builtin.shell:
    cmd: nginx -t && systemctl reload nginx
  changed_when: true

A fragment of the template, to see how it is parameterised:

{# roles/web/templates/site.conf.j2 (fragment) #}
{{ ansible_managed | comment }}
upstream tramontana_app {
{% for node in web_backends | default([{'host': web_upstream_host, 'port': web_upstream_port}]) %}
    server {{ node.host }}:{{ node.port }} max_fails=3 fail_timeout=30s;
{% endfor %}
    keepalive {{ web_upstream_keepalive }};
}

server {
    listen 443 ssl default_server;
    http2 on;
    server_name {{ web_domain }};

    ssl_certificate     /etc/letsencrypt/live/{{ web_domain }}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/{{ web_domain }}/privkey.pem;

    add_header Strict-Transport-Security "max-age={{ web_hsts_max_age }}; includeSubDomains" always;

    location = /health {
        access_log off;
{% for network in web_health_networks %}
        allow {{ network }};
{% endfor %}
        deny all;
        proxy_pass http://tramontana_app;
    }
    ...
}

That loop over web_backends is the reason for having used an upstream from the start even though there is only one server today: adding the second node from 07-07 will mean changing one inventory variable.

And the application cycle, always the same:

$ cd ~/tramontana-infra
$ ansible-playbook site.yml --limit srv-tramontana-test --tags web
$ ansible-playbook site.yml --limit srv-tramontana --tags web --check --diff
$ ansible-playbook site.yml --limit srv-tramontana --tags web

Test environment first, --check --diff in production to see exactly what would change, and only then for real.

Operation and maintenance

What to look at in the logs every day. Three queries that fit into two minutes:

# 1. Status codes over the last 24 h
$ journalctl -t nginx_access --since "24 hours ago" -o cat | \
      awk '{print $9}' | sort | uniq -c | sort -rn
   3812 200
    241 304
     58 301
     19 404
      6 429
      2 502

# 2. The slowest paths (urt = the application's time)
$ journalctl -t nginx_access --since today -o cat | \
      grep -oP '"[A-Z]+ \K[^ ]+(?=.*urt=\K)?' >/dev/null; \
  journalctl -t nginx_access --since today -o cat | \
      awk '{for(i=1;i<=NF;i++) if($i ~ /^urt=/){split($i,a,"=");
           print a[2], $7}}' | sort -rn | head -5
3.812 /reports/billing
1.204 /houses/mas-figueres/availability
0.890 /reports/occupancy
0.412 /bookings/1023
0.388 /houses

# 3. Proxy errors
$ journalctl -t nginx_error --since "24 hours ago" -p warning

Signal	What it means	What to do
502 Bad Gateway	The application is not responding	`systemctl status tramontana`; look at `errors.log`
504 Gateway Timeout	The application takes longer than `proxy_read_timeout`	A slow query: 08-02, do not just raise the timeout
A spike of 429s	Rate limiting active	Real abuse, or a badly calibrated threshold?
A spike of 404s on odd paths	Automated scanning	Normal on the Internet; watch it if it escalates
`upstream prematurely closed`	The application closed the connection	Usually a restart or an application failure

Nginx during deployment. deploy.sh from 04-07 changes the /opt/tramontana/app link and restarts the application. Now a step has to be added: if the new version brings different static files, Nginx must be reloaded so that alias points at the newly resolved destination.

# Fragment to add to deploy.sh, after the atomic ln -sfn
reload_proxy() {
    if command -v nginx >/dev/null 2>&1; then
        if sudo nginx -t >/dev/null 2>&1; then
            sudo systemctl reload nginx
            log "proxy reloaded"
        else
            error "the nginx configuration is not valid; not reloading"
            return 1
        fi
    fi
}

The caching problem, which is the classic post-deployment incident. With expires 30d and Cache-Control: immutable, a browser that has already downloaded /static/style.css will not ask for it again for a month, even though the file has changed. The user sees the new version of the application with the previous version's styles, and the result is a broken site that fixes itself in a few days. Worse still: the users who do not suffer from it cannot reproduce the problem.

The correct solution is not to lower expires, which would waste the cache forever because of a one-day problem. It is to version the file name:

Approach	How	Assessment
`style.css?v=3.2.1`	Query parameter	It works, but some intermediate caches ignore it
`style.a3f19c.css`	The content hash in the name	The right one: a new URL = a new download
Lowering `expires` to 5 min	—	Loses the benefit of the cache
Purging the browser cache	—	Impossible: you do not control the client

With the hash in the name, immutable is literally true and there is no conflict: the URL never changes content. It is the job of the application's build, and therefore a request for Luis.

Certificate renewal. certbot.timer renews; the hook reloads. check_certificate.sh from 06-05 warns at 20 days. The only thing that changes is that now you have to verify that the hook still exists after every update of the certbot package:

$ ls -l /etc/letsencrypt/renewal-hooks/deploy/
-rwxr-xr-x 1 root root 46 Aug 18 11:02 10-reload-nginx.sh

Nginx updates. needrestart from 06-06 will detect that the binary changed and will ask to restart the service. Here restart genuinely is necessary — a reload does not change the running binary — and it is a few hundred milliseconds. With two nodes and draining, zero, which is another argument for option B from 07-07.

Common Mistakes and Tips

Using cert.pem instead of fullchain.pem. It works in your browser, which already has the intermediate cached, and it fails on old mobile clients, in curl on other systems and in integrations. It is an intermittent, baffling failure. Always fullchain.pem.
Forgetting X-Forwarded-Proto. An infinite redirect loop if the application forces HTTPS, and Secure cookies that are never issued.
Forgetting X-Real-IP. access.log records 127.0.0.1 on every request: goodbye to analytics, to diagnosis and to any IP-based blocking.
Leaving 8080 reachable from outside. The proxy contributes nothing if it can be bypassed. Verify it from another machine, not from the server itself.
restart instead of reload. It cuts the requests in flight and, with an invalid configuration, the service does not come back up.
Reloading without nginx -t. With reload you break nothing, but you lose the change without noticing and then you cannot understand why it is not being applied.
HSTS with a two-year max-age on day one. It is irrevocable from the server. Start with 300 seconds.
Confusing location precedence. A regex beats a more specific prefix. Static directories go with ^~.
Putting text/html in gzip_types or compressing JPEGs. The first gives a warning, the second burns CPU for nothing.
limit_req without burst. A normal page loads twenty resources at once and all but the first get a 429.
Leaving the default site enabled. Any request with an unknown Host ends up in it, revealing the Nginx version.
Letting certbot --nginx rewrite a configuration managed by Ansible. The real file and the template diverge, and the next ansible-playbook undoes the change at the worst possible moment.
Raising proxy_read_timeout to "fix" the 504s. You are hiding a slow query. Fix it in the database (08-02).
A tip on method. Save the output of curl -w before setting up the proxy. Without that starting point you cannot answer "has TLS slowed us down?" with a number.

Exercises

Exercise 1

A user reports that, after the deployment of version 3.2.1, the site "looks scrambled" on their laptop but not on Marta's. Diagnose the problem from the Nginx logs, explain the cause and propose the definitive solution with its technical justification.

Exercise 2

Write a script check_web.sh following the course conventions that verifies end to end that the web server is correctly configured, returning 0, 1 or 2 like health_check.sh. It must include a check that port 8080 is not reachable from outside.

Exercise 3

Marta asks why it has been necessary to set up "yet another server" if the application was already working, and whether this does not add a point of failure. Write the answer, saying what this measure protects against and what it does not.

Solutions

Solution 1

Diagnosis. The symptom — two users, different behaviour, the same server — points at something that depends on the state of the client, not the server. The candidates are the browser cache, an extension or an intermediate proxy. The logs confirm it:

# Static file requests over the last 2 h, with their code
$ journalctl -t nginx_access --since "2 hours ago" -o cat | \
      awk '$7 ~ /^\/static\// {print $1, $7, $9}' | sort | uniq -c
     14 198.51.100.23 /static/style.css 200
      1 203.0.113.77 /static/style.css 304

The reading is clear: Marta's laptop (198.51.100.23) received a 200 — it downloaded the new file — and the user's (203.0.113.77) received a 304 Not Modified, that is, it revalidated and kept its copy. And there is an even more revealing detail: if the user had not even requested the file, no line of theirs would appear at all, because expires 30d with immutable authorises the browser not to ask at all.

# Confirm the headers we are sending
$ curl -sI https://bookings.tramontana.example/static/style.css | \
      grep -iE 'cache-control|expires|etag|last-modified'
cache-control: public, immutable
expires: Thu, 17 Sep 2026 09:41:22 GMT
etag: "66c2a1b3-2118"
last-modified: Mon, 18 Aug 2026 09:12:35 GMT

The cause. The configuration declares /static/style.css immutable for 30 days. The deployment changed the content of the file while keeping the URL. Marta's browser had no previous copy (or had cleared it) and downloaded the new one; the user's had a copy from three days ago and, under the contract we gave it, has every right to use it until September. The server is doing exactly what we asked it to do; the mistake is in the design, not in the configuration.

Why the intuitive solutions are bad:

Tempting solution	Why not
Lower `expires` to 5 minutes	Gives up the cache forever because of a one-day problem. Every visit downloads 41 KB again
Remove `immutable`	Reduces the problem to a revalidation, but there is still a window and it adds one request per resource per visit
Ask the user to press Ctrl+F5	Does not scale: some users never get in touch and carry on seeing a broken site
Rename the file by hand at each deployment	It works and it is fragile: it will be forgotten

The definitive solution: a content fingerprint in the file name. The application's build generates style.a3f19c8d.css, where a3f19c8d are the first bytes of the SHA-256 hash of the content, and the HTML templates reference that name.

# The Nginx configuration does not change: it is still correct.
location ^~ /static/ {
    alias /opt/tramontana/app/static/;
    expires 1y;                 # now it CAN be a year
    add_header Cache-Control "public, immutable";
    access_log off;
    try_files $uri =404;
}

Why this resolves it completely, and it is a general principle of the web:

A URL never changes content. immutable stops being a risky promise and becomes literally true.
A content change produces a new URL, which no cache in the world has. The download is unavoidable and automatic.
The HTML must be ephemeral. It is what holds the references to the hashed names, so it is served with Cache-Control: no-cache (always revalidate). It is a small file and with ETag the revalidation costs a 200-byte 304.

# The HTML is not cached: it is the index that points at the versioned static files
location / {
    proxy_pass http://tramontana_app;
    include /etc/nginx/snippets/proxy-tramontana.conf;
    add_header Cache-Control "no-cache" always;
}

An immediate mitigation, while Luis implements the hash, applicable today:

# Interim solution: static paths carrying the release version.
# /static/3.2.1/style.css -> /opt/tramontana/releases/3.2.1/static/style.css
location ~ ^/static/(?<version>[0-9]+\.[0-9]+\.[0-9]+)/(?<resource>.*)$ {
    alias /opt/tramontana/releases/$version/static/$resource;
    expires 1y;
    add_header Cache-Control "public, immutable";
}

It fits the /opt/tramontana/releases/<version>/ structure that has existed since Module 5, and it gives the same result without touching the build: each version has its own URL space.

And the lesson on method: this incident is detected from the log by comparing 200 and 304 codes per IP. It deserves a place in the on-call notebook you will formalise in 08-06, because it will happen again with any new resource that is cached aggressively.

Solution 2

#!/usr/bin/env bash
#
# check_web.sh - End-to-end verification of the reverse proxy
#
# Exit codes (identical to health_check.sh):
#   0 = everything correct
#   1 = warning: something degraded but the service works
#   2 = critical: the service is not correct or is not secure
#
set -euo pipefail

readonly SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"

readonly TRAMONTANA_DOMAIN="${TRAMONTANA_DOMAIN:-bookings.tramontana.example}"
readonly TRAMONTANA_IP="${TRAMONTANA_IP:-10.0.2.15}"
readonly TRAMONTANA_APP_PORT="${TRAMONTANA_APP_PORT:-8080}"
readonly TRAMONTANA_EXTERNAL_HOST="${TRAMONTANA_EXTERNAL_HOST:-192.168.122.104}"
readonly TRAMONTANA_CERT_DAYS="${TRAMONTANA_CERT_DAYS:-20}"

umask 027

# Accumulated state: it always keeps the worst result seen
overall_status=0

report() {
    local level="$1" message="$2"
    case "$level" in
        ok)       log "OK       $message" ;;
        warn)     error "WARN     $message"; (( overall_status < 1 )) && overall_status=1 ;;
        critical) error "CRITICAL $message"; overall_status=2 ;;
    esac
    return 0
}

check_configuration() {
    if sudo nginx -t >/dev/null 2>&1; then
        report ok "the nginx configuration is valid"
    else
        report critical "nginx -t fails: the configuration on disk is not valid"
    fi
}

check_service() {
    if systemctl is-active --quiet nginx; then
        report ok "nginx active"
    else
        report critical "nginx is NOT active"
        return 0
    fi
    # Configuration on disk differing from the loaded one: somebody edited without reloading
    local started
    started="$(systemctl show nginx -p ActiveEnterTimestampMonotonic --value)"
    local mtime
    mtime="$(stat -c %Y /etc/nginx/sites-enabled/"${TRAMONTANA_DOMAIN}" 2>/dev/null || echo 0)"
    local start_epoch
    start_epoch="$(date -d "$(systemctl show nginx -p ActiveEnterTimestamp --value)" +%s 2>/dev/null || echo 0)"
    if (( mtime > start_epoch )); then
        report warn "the configuration was modified after the last reload"
    fi
}

check_redirect() {
    local code target
    code="$(curl -s -o /dev/null -w '%{http_code}' -m 5 \
        "http://${TRAMONTANA_DOMAIN}/houses")" || {
        report critical "port 80 is not responding"
        return 0
    }
    target="$(curl -s -o /dev/null -w '%{redirect_url}' -m 5 \
        "http://${TRAMONTANA_DOMAIN}/houses")"
    if [[ "$code" == "301" && "$target" == https://* ]]; then
        report ok "HTTP redirects to HTTPS ($target)"
    else
        report critical "HTTP does not redirect to HTTPS (code $code)"
    fi
}

check_https_and_headers() {
    local headers
    headers="$(curl -sI -m 10 "https://${TRAMONTANA_DOMAIN}/" | tr 'A-Z' 'a-z')" || {
        report critical "HTTPS is not responding"
        return 0
    }
    grep -q '^http/2 200' <<<"$headers" \
        && report ok "HTTPS responds 200 over HTTP/2" \
        || report warn "HTTPS responds but not over HTTP/2"

    local -a required=(
        strict-transport-security
        x-content-type-options
        x-frame-options
        referrer-policy
        content-security-policy
    )
    local h missing=0
    for h in "${required[@]}"; do
        grep -q "^${h}:" <<<"$headers" || { report warn "the $h header is missing"; missing=1; }
    done
    (( missing == 0 )) && report ok "all 5 security headers are present"

    grep -q '^server: nginx$' <<<"$headers" \
        || report warn "server_tokens looks enabled: the version is being revealed"
}

check_tls() {
    local output
    output="$(echo | timeout 10 openssl s_client -connect "${TRAMONTANA_DOMAIN}:443" \
        -servername "${TRAMONTANA_DOMAIN}" -status 2>/dev/null)" || {
        report critical "TLS could not be negotiated"
        return 0
    }
    grep -q 'Verify return code: 0 (ok)' <<<"$output" \
        && report ok "certificate chain valid" \
        || report critical "the certificate chain does NOT validate (check fullchain.pem)"

    grep -q 'OCSP Response Status: successful' <<<"$output" \
        && report ok "OCSP stapling active" \
        || report warn "no OCSP stapling"

    # Old TLS must be rejected
    if echo | timeout 5 openssl s_client -connect "${TRAMONTANA_DOMAIN}:443" \
            -tls1_1 >/dev/null 2>&1; then
        report critical "TLS 1.1 is still accepted"
    else
        report ok "TLS 1.0 and 1.1 rejected"
    fi

    # Days until expiry (reuses the threshold from 06-05)
    local expiry days
    expiry="$(echo | openssl s_client -connect "${TRAMONTANA_DOMAIN}:443" \
        -servername "${TRAMONTANA_DOMAIN}" 2>/dev/null | \
        openssl x509 -noout -enddate | cut -d= -f2)"
    days=$(( ( $(date -d "$expiry" +%s) - $(date +%s) ) / 86400 ))
    if (( days < 7 )); then
        report critical "the certificate expires in $days days"
    elif (( days < TRAMONTANA_CERT_DAYS )); then
        report warn "the certificate expires in $days days"
    else
        report ok "certificate valid for another $days days"
    fi
}

# The most important check: the application's port must NOT be
# reachable from outside. It is tested FROM ANOTHER MACHINE, because
# from the server itself 127.0.0.1:8080 will always respond.
check_app_port_closed() {
    if ! ss -tlnp 2>/dev/null | grep -q "127.0.0.1:${TRAMONTANA_APP_PORT}"; then
        report warn "the application is not listening on 127.0.0.1:${TRAMONTANA_APP_PORT}"
    fi
    # Listening on 0.0.0.0 = exposed even if the firewall covers it
    if ss -tlnp 2>/dev/null | grep -qE "0\.0\.0\.0:${TRAMONTANA_APP_PORT}|\*:${TRAMONTANA_APP_PORT}"; then
        report critical "the application is listening on ALL interfaces"
    fi

    if ! ssh -o BatchMode=yes -o ConnectTimeout=5 \
            "operator@${TRAMONTANA_EXTERNAL_HOST}" true 2>/dev/null; then
        report warn "no access to ${TRAMONTANA_EXTERNAL_HOST}: could not test from outside"
        return 0
    fi
    if ssh -o BatchMode=yes "operator@${TRAMONTANA_EXTERNAL_HOST}" \
            "nc -z -w3 ${TRAMONTANA_IP} ${TRAMONTANA_APP_PORT}" 2>/dev/null; then
        report critical "port ${TRAMONTANA_APP_PORT} IS REACHABLE from outside"
    else
        report ok "port ${TRAMONTANA_APP_PORT} is closed from outside"
    fi
}

check_rate_limit() {
    local codes=""
    local i
    for i in $(seq 1 12); do
        codes+="$(curl -s -o /dev/null -w '%{http_code}' -m 5 \
            "https://${TRAMONTANA_DOMAIN}/login") "
    done
    if grep -q '429' <<<"$codes"; then
        report ok "the /login rate limit returns 429 on bursts"
    else
        report warn "the /login rate limit did not trigger: [$codes]"
    fi
}

check_certbot_hook() {
    local hook=/etc/letsencrypt/renewal-hooks/deploy/10-reload-nginx.sh
    if [[ -x "$hook" ]]; then
        report ok "post-renewal reload hook present"
    else
        report warn "the reload hook is missing: after renewing, the old certificate would remain"
    fi
}

main() {
    require_command curl
    require_command openssl
    require_command ss

    check_configuration
    check_service
    check_redirect
    check_https_and_headers
    check_tls
    check_app_port_closed
    check_rate_limit
    check_certbot_hook

    case "$overall_status" in
        0) log "verification complete: everything correct" ;;
        1) error "verification complete with WARNINGS" ;;
        2) error "verification complete: CRITICAL state" ;;
    esac
    return "$overall_status"
}

main "$@"

$ chmod 0750 ~/scripts/check_web.sh
$ shellcheck ~/scripts/check_web.sh && echo "no warnings"
no warnings

$ ~/scripts/check_web.sh; echo "status: $?"
[2026-08-18 12:04:11] OK       the nginx configuration is valid
[2026-08-18 12:04:11] OK       nginx active
[2026-08-18 12:04:12] OK       HTTP redirects to HTTPS (https://bookings.tramontana.example/houses)
[2026-08-18 12:04:12] OK       HTTPS responds 200 over HTTP/2
[2026-08-18 12:04:12] OK       all 5 security headers are present
[2026-08-18 12:04:13] OK       certificate chain valid
[2026-08-18 12:04:13] OK       OCSP stapling active
[2026-08-18 12:04:14] OK       TLS 1.0 and 1.1 rejected
[2026-08-18 12:04:14] OK       certificate valid for another 71 days
[2026-08-18 12:04:15] OK       port 8080 is closed from outside
[2026-08-18 12:04:17] OK       the /login rate limit returns 429 on bursts
[2026-08-18 12:04:17] OK       post-renewal reload hook present
[2026-08-18 12:04:17] verification complete: everything correct
status: 0

Four design decisions in the script:

The state accumulates to the worst, not to the last. A WARN followed by an OK must not erase the warning. Hence the overall_status variable and the report function that only raises the level.
An explicit return 0 in every function. With set -euo pipefail, a function whose last expression is false aborts the script. The return 0 at the end of report is indispensable, and it is a mistake shellcheck does not catch.
The external check degrades to a warning, not to critical, if there is no access. Not being able to test something is not the same as testing it and having it fail. Confusing the two produces false alerts when the test machine is switched off, and that leads straight to the alert fatigue of 08-06.
It distinguishes "listening on 0.0.0.0" from "is reachable". The first is an application configuration fault covered up by the firewall; the second is a real exposure. Both are critical, but the cause and the fix are different.

To make it operational, a timer that runs it after every renewal and every morning, along the lines of check_backup.sh:

# /etc/systemd/system/check-web.timer
[Unit]
Description=Daily verification of the web server

[Timer]
OnCalendar=*-*-* 08:15:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

Silence if all is well: the service unit only writes to the journal, and only a status of 2 triggers the notification.

Solution 3

Why we have put a web server in front of the application To: Marta Vidal · From: Systems Operations · 18 August 2026

Summary: as of today, everything that travels between a customer's browser and our server is encrypted. Until this morning it was not, and that included the names, phone numbers, email addresses and stay dates of the people who book with us. It was the most serious open problem we had and it is now resolved.

What we have set up, without jargon. We have put a program called Nginx in charge of receiving the visits to the website. Before, the booking application answered directly; now Nginx answers, checks and tidies up the request, and passes it on to the application, which carries on exactly as it was, with no changes at all. For the customer it is invisible; for us it changes quite a lot.

What this measure protects against:

Protects against How

Somebody reading a booking's data in transit All the traffic is encrypted with the certificate we prepared in June

Somebody altering what the customer sees The encryption also guarantees that nobody changes the content along the way

A browser arriving unencrypted by mistake Automatic redirection, plus an instruction to the browser not even to try

Somebody hammering the login form trying passwords At most 5 attempts per minute and address

A single address saturating the service A limit on simultaneous connections

An application failure exposing technical information Nginx shows an error page of our own

The site being loaded in disguise inside another, fraudulent site Security headers that the browser honours

What it does NOT protect against, and it is worth being clear about it:

It does not protect the stored data. The encryption covers the journey, not the warehouse. The data is still in the database as it was; that is addressed in next week's work.

It does not protect against a failure in the application itself. If there is a bug in the booking code, Nginx passes it through just the same.

It does not protect against an employee's weak password. That is covered by other measures we already have.

It does not replace the backups, nor does it prevent an accidental deletion.

It does not make the site faster on the first visit. In fact the first connection takes around 20 milliseconds longer because of the encryption — a twentieth of a blink.

On whether it adds a point of failure: yes, and it is a good question. It is literally true that there is now one more piece that can break. Three qualifications:

Nginx is one of the most thoroughly tested pieces of software in existence: it drives an enormous share of the world's websites and it is extraordinarily stable. The probability of it failing before our application does is very low.

It also removes failures. Before, every update of the application cut the service for a few seconds and the customer saw a connection error. Now Nginx is still there and shows a decent page. And it serves the images and styles itself, so if the application gets overloaded, at least the site does not look completely broken.

We have measured it: the site transfers 78% less data than before, because Nginx compresses it. On a phone with poor coverage that is more noticeable than the 20 milliseconds of encryption.

What it cost. One morning's work, and zero euros: the certificate is free and renews itself every three months, with an automatic warning twenty days beforehand in case anything went wrong. The configuration is saved as code, so if we had to rebuild the whole server tomorrow, this is reproduced in minutes along with everything else.

What is still outstanding and the order I propose:

This week: raise to two years the instruction that stops browsers using unencrypted connections. We have deliberately set it to five minutes, so that we can verify over a few days that everything works before committing — once sent, it cannot be withdrawn.

Next week: go over one detail of the site with Luis (the style files), because after each update some customers may see the previous version stored in their browser for a few days. We already know how to fix it.

This quarter: the database, which is the next piece I want to get into shape.

A final note, important for compliance. Serving personal data unencrypted is hard to defend under the General Data Protection Regulation, which requires appropriate technical measures. With this we move from a questionable situation to a correct one and, on top of that, a demonstrable one: I can generate an automatic report at any time certifying that the encryption is active and correctly configured. I will fold it into the daily review procedure.

Protects against	How
Somebody reading a booking's data in transit	All the traffic is encrypted with the certificate we prepared in June
Somebody altering what the customer sees	The encryption also guarantees that nobody changes the content along the way
A browser arriving unencrypted by mistake	Automatic redirection, plus an instruction to the browser not even to try
Somebody hammering the login form trying passwords	At most 5 attempts per minute and address
A single address saturating the service	A limit on simultaneous connections
An application failure exposing technical information	Nginx shows an error page of our own
The site being loaded in disguise inside another, fraudulent site	Security headers that the browser honours

Conclusion

The oldest debt in the course is closed. The certificate you issued in 06-05, which had spent two modules waiting on the disk, is now in use: https://bookings.tramontana.example responds encrypted, with TLS 1.2 and 1.3, suites taken from a maintained source rather than invented, OCSP stapling, the complete chain from fullchain.pem, and the five security headers that turn the customer's browser into an ally. And you verified it with openssl s_client instead of trusting that it works.

You understand the reverse proxy as what it is: a single point where the policy is applied — encryption, headers, compression, rate limiting, logging — without the application having to know anything about it. You know how Nginx picks the server by listen and server_name, and how it picks the location with a precedence in which a regex beats a more specific prefix, which is the source of half the bewilderment. You know why every proxy_set_header is where it is: without Host the links come out broken, without X-Real-IP the logs are useless, and without X-Forwarded-Proto the application goes into a redirect loop. And you know that nginx -t goes always before a reload, and that reload goes always instead of restart.

You have also measured. The 19 ms of the TLS handshake, the 78% fewer bytes from compression, the static requests the application has stopped serving, and the p99 of $upstream_response_time that separates "the client has a bad connection" from "the application is slow". That last figure points straight at the next lesson, because when urt goes up, the culprit is almost never the application: it is a query.

In 08-02 you will set up the database server properly. PostgreSQL 16 has been running since Module 5 with the default configuration, which is designed to start anywhere rather than for srv-tramontana's 3.8 GB. You will tune the memory with reasoned formulas, connect it with what you learned about transparent huge pages in 07-03, resolve once and for all the mismatch between the application's max_connections=80 and the server's max_connections=100 — which already caused an incident in 07-02 — by putting PgBouncer in transaction mode, lock down pg_hba.conf field by field, and set up physical backups with WAL archiving and point-in-time recovery, which is the only thing that lets you undo a DELETE with no WHERE. That is where Tramontana's business really lives.

Setting Up a Web Server

Contents

Objective, prerequisites and starting state

What a reverse proxy is and why it goes in front

Choosing the web server: Nginx, Apache or Caddy

Installation and the structure of /etc/nginx

Anatomy of the configuration: contexts, server and location

How Nginx picks the `server` block

How Nginx picks the `location` block

Building Tramontana's virtual server

Static content, compression and rate limiting

Compression

`sendfile` and company

Rate limiting: how it really works

Logging and rotation

Integrating the already-issued certificate with certbot

Verification and measurement, before and after

The battery of checks

Measuring before and after

Automation with Ansible: the web role

Operation and maintenance

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

Setting Up a Web Server

Contents

Objective, prerequisites and starting state

What a reverse proxy is and why it goes in front

Choosing the web server: Nginx, Apache or Caddy

Installation and the structure of /etc/nginx

Anatomy of the configuration: contexts, server and location

How Nginx picks the server block

How Nginx picks the location block

Building Tramontana's virtual server

Static content, compression and rate limiting

Compression

sendfile and company

Rate limiting: how it really works

Logging and rotation

Integrating the already-issued certificate with certbot

Verification and measurement, before and after

The battery of checks

Measuring before and after

Automation with Ansible: the web role

Operation and maintenance

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

How Nginx picks the `server` block

How Nginx picks the `location` block

`sendfile` and company