The previous lesson ended by pointing at two debts you have been carrying since Module 5, and both are cryptographic. The first: db_password is written in the clear inside /etc/tramontana/app.conf, and you have already seen that any permission failure turns that into a credential leak no amount of watching can prevent. The second: the booking traffic — with the guests' names you have taken such care to protect inside the backups — travels unencrypted over port 8080. This lesson resolves both, because they are the same problem seen from two angles: keeping a secret and proving an identity.

They are two clearly separate halves. In the first you will learn how to take a secret out of a configuration file and manage it with the tools the system already gives you: strict permissions, systemd's EnvironmentFile and LoadCredential, GPG, pass and LUKS. In the second, what a certificate really is, how it is generated, read and verified, how to obtain a free and automatically renewed one from Let's Encrypt, and how to watch its expiry — which is the silliest and most frequent production failure of all.

A note on scope. Here you obtain, install and verify the cryptographic material for bookings.tramontana.example. The web server that will serve it — Nginx as a reverse proxy in front of the application on port 8080 — is built in project 08-01. That is deliberate: encryption in transit is prepared and tested now so that in Module 8 it is a configuration detail and not a new topic.

Compliance warning. The encryption of personal data, in transit and at rest, is not an optional improvement: the GDPR requires it as an appropriate technical measure (art. 32). bookings.csv contains guests' names. In a real environment, the design of key management and key custody must be reviewed by the security officer, and decisions about personal data by the data protection officer.

1. What a secret is and where it never goes
2. The decent minimum: permissions, EnvironmentFile and LoadCredential
3. Encrypting secrets at rest: GPG and pass
4. Disk encryption with LUKS
5. Centralised secret managers
6. Credential rotation
7. What TLS guarantees and what a certificate contains
8. Generating, reading and verifying cryptographic material with OpenSSL
9. Let's Encrypt and automatic renewal
10. TLS good practice, key custody and expiry monitoring

A secret is a piece of data whose value depends on only the right people knowing it: a password, an API key, a token, a private key, the passphrase of a backup repository. It differs from any other piece of sensitive data in that its compromise is not a privacy problem but a control problem: whoever has it can act in your name.

The first rule is a negative one, and it is worth understanding the why of each entry because each corresponds to a real and frequent leak:

The first two deserve a demonstration, because people underestimate them:

It is worth being precise about the second case, because there is a nuance that is often misread: /proc/<pid>/environ has 0400 permissions and only the process's owner and root can read it. That makes environment variables much better than the command line, but it does not make them a good place: they are inherited by every child process, they show up in memory dumps and in the error traces of many frameworks, and any escalation to root exposes them all at once.

You do not need infrastructure to improve substantially. With what you already know about permissions (02-07), service users (05-01) and systemd units (05-05), there are three levels, each better than the last.

The idea is to separate the secret from the configuration: app.conf stops containing the password and becomes a file readable by the group, while the secret lives in a separate file with minimal permissions.

And the unit loads it into the service's environment, without it ever appearing on the command line:

Remember the course's convention: app.conf is chattr +i, so editing it means removing the attribute, making the change and putting it back — with its prior copy:

This already closes the exposure from the last exercise of 06-04: an accidental chmod 644 on app.conf no longer leaks anything. But the secret is still in the clear on disk and in the process's environment.

Ubuntu 24.04 ships systemd's credential mechanism, which is clearly superior: the secret is delivered to the service in a file inside a private tmpfs accessible only to that service, it is not inherited by children and it does not appear in the environment.

The service receives the path in the CREDENTIALS_DIRECTORY variable and reads the file:

The difference from EnvironmentFile in a table, because it decides the choice:

The step that closes the circle: systemd-creds encrypts the secret with a key derived from the system (and from the TPM if there is one), so that the file on disk no longer contains the secret in the clear.

That 0 is the result you were after. The third open incident is now closed: the password is no longer in the clear in any configuration file, it is not in the process's environment, it is not readable by any user on the system and — an important consequence — nor does it travel in the clear inside the restic backup, because what gets copied is the encrypted file.

One honest limitation: systemd-creds's encryption key is tied to the machine (/var/lib/systemd/credential.secret). That is exactly what you want against the leak of a file, but it means the .cred file cannot be restored on a rebuilt server. The original secret has to be somewhere you can recover it from as well: that is the next section, and it is what has to be recorded in the runbook from 05-08.

You need a place where the original secret lives encrypted, recoverable by an authorised person, with a change history and off the server. The classic answer on Linux is GPG, and pass built on top of it.

The practical difference is one of management: the symmetric mode forces you to transmit the passphrase through another channel and to change it when somebody leaves the team; the asymmetric one lets you encrypt for several people and revoke one person's access without re-encrypting for the rest. In a team, always asymmetric.

Notice the > /dev/shm/recovered.txt in the decryption: /dev/shm is memory, not disk. Writing a decrypted secret to the filesystem leaves recoverable traces even after you delete it.

pass keeps each secret in a GPG-encrypted file inside a git repository. It is simple, auditable and depends on no service.

Three details that make it fit for production:

• A full history with git. pass git log --oneline shows every addition, change and removal, with a date and an author. It is the traceability of rotation.

• The files are encrypted, so the repository can be synchronised to a remote without exposing anything: pass git remote add origin ... and pass git push. That solves the "off the server" requirement.

• Composition with scripts. pass writes to stdout, so it fits with everything from Module 4:

  # Generate the .cred file from pass, without the secret ever touching the disk in the clear
  $ pass tramontana/production/db_password \\
      | sudo systemd-creds encrypt --name=db_password - \\
             /etc/tramontana/secrets/db_password.cred
  

  That pipeline is the answer to the recoverability problem from the previous section: pass is the source of truth, systemd-creds is the operational copy tied to the machine, and rebuilding the server consists of running this line again.

And the obvious warning: the private GPG key and its passphrase are now the master key. Its backup (gpg --export-secret-keys, encrypted, on separate physical media) and its custody are part of the runbook, and losing it amounts to losing every secret.

The secrets are encrypted now, but /srv/tramontana/backups contains bookings.csv with guests' names. If somebody takes the disk away — or the VM image, or the disk of a decommissioned server that was never wiped — the filesystem's permissions protect nothing: it gets mounted on another machine and everything is readable.

LUKS is the block encryption standard on Linux. It encrypts the whole device, underneath the filesystem.

And here is the real problem, which has to be stated without decoration: a LUKS volume needs a passphrase in order to open, and a server has to boot on its own at three in the morning. The options, with their trade-offs:

For the lab, the key file option with nofail is reasonable, and it is important to understand exactly what it protects:

What it protects and what it does not, stated precisely: it protects against the theft or removal of the data disk, against access to the image of the stopped VM, and against data recovery from a discarded disk. It does not protect against an attacker who compromises the running server, because as far as they are concerned the volume is mounted and readable just as it is for anybody else. It is encryption at rest, not encryption in use, and presenting it as anything else to Marta would be misleading her.

Notice too how it composes with what came before: restic already encrypts its repository with its own password — which now lives in pass — so the backups have two independent layers. And for the off-site destination of the 3-2-1 rule, restic's encryption is the one that really matters, because you do not control the remote medium.

Everything above is correct for one machine. When there are several, a new problem appears: distributing and rotating secrets across N servers without copying them by hand.

The point where it starts paying off has a fairly identifiable threshold: when rotating a credential stops being a five-minute task. With one server, rotating means changing pass, regenerating the .cred and restarting the service. With twenty, without a central manager, it is a manual operation prone to leaving machines half done — which is the worst of all worlds, because you have the cost of the rotation without the guarantee.

sops deserves a specific mention because it will be relevant in 07-06: it lets you keep an Ansible variables file in git where the keys are readable and the values are encrypted, so that the diff of a configuration change remains reviewable without exposing the secrets.

Vault's other advantage, which the rest do not have, is dynamic secrets: instead of a fixed database password, it generates credentials valid for one hour, specific to each process. A secret that expires on its own needs no rotation, and a leak has a minimal exploitation window. It is the model the industry is heading towards.

Rotating means replacing a secret with a new one. It is compulsory because the probability that a secret has been compromised only grows with time: every copy, every log, every person who saw it, every machine it has been on. Rotating limits the exploitation window of a leak that may already have happened without your knowing.

When to rotate, in order of urgency:

1. Immediately, on any suspicion of exposure. It is what you did with db_password when it turned up in the backup with 644 permissions, and what outcome B of the last exercise in 06-04 would do.
2. When somebody with access leaves the team. It is not distrust: it is that control over who knows the secret has been lost.
3. Periodically, with a written interval. For a service credential like this one, between 90 days and a year is usual, depending on the exposure.

The procedure without interrupting the service is the part with technical substance, and it depends on a property of the target system: that it accepts two valid credentials at once. PostgreSQL does not do that with the same account, so the pattern uses two users:

The order is what matters: create, apply, verify, withdraw. Swapping the last two steps leaves the service down until somebody notices, and at three in the morning that means hours. It is the same "measure before and after" discipline you applied in 05-07.

A decent way to generate passwords, so as not to give in to the temptation of inventing them:

The second half of the lesson. TLS (Transport Layer Security, SSL's successor) gives three guarantees, and it is worth separating them because the third is the one almost everybody forgets:

The third is the reason certificates exist. Encryption without authentication is useless against a man in the middle: if an attacker interposes themselves, negotiates one encrypted connection with you and another with the real server, they read everything. It is the same reasoning as the SSH fingerprint warning in 06-02.

And what TLS does not guarantee: that the server is honest, that the application is secure, that the data is encrypted at rest, or that whoever connects is who they say they are (for that you need to authenticate the client).

The problem to solve is one of bootstrapping: how to trust the public key of a server you have never seen before. The solution is to delegate to a third party you already trust.

Verification consists of checking, upwards, that each certificate is signed by the next one, until you reach a root that is already in the local store (/etc/ssl/certs/, from the ca-certificates package). The root CA signs little and lives heavily protected; the intermediates do the day-to-day work and can be revoked without invalidating the root.

A certificate is the server's public key plus a set of assertions, all signed by the CA:

The point about the SAN is practical, not anecdotal: for years now browsers and TLS libraries have ignored the CN and validated exclusively against the SAN. A certificate with CN=bookings.tramontana.example and no SAN is rejected. It is a classic source of lost hours when generating certificates by hand.

The TLS 1.3 handshake, in five lines and without going into cryptography:

1. The client sends the algorithms it supports and its contribution to the key exchange.
2. The server chooses, sends its own contribution and its certificate.
3. The client verifies the certificate against its CA store and checks that the name is in the SAN.
4. Both derive the same session key without ever having transmitted it.
5. From then on, everything travels encrypted with that key. A single round trip.

openssl is the Swiss army knife of this field. Four operations cover 90% of the work.

The CSR (Certificate Signing Request) contains the public key and the requested names, and it is signed with the private key to prove that you hold it. The private key never leaves the server.

That -copy_extensions copyext is essential: without it, the CSR's SAN is not copied into the certificate and you get a useless one. It is the most frequent mistake when generating test certificates by hand.

A self-signed certificate is good for testing the server's configuration, and for that alone: no CA vouches for it, so every client will warn. Never in production.

That last check resolves a baffling and frequent failure: the server does not start or refuses connections because the key and the certificate are not a pair, typically after regenerating one and forgetting the other. If the two sums match, they are a pair.

And inspecting a running server:

The -servername is compulsory when there are several sites on the same IP: it is the SNI extension, which says which certificate you want. Without it you will receive the default certificate and believe there is a problem where there is none.

Let's Encrypt is a free, automated CA. Its protocol, ACME, works on a simple idea: to prove that you control a domain, the CA asks you to do something only its owner could do.

DNS-01 is essential in two cases: wildcard certificates (*.tramontana.example) and servers that do not expose port 80 — and it requires the DNS provider to have an API, or doing it by hand at every renewal, which is not viable.

And here the note on scope from the beginning applies: Nginx is built in 08-01, so for now we use --standalone mode, in which certbot brings up its own temporary server on port 80. Remember that in 06-03 you left 443 open but not 80: it has to be opened first.

The four files it generates, because each server asks for a different one:

Serving cert.pem instead of fullchain.pem is a classic mistake: it works in the desktop browser, which usually has the intermediate cached, and fails on mobile clients and in curl. It is detected with openssl s_client by looking at whether the chain is complete.

Let's Encrypt certificates last 90 days. That forces automation, which is intentional: an automated and tested renewal fails less often than an annual one done by hand.

--dry-run is the most important line in this section. Renewal typically fails because port 80 is closed by a new firewall rule, because a service is occupying that port, or because a --deploy-hook is broken. And it fails silently, 60 days after you made the change, when nobody remembers making it. Running a dry run after every network or firewall change is the discipline that avoids that scenario.

The deploy hook is what reloads the service after renewing:

A renewed certificate the service has not reloaded is no use: the process goes on presenting the old one until it restarts. It is another classic silent failure.

On mTLS (mutual authentication), briefly: the server also demands a client certificate. It is excellent for service-to-service communication — much better than an API key — and inadvisable for people, because of the cost of distributing and renewing certificates on every device.

The current criteria, which will be applied in 08-01 when configuring Nginx:

• TLS 1.2 and TLS 1.3 only. SSL 3.0, TLS 1.0 and 1.1 have been withdrawn because of known vulnerabilities and no current client needs them.
• Suites with forward secrecy (ECDHE), so that a future compromise of the private key does not allow traffic captured in the past to be decrypted.
• HSTS, the Strict-Transport-Security header, which tells the browser to always use HTTPS for this domain. Careful: it is hard to reverse, so start with a short max-age.
• Do not invent the cipher suite list. Use the generator at ssl-config.mozilla.org and its intermediate profile, which is the reasonable balance between security and compatibility, and review it periodically.

640 root:ssl-cert is the pattern: root writes, the ssl-cert group reads, nobody else. The process that needs the key is added to that group — the key is never made world-readable.

If the private key leaks, generating a new one is not enough: you have to revoke the old one, because it remains valid until it expires and would let whoever holds it impersonate your service:

Honestly about its effectiveness: revocation depends on the client checking it, and many do not, or fail permissively. It is necessary but not sufficient; the response plan has to assume that the leaked key remains usable for a while.

The silliest and most frequent production failure is an expired certificate. It happens even with automatic renewal, because automation breaks too. The measure is an independent control that does not trust the renewal mechanism, along the lines of check_backup.sh from 05-08:

The 20-day threshold is not arbitrary: with 90-day certificates, certbot renews at 60 days, so 30 days of margin remain. A warning at 20 means the automatic renewal has failed twice and there are still nearly three weeks to fix it calmly.

And since the script returns 0/1/2 just like health_check.sh, it integrates without friction with the general check and with the monitoring.

• Generating a certificate with no SAN. The CN is no longer validated. Without -addext "subjectAltName=..." in the CSR and without -copy_extensions copyext when self-signing, you get a certificate every client rejects.
• Serving cert.pem instead of fullchain.pem. It works in your browser — which has the intermediate cached — and fails on mobiles and in curl. Always verify with openssl s_client from another machine.
• Not running the renewal dry run. certbot renew --dry-run after every firewall or web server change. Otherwise you find out 60 days later, when the certificate genuinely expires.
• Forgetting the deploy hook. A renewed certificate the service has not reloaded is worth nothing: the process goes on presenting the old one.
• Passing a secret on the command line. --password=X is visible in ps to the whole system. Use a file, standard input or the program's credential mechanism.
• Decrypting a secret to a file on disk. It leaves recoverable traces. Use /dev/shm, a pipe, or pass -c with the temporary clipboard.
• Believing that LUKS protects a running server. It protects a stolen or discarded disk, not a compromised system, where the volume is already mounted. Getting that wrong in a report is worse than leaving it out.
• Rotating by withdrawing the old credential before verifying the new one. Create, apply, verify, withdraw. That order, always.
• Losing pass's GPG key with no backup. It is the master key to all your secrets. Its encrypted backup and its custody are part of the runbook, not a detail.
• A tip on method. Record in the runbook, for each secret: where the source of truth is, who can recover it, when it was last rotated and how often it is due. An inventory of secrets with no dates is an inventory nobody maintains.

restic needs its repository password in order to run the nightly backup with no human intervention. Today it is in a 600 file owned by root and read by backup_tramontana.sh. Redesign the solution using pass as the source of truth and systemd's credential mechanism, and explain the recoverability problem that appears and how you resolve it.

You receive this warning and have to diagnose it:

certbot.timer is active and certbot renew has given no visible error in the journal. List the possible causes in order of likelihood, with the command that confirms or rules out each one.

Marta asks you in writing "for the encryption to protect the guests' data". Write the report with the three layers of encryption you have built — secrets with systemd-creds and pass, disk with LUKS, transit with TLS — stating for each one what it protects and what it does not protect, and which risk to the guests' data remains open despite all three.

Solution 1

And in the script, restic reads the password from a file, which is exactly what the credential hands it:

Three decisions worth justifying:

• RESTIC_PASSWORD_FILE instead of RESTIC_PASSWORD: the password does not go through the environment, only the path to the file does.
• ${CREDENTIALS_DIRECTORY:?...} with a message: if the script is run by hand instead of by systemd, it fails immediately and with a clear message, instead of trying to back up without a credential. It is the expansion from 04-02 applied to something real.
• systemd mounts the credential in a tmpfs private to the service: neither luis, nor intern, nor svc-tramontana itself from another process can read it.

The recoverability problem, which is the substance of the exercise: the .cred file is encrypted with a key derived from this machine (/var/lib/systemd/credential.secret). If the server is lost — scenario (c) of the restore in 05-08, or a reinstall after a compromise — that file cannot be decrypted. And without restic's password there is no way to read the backups: you would have the backups and be unable to restore them, which is the worst possible failure.

It is resolved on two fronts:

1. pass is the source of truth, and it lives elsewhere. The pass repository is synchronised to a remote (pass git push), and the private GPG key has an encrypted backup on separate physical media. Rebuilding the server means: install, restore pass, and run the pipeline from step 2 again.
2. The runbook documents it explicitly, with that pipeline written out literally and the location of the GPG key. The runbook is already off the server thanks to the discipline of 05-08.

The general rule to take away: a secret tied to the machine can never be the only copy. The systemd credential is an operational cache; the source of truth is somewhere else and is recoverable by an authorised person.

Solution 2

The fact that certbot renew gave no error is the main clue: you have to distinguish "it renewed and something afterwards failed" from "it did not renew and nobody noticed". The first check separates the two branches:

Branch A — the certificate on disk is new (expires in ~90 days). The renewal worked; the problem is downstream.

1. The service was not reloaded (the most likely). The process is still in memory with the old certificate:

   $ ls -l /etc/letsencrypt/renewal-hooks/deploy/
   $ sudo journalctl -u certbot --since "40 days ago" | grep -iE 'hook|deploy|reload'
   $ systemctl show nginx -p ActiveEnterTimestamp
   

   If the service's start timestamp is earlier than the renewal, this is it. It is fixed by reloading, and fixed at the root by reviewing the deploy hook.
2. The server points at the wrong file: at a cert.pem copied to another path at some point, instead of at the link under live/ that certbot updates.

   $ grep -rE 'ssl_certificate' /etc/nginx/
   

3. You are looking at another certificate: without -servername, openssl s_client returns the default site. The script itself passes it, but check if you have been testing by hand.

Branch B — the certificate on disk also expires in 6 days. Nothing has been renewed. certbot renew does not fail loudly when the challenge cannot be completed in a non-interactive run, so you have to provoke the error:

Causes in order of likelihood:

1. Port 80 is closed. It is cause number one, because any firewall review carries away that rule "that did not look necessary":

   $ sudo ufw status verbose | grep -E '^80|http'
   $ sudo nft list ruleset | grep -E 'dport 80|dport { .*80'
   

2. Another process occupies 80 and --standalone cannot bring up its server:

   $ sudo ss -tulpn | grep ':80 '
   

3. The timer does not actually run, even though it is active:

   $ systemctl list-timers certbot.timer --no-pager
   $ sudo journalctl -u certbot.service --since "70 days ago" | tail -20
   

   An empty LAST with an active timer means it has never fired.
4. DNS: the name no longer resolves to this machine, so the HTTP-01 challenge lands somewhere else:

   $ dig +short bookings.tramontana.example
   $ getent hosts bookings.tramontana.example
   

5. The CA's rate limit, if there were many failed attempts. The --dry-run output says so, and it is distinguishable because it uses the staging environment and consumes no quota.

The lesson on method: the warning arrived with 6 days of margin because the threshold was 20 and the control is independent of the renewal mechanism. If the monitoring had consisted of trusting certbot, you would have found out with the service down. That is why verification controls must not share a mechanism with what they verify.

Solution 3

Report for Marta Vidal — Encryption of guests' data on srv-tramontana 18 August 2026

Three layers of encryption have been put in place. Each covers a different risk and none replaces the others. I set out what each one protects and does not protect, and the risk that remains open.

Layer 1 — Secrets (systemd-creds + pass) The database password and the backup repository password are no longer in the clear in any configuration file or in any process's environment. They are encrypted on disk and systemd delivers them to the service in a private space in memory. The source of truth is in a manager with history, off the server. Protects: credential leakage through a permission mistake, through a badly made backup, through a log entry or through reading a process's environment. It closes the incident that has been open since the backup with 644 permissions. Does not protect: against an attacker with root privileges on the running machine, who can read the credential in the same way as the service.

Layer 2 — The backup disk (LUKS) The /srv/tramontana/backups volume is encrypted at block level. In addition, the restic repository has its own encryption, so the backups have two independent layers. Protects: theft or physical loss of the disk, access to the image of the powered-off machine, and data recovery from a disk withdrawn without secure erasure. Does not protect: against an attacker who compromises the running server. For them, the volume is mounted and readable. It is encryption at rest, not in use.

Layer 3 — Transit (TLS) The cryptographic material for bookings.tramontana.example has been issued by Let's Encrypt, verified, with automatic renewal tested and independent expiry monitoring warning 20 days ahead. Protects: the reading and modification of booking data on its way across the network, and impersonation of the service. Does not protect: the data once it reaches the server, nor the security of the application itself.

Open risk, and it is important that it is on record Guests' data currently travels unencrypted: the application listens over HTTP on port 8080. The certificate is ready and verified, but the component that will present it — the reverse proxy in front of the application — is scheduled for the next phase of the project. Until it is deployed, encryption in transit is not in force, even though the material is ready. Interim mitigation: 8080 is not reachable from the Internet, so the exposure is limited to the internal network.

Residual risk common to all three layers No layer protects against a compromise of the running server with root privileges. The controls from the previous phase act against that scenario: change detection with AIDE, access auditing with auditd, and the incident response procedure with a rebuild.

Compliance. The encryption of personal data in transit and at rest is a GDPR obligation (art. 32), not an optional improvement, and the open risk in the previous point is relevant in that respect. I recommend that the design of key custody and the decision on the timescale for deploying the reverse proxy be reviewed by the data protection officer.

The course's two cryptographic debts are settled, each in its own way. The db_password no longer exists as text in any configuration file: it lives encrypted in a .cred that only systemd can decrypt and only the service can read, with pass as the versioned source of truth synchronised off the server. The backup volume is encrypted with LUKS, with precise knowledge of what that protects and what it does not. And the material for bookings.tramontana.example is issued, verified with openssl, with automatic renewal tested in a dry run and an independent expiry control that warns twenty days ahead. The third open incident is now closed, and you know why LoadCredentialEncrypted is better than EnvironmentFile, why the SAN outranks the CN, and why the order of a rotation is create, apply, verify and withdraw.

One is left. On 18 August at 06:12, unattended-upgrades applied the fix for a TLS CVE to openssl and libssl3t64 — the library that underpins everything you have just built in the second half of this lesson — and left running services still using the old version loaded in memory. An installed patch nobody has genuinely applied is a vulnerability with paperwork. In lesson 06-06: Securing Linux Systems that incident is closed with needrestart and something more ambitious is done: bringing everything you have learned together into an explicit threat model and a coherent security posture. You will reduce the attack surface, harden authentication with PAM — the debt you left outstanding in 05-01 — write an AppArmor profile for the application, apply the kernel hardening sysctl settings and the noexec mounts, raise the systemd-analyze security score with judgement, and finish with a hardening checklist for srv-tramontana that says honestly what is done, what has been consciously accepted and what falls outside an administrator's remit.