The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Automation with Ansible

Next ►
►

At the end of the previous lesson the problem was laid out, and it is worth facing head-on: the entire configuration of srv-tramontana lives in the administrator's memory and in a runbook written in prose. The users and groups from 05-01, the sudoers rules from 05-02, the package pinning from 05-03, the LVM from 05-04, the units and timers from 05-05, the log rotation from 05-06, the netplan configuration from 06-01, the hardened sshd_config from 06-02, the ufw rules from 06-03, AIDE and auditd from 06-04, the secrets from 06-05, PAM and AppArmor from 06-06, and the sysctl settings from 07-03. All of it was applied by hand, command by command, over weeks.

The consequences are concrete:

  • The 8-hour RTO that Marta approved depends on somebody remembering the right order for all those steps, under pressure and probably in the middle of the night.
  • The rule from 06-04 — "a compromised server is reinstalled, not cleaned" — is easy to state and very expensive to honour. If reinstalling costs a day's work, the temptation to "clean" is enormous.
  • srv-tramontana-test is not the same as production, however careful the cloud-init user-data was. And a test environment that differs does not test what you think it tests.
  • Nobody but you can rebuild the server.

This lesson turns all of that into versioned, reviewable, executable code. It is not just one more tool: it is the shift from "I know how the server is configured" to "the repository defines how the server is configured".

Contents

  1. Desired state versus instructions
  2. Idempotence, and why it changes everything
  3. Why Ansible: agentless, over SSH, in YAML
  4. Installation, inventory and variables
  5. Modules and tasks
  6. Playbooks: handlers, conditions, loops and blocks
  7. Jinja2 templates
  8. Roles: organising for reuse
  9. Ansible Vault and secrets
  10. Running it: --check, --diff, --tags and ansible-lint
  11. The Tramontana case: rebuilding the server and measuring the RTO

Desired state versus instructions

deploy.sh is a good script. It has a simulation mode, locking, a prior backup, verification with curl and automatic rollback. And even so it represents a different model from the one needed here.

# IMPERATIVE model: a sequence of instructions
sudo useradd -r -u 997 -s /usr/sbin/nologin svc-tramontana
sudo groupadd -g 1002 tramontana
sudo usermod -aG tramontana operator

Run that twice and the second run fails: the user already exists. To make it repeatable you have to write the checks by hand:

getent passwd svc-tramontana >/dev/null || \
    sudo useradd -r -u 997 -s /usr/sbin/nologin svc-tramontana
getent group tramontana >/dev/null || sudo groupadd -g 1002 tramontana
id -nG operator | grep -qw tramontana || sudo usermod -aG tramontana operator

It works, and it is what you learned to do in 04-06. But multiply it by the two hundred operations that configure srv-tramontana and you have two thousand lines of Bash full of checks, each one an opportunity to get something wrong.

# DECLARATIVE model: describing the desired state
- name: Create the tramontana group
  ansible.builtin.group:
    name: tramontana
    gid: 1002
    state: present

- name: Create the service account
  ansible.builtin.user:
    name: svc-tramontana
    uid: 997
    group: tramontana
    system: true
    shell: /usr/sbin/nologin
    create_home: false
    state: present

The difference is not one of syntax: it is one of what you write. In the first model you describe how to get there; in the second, where you want to be. The tool checks the current state and does only what is missing. And the practical consequence is enormous: the file reads as a description of the server, not as a procedure. Somebody who opens it a year from now understands how the machine is configured without running it.

Imperative (Bash) Declarative (Ansible)
What you write The steps The result
Running it twice Fails, unless you program around it No effect the second time
Starting point Must be known Any
It reads as A procedure A description
Good for One-off operations, deployment Configuration

And an important clarification: Ansible does not replace deploy.sh. Deploying a specific version with rollback is an imperative operation and the script does it well. What it replaces is the server's configuration.

Idempotence, and why it changes everything

In 04-06 you defined idempotence: an operation that can be run several times with the same result. There it was a good practice; in configuration management it is the property that makes the whole model viable.

$ ansible-playbook -i inventory.yml site.yml

PLAY RECAP *******************************************************************
srv-tramontana  : ok=47  changed=12  unreachable=0  failed=0  skipped=3

# And running it again, with nothing changed
$ ansible-playbook -i inventory.yml site.yml

PLAY RECAP *******************************************************************
srv-tramontana  : ok=47  changed=0   unreachable=0  failed=0  skipped=3

That changed=0 on the second run is the goal, and it has three consequences worth stating:

  1. It can be run without fear. You do not have to wonder whether it has already been run. If the server is as it should be, nothing happens.
  2. It detects drift. If tomorrow you run the playbook and it reports changed=3, something changed outside the code: somebody edited a file by hand, an update overwrote a configuration. The playbook itself is an integrity check, complementary to the AIDE from 06-04.
  3. It can be run periodically so that the configuration converges again on its own.

The changed that each task reports is a statement about the world, not about what the tool did. And that is why the modules that cannot guarantee it are flagged as problematic, which is the reason for the warning about command and shell you will see later.

Why Ansible: agentless, over SSH, in YAML

Ansible Puppet / Chef Salt Terraform
Model Push, agentless Pull, with an agent Both Push, API
Transport SSH Its own HTTPS ZeroMQ or SSH The provider's API
Language YAML Its own DSL / Ruby YAML HCL
What it manages Configuration Configuration Configuration Provisioning
Learning curve Gentle Steep Medium Medium
Large scale Hundreds, with tuning Thousands Thousands N/A

The three reasons Ansible fits here:

  • Agentless. Nothing has to be installed on the managed server and there is no extra daemon to maintain. Less attack surface, in line with 06-06.
  • Over SSH. It uses exactly the infrastructure you hardened in 06-02: ed25519 keys, ~/.ssh/config, an sshd_config with no passwords. It adds no new channel to secure, and the authentication and auditing are the ones you already have.
  • YAML. It can be read without knowing the language, which matters when somebody else has to understand the configuration.

The distinction from Terraform is worth having clear because the two get confused: Terraform provisions — it creates the virtual machine, the network, the disk — and Ansible configures what is inside. They are complementary: in 07-04 you provisioned with virt-install and cloud-init; now you configure with Ansible.

Installation, inventory and variables

Ansible is installed on the control machine, not on the managed server:

# On laptop-student
$ sudo apt install ansible ansible-lint
$ ansible --version | head -2
ansible [core 2.16.3]
  config file = /home/student/tramontana-infra/ansible.cfg

The repository, versioned in git:

$ mkdir -p ~/tramontana-infra/{inventory,group_vars,host_vars,roles,templates}
$ cd ~/tramontana-infra && git init
# ansible.cfg
[defaults]
inventory = inventory/production.yml
roles_path = roles
host_key_checking = True
# See the file changes on every run: applies the course convention of
# "diff -u after editing"
diff = True
stdout_callback = yaml
callbacks_enabled = profile_tasks, timer
interpreter_python = auto_silent
retry_files_enabled = False

[ssh_connection]
# Reuse the SSH connection between tasks: it cuts the total time a lot
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=300s

host_key_checking = True is deliberate and it goes against what many tutorials recommend: turning it off avoids the unknown-fingerprint warning and, with it, the protection against man-in-the-middle attacks you studied in 06-02. It is left enabled and the fingerprint is accepted the first time.

pipelining = True deserves a mention: without it, Ansible copies a script to the server and runs it, for every task. With it, it sends it over the already-open connection. On a fifty-task playbook the difference is minutes.

The inventory

# inventory/production.yml
all:
  children:
    web_servers:
      hosts:
        srv-tramontana:
          ansible_host: 10.0.2.15
          env_name: production
    test_servers:
      hosts:
        srv-tramontana-test:
          ansible_host: 192.168.122.104
          env_name: test
  vars:
    ansible_user: operator
    ansible_ssh_private_key_file: ~/.ssh/id_ed25519
    ansible_python_interpreter: /usr/bin/python3
$ ansible-inventory --graph
@all:
  |--@test_servers:
  |  |--srv-tramontana-test
  |--@web_servers:
  |  |--srv-tramontana

$ ansible all -m ping
srv-tramontana | SUCCESS => {"changed": false, "ping": "pong"}
srv-tramontana-test | SUCCESS => {"changed": false, "ping": "pong"}

Variables by group and by machine

Variable precedence lets you have one common definition and differences per environment, which is what makes it possible for testing and production to be almost identical in a controlled way:

# group_vars/all.yml — common to every machine
tramontana_service_user: svc-tramontana
tramontana_service_uid: 997
tramontana_group: tramontana
tramontana_gid: 1002
tramontana_port: 8080
tramontana_version: "3.2.1"

tramontana_paths:
  - { path: /opt/tramontana,                 mode: '0755', owner: root,             group: root }
  - { path: /opt/tramontana/releases,        mode: '0755', owner: root,             group: root }
  - { path: /opt/tramontana/shared/uploads,  mode: '2770', owner: svc-tramontana,   group: tramontana }
  - { path: /etc/tramontana,                 mode: '0750', owner: root,             group: tramontana }
  - { path: /var/log/tramontana,             mode: '0750', owner: svc-tramontana,   group: adm }
  - { path: /srv/tramontana/backups,         mode: '2770', owner: operator,         group: tramontana }

base_packages:
  - postgresql-16
  - restic
  - ufw
  - fail2ban
  - auditd
  - aide
  - libpam-pwquality
  - needrestart
  - sysstat

sysctl_performance:
  net.core.default_qdisc: fq
  net.ipv4.tcp_congestion_control: bbr
  vm.swappiness: 10
  vm.dirty_background_ratio: 5
  vm.dirty_ratio: 10
  fs.inotify.max_user_watches: 524288

sysctl_security:
  kernel.dmesg_restrict: 1
  kernel.kptr_restrict: 2
  kernel.randomize_va_space: 2
  fs.protected_hardlinks: 1
  fs.protected_symlinks: 1
  fs.protected_fifos: 2
  fs.protected_regular: 2
  kernel.unprivileged_bpf_disabled: 1
  kernel.yama.ptrace_scope: 1
# group_vars/web_servers.yml — production only
tramontana_log_level: info
tramontana_max_connections: 80
tramontana_internal_network: 10.0.2.0/24
backup_time: "02:30"
# group_vars/test_servers.yml — test only
tramontana_log_level: debug
tramontana_max_connections: 20
tramontana_internal_network: 192.168.122.0/24
backup_time: "22:00"
# Tools that are not needed in production
extra_packages:
  - strace
  - bpfcc-tools
  - bpftrace
  - linux-tools-generic

There is the concrete value: the differences between environments are explicit and fit in ten lines. In 07-04, with cloud-init, you had to maintain two parallel files and trust yourself not to forget anything. Here, whatever does not appear in group_vars/test_servers.yml is identical by construction.

Facts are the variables Ansible discovers about each machine:

$ ansible srv-tramontana -m ansible.builtin.setup \
      -a 'filter=ansible_distribution*,ansible_memtotal_mb,ansible_processor_vcpus'
srv-tramontana | SUCCESS => {
    "ansible_facts": {
        "ansible_distribution": "Ubuntu",
        "ansible_distribution_version": "24.04",
        "ansible_memtotal_mb": 3891,
        "ansible_processor_vcpus": 2
    }
}

Modules and tasks

A module is an idempotent unit of work. The ones that actually get used:

Module What for Note
ansible.builtin.apt Packages on Debian/Ubuntu state: present, latest, absent
ansible.builtin.copy Copying a file With mode, owner, group, backup
ansible.builtin.template Copying with Jinja2 processing The most used one
ansible.builtin.file Directories, links, permissions state: directory, link, absent
ansible.builtin.lineinfile One line in a file A last resort: template is better
ansible.builtin.user / group Identities Explicit uid/gid
ansible.builtin.systemd_service Services and units enabled, state, daemon_reload
ansible.posix.sysctl Kernel parameters With sysctl_file
community.general.ufw Firewall
community.general.lvol / filesystem LVM and filesystems
ansible.builtin.command / shell Running a command A last resort

And the warning about those last two, which is one of the things that separates a good playbook from a bad one:

# WRONG: not idempotent. It always runs and always reports 'changed'.
- name: Initialise the AIDE database
  ansible.builtin.command: aideinit

# RIGHT: 'creates' makes the task skip if the file already exists
- name: Initialise the AIDE database
  ansible.builtin.command:
    cmd: aideinit -y -f
    creates: /var/lib/aide/aide.db

The three ways of making a command idempotent:

Option When
creates: <path> The command produces a file: if it exists, it does not run
removes: <path> It only runs if the file exists
changed_when: You decide, based on the output or the return code 
- name: Check whether any services need restarting
  ansible.builtin.command: needrestart -r l -p
  register: pending
  changed_when: false          # it never modifies anything: it is a query
  failed_when: pending.rc > 2

changed_when: false on queries is what keeps the PLAY RECAP clean. A playbook that always reports changed on five tasks loses its value as a drift detector, because the noise cannot be told from the signal.

Playbooks: handlers, conditions, loops and blocks

# site.yml
- name: Configure the Tramontana servers
  hosts: all
  become: true
  gather_facts: true

  tasks:
    # ---------- Identities ----------
    - name: Create the tramontana group
      ansible.builtin.group:
        name: "{{ tramontana_group }}"
        gid: "{{ tramontana_gid }}"
        state: present
      tags: [users]

    - name: Create the service account
      ansible.builtin.user:
        name: "{{ tramontana_service_user }}"
        uid: "{{ tramontana_service_uid }}"
        group: "{{ tramontana_group }}"
        system: true
        shell: /usr/sbin/nologin
        create_home: false
        state: present
      tags: [users]

    # ---------- Paths: a loop over the list from group_vars ----------
    - name: Create the directory tree with its permissions
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: directory
        mode: "{{ item.mode }}"
        owner: "{{ item.owner }}"
        group: "{{ item.group }}"
      loop: "{{ tramontana_paths }}"
      loop_control:
        label: "{{ item.path }}"       # readable output instead of the whole dict
      tags: [paths]

    # ---------- Packages ----------
    - name: Install the base packages
      ansible.builtin.apt:
        name: "{{ base_packages + (extra_packages | default([])) }}"
        state: present
        update_cache: true
        cache_valid_time: 3600
      tags: [packages]

    - name: Pin the PostgreSQL version
      ansible.builtin.dpkg_selections:
        name: postgresql-16
        selection: hold
      tags: [packages]

    # ---------- sysctl ----------
    - name: Apply the kernel security parameters
      ansible.posix.sysctl:
        name: "{{ item.key }}"
        value: "{{ item.value }}"
        sysctl_file: /etc/sysctl.d/60-hardening.conf
        sysctl_set: true
        reload: true
      loop: "{{ sysctl_security | dict2items }}"
      loop_control:
        label: "{{ item.key }}"
      tags: [kernel, security]

    - name: Apply the performance parameters
      ansible.posix.sysctl:
        name: "{{ item.key }}"
        value: "{{ item.value }}"
        sysctl_file: /etc/sysctl.d/70-performance.conf
        sysctl_set: true
        reload: true
      loop: "{{ sysctl_performance | dict2items }}"
      loop_control:
        label: "{{ item.key }}"
      tags: [kernel]

    # ---------- Configuration from a template ----------
    - name: Generate app.conf from the template
      ansible.builtin.template:
        src: templates/app.conf.j2
        dest: /etc/tramontana/app.conf
        owner: root
        group: "{{ tramontana_group }}"
        mode: '0640'
        backup: true                 # leaves a dated copy, like .bak-
        validate: "grep -q '^db_host=' %s"
      notify: Restart tramontana
      tags: [config]

    # ---------- SSH, with the safety net from 06-02 ----------
    - name: Harden the SSH configuration
      ansible.builtin.template:
        src: templates/sshd_tramontana.conf.j2
        dest: /etc/ssh/sshd_config.d/60-tramontana.conf
        owner: root
        group: root
        mode: '0600'
        # It is NOT applied if the syntax is wrong: this stops you locking yourself out
        validate: /usr/sbin/sshd -t -f %s
      notify: Reload ssh
      tags: [ssh, security]

    # ---------- Firewall, in the RIGHT order ----------
    - name: Default policy of denying incoming traffic
      community.general.ufw:
        direction: incoming
        policy: deny
      tags: [firewall]

    - name: Allow SSH BEFORE enabling the firewall
      community.general.ufw:
        rule: limit
        port: '22'
        proto: tcp
        comment: 'SSH with rate limiting'
      tags: [firewall]

    - name: Allow PostgreSQL only from the internal network
      community.general.ufw:
        rule: allow
        port: '5432'
        proto: tcp
        src: "{{ tramontana_internal_network }}"
      tags: [firewall]

    - name: Enable the firewall
      community.general.ufw:
        state: enabled
      tags: [firewall]

    # ---------- Service ----------
    - name: Install the systemd unit
      ansible.builtin.template:
        src: templates/tramontana.service.j2
        dest: /etc/systemd/system/tramontana.service
        owner: root
        group: root
        mode: '0644'
      notify:
        - Reload systemd
        - Restart tramontana
      tags: [service]

    - name: Enable the service
      ansible.builtin.systemd_service:
        name: tramontana.service
        enabled: true
        state: started
        daemon_reload: true
      tags: [service]

    # ---------- A block with error handling ----------
    - name: Initialise AIDE
      block:
        - name: Generate the integrity database
          ansible.builtin.command:
            cmd: aideinit -y -f
            creates: /var/lib/aide/aide.db.new
          register: aide_init

        - name: Put the database in place
          ansible.builtin.copy:
            src: /var/lib/aide/aide.db.new
            dest: /var/lib/aide/aide.db
            remote_src: true
            mode: '0600'
          when: aide_init.changed
      rescue:
        - name: Warn that AIDE could not be initialised
          ansible.builtin.debug:
            msg: "AIDE was not initialised. Check by hand; it does not block the rest."
      always:
        - name: Record the attempt
          ansible.builtin.lineinfile:
            path: /var/log/tramontana/ansible.log
            line: "{{ ansible_date_time.iso8601 }} AIDE: {{ aide_init.rc | default('not run') }}"
            create: true
            mode: '0640'
            owner: "{{ tramontana_service_user }}"
            group: adm
      tags: [security, aide]

  handlers:
    - name: Reload systemd
      ansible.builtin.systemd_service:
        daemon_reload: true

    - name: Restart tramontana
      ansible.builtin.systemd_service:
        name: tramontana.service
        state: restarted

    - name: Reload ssh
      ansible.builtin.systemd_service:
        name: ssh.service
        state: reloaded

The mechanisms that appear there and deserve an explanation:

notify and handlers. A handler runs only if the task that notifies it reported changed, and only once, at the end of the play, even if five tasks notify it. That is exactly what you want: if app.conf and the unit both change, the service restarts once, not twice. And if nothing changes, it does not restart at all.

validate. The module writes the file to a temporary location, runs the validation command with %s replaced by that path, and only if it succeeds does it put the file in place. On the SSH task this is the safety net from 06-02, automated: an sshd_config with a syntax error never gets installed, so you do not lock yourself out.

backup: true. It leaves a dated copy before overwriting. It is the course's .bak-$(date +%F) convention, applied by the tool.

The order of the ufw tasks. Allow SSH before enabling the firewall, just as in 06-03. In a playbook the order is explicit and documented, which is an advantage over a prose runbook where that detail can be overlooked.

block/rescue/always. The equivalent of the trap from 04-06: rescue runs if something in the block fails, always runs regardless. Here it stops an AIDE failure from aborting the whole configuration.

Jinja2 templates

This is where configuration management stops being "copying files" and becomes useful:

{# templates/app.conf.j2 #}
# FILE GENERATED BY ANSIBLE — DO NOT EDIT BY HAND
# Any manual change will be lost on the next run.
# Source: {{ template_path | default('templates/app.conf.j2') }}
# Environment: {{ env_name }}
# Generated: {{ ansible_date_time.iso8601 }}

db_host=127.0.0.1
db_port=5432
db_name=tramontana_bookings
max_connections={{ tramontana_max_connections }}
query_timeout={{ tramontana_timeout | default(30) }}
log_level={{ tramontana_log_level }}
listen=127.0.0.1
port={{ tramontana_port }}

{% if env_name == 'test' %}
# Test only: detailed tracing and synthetic data
sql_trace=true
synthetic_data=true
{% endif %}

{% for house in houses | default([]) %}
active_house={{ house }}
{% endfor %}
$ ansible-playbook site.yml --tags config --diff --limit srv-tramontana-test

TASK [Generate app.conf from the template] ************************************
--- before: /etc/tramontana/app.conf
+++ after: /etc/tramontana/app.conf
@@ -1,10 +1,14 @@
-# FILE GENERATED BY ANSIBLE — DO NOT EDIT BY HAND
-# Environment: test
-max_connections=20
+# FILE GENERATED BY ANSIBLE — DO NOT EDIT BY HAND
+# Environment: test
+max_connections=20
+sql_trace=true
+synthetic_data=true
changed: [srv-tramontana-test]

That "do not edit by hand" header is not decorative: without it, somebody will edit the file, it will work for a month, and the next run of the playbook will revert it with no warning. With it, at least they know what happened.

And one more template, the systemd unit, which shows the value of generating configuration from variables:

{# templates/tramontana.service.j2 #}
# GENERATED BY ANSIBLE — DO NOT EDIT
[Unit]
Description=Tramontana Bookings ({{ env_name }})
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=exec
User={{ tramontana_service_user }}
Group={{ tramontana_group }}
ExecStart=/opt/tramontana/app/tramontana --config /etc/tramontana/app.conf
Restart=on-failure
RestartSec=5s
LoadCredentialEncrypted=db_password:/etc/tramontana/secrets/db_password.cred

# Hardening (06-06). Measured exposure: 1.6 OK
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @obsolete
SystemCallFilter=getrandom
CapabilityBoundingSet=
ReadWritePaths=/var/log/tramontana /opt/tramontana/shared/uploads

MemoryMax={{ tramontana_memory_max | default('512M') }}
TasksMax={{ tramontana_tasks_max | default(64) }}
LimitNOFILE=8192

[Install]
WantedBy=multi-user.target

Note SystemCallFilter=getrandom: it is the exception you discovered while debugging the SIGSYS in the 06-06 exercise. Now it is in the code, with the complete unit, instead of in a drop-in somebody would have to remember.

Roles: organising for reuse

A three-hundred-line playbook is unmanageable. Roles split it into pieces with a fixed structure:

$ ansible-galaxy init roles/tramontana
$ tree roles/ -L 2
roles/
├── common
│   ├── defaults/main.yml      # default variables (low precedence)
│   ├── files/                 # files to copy verbatim
│   ├── handlers/main.yml
│   ├── meta/main.yml          # dependencies on other roles
│   ├── tasks/main.yml         # the tasks
│   ├── templates/             # Jinja2 templates
│   └── vars/main.yml          # role variables (high precedence)
├── security
└── tramontana
# site.yml, refactored
- name: Configure the Tramontana servers
  hosts: all
  become: true
  roles:
    - role: common        # users, packages, sysctl, time zone
    - role: security      # ssh, ufw, fail2ban, auditd, aide, pam, apparmor
    - role: tramontana    # paths, app.conf, unit, timers, logrotate

Three lines, and each role is reusable. The advantage shows up when there is a second server: common and security apply as they are, and only the third one changes.

# roles/security/meta/main.yml
dependencies:
  - role: common

And Ansible Galaxy, so as not to rewrite what has already been written:

$ ansible-galaxy collection install community.general ansible.posix
$ ansible-galaxy role install geerlingguy.postgresql

With the same warning as in 05-03 about third-party repositories: installing a Galaxy role means running somebody else's code with root privileges on your server. Review it first, and pin the version.

Ansible Vault and secrets

Secrets cannot sit in the clear in a repository. Vault encrypts them inside the repository itself:

$ ansible-vault create group_vars/all/secrets.yml
New Vault password:
Confirm New Vault password:
# Contents (decrypted)
vault_db_password: "Zx9K2pQ7vLm4RtWn"
vault_restic_password: "R3st1c-Tr4m0nt4n4-2026"
vault_aide_key: "..."
$ head -3 group_vars/all/secrets.yml
$ANSIBLE_VAULT;1.1;AES256
36613264313632373933353436646138613831306638386566613864333331373766373530353137
6134663137363134333661386133343338326238623166390a3833326566...

$ ansible-vault view group_vars/all/secrets.yml
$ ansible-vault edit group_vars/all/secrets.yml
$ ansible-vault rekey group_vars/all/secrets.yml    # change the password

Encrypting a single value, which lets the rest of the file stay readable in a diff:

$ ansible-vault encrypt_string 'Zx9K2pQ7vLm4RtWn' --name 'vault_db_password'
vault_db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          36613264313632373933353436646138613831306638386566613864333331...

And the integration with the pass from 06-05, which avoids having two sources of truth for the secrets:

$ cat ~/.ansible-vault-pass.sh
#!/usr/bin/env bash
# Fetches the Vault password from pass, so as not to keep it in a file.
set -euo pipefail
pass tramontana/ansible/vault
$ chmod 700 ~/.ansible-vault-pass.sh
# ansible.cfg
[defaults]
vault_password_file = ~/.ansible-vault-pass.sh
$ ansible-playbook site.yml    # it no longer asks for the password

That way pass stays the source of truth for every secret, with its git history and its copy off the server, and Vault is only the transport mechanism into the playbook.

The vault_* naming convention for encrypted variables, and referencing them from ordinary variables, lets you see at a glance what is secret:

# group_vars/all/main.yml
tramontana_db_password: "{{ vault_db_password }}"

And a warning: a Vault variable used in a template ends up in the clear on the server. Vault protects the repository, not the destination. The database credential still goes through systemd-creds as in 06-05:

- name: Install the encrypted database credential
  ansible.builtin.shell:
    cmd: |
      set -o pipefail
      printf '%s' '{{ tramontana_db_password }}' \
        | systemd-creds encrypt --name=db_password - \
              /etc/tramontana/secrets/db_password.cred
    creates: /etc/tramontana/secrets/db_password.cred
  no_log: true          # <- ESSENTIAL: without this, it appears in the output
  notify: Restart tramontana

no_log: true is mandatory on any task that handles a secret. Without it, Ansible prints the complete command — password included — in the output and in any log that gets kept. It is the same principle as in 06-05: a secret on the command line is an exposed secret.

Running it: --check, --diff, --tags and ansible-lint

# --check: SIMULATION MODE. It is the course's --dry-run, applied to everything.
$ ansible-playbook site.yml --check --diff --limit srv-tramontana-test

TASK [common : Install the base packages] *************************************
changed: [srv-tramontana-test]

TASK [tramontana : Generate app.conf from the template] ***********************
--- before
+++ after
@@ -5,7 +5,7 @@
-max_connections=200
+max_connections=20
changed: [srv-tramontana-test]

PLAY RECAP ********************************************************************
srv-tramontana-test : ok=44  changed=7  unreachable=0  failed=0

--check --diff together are the most valuable tool in day-to-day work: they say exactly what would change and which lines of each file, without touching anything. It is the course's convention — simulate before acting — applied to the server's entire configuration.

With one honest limitation: in --check mode, tasks that depend on the result of an earlier one can fail or report incorrectly, because the earlier one was not really run. A command with creates on a file that would be created by a previous task will give a false positive.

# Limiting the scope
$ ansible-playbook site.yml --limit srv-tramontana-test
$ ansible-playbook site.yml --tags firewall,ssh
$ ansible-playbook site.yml --skip-tags packages
$ ansible-playbook site.yml --start-at-task "Install the systemd unit"

# Increasing detail
$ ansible-playbook site.yml -v      # results
$ ansible-playbook site.yml -vvv    # + SSH connection and modules

And static analysis, which is to Ansible what shellcheck is to Bash:

$ ansible-lint
WARNING  Listing 3 violation(s) that are fatal
risky-file-permissions: File permissions unset or incorrect
roles/tramontana/tasks/main.yml:24 Task/Handler: Copy the backup script

command-instead-of-module: apt-mark used in place of dpkg_selections module
roles/common/tasks/main.yml:41 Task/Handler: Pin the postgresql version

no-changed-when: Commands should not change things if nothing needs doing
roles/security/tasks/main.yml:88 Task/Handler: Check for pending restarts

$ ansible-lint --write     # fixes automatically what it can

All three warnings are real and of the kind that matters: undeclared permissions — which would leave the file with the default umask — a command where there is an idempotent module, and a query with no changed_when: false that dirties the PLAY RECAP. ansible-lint in git's pre-commit hook is how you stop them slipping through.

The Tramontana case: rebuilding the server and measuring the RTO

The module's goal: making the rebuild of srv-tramontana from scratch an operation of code, not of memory.

The experiment

# 1. A clean machine, with nothing configured
$ virsh destroy srv-tramontana-test 2>/dev/null
$ virsh undefine srv-tramontana-test --remove-all-storage
$ cd /var/lib/libvirt/images
$ sudo qemu-img create -f qcow2 -F qcow2 \
      -b noble-server-cloudimg-amd64.img test.qcow2 20G
$ sudo virt-install --name srv-tramontana-test \
      --memory 2048 --vcpus 2 --cpu host-passthrough \
      --disk path=/var/lib/libvirt/images/test.qcow2,bus=virtio \
      --disk path=/var/lib/libvirt/images/seed.iso,device=cdrom \
      --network network=default,model=virtio --os-variant ubuntu24.04 \
      --graphics none --import --noautoconsole

# The minimal cloud-init: only the user, the SSH key and python3. Nothing else.
# Everything else is Ansible's job.
$ sleep 90 && ansible srv-tramontana-test -m ping
srv-tramontana-test | SUCCESS => {"changed": false, "ping": "pong"}

The rebuild, timed

$ time ansible-playbook site.yml --limit srv-tramontana-test

PLAY [Configure the Tramontana servers] ***************************************

TASK [common : Create the tramontana group] ***********************************
changed: [srv-tramontana-test]
...
TASK [tramontana : Enable the backup timer] ***********************************
changed: [srv-tramontana-test]

RUNNING HANDLER [Reload systemd] **********************************************
changed: [srv-tramontana-test]

RUNNING HANDLER [Restart tramontana] ******************************************
changed: [srv-tramontana-test]

PLAY RECAP ********************************************************************
srv-tramontana-test : ok=118  changed=94  unreachable=0  failed=0  skipped=6

real	6m41.882s
user	0m48.204s
sys	0m12.118s

The verification

A playbook that finishes without error does not mean the server works. It has to be checked with the same criteria as always:

$ ansible srv-tramontana-test -m shell -a '~/scripts/health_check.sh; echo "status: $?"'
srv-tramontana-test | CHANGED | rc=0 >>
status: 0

$ ansible srv-tramontana-test -b -m shell -a 'systemd-analyze security tramontana.service | tail -1'
→ Overall exposure level for tramontana.service: 1.6 OK 🙂

$ ansible srv-tramontana-test -b -m shell -a 'ufw status verbose | head -6'
$ ansible srv-tramontana-test -b -m shell -a 'sshd -T | grep -E "permitrootlogin|passwordauth"'
permitrootlogin no
passwordauthentication no

$ ansible srv-tramontana-test -b -m shell -a 'sysctl -n kernel.yama.ptrace_scope net.ipv4.tcp_congestion_control'
1
bbr

$ ansible srv-tramontana-test -b -m shell -a 'aa-status --enabled && echo "apparmor active"'
apparmor active

Exposure 1.6, SSH hardened, sysctl applied, AppArmor active. Identical to production, because it comes out of the same code.

And the second run

$ ansible-playbook site.yml --limit srv-tramontana-test

PLAY RECAP ********************************************************************
srv-tramontana-test : ok=118  changed=0   unreachable=0  failed=0  skipped=6

real	1m14.402s

changed=0: the playbook really is idempotent. And that 74-second second run is now also a drift detector that can be launched weekly.

The RTO

Phase Before (manual) With Ansible
Provisioning the machine 20-30 min 2 min (cloud-init)
Configuring the system 6-7 hours 7 min
Restoring the data 30-45 min 30-45 min (restic)
Verifying 30 min 5 min (automated checks)
Total ~8 hours ~50 minutes

From 8 hours to under one. And the improvement is not only the time: the manual rebuild is error-prone and depends on one specific person; the automated one always produces the same result and can be launched by anybody on the team with access to the repository.

That is what turns the rule from 06-04 into a procedure that can actually be followed: "a compromised server is reinstalled, not cleaned" stops being an uncomfortable piece of advice when reinstalling costs fifty minutes. And it is worth updating the runbook and talking to Marta: the agreed RTO can come down from 8 hours to 2, with room to spare.

What is left out, and it has to be said

Not automated Why
The pass GPG key It is the master key: it is restored by hand from its physical custody
The Let's Encrypt certificate It is reissued with certbot; there is no point in copying it
The AIDE database It must be generated on the rebuilt machine, not copied
The data (restic) It is a separate process, with its own verification
The decision to rebuild It is a human one

Common Mistakes and Tips

  • Using command or shell where there is a module. It loses idempotence and state checking. ansible-lint detects it.
  • Forgetting changed_when: false on queries. The playbook always reports changed and loses its value as a drift detector.
  • Forgetting no_log: true on a task with secrets. The secret appears in the output and in any log that is kept.
  • Disabling host_key_checking. It is what many tutorials recommend, and it removes the man-in-the-middle protection from 06-02.
  • Editing a template-generated file by hand. It gets reverted on the next run. Hence the "DO NOT EDIT BY HAND" header.
  • Not using validate on critical configurations. An sshd_config with a syntax error locks you out. validate: /usr/sbin/sshd -t -f %s prevents it.
  • Enabling ufw before allowing SSH. The same mistake as in 06-03, now in code. The order of the tasks matters.
  • Running against production without testing on the test machine. That is what --limit is for, and what srv-tramontana-test exists for.
  • Trusting --check blindly. Tasks that depend on an earlier one can give false positives because the earlier one was not run.
  • Installing Galaxy roles without reviewing them. It means running somebody else's code as root on your server. Review it and pin the version.
  • Believing that a playbook with no errors means a working server. Always verify with real checks: health_check.sh, sshd -T, systemd-analyze security.
  • A tip on method. The playbook is executable documentation: the only kind that cannot go out of date, because if it differs from reality the changed tells you. Treat it like code: git, change review, ansible-lint in pre-commit, and tested on the test machine before production.

Exercises

Exercise 1

Write the security role that applies the hardening from 06-06: PAM with pam_pwquality and pam_faillock, the security sysctl settings, the noexec mounts, and the blocked kernel modules. Pay particular attention to not leaving the machine inaccessible, and explain every precaution.

Exercise 2

ansible-playbook site.yml --check against production reports changed=5, when it should be 0. Design the procedure for investigating where that drift comes from and deciding what to do in each case.

Exercise 3

Marta asks whether Ansible can bring the agreed 8-hour RTO down. Write the answer with the new number, what is still not automated, and what it would take to go lower.

Solutions

Solution 1

# roles/security/defaults/main.yml
security_pam_minlen: 12
security_pam_minclass: 3
security_faillock_deny: 5
security_faillock_unlock: 600
security_noexec_mounts: true

security_blocked_modules:
  - cramfs
  - freevxfs
  - jffs2
  - hfs
  - hfsplus
  - udf
  - dccp
  - sctp
  - rds
  - tipc
# roles/security/tasks/main.yml
---
# =====================================================================
# GENERAL WARNING: this role can leave the machine inaccessible.
# It is ALWAYS tested against srv-tramontana-test before production.
# =====================================================================

# ---------- Security sysctl ----------
# Low risk: an invalid value fails in the task, not at boot.
- name: Apply the kernel security parameters
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/60-hardening.conf
    sysctl_set: true
    state: present
    reload: true
  loop: "{{ sysctl_security | dict2items }}"
  loop_control:
    label: "{{ item.key }}"
  tags: [kernel]

# ---------- Blocked modules ----------
- name: Block unused kernel modules
  ansible.builtin.template:
    src: blacklist-tramontana.conf.j2
    dest: /etc/modprobe.d/blacklist-tramontana.conf
    owner: root
    group: root
    mode: '0644'
  notify: Regenerate initramfs      # <- ESSENTIAL (the lesson from 07-01)
  tags: [modules]

# ---------- PAM: THE DANGEROUS PART ----------
# A syntax error in PAM prevents EVERY login, the console included.
# The precautions, in order:

- name: "PAM | Precaution 1: check that there is alternative access"
  ansible.builtin.assert:
    that:
      - ansible_connection != 'local'
      - inventory_hostname in groups['test_servers'] or pam_confirmed | default(false)
    fail_msg: >-
      Configuring PAM in production requires -e pam_confirmed=true and a root
      session open in another terminal. Test on the test machine first.
  tags: [pam]

- name: "PAM | Precaution 2: dated backup"
  ansible.builtin.copy:
    src: "{{ item }}"
    dest: "{{ item }}.bak-{{ ansible_date_time.date }}"
    remote_src: true
    mode: preserve
    force: false            # does not overwrite an earlier copy from the same day
  loop:
    - /etc/pam.d/common-auth
    - /etc/pam.d/common-password
  tags: [pam]

- name: "PAM | Install libpam-pwquality"
  ansible.builtin.apt:
    name: libpam-pwquality
    state: present
  tags: [pam]

- name: "PAM | Password quality policy"
  ansible.builtin.template:
    src: pwquality.conf.j2
    dest: /etc/security/pwquality.conf
    owner: root
    group: root
    mode: '0644'
  tags: [pam]

- name: "PAM | faillock configuration"
  ansible.builtin.template:
    src: faillock.conf.j2
    dest: /etc/security/faillock.conf
    owner: root
    group: root
    mode: '0644'
  tags: [pam]

# The PAM stack is touched with pam_auth_update, which validates and keeps
# things consistent, rather than editing common-auth by hand with lineinfile.
- name: "PAM | Enable the modules through pam-auth-update"
  ansible.builtin.command:
    cmd: pam-auth-update --enable pwquality faillock
  register: pam_update
  changed_when: "'Nothing to do' not in pam_update.stdout"
  tags: [pam]

# ---------- Precaution 3: VERIFY that authentication still works ----------
# If this fails, it has to be reverted IMMEDIATELY using the session that is
# still open.
- name: "PAM | Verify that authentication still works"
  ansible.builtin.command:
    cmd: "runuser -u {{ ansible_user }} -- /usr/bin/true"
  changed_when: false
  register: pam_verification
  failed_when: pam_verification.rc != 0
  tags: [pam]

- name: "PAM | Verify that sudo still works"
  ansible.builtin.command:
    cmd: sudo -n -l
  become: false
  changed_when: false
  tags: [pam]

# ---------- Safe mounts: the OTHER dangerous part ----------
# A badly written fstab prevents the machine from booting (07-01). Hence:
# a backup, a validated template, and mount -a BEFORE calling the task done.

- name: "Mounts | Back up fstab"
  ansible.builtin.copy:
    src: /etc/fstab
    dest: "/etc/fstab.bak-{{ ansible_date_time.date }}"
    remote_src: true
    mode: preserve
    force: false
  when: security_noexec_mounts
  tags: [mounts]

- name: "Mounts | Check live that noexec breaks nothing"
  block:
    - name: "Mounts | Remount /tmp with noexec temporarily"
      ansible.posix.mount:
        path: /tmp
        state: remounted
        opts: defaults,noatime,noexec,nosuid,nodev
        fstype: ext4
        src: "{{ ansible_mounts | selectattr('mount','equalto','/tmp')
                 | map(attribute='device') | first }}"

    - name: "Mounts | Check that apt still works"
      ansible.builtin.apt:
        name: tree
        state: present
        force_apt_get: true
      changed_when: false

    - name: "Mounts | Check that deployment still works"
      ansible.builtin.command:
        cmd: /home/operator/scripts/deploy.sh --dry-run {{ tramontana_version }}
      changed_when: false
      become_user: operator
  rescue:
    - name: "Mounts | Revert: noexec breaks something"
      ansible.posix.mount:
        path: /tmp
        state: remounted
        opts: defaults,noatime
        fstype: ext4
        src: "{{ ansible_mounts | selectattr('mount','equalto','/tmp')
                 | map(attribute='device') | first }}"

    - name: "Mounts | Abort the mount hardening"
      ansible.builtin.fail:
        msg: "noexec on /tmp breaks apt or the deployment. Investigate before persisting."
  when: security_noexec_mounts
  tags: [mounts]

- name: "Mounts | Persist in fstab (only if the live test passed)"
  ansible.posix.mount:
    path: "{{ item.path }}"
    src: "{{ item.source }}"
    fstype: "{{ item.type }}"
    opts: "{{ item.options }}"
    state: mounted
  loop:
    - { path: /var/tmp,  source: /tmp,  type: none,  options: 'bind,noexec,nosuid,nodev' }
    - { path: /dev/shm,  source: tmpfs, type: tmpfs, options: 'defaults,noexec,nosuid,nodev' }
  loop_control:
    label: "{{ item.path }}"
  when: security_noexec_mounts
  tags: [mounts]

- name: "Mounts | VERIFY fstab before anybody reboots"
  ansible.builtin.command:
    cmd: mount -a
  changed_when: false
  tags: [mounts]

# ---------- Final check ----------
- name: "Verify that the service is still up after the hardening"
  ansible.builtin.command:
    cmd: /home/operator/scripts/health_check.sh
  become_user: operator
  changed_when: false
  register: health
  failed_when: health.rc != 0
  tags: [verification]
# roles/security/handlers/main.yml
- name: Regenerate initramfs
  ansible.builtin.command:
    cmd: update-initramfs -u -k all

The precautions, which is what the exercise asks for:

Precaution What it protects against
An assert requiring pam_confirmed=true in production Running the PAM role against production by accident, with no rescue session open
force: false on the backups Overwriting a good copy with an already-broken one if it is run twice on the same day
pam-auth-update instead of lineinfile Editing common-auth by hand is the classic way of breaking the stack order. The tool validates and keeps things consistent
Verification with runuser and sudo -n -l Detecting that authentication has broken while the connection is still open, which is the only window in which to fix it
block/rescue on the mounts noexec on /tmp breaks installers; the rescue reverts to the previous state instead of leaving the machine half done
A live test before touching fstab A broken fstab prevents booting (07-01). It is tested with remounted, which is reversible
mount -a at the end The safety net from 05-04 and 07-01, now automated
notify: Regenerate initramfs Touching modprobe.d without regenerating the initramfs produces the (initramfs) prompt
health_check.sh at the end Hardening and breaking the service is not hardening

And the precaution that is not in the code and has to be written in the runbook: before running this role against production, virsh snapshot-create-as or the equivalent. The code minimises the risk; it does not eliminate it.

Solution 2

changed=5 in --check means the real server differs from the code. There are three possible causes and they lead to different decisions, so the first job is to identify which is which.

# Step 1: WHAT exactly would change. --diff gives the specific lines.
$ ansible-playbook site.yml --check --diff --limit srv-tramontana \
    2>&1 | tee /tmp/drift-$(date +%F).txt

$ grep -E '^(TASK|changed:)' /tmp/drift-$(date +%F).txt | grep -B1 changed
TASK [common : Install the base packages]
changed: [srv-tramontana]
TASK [tramontana : Generate app.conf from the template]
changed: [srv-tramontana]
TASK [security : Apply the kernel security parameters]
changed: [srv-tramontana]
TASK [security | faillock configuration]
changed: [srv-tramontana]
TASK [tramontana : Install the systemd unit]
changed: [srv-tramontana]
# Step 2: WHEN it changed and WHO changed it. This is where the Module 6 work pays off.
$ ansible srv-tramontana -b -m shell -a 'aide --check 2>&1 | head -30'
$ ansible srv-tramontana -b -m shell -a 'ausearch -k tramontana_conf -ts recent -i | tail -20'
$ ansible srv-tramontana -b -m shell -a 'journalctl --since "7 days ago" | grep -E "sudo:.*COMMAND" | tail -20'
$ ansible srv-tramontana -b -m shell -a 'grep -E "install|upgrade" /var/log/apt/history.log | tail -10'
$ ansible srv-tramontana -b -m shell -a 'ls -la /etc/tramontana/*.bak-* /etc/systemd/system/tramontana.service.d/ 2>/dev/null'

The three causes and how to treat them, which is the heart of the exercise:

Cause How you recognise it What to do
A. A legitimate manual change never taken back into the code auditd shows a sudo from somebody on the team, there is a dated .bak-, and the change makes sense Take the change into the code, do not revert it
B. Unauthorised drift Nobody recognises it, there is no trace in auditd, or the auid is unexpected Investigate as an incident (06-04) before touching anything
C. The code is out of date The server is fine and the playbook reflects an old state Fix the playbook

Applying that to the five changes in this case:

# --- Change 1: packages ---
$ grep -A2 'Install the base packages' /tmp/drift-*.txt
# The diff shows it would install 'sysstat', which is already installed...
$ ansible srv-tramontana -b -m shell -a 'dpkg -l sysstat | tail -1'
ii  sysstat  12.6.1-2  amd64  system performance tools

A --check false positive: apt with update_cache cannot check the real state without updating the index, so it conservatively reports changed. This is not drift. It is confirmed by running without --check on the test machine and seeing it report ok.

# --- Change 2: app.conf ---
--- before: /etc/tramontana/app.conf
+++ after: /etc/tramontana/app.conf
@@ -5,7 +5,7 @@
-query_timeout=45
+query_timeout=30

$ ansible srv-tramontana -b -m shell -a 'ausearch -k tramontana_conf -i | grep -A2 "success=yes" | tail -6'
type=SYSCALL ... auid=luis uid=root euid=root comm="vim" key=tramontana_conf

Cause A. Luis raised the timeout to 45 s to diagnose some errors. A legitimate change, but an undocumented one. The decision: do not revert it silently. You talk to Luis, decide whether 45 is right, and if it is it goes into group_vars:

# group_vars/web_servers.yml
tramontana_timeout: 45   # raised on 2026-08-15 because of slow queries (see #142)
# --- Change 3: security sysctl ---
-kernel.yama.ptrace_scope = 0
+kernel.yama.ptrace_scope = 1

Serious. Somebody lowered ptrace_scope, exactly what was advised against in 07-02, and that allows the memory of other processes belonging to the same user to be read — the database credential included.

$ ansible srv-tramontana -b -m shell -a 'grep -rn ptrace /etc/sysctl.d/'
$ ansible srv-tramontana -b -m shell -a 'ausearch -k privileges -ts recent -i | tail'

If there is a trace of somebody on the team, it is cause A with a decision that has to be reverted and explained. If there is no trace, it is cause B: it is handled as an incident under the 06-04 procedure, and the db_password is considered compromised and rotated.

# --- Change 4: faillock ---
-deny = 3
+deny = 5

Cause C, most likely: somebody adjusted the value after several accidental lockouts and it was a reasonable decision. It is confirmed and the code is corrected, since that is what was out of date.

# --- Change 5: the systemd unit ---
$ ansible srv-tramontana -b -m shell -a 'ls -la /etc/systemd/system/tramontana.service.d/'
-rw-r--r-- 1 root root 214 Aug 18 16:12 hardening.conf

There is a drop-in the playbook does not know about. The playbook generates the complete unit, and the drop-in modifies it: the two coexist and the effective result is not the one in the code. Cause A, and the model has to be decided: either the playbook generates the drop-in too, or its directives are folded into the unit template. The second is cleaner, and in fact the template already includes them — the drop-in is a leftover from the manual work in 06-06 and has to be removed:

- name: Remove manual drop-ins already folded into the template
  ansible.builtin.file:
    path: /etc/systemd/system/tramontana.service.d/hardening.conf
    state: absent
  notify:
    - Reload systemd
    - Restart tramontana

The procedure, generalised:

  1. --check --diff, saving the output with a date.
  2. For each change, correlate it with aide --check, ausearch, journalctl and /var/log/apt/history.log.
  3. Classify it as A, B or C.
  4. A: take it into the code after confirming with whoever made it. B: incident procedure, touch nothing yet. C: fix the code.
  5. Re-run until changed=0.
  6. Record in the change log what each one was.

And the improvement that stops it happening again: automate the detection with a weekly timer that runs --check and alerts only if there are changes, following the silence-if-all-is-well principle:

$ cat ~/tramontana-infra/check_drift.sh
#!/usr/bin/env bash
# Detects configuration drift. Silence if there is none.
set -euo pipefail
output="$(mktemp)"; trap 'rm -f "$output"' EXIT

ansible-playbook site.yml --check --diff --limit srv-tramontana >"$output" 2>&1 || true
changes="$(grep -oP 'changed=\K[0-9]+' "$output" | head -1)"

if [[ "${changes:-0}" -gt 0 ]]; then
    mail -s "[srv-tramontana] Configuration drift: ${changes} changes" \
        [email protected] <"$output"
    exit 1
fi

Note that this turns the playbook into a third integrity check, complementary to AIDE (which watches files) and auditd (which watches accesses): this one watches the logical state of the configuration, which neither of the other two sees.

Solution 3

RTO review following the automation of the configuration To: Marta Vidal · From: Systems Operations · 18 August 2026

Short answer: yes. I propose bringing the agreed RTO down from 8 hours to 2 hours, with a comfortable margin over the measured time.

What has changed. Until now, the configuration of srv-tramontana existed in two places: on the server itself and in my head, with a prose runbook as backup. Rebuilding it meant following several hundred steps by hand — users, permissions, packages, disks, services, firewall, hardening, logs, backups — in the right order and without missing any.

As of this week, all of that configuration is written as versioned code, reviewable and executable. Rebuilding the server consists of running one command.

The measurement. I have rebuilt the server from scratch in the test environment, timing it:

Phase Before (manual) Now (automated)
Creating the machine 20-30 min 2 min
Configuring the complete system 6-7 hours 7 min
Restoring the data 30-45 min 30-45 min
Verifying that everything works 30 min 5 min
Total ~8 hours ~50 minutes

And I have checked that the result is identical to production using the same objective measures we normally use: the service's security score, the effective SSH configuration, the system parameters and the state of the firewall.

Why I propose 2 hours and not 1. The measured time is 50 minutes under laboratory conditions. A real incident adds things that cannot be measured in advance:

  • Decision time: noticing, diagnosing and deciding to rebuild.
  • Hardware or the provider: waiting for a machine to be available.
  • The unexpected: a restore that fails on the first attempt, a network problem.
  • Communication and verification with the team.

A commitment of 2 hours leaves us more than double the measured time as a cushion. Committing to 1 hour would be tight, and an RTO that cannot be met is worse than a conservative one.

What is still not automated, and is on the record:

Item Why Time
The master secrets key It is the key to everything else; it is restored by hand from its physical custody, deliberately 5 min
The website certificate It is reissued; there is no point in copying it 2 min
The integrity database It has to be generated on the new machine, not copied Included
The bookings data It is a separate process, with its own verification 30-45 min
The decision to rebuild It is human, and it should be Variable

Additional benefits, beyond the time. Three that seem to me as important as the RTO:

  1. We no longer depend on one person. Anybody on the team with access to the repository can rebuild the server. Before, if I was not available, the RTO was open-ended.
  2. The test environment is now identical to production, because it comes out of the same code. That means what we test there is representative, which we could not guarantee before.
  3. We detect unauthorised changes. By running the code in simulation mode we know whether the server differs from what it should be. I have scheduled it weekly, and it only alerts if there is something. In fact the first run already picked up five differences, four of them legitimate changes nobody had documented.

And something I want to highlight. A few weeks ago I explained to you that a compromised server should be reinstalled, not cleaned. It is the right recommendation and it was, honestly, hard to honour: nobody cheerfully decides to lose a day's work. With a 50-minute rebuild, that recommendation goes from being an uncomfortable piece of advice to being the obvious procedure. Automation is not only efficiency: it is what makes a security decision possible to follow.

What it would take to go lower. If at some point the target were an RTO under 30 minutes, the options would be:

Measure Effect Cost
A second server already configured and on standby Eliminates the rebuild phase Doubling the server
Continuous restoration to a standby server Also reduces the data time A server plus maintenance work
Two active servers sharing the load The failure of one does not interrupt the service Doubling, plus a load balancer

All three change the problem: we stop talking about recovery and start talking about redundancy. It is the matter I still owe you with numbers, and I think it is the natural conversation to have after this one.

Concrete proposal. Update the service level agreement to an RTO of 2 hours and an RPO of 4 hours (the latter unchanged), and schedule a full recovery rehearsal every six months to verify that the number holds. The first one in September.

Conclusion

The configuration of srv-tramontana no longer lives in your head. It lives in a git repository, written as desired state rather than as a sequence of instructions, and that difference is what makes the file read as a description of the server rather than as a procedure. You know why idempotence is the central property — it lets you run without fear, it detects drift and it makes periodic convergence possible — and why command and shell are the last resort: they break precisely that. You have inventories with per-group variables that make the differences between environments explicit and minimal, Jinja2 templates that generate app.conf and the systemd unit — with the getrandom exception you discovered debugging a SIGSYS, now in the code and not in anybody's memory — roles that organise the whole thing, and Vault integrated with the pass from 06-05 so that there are not two sources of truth for the secrets.

And you have the safeguards where they matter: validate: sshd -t -f %s prevents installing an SSH configuration that would lock you out, backup: true leaves the dated copy the course's convention calls for, the handlers restart the service only if something changed, the order of the ufw tasks allows SSH before enabling the firewall, and --check --diff is the course's --dry-run taken to the server's entire configuration. With the honesty of knowing that --check has false positives and that a playbook with no errors does not mean a working server: that is why every run ends by verifying with health_check.sh, sshd -T and systemd-analyze security.

The number that sums up the module is from eight hours to fifty minutes. And its consequence matters more than the number: the rule from 06-04 — "a compromised server is reinstalled, not cleaned" — stops being advice nobody wants to follow and becomes the obvious procedure. Automation is not only efficiency; it is what makes a security decision that used to be theoretical possible to follow. As a side effect, the playbook has become a third integrity check alongside AIDE and auditd: it watches the logical state of the configuration, which neither of the other two sees.

But notice what is still unsolved, and what the last three answers to Marta have been pointing at ever more clearly. You can rebuild the server in fifty minutes. You can test any change before applying it. You can detect an intrusion, encrypt the secrets and measure performance. And even so, if srv-tramontana goes down, Tramontana Bookings is down. A disk, a power supply, a network outage, a kernel that will not boot after an update: fifty minutes of interruption in the best case, and that only if somebody is awake to launch the playbook. All the work of seven modules rests on a single machine.

Lesson 07-07: High Availability and Load Balancing attacks that directly. You will learn the vocabulary precisely — availability and the "nines" with the minutes of downtime a year they really represent, vertical versus horizontal scaling, MTBF and MTTR and their relationship with the RPO and RTO you already have agreed — and you will see why state is the hard problem: replicating processes is easy, replicating data and sessions is not. You will set up a load balancer with HAProxy in front of two instances, with health checks that reuse the 0/1/2 codes from health_check.sh and connection draining so you can deploy without cutting anybody off; you will give high availability to the load balancer itself with keepalived and a floating virtual IP, avoiding the most common design error in the field; you will see PostgreSQL replication, why automatic failover is dangerous without quorum, and what split-brain is. And you will finish with the analysis Marta has been asking for over three lessons: what this architecture really costs, what an hour of downtime costs, and whether Tramontana needs it — because the professional answer is not always "yes".

◄ Previous
◄
Next ►
►

Linux Course: From Beginner to System Administrator

Module 1: Introduction to Linux

Module 2: Basic Linux Commands

Module 3: Advanced Command-Line Skills

Module 4: Shell Scripting

Module 5: System Administration

Module 6: Networking and Security

Module 7: Advanced Topics

Module 8: Practical Projects

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved