You have already used SSH dozens of times in this course: you log in to srv-tramontana from laptop-student, you copy files with scp and you synchronise with rsync. You have used it the way people use a lift, without wondering how it works. This lesson opens the box, because SSH is your server's front door and right now that door accepts passwords, allows root logins and has racked up 47 failed attempts from 203.0.113.44 that nobody has looked at.

By the end you will have ed25519 keys instead of passwords, an sshd_config hardened directive by directive, a ~/.ssh/config that does the heavy lifting for you, tunnels to reach Tramontana's database without exposing it to anybody, and the habit — which here is not optional — of never closing the door you are coming in through.

1. What SSH solves and why Telnet and FTP are dead
2. Architecture and the protocol's three phases
3. Server authentication: host keys and known_hosts
4. Key authentication: generating, installing and protecting
5. ssh-agent, agent forwarding and ProxyJump
6. The client configuration file
7. Hardening /etc/ssh/sshd_config
8. The safe procedure for not locking yourself out
9. Tunnels: local, remote and SOCKS
10. File transfer: scp, sftp and rsync
11. Persistent sessions with tmux
12. Auditing: the 47 attempts from 203.0.113.44

Before SSH, administering a remote machine meant Telnet: a protocol that sends the username, the password and every command in plain text over the network. Anybody with access to the same segment — or to the router in between — read the whole session. The same goes for FTP, rlogin and rsh.

SSH solves three problems at once, and it is worth telling them apart because each one is attacked differently:

The third is the most forgotten and the most important: an encrypted channel to the wrong attacker protects absolutely nothing. That is why this lesson devotes a whole section to known_hosts.

srv-tramontana runs the sshd daemon, managed by systemd. In Ubuntu 24.04 there is a detail that surprises a lot of people: the service is socket-activated.

ssh.socket listens on the port and starts ssh.service when a connection arrives. The practical consequence is important and we come back to it in section 7: the Port directive in sshd_config is ignored while socket activation is in place.

An SSH connection goes through three phases in this order:

1. Key exchange and the encrypted channel. Client and server negotiate algorithms and derive a session key with ephemeral Diffie-Hellman. From here on everything is encrypted, phase 3 included.
2. Server authentication. The server proves it holds the private key matching its host key; the client checks that key against known_hosts.
3. Client authentication. Now, and only now, the user identifies themselves — with a key or with a password. It is essential that this comes afterwards: that is why the password never travels in the clear.

The server has its own keys in /etc/ssh/, generated at installation time:

The first time you connect you see this:

What that warning really means, as opposed to what everybody does with it: SSH is telling you "I have no way of knowing whether this machine is the one you are after". The correct answer is not to type yes blindly, but to compare that fingerprint with the one you obtained through a different channel — the VM console, the cloud provider's panel, a signed email from whoever installed the server. If you accept without checking, you accept the first machine that answers, and that is exactly the gap that the man-in-the-middle attack exploits: somebody places themselves between you and the server, presents you with their host key, you accept, and from then on they decrypt your session, see your password and forward it to the real server. All with the green padlock in place.

Once accepted, the key is saved in ~/.ssh/known_hosts and later connections verify themselves. If it changes one day, you will see a huge warning with REMOTE HOST IDENTIFICATION HAS CHANGED! and the connection is refused. There are three legitimate causes: the server was reinstalled, its host keys were regenerated, or the IP was reused for another machine. The illegitimate cause is a man in the middle. Work out which it is before deleting anything, and once you know, delete only that entry:

Never rm ~/.ssh/known_hosts: that destroys the trust built up with every machine and leaves you accepting new fingerprints blindly for weeks.

Asymmetric cryptography uses a pair of mathematically related keys: what the private one signs, only its matching public one verifies, and the private key cannot be derived from the public one in any reasonable time. In SSH, you keep the private key on your laptop and upload the public one to the server. To authenticate, the server throws you a challenge, you sign it with the private key and it verifies the signature with the public key it holds in authorized_keys. The private key never leaves your machine, not even encrypted: that is what makes it radically better than a password, which does travel (even if through an encrypted channel) and which can be guessed by brute force.

Why ed25519 and not RSA: ed25519 offers security equivalent to 3072-bit RSA with 68-character keys, signs and verifies faster, does not depend on a good random number generator for every signature and has no parameters you can choose badly. Use 4096-bit RSA only if you need compatibility with old equipment that does not support ed25519. The -C comment is not decorative: it identifies whose key it is and when it was made, for when you review authorized_keys two years from now.

The passphrase encrypts the private key on disk. It is the second layer: if your laptop is stolen, they have a useless file without it. The cost — typing it every time you use it — is solved by ssh-agent.

The permissions are compulsory, not a recommendation. SSH refuses to use a private key that others can read:

Installing the public key on the server, with the tool or by hand:

Each line of authorized_keys accepts per-key options in front of the type, and they are a layer of control almost nobody uses:

The restrict,command= combination is the right way to give access to an automated process — a backup, a deployment — without giving it a shell.

ssh-agent holds your decrypted key in memory so you do not have to type the passphrase every time:

-t 4h makes the key expire in the agent: if you leave your laptop unlocked in a café, the exposure window is limited. ssh-add -D removes them all at once.

Agent forwarding (ssh -A) lets you use your keys from the server you have logged in to, in order to hop to a third one. Its real risk: while the session is open, anybody with root on that intermediate machine can use your agent — talk to the socket in /tmp/ssh-*/agent.* and sign with your keys to log in anywhere they are valid. It does not steal your key, but it can use it, which in practice is the same thing.

The correct alternative is ProxyJump, which does not expose the agent: the client opens a tunnel through the hop and negotiates end-to-end encryption with the final destination.

Practical rule: use ProxyJump always; use -A only if there is no alternative, and then with ssh-add -c (which asks for confirmation on every use).

Everything above is written once in ~/.ssh/config (mode 600) and disappears from your muscle memory:

Multiplexing is the one you notice most: repeated rsync and scp runs stop renegotiating the encryption every time.

This is the heart of the lesson. Ubuntu 24.04 reads /etc/ssh/sshd_config and, inside it, an Include /etc/ssh/sshd_config.d/*.conf at the top of the file. Since in SSH the first value found wins, whatever you put in a file under sshd_config.d/ takes priority over the main file. That is the clean way to harden without touching the original or fighting the next package update.

And the honest debate about changing the port: moving SSH from 22 to 2222 hugely reduces the noise from automated scans and, with it, the size of your logs. What it does not do is protect you: any ten-second nmap finds the new port, and in exchange you complicate life for your tools and your colleagues. It is log hygiene, not security; never count it as a control. If you do it anyway on Ubuntu 24.04, remember socket activation: Port 2222 in sshd_config has no effect and you have to touch the socket.

The first empty line clears the inherited list; without it, the service would listen on both ports.

This module's golden rule: never close the door you are coming in through. In SSH that translates into five steps that are never skipped.

sshd -t validates the syntax; sshd -T shows you the result after resolving the Includes, which is the only thing that counts. reload re-reads the configuration without touching established sessions: if you have made a mess, your two open sessions stay alive and you can revert. And above all there is the VM console in VirtualBox, which does not go over the network and is your last resort. With socket activation, a port change also needs sudo systemctl restart ssh.socket.

An SSH tunnel carries an arbitrary TCP connection inside the encrypted channel. Three forms, three real use cases at Tramontana:

Read -L 15432:127.0.0.1:5432 like this: "open port 15432 on my machine; whatever arrives there, take it out of the other end of the tunnel and deliver it to 127.0.0.1:5432 as seen from the server". It is exactly what you need in order to administer the database with a graphical tool without opening 5432 in the firewall. -N means "do not run any command, just the tunnel".

An important warning: by default remote tunnels (-R) listen only on the server's localhost, and that is a good thing. Opening them on all interfaces requires GatewayPorts yes in sshd_config, which is a splendid way of bypassing your own firewall without noticing. Leave it as it is.

--partial keeps what has been transferred if the connection drops, and --progress tells you whether it is worth waiting. Combined with the ControlPersist from section 6, a repeated rsync does not even renegotiate the encryption.

If your connection drops halfway through an apt upgrade or a deploy.sh, the process receives SIGHUP and dies half-finished. A deployment interrupted between the release rsync and the systemctl restart leaves the application in a state nobody designed.

tmux (or screen, which is older) keeps the session on the server, independent of your connection. Professional rule: every long operation in production is launched inside tmux. It is free and it prevents the kind of incident people talk about for years.

Everything sshd does ends up in the journal and in /var/log/auth.log.

lastb reads /var/log/btmp, the record of failed attempts (last reads wtmp, the successful ones). With what you learned in Module 3, the summary by IP and by attempted username comes out in one line:

Interpretation: 47 attempts from a single IP, trying root, admin, postgres and ubuntu in three minutes. This is not somebody getting their password wrong: it is a brute-force bot, one of the thousands that sweep the Internet. The two attempts from 10.0.2.77 come from the internal network and are probably Luis mistyping.

The good news: with PasswordAuthentication no you have just turned those 47 attempts into harmless noise, because the route they were trying no longer exists. The bad news: the bot keeps calling, it consumes resources, it dirties your logs and it will still be there tomorrow with another technique. Blocking it at the door — and automating the blocking — is exactly the job of the next lesson.

Security and compliance warning: srv-tramontana processes guests' personal data, so remote access is a control subject to the GDPR. In a real environment that means named accounts (never shared ones), a record of who logs in and when with an agreed retention period, periodic review of authorized_keys to withdraw the keys of people who have left, and review of the change by the security officer. And the analysis of access logs, which identifies working people, has legal limits: it is done for security purposes and under the notified policy, not to keep anybody under surveillance.

• Enabling PasswordAuthentication no without having tested the key. The classic that leaves people locked out of production servers. Test with ssh -o PreferredAuthentications=publickey first, and only then reload.
• Wrong permissions in ~/.ssh. SSH silently (or almost silently) ignores keys and authorized_keys files that are too open. 700 for the directory, 600 for private keys and authorized_keys. If something "does not work for no reason", ssh -vvv tells you in the Authentications that can continue line.
• Deleting the whole known_hosts when faced with a changed-fingerprint warning. Investigate the cause and use ssh-keygen -R <host>.
• systemctl restart ssh instead of reload. With a broken configuration, restart locks you out; reload keeps existing sessions alive.
• Copying the private key to the servers "so I can hop from one to the other". The private key does not leave your laptop: that is what ProxyJump is for and, with reservations, the agent.
• Forgetting that the AllowGroups group has to exist. If you write AllowGroups sshusers and the group is empty, nobody gets in. Create it and add the people before reloading.
• Tip: document every key in authorized_keys with its -C comment (person, machine, date) and review it every quarter. An orphaned key is an open door with a forgotten name on it.

1. Restricted access for backups. Tramontana's NAS (10.0.2.90) must run /home/operator/scripts/backup_tramontana.sh --read-only on the server, and nothing else. Write the exact authorized_keys line and justify each option.
2. A tunnel to the database. Marta wants Luis to query PostgreSQL from laptop-luis with a graphical tool, without 5432 being exposed. Give the command, explain each parameter and say what is configured in the tool.
3. Recovering from a disastrous change. You apply AllowGroups admins (a group that does not exist) and reload. You have one SSH session open and the VM console. Detail the diagnosis and the repair, and which step of the procedure would have saved you the scare.

1. A single line, with a double lock — who and from where — plus confinement of the command:

• from="10.0.2.90" — even if the NAS's key is stolen, it only works from that IP. It is an extra layer that costs nothing.
• restrict — disables tunnels, agent forwarding, X11 and pty allocation in one go. It is the safe default: instead of forbidding things one by one, everything is forbidden and only what is needed is enabled.
• command="..." — that command runs whatever happens; if the NAS asks for bash, it gets the script all the same. The original stays in SSH_ORIGINAL_COMMAND, so the script could inspect it, but never run it blindly.
• The final comment identifies the key for the quarterly review.

With those three options, a compromise of the NAS does not give you a shell on srv-tramontana: at most it gives a read-only backup run from one specific IP.

2. A local tunnel from Luis's laptop:

• -L 15432:127.0.0.1:5432 — opens 15432 on laptop-luis; whatever arrives there goes out through the tunnel and is delivered to 127.0.0.1:5432 from the server's point of view. The database still listens only on localhost.
• -N — do not run any remote command: we only want the tunnel.
• -f — go into the background once authenticated, so you get your terminal back.

In the graphical tool, Luis configures host 127.0.0.1, port 15432, his username and his database. What this protects: PostgreSQL is not exposed to the network, authentication is controlled by SSH (with a key, not a password) and the traffic is encrypted. What it does not protect: if Luis's laptop is compromised, the attacker has the same access; and any local user of that laptop can use 15432, so the strict form would be -L 127.0.0.1:15432:127.0.0.1:5432.

3. Diagnosis and repair from the session that is still open (or from the VM console):

Two possible repairs, depending on what you meant to do:

The step that would have saved you the scare is step 3 of the procedure: sudo sshd -T shows the effective configuration, and checking getent group admins before reloading would have revealed in two seconds that the group did not exist. sshd -t does not catch it, because the syntax is perfectly valid: the file is well written and says exactly what you did not want. That is the difference between validating the syntax and validating the intent — and the reason the second open session is not negotiable.

srv-tramontana's front door now has a lock. You have understood the protocol's three phases and why the password never travels in the clear; you know what the fingerprint you are asked to accept the first time really means and how to react when it changes; you have an ed25519 pair with a passphrase, an agent that expires it after four hours and a ~/.ssh/config that multiplexes connections and hops with ProxyJump instead of exposing the agent. On the server, a file in sshd_config.d/ forbids root, removes password authentication, restricts access to the sshusers group and records in VERBOSE mode the fingerprint of every key that logs in — all applied with sshd -t, sshd -T, reload and three open sessions, because you never close the door you are coming in through. And you reach the database through a -L tunnel without exposing a single extra port.

You have also put a name to the noise: 47 attempts from 203.0.113.44 trying root, admin, postgres and ubuntu in three minutes. Against that particular IP you have won — there are no passwords left to guess — but the bot keeps ringing the bell, and tomorrow it will try 8080, which is still wide open to anybody who reaches the machine, just as PostgreSQL would be if it ever listened outside localhost. SSH was one service; what is missing is a policy about what may even speak to this server. That is lesson 06-03: Firewalls and Perimeter Security, where you will get to know netfilter and nftables from the inside, bring up ufw in the right order so as not to lose your session, write a complete ruleset for Tramontana, tune the network sysctl settings with an impact on security and set fail2ban to block 203.0.113.44 automatically, along with everyone who comes after it.