The previous lesson ended with a question that has been postponed four times: all the security of TLS rests on the validation of a certificate — and who vouches for that certificate? This lesson opens that box and, with it, what module 2 announced as the really hard problem. Its thesis can be stated in one sentence: algorithms almost never fail; key management does. AES has not been broken, RSA has not been broken, SHA-256 has not been broken — but keys leak into repositories, certificates nobody was watching expire, keys needed to restore a backup get lost, and credentials are left active years after their owner walked out. You are going to see the complete life cycle of a key, where it must live, how it is rotated without interrupting the service, what an X.509 certificate contains, how the chain of trust works and what a certificate authority really does.

1. The thesis: algorithms do not fail, management does
2. The life cycle of a key
3. Where a key lives: from the .env to the HSM
4. The secret in the repository: why deleting the commit is not enough
5. Rotation without interrupting the service
6. X.509 certificates: exactly what they contain
7. The chain of trust and what the browser validates
8. Certificate authorities and types of validation
9. ACME and Let's Encrypt: automatic issuance and renewal
10. Revocation: CRL, OCSP, stapling and their limits
11. Certificate transparency
12. Internal PKI: when to set up your own CA
13. Envelope encryption
14. Custody, key backups and the dilemma of losing them

Run mentally through the incidents you have studied on the course. At Equifax (02-06) the breach came in through an outstanding patch, but what made it historic was that an expired certificate blinded the system inspecting the traffic for ten months. In the Nimbus ransomware (02-06) the attacker broke nothing: they found a secret forgotten in a .env and escalated with it. In the leak through a misconfigured bucket no cryptography was even involved.

Hence this module, which has devoted five lessons to the primitives, devotes the most operational one to management. The professional question is not "which algorithm do I use?", but "where does this key live, who can read it, when is it rotated and what happens if I lose it?"

A key is not a value: it is an object with states, and every transition has questions that must be answered in writing.

The phase almost nobody documents is the last one, and it is the one that produces the classic finding: a key that was decommissioned "years ago" and still works in some forgotten system.

The conceptual difference between a secrets manager and a KMS deserves precision, because they get confused: a secrets manager hands you the secret in a controlled, audited way (the application ends up with the value in memory); a KMS or an HSM hand you nothing, but operate on your behalf — "encrypt this", "sign this" — and the master key never leaves. That is why the KMS is the right piece for the master key of envelope encryption (section 13).

A concrete recommendation for Nimbus, in keeping with its size and budget: a secrets manager for database credentials, API keys, the webhook secret and the token signing key, injected at container start-up and with zero secrets in the repository; the cloud provider's KMS for the master key that protects the data keys of attachments, clinical notes and backups; the device secure module for the staff's passkeys, which already use it without anybody configuring anything; and no HSM, unless a customer or a sector standard demands one, because its cost is not justified against Nimbus's real risk.

This is the scenario that opened the escalation in the ransomware case of 02-06: an old .env file, with a non-expiring API token, versioned by mistake.

How it is detected:

As well as scanning in CI (02-04), it is worth having an alert that watches the organisation's public repositories and their forks.

Why deleting the commit is not enough. This is the point almost everybody gets wrong:

What rotating really means. It is not "changing the value in the .env". It is: (1) issue a new credential; (2) deploy it to every consumer; (3) effectively revoke the old one, checking that it no longer works; (4) review the logs of the old one's use from the date it was exposed, looking for unrecognised access; (5) document the incident. Until step 3 is carried out, the credential remains valid however thoroughly you have deleted it from the code.

And the correct order: rotate first, clean the history afterwards. Cleaning the history is hygiene; rotating is containment. Reversing the order leaves the window open while the long job is being done.

Rotating is frightening because it looks as though it forces an outage: if you change the key, everything signed with the previous one stops validating. The solution is an overlap period with two active keys and a key identifier, the kid, which you already saw working in the JWT validation with JWKS in 02-05.

The idea: every signed or encrypted object carries written into it which key it was made with, so the verifier can have several keys loaded and pick the right one.

The four phases of a clean rotation:

Important points: the kid comes from an untrusted message, so it serves only to select the key, never to decide whether to validate; if the kid is not on your list, the token is rejected, no exceptions. And a rotation that is only carried out manually "when it is due" does not get carried out: automate the process and rehearse it at least once calmly before you need it in a hurry.

Reasonable cadences for Nimbus:

A digital certificate is a document that binds a public key to an identity, signed by an authority the verifier trusts. It is the answer to the question left open in 03-03: whose public key is this?

Field by field, with what to look at in a review:

The most frequent misunderstanding is to believe that the browser compares the domain with the CN. It has not done so for years: it compares with the SAN. A certificate without the domain in the SAN produces a name error however correct the CN may be.

Why the intermediate exists. The root key is too valuable to be used every day: it lives disconnected, in an HSM, and only comes out in audited ceremonies. The intermediate does the daily work and, if it is compromised, it is revoked and replaced without invalidating the root — which is installed on billions of devices and cannot easily be changed.

Exactly what the client checks when it validates, in order:

1. The signature of each link: the leaf is signed by the intermediate, the intermediate by the root.
2. The root is in the trust store of the system or the browser.
3. Validity: the current date falls within the Validity of every certificate in the chain.
4. Name: the requested domain matches some entry in the leaf's SAN.
5. Uses: Key Usage and Extended Key Usage permit TLS server authentication.
6. Basic Constraints: the intermediates are CA:TRUE and the leaf CA:FALSE.
7. Revocation: an OCSP query or stapling (section 10).
8. Transparency: browsers require valid SCTs (section 11).
9. The handshake signature: the CertificateVerify of 03-05 proves that the server holds the private key. Without this step, presenting somebody else's certificate would be enough.

The most common operational error is to send only the leaf and forget the intermediate. Desktop browsers usually fetch it and paper over the failure; many API clients, the mobile app and HTTP libraries do not, and the result is the classic incident of "it works in the browser but fails in the app". That is why you serve fullchain.pem, not cert.pem.

A CA is an organisation that operating system and browser vendors have decided to trust, after periodic audits. Its business is verifying identities before signing.

The conclusion, which surprises many people: from the technical and user-experience point of view, DV, OV and EV are equivalent. Browsers stopped highlighting EV visually precisely because the studies showed that users did not perceive the difference, while the cost and the friction were real enough.

For Nimbus: DV certificates issued with ACME. OV is only worth considering if a corporate customer demands it by contract, and it is worth being able to argue that such a demand rarely adds technical security.

And one honest limitation of the model: any of the CAs in the trust store can issue a certificate for any domain. It has happened, through compromise or through a CA's mistake. The answers to that risk are certificate transparency (section 11), the CAA record in the DNS — which declares which CAs may issue for your domain — and, in specific cases, pinning with its risks (03-05).

ACME is the protocol that automates issuance: the client proves control of the domain by passing a challenge and receives the certificate with no human involvement.

Two challenges and when to use each: HTTP-01 serves a file under /.well-known/acme-challenge/ and is the simplest, but it requires port 80 to be reachable and does not allow wildcards; DNS-01 publishes a TXT record, works without exposing anything and does allow wildcards, at the price of needing credentials for your DNS provider — which are in turn a secret that has to be managed.

An expired certificate is an availability incident. The service does not degrade: it goes down, with a full-screen security warning that also destroys user trust. And remember Equifax (02-06): there the expired certificate did not bring a website down, it blinded the traffic inspection system for ten months, which let the exfiltration go unnoticed. Hence the transferable rule from that lesson: every control needs a control that verifies it is still working.

Applied to Nimbus: automatic renewal is not a sufficient control; the control is the external monitoring of the expiry date, which alerts at 30 and 7 days and works even if the renewal process has broken silently. The three usual reasons it breaks: the timer was disabled after a migration, the HTTP-01 challenge stopped resolving because of a configuration change, or the certificate renews but nobody reloads the service that serves it.

If a private key leaks, the certificate remains cryptographically valid until it expires. A mechanism is needed to say "this one is no longer good".

The uncomfortable consequence has to be stated plainly: revocation works worse than it appears to. Since clients fail open so as not to break browsing when the CA does not respond, an attacker who controls the network can simply block the OCSP query. That is why the industry has moved towards short-lived certificates — 60 or 90 days, and falling — as a practical substitute for revocation: if the certificate expires soon, the window of damage is short by construction.

What Nimbus must do if the private key of its certificate leaks: (1) issue a new certificate with a new key — never reuse the compromised key; (2) deploy it; (3) revoke the previous one, in the knowledge that revocation is a partial measure; (4) review logs for improper use; (5) analyse how the key leaked.

Certificate transparency (CT) answers the risk from section 8: that a CA issues a certificate for your domain without your knowledge. Every public CA must publish each certificate it issues in public, append-only, auditable logs — the same idea as Nimbus's audit table, with INSERT but no UPDATE (01-04) — and the browser demands proofs of that publication (the SCTs you saw in the certificate in section 6).

This does not prevent an improper issuance, but it makes it impossible to hide. And there lies its value for Nimbus: it turns an invisible attack into a detectable one.

How Nimbus uses it, specifically: (1) subscribe to CT monitoring for nimbusreservas.example and all its subdomains, with notification to Lucía and Marta; (2) review every alert, because a certificate Nimbus has not requested is an incident that is triggered immediately; (3) publish a CAA record in the DNS declaring the only authorised CA, which also reduces the monitoring noise; and (4) use CT as an inventory too, because the list of issued certificates reveals forgotten subdomains nobody remembered, some with exposed services — that is, it is a tool for reducing the attack surface (01-04) as well as for detection.

Public CAs are no use for mTLS between internal services (03-05), because they do not issue certificates for internal names or for service identities. The alternative is a private CA whose root is installed on your own systems.

The risks of doing it badly, which are serious: an internal root installed on the laptops can issue valid certificates for any domain, including the employee's bank; if its key leaks, the attacker can impersonate any site to those machines. Besides, a CA without working revocation, without automated renewal or without an inventory becomes a source of outages, and a root with twenty years of validity and no replacement plan is a debt inherited by whoever comes next.

Minimum rules if Nimbus sets one up: an offline root with its key protected and backed up; an intermediate for day-to-day issuance; short lifetimes and automated issuance (for example, with the PKI feature of a secrets manager, which does this job well); a scope limited to internal names, never to public domains; an inventory of every certificate issued; and a written plan for rotating the root.

Envelope encryption is the standard pattern for encrypting a lot of data with few master keys, and it is the application of the hybrid encryption of 03-03 to storage.

How it works in Nimbus, step by step: (1) when an attachment is uploaded, the API asks the KMS for a data key and receives two versions, one in the clear to use now and one encrypted with the master key; (2) it encrypts the attachment with the plaintext key using AES-256-GCM with a unique nonce and AAD containing the tenant_id (03-02); (3) it discards the plaintext key from memory and stores next to the object the encrypted data key, the nonce and the identifier of the master key; (4) to decrypt, it asks the KMS to decrypt the data key, uses it and discards it again.

The four advantages that explain why it is the dominant pattern:

That last row is an underrated detection control: if somebody tries to decrypt ten thousand attachments in an hour, the KMS log shows it even if the application is compromised.

Here is the most uncomfortable tension in the whole of key management, and it has no elegant solution:

There is no automatic middle ground: it is a business decision that has to be taken explicitly and in writing, differently for each type of key.

Good custody practices, when the answer is yes:

• An encrypted copy in a different location from the data it protects, with independent access control.
• Splitting between several people when the key is critical: reconstruction requires, for example, two of three custodians. It avoids the single point of failure and also the single point of abuse.
• A recovery rehearsal at least once a year, documented. Custody that has never been tested is an assumption, exactly like a backup that has never been restored (02-04).
• A written procedure for who may request the recovery, who approves it and how it is recorded.

Professional validation note. When the key protects data that reveals health information — such as the clinical notes of the clinics that are Nimbus customers — the custody design has legal implications: who may access the data, on what lawful basis, how long it is retained and what happens to the keys at the end of the retention period. Destroying the key may in fact be a valid mechanism for erasing data, but that must be validated with the Data Protection Officer and with legal advice before you rely on it. The regulatory framework is covered in 06-03.

1. Secrets in the repository. The most frequent and the most expensive. A scanner in CI and pre-commit hooks.
2. Believing that deleting the commit solves anything. Only effective rotation solves it; and you rotate first, then clean up.
3. Not knowing how many keys there are. Without an inventory — key, purpose, owner, location, rotation date — no management is possible.
4. Never rotating "because it works". A key that is never rotated accumulates exposure: everybody who has passed through the team could have seen it.
5. Rotating without a kid or an overlap. It turns a routine operation into a service outage, and that is why it stops being done.
6. Storing the key next to the encrypted data. Decorative encryption. Especially serious with backups.
7. Relying only on automatic certificate renewal. External monitoring of the expiry is needed, with alerts at 30 and 7 days.
8. Serving only the leaf without the intermediate. It works in the browser and fails in the mobile app.
9. Validating the CN instead of the SAN.
10. Assuming that revocation works. It fails open. Short lifetimes and rotation are the real defence.
11. Setting up an internal CA without maintaining it. A badly guarded internal root is a master key for attacking your own machines.
12. Not backing up the master data key, or backing it up and never testing it. Both end equally badly.

Tips

• Start with the inventory: list every key, secret and certificate with its owner, purpose, location and rotation date. It is an afternoon's work and it completely changes the conversation.
• Automate what you can: ACME issuance, renewal, rotation with overlap, secret scanning. What is manual does not get done.
• Rehearse a complete rotation and a recovery from custody before you need them.
• Apply the single purpose principle: one key, one function. Derive with HKDF (03-02) instead of reusing.
• And remember the thesis: if one day you have a cryptographic incident, the cause is most likely a badly managed key, not a broken algorithm.

A scan detects that fourteen months ago a deploy/.env.production file was pushed to the repository with the PostgreSQL password, the gateway API key and the token signing secret. The repository is private, with access for the five people on the team and two people from the consultancy (A-19).

You are asked to: (a) order the actions for the first four hours and justify the order; (b) explain exactly what rotating each of the three secrets means and what outage risk each rotation carries; (c) state which logs you would review and over what time window; (d) say whether this is a notifiable security incident and what information you need in order to decide; (e) propose the three preventive measures that stop it happening again.

The Nimbus mobile app fails with "invalid certificate" while the website works correctly in the browser. The output of openssl s_client shows:

You are asked to: (a) diagnose the exact cause; (b) explain why the browser works and the app does not; (c) state the specific fix on the server; (d) propose the automatic check that would detect this failure before a customer reports it; (e) list three other different reasons why a valid certificate can fail in a client.

Marta asks for a one-page document with the key management scheme.

You are asked to: build a table with at least six Nimbus keys or secrets (token signing, attachment encryption, backup encryption, database credentials, the webhook secret, the TLS certificate) stating for each one: where it lives, who can access it, rotation cadence, whether it has a backup and where, and what happens if it is lost. Then justify the two most debatable decisions in your table.

Exercise 1

(a) The first four hours, in order:

1. Consider all three secrets compromised. No discussion: fourteen months and nine people with access, two of them external.
2. Rotate the payment gateway key first. It has the greatest direct financial impact and the rotation is the least disruptive (it is coordinated with the provider).
3. Rotate the token signing secret using the overlap mechanism of section 5, so as not to throw every user out.
4. Rotate the PostgreSQL password, coordinated with a deployment, since it affects every process that connects.
5. In parallel, review the logs of the use of all three secrets.
6. Afterwards, clean up the repository history and communicate it to the team and the consultancy.

The order reflects the fact that rotating is containment and cleaning the history is hygiene: while the credential remains valid, deleting the commit changes nothing.

(b) What rotating each one means:

(c) Logs and window. Window: from the date of the commit (fourteen months) until today, not just from detection. Logs: access and transactions in the gateway console; PostgreSQL connections by origin, user and unusual hour; token issuance and use with anomalous signatures; access to the A-05 bucket; and repository activity (clones, forks, access by the consultancy's accounts). You look for unknown origins, times outside the pattern and anomalous volumes, just as in the analysis of 02-06.

(d) Is it notifiable? It is certainly a security incident. It is notifiable if there are indications of unauthorised access to personal data. To decide, you need: logs that are sufficient and with adequate retention to cover fourteen months (if there are none, that in itself is a serious finding and pushes you towards caution), evidence of anomalous access, and the scope of the data reachable with those credentials — which here includes data that indirectly reveals health information. The decision is taken with legal advice and with the Data Protection Officer, within the GDPR's 72-hour deadline (06-03).

(e) Three preventive measures: (1) a secret scanner in CI and a pre-commit hook that blocks the commit, plus a correct .gitignore; (2) a secrets manager with injection at start-up, so that no .env with real values exists; (3) scheduled automatic rotation with overlap, so that no credential accumulates fourteen months of life and so that rotating is a routine operation rather than an emergency.

Exercise 2

(a) The exact cause. The server is sending only the leaf certificate (0 s:), without the Let's Encrypt intermediate certificate. Code 20 literally means the client cannot obtain the local issuer: it cannot build the chain up to a trusted root.

(b) Why the browser does work. Desktop browsers usually have the intermediate cached from previous visits or download it via the certificate's Authority Information Access extension. That retrieval is a courtesy, not part of standard validation, and many HTTP libraries and mobile clients do not implement it: they require the server to send the complete chain, as the standard mandates.

(c) Fix. Configure ssl_certificate to point at fullchain.pem (leaf + intermediate), not at cert.pem, and reload Nginx. Then verify with openssl s_client, where both links must appear along with Verify return code: 0 (ok).

(d) Automatic check. An external scheduled job that, for each domain, runs openssl s_client without using the local intermediate store and verifies three things: return code 0, a number of certificates in the chain greater than one, and remaining days of validity above the threshold. It alerts Lucía if any of them fails. It is the "control that verifies the control works" of Equifax applied here.

(e) Three other reasons: (1) the domain is not in the SAN (even though the CN is correct); (2) the client's clock is out, which makes a valid certificate look expired or not yet valid — common on phones and on freshly installed devices; (3) an out-of-date pin in the app after a certificate renewal (03-05). A common fourth: the corresponding root is not in the trust store of an old, unpatched system.

Exercise 3

The two most debatable decisions:

1. Not backing up the token signing key. Debatable because losing it throws every user out at once. It is justified because the impact is inconvenience, not data loss: users log back in. A copy, on the other hand, would be a permanent surface for a secret that allows any user to be impersonated. The risk of the copy outweighs the risk of the outage.
2. Keeping the backup key in a different account, with two of three custodians. Debatable because of the friction it adds to an urgent restore: if Lucía is off sick and a custodian is missing, the restore is delayed. It is justified by the direct lesson of the ransomware of 02-06: the key and the backups in the same environment mean that whoever compromises production takes both. The friction is mitigated by having three custodians and rehearsing the recovery once a year, so that the procedure is proven before it is needed.

You have entered the hard problem and you come out with its thesis internalised: algorithms almost never fail; key management does. You have seen it in the course's incidents — the expired Equifax certificate that blinded detection for ten months, the Nimbus .env that opened the escalation — and in the contrast between what people fear and what actually happens. You have the complete life cycle of a key, with the questions to be answered at each phase and the realisation that the phase nobody documents — destruction — is the one that produces credentials still active years later. You know where a key can live, from the source code to the HSM, with the precise distinction between a secrets manager, which hands you the value, and a KMS or HSM, which operate on your behalf without handing you anything; and with the concrete recommendation for Nimbus: a secrets manager for credentials, a KMS for the master data key, the device module for the passkeys and an HSM only if somebody demands one.

You have learned what rotating really means — issue, deploy, effectively revoke, review logs and document — why deleting the commit revokes nothing and why you rotate before cleaning the history; and you have seen how to rotate without interrupting the service with overlap and a kid, the same piece that underpinned the JWT validation of 02-05. On the PKI side you have opened an X.509 certificate field by field and you know that what the browser validates is the SAN and not the CN; you understand the chain of trust and why the intermediate exists, the nine steps the client checks and the classic failure of serving the leaf without the intermediate; you know that DV, OV and EV are technically equivalent and that any CA in the trust store can issue for your domain; you can handle ACME and you know that automatic renewal is not a sufficient control — the control is the external monitoring of the expiry; you know the real limits of revocation, which fails open and has in practice been replaced by short-lived certificates; and you know how to use certificate transparency to detect improper issuance and, along the way, as an inventory of forgotten subdomains. You add when an internal PKI makes sense and its risks, envelope encryption with its data key and master key — which also leaves per-object auditing in the KMS — and the honest dilemma of custody: copying too much multiplies the surface, copying too little turns encryption into permanent loss, and there is no automatic middle ground, only a written decision per type of key.

You now have all the pieces of the module: the primitives, the protocols that assemble them and the management that holds them up. In Applications of Cryptography (03-07) we close by walking through the Nimbus system end to end and pointing out what protects what: encryption at rest with its three levels — disk, database and application — and the key point that disk encryption does not protect against a compromised API; field-level encryption of the clinical notes with the practical problem it introduces, namely that they can no longer be searched or indexed; encrypted backups with the key outside the environment; artefact signing in the CI/CD and its connection with SolarWinds; tokens and the 120-second signed URLs explained from the inside; and the real difference between pseudonymisation and anonymisation, with the reason why hashing a DNI is not anonymisation.