The incident we closed out module 5 with said it all: MediNube's TLS private key didn't fall because of a mathematical flaw — it fell because of a git push. All the cryptography we've built throughout the course — AEAD, Argon2id, Ed25519 signatures, mTLS — shares a single point of failure: keys. If a key lives in the wrong place, it doesn't matter how perfect the primitive using it is. This lesson answers the question we've been deferring since module 1: where should MediNube's secrets actually live? And along the way it settles several outstanding debts: the API_KEYS dictionary from 01-04, the medical-records master key from 02-03, the pepper from 03-03, and the envelope encryption we previewed in 04-05.

1. The real problem: secrets leak through carelessness
2. MediNube's secrets inventory
3. Key hierarchy: envelope encryption with DEK and KEK
4. The secrets-management maturity scale
5. Environment variables: correct use and limits
6. Encrypted files in the repository: sops and age
7. Secrets managers: Vault and KMS in practice
8. Rotating the master key: crypto-agility cashes in
9. The pepper, finally done right
10. Prevention: keeping the secret out of the repository

No realistic attacker is going to factor MediNube's RSA-3072 modulus. What MalloryClinic will do is search the places where developers accidentally leave secrets lying around:

• Git repositories: the privkey.pem from the 05-03 incident. The commit got deleted, but git history — and the bots that scan GitHub — don't forgive (a secret pushed to a public repo is considered compromised within minutes).
• Logs: a logger.debug(f"Connecting with key {api_key}") written during a debugging session and forgotten.
• Shared .env files: over email, over Slack, "just this once so you can get the project running."
• Docker images: a COPY .env /app/ freezes the secret into every layer of the image, even if the file is deleted afterward.
• Source code, directly: the classic. Our old acquaintance from 01-04:

This dictionary breaks golden rule 2 (the enemy knows the system: anyone with repo access knows the keys) and makes rotation impossible without a new deployment. Today, we retire it.

Before deciding where to store each secret, you need to know which ones you have. Here's the inventory accumulated over the course — building this table for your own project is the real first step of any secrets-management plan:

Notice one detail: not all secrets are equal. The medical-records master key protects health data with decades-long GDPR obligations; a webhook key protects notifications. The level of protection can — and should — be proportional to the impact.

In 04-05 we built the hybrid envelope and previewed that the DEK/KEK pattern was "exactly what KMSes do." It's time to settle that promise. The idea behind envelope encryption is to organize keys into a hierarchy:

• DEK (data encryption key): the key that encrypts the data. It lives right next to the data, but encrypted.
• KEK (key encryption key): the key that encrypts the DEKs. It lives in the KMS and never leaves it.

The operational flow is the key point, because it's often misunderstood:

1. The app wants to decrypt a medical record. Alongside the data it has the encrypted DEK stored.
2. It sends the encrypted DEK to the KMS: "decrypt this for me" (a decrypt operation).
3. The KMS checks that the app is authorized, decrypts the DEK with the KEK, and returns the DEK in the clear.
4. The app uses the DEK in memory for medinube.crypto's AES-GCM, and discards it when done.

The data never passes through the KMS. Only 32-byte keys travel through the KMS, not megabyte-sized records. This makes the pattern fast, cheap, and auditable: the KMS logs every operation (who asked to decrypt which DEK, and when), and rotating the KEK only requires re-encrypting DEKs (a handful of them), not terabytes of data.

Not every organization needs an HSM. What matters is knowing which rung you're on and what the next one is:

MediNube, with medical records and GDPR obligations, must be at level 3, with the KEK ideally backed by an HSM (cloud KMSes already do this under the hood). Level 1's .env is still legitimate... for a developer's machine with test data.

Environment variables are the standard mechanism for delivering a secret to a process, and they're fine for that. But it's worth knowing their limits:

Explanation: instead of having the values in the code, the code only knows the secret's name. Failing loudly (RuntimeError) when it's missing is deliberate: an absent secret should block startup, not degrade into an insecure default.

Limits you should know about:

• Visible in /proc: on Linux, cat /proc/<pid>/environ shows a process's environment. Any user who can read that file (root, or the process's own user) sees the secrets.
• They're inherited: every child process gets the full environment. That third-party script you run also sees MEDINUBE_MASTER_KEY.
• They end up in logs and crash dumps: many frameworks print the full environment when reporting an error. Configure your error handler to redact variables with names like *_KEY, *_SECRET, *_PASSWORD.
• They don't rotate on their own: changing the value requires restarting the process.

Sometimes you want to version configuration with secrets included (for infrastructure, say). The answer isn't "commit the secret," it's committing it encrypted. The sops + age pairing is the lightweight standard:

The elegant part of sops is that it encrypts value by value: the git diff still shows which key changed, without revealing its content. And it can use age, a cloud KMS, or PGP as a backend — crypto-agility here too. Its limit: it decentralizes rotation (re-encrypt and redeploy) and gives no access auditing. It's a level 2.5 on our scale: perfect for infrastructure configuration, insufficient as MediNube's central secrets manager.

A secrets manager (HashiCorp Vault, OpenBao, or your cloud's KMS/Secrets Manager) provides what no earlier level has: application authentication, secrets with expiration, and central auditing. The conceptual flow, common to every provider:

1. The app authenticates to the manager. Not with "yet another hardcoded secret" (that would be the same old problem!), but with a platform identity: the Kubernetes service account, the cloud instance's IAM role, an mTLS certificate from our 05-01 PKI.
2. The manager returns a token with a TTL (time to live) and, for many secrets, a lease: the secret itself expires (think database credentials generated on the fly, valid for 1 hour).
3. The app renews the lease while it's alive; when it dies, the secret expires on its own. It's the same lesson as certificates' short lifetimes from 05-03, applied to every secret.

With the hvac library (Vault's Python client), the code looks like this — treat this example as a flow, not as lock-in to one provider:

Details worth savoring: the transit engine's ciphertext starts with vault:v2: — a versioned format, exactly like our 0x01 byte in medinube.crypto. Everyone arrives at the same conclusion: without a version, there's no rotation.

Compliance note: for MediNube, the secrets manager isn't just good engineering: the GDPR requires "appropriate technical measures" (Art. 32), and in an audit, the manager's access log is the evidence of who could have touched the medical-records key. Coordinate the choice of level with your security lead and your DPO.

In 02-03 we designed the format v1 = 0x01 || nonce(12) || ciphertext+tag with AAD f"paciente={patient_id};formato=v1" and said that version byte would "save us someday." Today is that day: it's time to rotate the medical-records master key (good periodic hygiene, and mandatory if you suspect compromise). The plan, thanks to versioning:

1. Create the new key in the manager (or a new version of the KEK, which re-encrypts the DEKs).
2. Define v2: identical format, byte 0x02, AAD formato=v2, new key.
3. Writes: all new encryption goes out as v2.
4. Reads: decrypt_medical_record looks at the first byte and picks the key. v1 remains readable.
5. Progressive re-encryption: a background job reads each v1 record, decrypts it, and rewrites it as v2. No service downtime.
6. Once no v1 remains in the DB, the old key is destroyed (or archived per retention policy).

Without that leading byte, rotation would have required either guessing which key encrypted each row (try-and-catch: slow and ugly) or re-encrypting everything at once with the service stopped. Golden rule 8 — crypto-agility — just paid for itself.

In 03-03 we left the pepper "for when we had somewhere safe to keep it." We have it now. Reminder: the salt lives alongside the hash in the DB (it's public); the pepper is a global secret that lives outside the DB, so that a database dump alone isn't enough to attack the hashes offline. The correct way isn't concatenation, it's HMAC (03-02):

Now MalloryClinic needs two breaches (the DB and the secrets manager) to mount a dictionary attack. Watch out: the pepper is hard to rotate (you'd have to re-hash on each user's next login, like the migration in 03-03), so protect it well from the start.

The 05-03 incident's runbook was reactive. The mature version is preventing the leak before the commit, and catching it in CI:

Essential companions:

• Defensive .gitignore: .env, *.pem, *.key, credentials.json should be ignored before they exist.
• Scanning already-published history: running gitleaks over the whole history uncovers secrets leaked months ago. If something shows up, apply the generalized 05-03 runbook: rotate first (the secret is already considered compromised), then clean up the history, notify [email protected], and look for suspicious usage in the logs. Cleaning up history without rotating is theater.
• Log redaction: logger filters that mask values that look like secrets.

• Mistake: deleting the commit and calling it fixed. The secret is still in the reflog, in forks, in scanners' caches. The only valid response to a leak is rotation.
• Mistake: authenticating the app to Vault with a hardcoded token. You've moved the problem, not solved it. Use platform identity (IAM, service account, mTLS).
• Mistake: sending whole records through the KMS. The KMS encrypts keys (DEKs), not data. That's exactly what envelope encryption is for.
• Mistake: a single secret "for everything." One key per purpose (a rule we already applied with HKDF in 02-04): that way a leak doesn't drag down the whole system, and each key can be rotated separately.
• Tip: measure your maturity with the levels table and climb one rung at a time. Going from level 0 to level 1 in an afternoon eliminates 80% of the immediate risk.
• Tip: treat the secrets inventory as living documentation. Every new secret enters the table with its owner, its impact, and its rotation plan.

1. Express audit: clone (mentally or for real) one of your own projects and locate its secrets. Classify them with the maturity-levels table: which level is each one at? What's the most urgent level jump?
2. Rotation with versioning: starting from this lesson's multi-version decrypt_medical_record, write encrypt_medical_record_v2 and the skeleton of the progressive re-encryption job reencrypt_pending(batch=100), which finds blobs starting with 0x01, decrypts them, and rewrites them as v2.
3. Secret hunter: write a regular expression to catch MediNube's API keys (sk_live_ + 16 hex chars) and PEM private-key blocks in pre-commit. Why is it worth adding it to CI even though it's already in pre-commit?

1. Open-ended answer, but the typical pattern: API keys at level 0 or 1, DB password at level 1 or 2, certificates at level 2. The most urgent jump is almost always getting anything at level 0 out of the code (an afternoon's work) and adding gitleaks to CI (an hour).

2. def encrypt_medical_record_v2(data: bytes, patient_id: str) -> bytes:
       nonce = os.urandom(12)
       aad = f"paciente={patient_id};formato=v2".encode()
       ct = AESGCM(KEYS[0x02]).encrypt(nonce, data, aad)
       return bytes([0x02]) + nonce + ct
   
   def reencrypt_pending(batch: int = 100) -> int:
       rows = db.query("SELECT id, patient_id, record FROM medical_records "
                        "WHERE get_byte(record, 0) = 1 LIMIT %s", [batch])
       for row in rows:
           data = decrypt_medical_record(row.record, row.patient_id)
           new_blob = encrypt_medical_record_v2(data, row.patient_id)
           db.execute("UPDATE medical_records SET record = %s WHERE id = %s",
                      [new_blob, row.id])
       return len(rows)
   

   Design points: the job is idempotent (if interrupted, the next pass picks up what's left), it works in batches so it doesn't overload the DB, and it uses the correct AAD for each version (re-encrypting changes the AAD from v1 to v2, which is why you have to decrypt and re-encrypt, not "just flip the byte").

3. Patterns: sk_live_[0-9a-f]{16} and -----BEGIN( [A-Z]+)? PRIVATE KEY-----. It's also needed in CI because pre-commit hooks are optional and local: a teammate without the hook installed (or a git commit --no-verify) skips them; CI is the safety net nobody can bypass.

We've closed the wound the whole course had been bleeding from: MediNube's secrets no longer live in dictionaries in the code or shared .env files, but in a manager with authentication, TTLs, and auditing; the medical-records master key rotates thanks to the v1/v2 versioning we planted back in module 2; the pepper finally exists for real; and gitleaks watches so the privkey.pem story never repeats. Envelope encryption with DEK/KEK has gone from a promise (04-05) to an architecture.

But keeping keys well guarded is only half the map: now you have to decide what gets encrypted, where, and against which threat, in every corner of MediNube's architecture — from Ana Pérez's browser to the row in PostgreSQL, backups included. That full map of encryption at rest and in transit is the next lesson. See you there.