The Project | About Us | Contribute | Donations | License

HOME

The previous lesson left inventory demanding a signed token before moving a single unit of stock. But that token, Anna's phone number on order P-2026-000123, the order.created event travelling through orders.events and the backups of km0_inventory in MinIO still circulate and rest in the clear. Anyone with access to a network segment, a recycled disk, a Docker volume or a misconfigured bucket can read them or, worse, modify them without anyone noticing. Knowing who is who is no use if the conversation can be overheard and altered.

This lesson is about cryptography applied to a distributed platform: not how the algorithms are designed, but what each one guarantees, when to use which, and how to integrate it without making the mistakes that turn correct encryption into useless encryption. We will look at symmetric and asymmetric encryption, hashes, HMAC and signatures; TLS for data in transit, with gRPC between orders and inventory as the practical case; encryption at rest at disk, database and column level, with envelope encryption for Anna's phone number; server-side encryption in MinIO and in the backups; and personal data protection: minimisation, pseudonymisation for the data lake and encryption as the basis for the right to be forgotten. Mutual authentication with certificates (mTLS) and the operational custody of keys (Vault) are left for 06-04.

Three properties: confidentiality, integrity, authenticity
Symmetric encryption: AES-GCM, keys, nonces and tags
Asymmetric encryption and hybrid encryption
Hashes, HMAC and digital signatures
Summary table: which primitive for which problem
Data in transit: TLS
TLS in gRPC between orders and inventory
TLS in PostgreSQL, Kafka, Redis and MinIO
Data at rest: disk, database, column
Envelope encryption: Anna's phone number
Key management: rotation, environments and where keys do not belong
Personal data: minimisation, pseudonymisation, anonymisation and the right to be forgotten
Kilometre Zero protection map
Common Mistakes and Tips
Exercises
Conclusion

Three properties: confidentiality, integrity, authenticity

When we say "protecting a piece of data" we lump together three different guarantees that are achieved with different tools:

Property	Question	Threat	Tool
Confidentiality	Is it read only by those who should?	Network eavesdropping, stolen disk, public bucket	Encryption (symmetric, asymmetric, TLS)
Integrity	Has it changed along the way?	Tampering with an event, a file, a price in transit	Hash, MAC, signature; the AES-GCM tag
Authenticity	Was it produced by who it claims?	Impersonating `orders` in Kafka, a fake `inventory` server	MAC (with a shared key), digital signature, TLS certificate

Encrypting does not provide integrity on its own: with some classic cipher modes (AES-CBC without a MAC) an attacker can flip bits in the ciphertext and produce controlled changes in the plaintext without knowing the key. That is why modern practice is authenticated encryption (AEAD), which combines both in a single operation. And a fourth property is usually added to the three, non-repudiation (the author cannot deny authorship), which only a digital signature provides, because a MAC can be generated by everyone who shares the key.

Symmetric encryption: AES-GCM, keys, nonces and tags

In symmetric encryption the same key encrypts and decrypts. It is fast (AES has dedicated CPU instructions: several GB/s per core) and it is what is used for the bulk of the data: disks, files, columns, and the body of a TLS connection. Its problem is key distribution: both ends need to have the key, and getting it to them securely is precisely the problem asymmetric encryption solves.

The current standard is AES-256-GCM (or ChaCha20-Poly1305 where there is no AES acceleration). It has five pieces:

Key: 32 random bytes. Everything depends on its secrecy.
Nonce (number used once): 12 bytes that are never repeated with the same key. It is not secret: it is stored alongside the ciphertext. Its job is to make encrypting the same message twice give different results.
Ciphertext: the same size as the original.
Authentication tag: 16 bytes that guarantee integrity. On decryption, if a single bit of the ciphertext (or of the associated data) has changed, verification fails and nothing is returned.
Associated data (AAD): plaintext information that is not encrypted but is authenticated: for example, the order_id an encrypted phone number belongs to, so that nobody can copy Anna's encrypted phone number onto Mark's order.

# km0/services/common/crypto.py
# pip install cryptography
import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

def generate_key() -> bytes:
    return AESGCM.generate_key(bit_length=256)          # 32 bytes from os.urandom

def encrypt(key: bytes, plaintext: bytes, aad: bytes = b"") -> bytes:
    nonce = os.urandom(12)                              # NEW on every encryption
    ciphertext_and_tag = AESGCM(key).encrypt(nonce, plaintext, aad)
    return nonce + ciphertext_and_tag                   # the nonce travels in front, in the clear

def decrypt(key: bytes, blob: bytes, aad: bytes = b"") -> bytes:
    nonce, ciphertext_and_tag = blob[:12], blob[12:]
    return AESGCM(key).decrypt(nonce, ciphertext_and_tag, aad)   # InvalidTag if anything changed

if __name__ == "__main__":
    k = generate_key()
    blob = encrypt(k, b"+34 600 123 456", aad=b"P-2026-000123")
    print(len(blob))                                    # 12 + 15 + 16 = 43 bytes
    print(decrypt(k, blob, aad=b"P-2026-000123"))       # b'+34 600 123 456'
    try:
        decrypt(k, blob, aad=b"P-2026-000124")          # another order: the AAD does not match
    except Exception as e:
        print(type(e).__name__)                         # InvalidTag

Why you must never reuse a nonce. GCM internally generates a stream of bits (a keystream) from the key and nonce, and XORs it with the message. If two messages are encrypted with the same key-nonce pair, the XOR of the two ciphertexts equals the XOR of the two plaintexts: the attacker strips the encryption away without knowing the key, and can also recover the authentication key and forge tags. With random 96-bit nonces, the collision probability becomes worrying at around 2³² messages under the same key (some 4 billion): more than enough for one key per service per year, but it is the reason keys are rotated and why a counter (a sequential nonce) is preferable when there is a single writer that can guarantee it never repeats.

Asymmetric encryption and hybrid encryption

In asymmetric encryption there is a key pair: whatever is encrypted with the public key can only be decrypted with the private key, and whatever is signed with the private key can be verified by anyone with the public key. It solves distribution: inventory publishes its public key and anyone can send it a secret that only it can open. Two families:

Family	Examples	Key size for ~128 bits of security	Use
RSA	RSA-2048, RSA-3072	2048-3072 bits	Signatures (RS256 from 06-01), certificates; encrypting small keys
Elliptic curves	ECDSA P-256, Ed25519 (signing), X25519 (key exchange)	256 bits	Signatures (ES256, EdDSA), key exchange in TLS 1.3; faster and more compact

What you do not do is encrypt bulk data with RSA: it is hundreds of times slower than AES and can only encrypt messages shorter than the key. The universal solution is hybrid encryption: a random symmetric key is generated for the message, the message is encrypted with AES-GCM, and that key (32 bytes) is encrypted with the recipient's public key. TLS does exactly this on every connection (section 6), and the envelope encryption of section 10 is the same pattern applied to storage.

Hashes, HMAC and digital signatures

A cryptographic hash (SHA-256, SHA-3, BLAKE2) condenses any input into a fixed-size value in such a way that it is infeasible to find the input from the hash, or two inputs with the same hash. It provides integrity only if the hash arrives over a trusted channel: an attacker who can change the file can change the hash that accompanies it. That is why there are two keyed constructions:

HMAC (hash-based message authentication code): HMAC(key, message). Only someone holding the key can generate or verify it. It provides integrity and authenticity between parties that share a secret; it does not provide non-repudiation (either party could have generated it). It is what signs MinIO's presigned URLs (04-03) and HS256 JWTs.
Digital signature (RSA-PSS, ECDSA, Ed25519): a hash of the message encrypted with the private key. Anyone can verify it with the public key; only the signer could have produced it. Integrity, authenticity and non-repudiation. It is RS256 in 06-01 and the foundation of TLS certificates.

A practical case for HMAC in Kilometre Zero: the event envelope from 02-04. Today any process with access to Kafka can publish a fake order.created to orders.events, or modify an event replayed from the lake. Adding an HMAC of the envelope, with a key held only by the legitimate producers and the consumers, lets inventory and analytics discard tampered events:

# km0/services/common/signed_events.py
import hmac, hashlib, json

def _canonical(event: dict) -> bytes:
    """Deterministic serialisation: same keys, same order, no whitespace.
    Without this, two equivalent JSON documents would give different HMACs."""
    unsigned = {k: v for k, v in event.items() if k != "signature"}
    return json.dumps(unsigned, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode()

def sign_event(event: dict, key: bytes) -> dict:
    event["signature"] = hmac.new(key, _canonical(event), hashlib.sha256).hexdigest()
    return event

def verify_event(event: dict, key: bytes) -> bool:
    expected = hmac.new(key, _canonical(event), hashlib.sha256).hexdigest()
    # compare_digest: constant time, so as not to leak how many bytes match
    return hmac.compare_digest(expected, event.get("signature", ""))

In the orders producer (02-04), before producer.send, you call sign_event(event, EVENTS_KEY); in every consumer, if not verify_event(event, EVENTS_KEY): discard_and_alert(). The key is distributed with the secrets manager from 06-04. When consumers must not be able to produce (for example, a third party reading delivery.dashboard), the envelope is signed with Ed25519 instead of HMAC: the same code with cryptography.hazmat.primitives.asymmetric.ed25519.

Summary table: which primitive for which problem

Need	Primitive	Key	Example in Kilometre Zero
Encrypting bulk data	AES-256-GCM (AEAD)	Symmetric	Anna's phone number in `km0_orders`; volumes; backups
Sending a key to someone / agreeing on one	RSA-OAEP, X25519 (ECDH)	Asymmetric	TLS handshake; envelope encryption with a KMS
Detecting tampering with a shared secret	HMAC-SHA256	Symmetric	Envelope of `orders.events`; presigned URLs
Detecting tampering without a shared secret, with authorship	Ed25519 / ECDSA / RSA-PSS signature	Asymmetric	RS256 JWT (06-01); certificates; events to third parties
Fingerprint of some content	SHA-256	None	Deduplicating photos in `km0-photos`; `ETag`; chained audit hash (06-05)
Storing passwords	Argon2id / bcrypt	None (salt)	`identity` (06-01)
Deriving keys from a master key	HKDF	Symmetric	One key per service and purpose from a root

Data in transit: TLS

TLS (Transport Layer Security) is the protocol that encrypts and authenticates a TCP connection: HTTPS is HTTP over TLS, and gRPC, PostgreSQL, Kafka and Redis support it natively. It combines everything above: asymmetric encryption to agree on a key, symmetric encryption for the traffic, signatures to authenticate the server, hashes for integrity.

The TLS 1.3 handshake, in brief:

sequenceDiagram
    participant C as Client (orders)
    participant S as Server (inventory)
    C->>S: ClientHello: versions, suites, ephemeral public key (X25519)
    S->>C: ServerHello: chosen suite, ephemeral public key
    Note over C,S: Both derive the same symmetric key (ECDH).<br/>From here on everything is encrypted.
    S->>C: Certificate (chain) + CertificateVerify (signature with the private key) + Finished
    C->>C: Verifies the chain up to a trusted CA,<br/>that the name matches (SAN), dates, revocation
    C->>S: Finished
    C->>S: Application data (gRPC) encrypted with AES-GCM / ChaCha20

Concepts worth pinning down:

Certificate: the server's public key plus its identity (DNS name in the Subject Alternative Name field, SAN), signed by a certificate authority (CA). The client trusts the CA (it has its root certificate in its trust store) and, transitively, the server.
Chain of trust: root → intermediate → server. The root is kept offline; intermediates sign the servers' certificates. On the Internet, roots are distributed by operating systems and browsers; inside Kilometre Zero, the CA will be our own (this lesson creates it with OpenSSL; 06-04 turns it into a PKI with automatic issuance and client certificates).
Forward secrecy: in TLS 1.3 the session key is agreed using ephemeral keys that are destroyed; stealing the server's private key tomorrow does not allow traffic captured today to be decrypted.
TLS 1.3 (2018) removed the weak suites, cut the handshake to a single round trip and encrypts the server's certificate. TLS 1.0 and 1.1 are obsolete; 1.2 is acceptable if properly configured. Always configure minimum 1.2, preferred 1.3.
Termination: the TLS connection can end at the service itself or at a component in front of it (the gateway from 06-05 will terminate TLS from the Internet). Between the terminator and the service, if there is a network, TLS is needed again.

What TLS does not do by default: authenticate the client. inventory will know the conversation is encrypted, and orders will know it is talking to the real inventory, but inventory does not know whether the client is orders or some random process. That is mTLS, and it is lesson 06-04.

TLS in gRPC between `orders` and `inventory`

We are going to replace the add_insecure_port that 02-03 left "without TLS for now". First, an internal CA and a server certificate with OpenSSL:

# km0/certs/generate.sh  (development; in production the PKI from 06-04 will do it)
set -e
mkdir -p km0/certs && cd km0/certs

# 1. Kilometre Zero root CA (EC P-256 key, valid for 10 years, development only)
openssl ecparam -name prime256v1 -genkey -noout -out km0-ca.key
openssl req -x509 -new -key km0-ca.key -sha256 -days 3650 \
    -subj "/CN=Kilometre Zero Internal CA" -out km0-ca.crt

# 2. Key and CSR for inventory, with SAN (DNS name in Compose and localhost for testing)
openssl ecparam -name prime256v1 -genkey -noout -out inventory.key
openssl req -new -key inventory.key -subj "/CN=inventory" -out inventory.csr

cat > inventory.ext <<EOF
subjectAltName = DNS:inventory, DNS:inventory.km0.internal, DNS:localhost
extendedKeyUsage = serverAuth
EOF

# 3. The CA signs inventory's certificate (90 days: short-lived certificates get rotated, not watched)
openssl x509 -req -in inventory.csr -CA km0-ca.crt -CAkey km0-ca.key -CAcreateserial \
    -days 90 -sha256 -extfile inventory.ext -out inventory.crt

openssl x509 -in inventory.crt -noout -subject -dates -ext subjectAltName
rm inventory.csr inventory.ext

The detail that causes the most failures is the SAN: the client compares the name it connected to (inventory, the service name in Compose) with those in the certificate; if they do not match, it rejects the connection even though the signature is valid. The CN no longer counts for that check.

Server (services/inventory/server.py), only the start-up changes:

import grpc
from concurrent import futures

def _read(path):
    with open(path, "rb") as f:
        return f.read()

credentials = grpc.ssl_server_credentials(
    private_key_certificate_chain_pairs=[(_read("certs/inventory.key"),
                                          _read("certs/inventory.crt"))],
    # root_certificates and require_client_auth arrive in 06-04 (mTLS)
)
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),
                     interceptors=[AuthInterceptor()])            # the one from 06-01 is still there
inventory_pb2_grpc.add_InventoryServicer_to_server(InventoryServicer(), server)
server.add_secure_port("0.0.0.0:50051", credentials)             # before: add_insecure_port

Client in orders:

# km0/services/orders/inventory_client.py (fragment)
credentials = grpc.ssl_channel_credentials(root_certificates=_read("certs/km0-ca.crt"))
channel = grpc.secure_channel("inventory:50051", credentials)   # the name must be in the SAN
stub = inventory_pb2_grpc.InventoryStub(channel)
# The JWT still travels in the metadata, now encrypted by TLS:
stub.ReserveStock(req, metadata=(("authorization", f"Bearer {token}"),), timeout=2)

The client is given only the CA, not inventory's certificate: that way, when inventory's certificate is renewed in 90 days' time, orders will keep trusting it without changing anything. If orders connects via localhost:50051 from the host, it works because localhost is in the SAN; if it connected by IP, it would fail (Ssl handshake failed ... Hostname mismatch).

In docker-compose.yml, inventory and orders mount ./certs read-only; the km0-ca.key file is not mounted in any service (it is only needed for signing):

  inventory:
    build: ./services/inventory
    volumes:
      - ./certs/inventory.crt:/app/certs/inventory.crt:ro
      - ./certs/inventory.key:/app/certs/inventory.key:ro
  orders:
    build: ./services/orders
    volumes:
      - ./certs/km0-ca.crt:/app/certs/km0-ca.crt:ro

You can check that the channel is encrypted by capturing the traffic: docker compose exec orders tcpdump -A port 50051 used to show aged-cheese and the JWT in the clear; now there are only unreadable bytes after the ClientHello.

TLS in PostgreSQL, Kafka, Redis and MinIO

Every data store from Modules 2 and 4 has its own way of enabling TLS. We will not develop it here, but we do pin down where it goes, because every Kilometre Zero client will have to change its connection string:

System	Server side	Client side (Python)	Note
PostgreSQL (`km0_analytics`, `km0_inventory`)	`ssl = on`, `ssl_cert_file`, `ssl_key_file` in `postgresql.conf`; `hostssl` in `pg_hba.conf` to enforce it	`psycopg.connect(..., sslmode="verify-full", sslrootcert="certs/km0-ca.crt")`	`sslmode=require` encrypts but does not verify the server; `verify-full` does
Kafka (`orders.events`, `delivery.positions`...)	Listener `SSL://:9093`, `ssl.keystore.location`, `ssl.truststore.location`; `SASL_SSL` if there is authentication as well	`KafkaProducer(security_protocol="SSL", ssl_cafile="certs/km0-ca.crt")`	Kafka uses JKS/PKCS12 keystores; convert them with `keytool`/`openssl pkcs12`
Redis (cache, sessions, revoked tokens)	`tls-port 6379`, `port 0`, `tls-cert-file`, `tls-key-file`, `tls-ca-cert-file`	`redis.Redis(ssl=True, ssl_ca_certs="certs/km0-ca.crt")` or `rediss://`	Redis ≥ 6; the Redis Cluster clients from 04-05 support it too
Cassandra (`km0_orders`)	`client_encryption_options: enabled: true` in `cassandra.yaml`	`Cluster(ssl_context=ctx)` with an `ssl.SSLContext` loading the CA	Node-to-node communication is encrypted separately (`server_encryption_options`)
MinIO (`km0-photos`...)	Certificates in `~/.minio/certs/public.crt` and `private.key`	`boto3.client("s3", endpoint_url="https://minio:9000", verify="certs/km0-ca.crt")`	The presigned URLs from 04-03 switch to `https://` with no further changes
HDFS (`/km0/events/...`)	`dfs.http.policy=HTTPS_ONLY`, `dfs.encrypt.data.transfer=true`	WebHDFS over `https://`	Kerberos for authentication is a world of its own, beyond the scope of this course

The common rule: always verify the server's certificate against the internal CA. A client that encrypts without verifying (sslmode=require, verify=False) protects against passive eavesdropping but not against an impostor server, which is the man-in-the-middle attack TLS exists to prevent.

Data at rest: disk, database, column

Data spends far more time at rest than in transit, and there the threats are different: a disk decommissioned without wiping, a cloned cloud volume, a backup in an accessible bucket, a curious database administrator, or a SELECT * from a compromised service. There are three levels, which stack:

Level	What it protects	Against what	Not against	Cost
Disk / volume (LUKS, dm-crypt, EBS/PD volume encryption)	Everything on the disk	Stolen or recycled disk, cloned volume, physical access	Anyone with access to the running system: the database sees the data in the clear	Almost nil (AES-NI); transparent
Database (TDE: transparent data encryption, encryption of data files and WAL)	The DB files and their backups	Copied files, leaked backups	DB administrators, any authorized `SELECT`	Low; requires managing the key outside the DB
Column / field (the application encrypts before the `INSERT`)	Specific fields: phone number, address, IBAN	DBAs, dumps, a compromised service reading the table, leaks from replicas and analytics	A compromise of the service that holds the key	High: the field cannot be indexed or searched; keys must be managed per service

Kilometre Zero applies all three: encrypted volumes across the whole infrastructure (one line of configuration in the cloud or in the installation), TDE wherever the engine offers it (Cassandra, and PostgreSQL with extensions or file encryption), and field encryption for the sensitive personal data that only one service needs in the clear. Anna's phone number is needed by delivery so the courier can call on arrival; analytics should never see it. With field encryption and a key of delivery's own, even if analytics reads the table, it sees bytes.

Envelope encryption: Anna's phone number

If each service encrypts its fields with a key, where does that key live? Storing it next to the data defeats the encryption; storing it in each process makes rotation impossible (the whole table would have to be re-encrypted). The standard pattern is envelope encryption:

There is a master key (KEK, key encryption key) that never leaves a KMS (key management service): AWS KMS, Google Cloud KMS, Azure Key Vault, or HashiCorp Vault's transit engine. The KMS offers two operations: encrypt(bytes) and decrypt(bytes), with access control and auditing.
For each piece of data (or each batch, or each customer) a random data key (DEK) is generated, the data is encrypted with AES-GCM and the DEK, and the DEK is encrypted with the KEK in the KMS.
The encrypted data and the encrypted DEK are stored together. The plaintext DEK is discarded.
To read: the KMS is asked to decrypt the DEK, and the data is decrypted with it.

flowchart LR
    P[Anna's phone number<br/>+34 600 123 456] -->|AES-GCM with DEK| PE[Encrypted phone number]
    DEK[Random DEK<br/>32 bytes] -->|KMS.encrypt with KEK| DEKE[Encrypted DEK]
    PE --> DB[(km0_orders<br/>phone_enc, dek_enc, kek_id)]
    DEKE --> DB
    KMS[KMS / Vault transit<br/>KEK 'km0-orders-2026' never leaves]
    DEK -.generated locally.-> DEK

The decisive advantage: rotating the KEK does not force you to re-encrypt the data, only to re-encrypt the DEKs (32 bytes each, and it can be done lazily), and revoking a service's access to the KEK leaves all its data unreadable at a stroke, even if it has the entire table. In Kilometre Zero we do not have a cloud KMS; in 06-04 Vault will take on that role with its transit engine. For this lesson, we simulate the KMS with a class that exposes only encrypt_dek/decrypt_dek and keeps the KEK out of reach of the business code:

# km0/services/orders/field_encryption.py
import os, base64, json
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

class LocalKMS:
    """Development stand-in for a KMS: the KEK lives here and is never exposed.
    In 06-04 this class is replaced by Vault's 'transit' engine."""
    def __init__(self, kek_by_id: dict[str, bytes], active_kek: str):
        self._keks = kek_by_id
        self.active_kek = active_kek

    def encrypt_dek(self, dek: bytes) -> tuple[str, bytes]:
        nonce = os.urandom(12)
        return self.active_kek, nonce + AESGCM(self._keks[self.active_kek]).encrypt(nonce, dek, b"dek")

    def decrypt_dek(self, kek_id: str, encrypted_dek: bytes) -> bytes:
        kek = self._keks[kek_id]                  # KeyError if the KEK was retired: crypto-shredding
        return AESGCM(kek).decrypt(encrypted_dek[:12], encrypted_dek[12:], b"dek")


class FieldEncryptor:
    """Encrypts fields of a record with envelope encryption and AAD = the record's id."""
    def __init__(self, kms: LocalKMS):
        self._kms = kms

    def encrypt(self, value: str, record_id: str) -> str:
        dek = AESGCM.generate_key(bit_length=256)
        nonce = os.urandom(12)
        ciphertext = AESGCM(dek).encrypt(nonce, value.encode(), record_id.encode())
        kek_id, encrypted_dek = self._kms.encrypt_dek(dek)
        envelope = {"v": 1, "kek": kek_id,
                    "dek": base64.b64encode(encrypted_dek).decode(),
                    "n": base64.b64encode(nonce).decode(),
                    "c": base64.b64encode(ciphertext).decode()}
        return json.dumps(envelope, separators=(",", ":"))   # stored as TEXT/BLOB

    def decrypt(self, envelope_json: str, record_id: str) -> str:
        s = json.loads(envelope_json)
        dek = self._kms.decrypt_dek(s["kek"], base64.b64decode(s["dek"]))
        return AESGCM(dek).decrypt(base64.b64decode(s["n"]), base64.b64decode(s["c"]),
                                   record_id.encode()).decode()


if __name__ == "__main__":
    # In development, the KEK comes from an environment variable loaded from the secrets manager (06-04)
    kms = LocalKMS({"km0-orders-2026": base64.b64decode(os.environ["KM0_KEK_ORDERS"])},
                   active_kek="km0-orders-2026")
    enc = FieldEncryptor(kms)
    envelope = enc.encrypt("+34 600 123 456", "P-2026-000123")
    print(envelope[:80], "...")
    print(enc.decrypt(envelope, "P-2026-000123"))       # +34 600 123 456
    # Copying the envelope onto Mark's order is no use: the AAD gives it away
    try:
        enc.decrypt(envelope, "P-2026-000124")
    except Exception as e:
        print("Rejected:", type(e).__name__)            # InvalidTag

In Cassandra (km0_orders), the phone column becomes phone_enc text, and in services/orders only the code that serves delivery calls decrypt. The query requirements change: you cannot run WHERE phone = ... on a field encrypted with a random nonce. If you need to search by phone number (for example, so that support can locate an order), you add a phone_hmac column with HMAC(index_key, normalised phone): deterministic, searchable, and not reversible. It is the same trick as the pseudonymisation in section 12.

For payment fields, the rule is not to have them: the card number (PAN) never reaches Kilometre Zero; the browser hands it directly to the payment gateway, which returns a payment token that payments stores. Encrypting a PAN yourself means taking on full PCI DSS; delegating is the only sensible option for a marketplace.

Key management: rotation, environments and where keys do not belong

Encryption shifts the problem: from protecting data to protecting keys, which are few and small. Principles:

Never in the code or in the repository. Not in a .py, not in a version-controlled docker-compose.yml, not in a container image. A secret in Git stays in the history forever. The environment variable in the previous example is an intermediate step that 06-04 replaces with injection from Vault.
One key per environment: development, testing and production share no keys. A dump of the test database must never be decryptable with the production key.
One key per service and per purpose: orders cannot decrypt delivery's fields, and the event HMAC key is not the field encryption key. With HKDF several can be derived from a root if necessary, but production KEKs live in the KMS.
Scheduled rotation: the KEK annually (or sooner if there is any suspicion), the per-record DEKs (no need to rotate them: they are regenerated on rewrite), TLS certificates every 90 days or less, the JWT signing key every few months with kid. Everything encrypted stores the identifier of the key it was made with, so it can be decrypted during the transition.
Separation of duties: whoever administers the database does not administer the KMS; whoever deploys does not see the production keys.
Backups of the keys, encrypted and somewhere other than the data: encrypted data whose key has been lost is deleted data.

What is principle here will be operations in 06-04: Vault, dynamic credentials, leases and auditing of access to every key.

Personal data: minimisation, pseudonymisation, anonymisation and the right to be forgotten

Anna, Mark and Lucy are people, and their names, addresses, phone numbers, emails, purchase history and (in delivery.positions) the couriers' positions are personal data subject to the GDPR. Cryptography is one of the tools, not the only one, and design decisions matter more than algorithms:

Minimisation: do not collect or keep what you do not need. The data lake from 04-02 stored the complete event in /km0/events/<day>/orders.jsonl, with email and address. analytics needs to know that the same customer bought three times, not who they are. And daily_sales does not even need that.
Pseudonymisation: replacing direct identifiers with a stable pseudonym that can only be reversed with information kept separately. customer_id instead of the email is the bare minimum; better still, an HMAC of the customer_id with a key analytics does not have, so that not even an accidental JOIN with km0_orders re-identifies anyone. Pseudonymised data is still personal, but the risk from a lake leak drops dramatically.
Anonymisation: transforming the data so that re-identification is reasonably impossible: aggregating (sales per product and day, not per customer), generalising (city instead of address; time band instead of instant), suppressing rare values (a single order of crianza-wine in Lleida on a Tuesday identifies someone even without a name). The aggregates in km0_analytics are anonymous; the pseudonymised events are not.
Tokenisation: replacing a value with a token that has no mathematical relationship to it, with a vault that stores the mapping; it is what the payment gateway does with the card.
Right to erasure ("to be forgotten"): when Lucy asks for her data to be deleted, it has to be deleted from km0_orders, from the replicas, from the backups, from the lake and from the Kafka topics with long retention. Deleting from an immutable backup or from a Kafka log is impracticable. Crypto-shredding solves it: if Lucy's personal data was encrypted with a DEK of Lucy's own (one per customer, stored encrypted by the KEK), destroying that DEK turns all her data, everywhere, into unrecoverable noise, without touching the backups. That is the compelling reason to choose "one DEK per customer" rather than "one per record" in section 10.

Pseudonymising events for the lake, in code:

# km0/services/analytics/pseudonymize.py
# Runs in the consumer that writes /km0/events/<day>/orders.jsonl (04-02),
# so that the lake never receives direct identifiers.
import hmac, hashlib

DIRECT_FIELDS = {"email", "name", "phone", "address"}                # removed
PSEUDONYM_FIELDS = {"customer_id"}                                   # replaced

def pseudonymize(event: dict, pseudonym_key: bytes) -> dict:
    data = dict(event["data"])
    for field in DIRECT_FIELDS:
        data.pop(field, None)
    for field in PSEUDONYM_FIELDS:
        if field in data:
            data[field] = hmac.new(pseudonym_key, data[field].encode(),
                                   hashlib.sha256).hexdigest()[:24]
    if "delivery_address" in data:                  # generalise: only the market
        data["market"] = data.pop("delivery_address").get("market")   # 'girona'
    return {**event, "data": data}

# Input:   {"type": "order.created", "data": {"order_id": "P-2026-000125", "customer_id": "u-lucy",
#           "email": "[email protected]", "phone": "+34 6...", "lines": [...],
#           "delivery_address": {"street": "...", "market": "lleida"}}}
# Output:  {"type": "order.created", "data": {"order_id": "P-2026-000125",
#           "customer_id": "9f1c2a...b7", "lines": [...], "market": "lleida"}}

pseudonym_key is held by identity (or the KMS), not by analytics: that way the data team can count returning customers and compute recommendations (05-03) without being able to know who 9f1c2a...b7 is. If one day that customer needs to be contacted (for example, for a product recall), the request goes through identity, which can recompute each customer's HMAC and find the match, with an audit trail of who asked for it (06-05).

Compliance warning. The GDPR (and the LOPDGDD in Spain) demands far more than encryption: a legal basis, information for the data subject, a record of processing activities, an impact assessment where positions are tracked, contracts with processors (the payment gateway, the cloud provider), and breach notification within 72 hours. PCI DSS regulates any contact with card data. What is described here is the technical design; the retention policy, the scope of pseudonymisation and the erasure procedure must be defined and reviewed with a data protection and security professional.

Kilometre Zero protection map

With all of the above, every piece of data on the platform is located and protected:

Data	Where it lives	In transit	At rest	Personal
Passwords	`identity` (PostgreSQL)	TLS	Argon2id (irreversible)	Yes
Access JWT	App memory; gRPC metadata / `Authorization`	TLS (gRPC and HTTP)	Not persisted	Contains `sub`
Catalogue, prices, photos	`catalog` (PostgreSQL), Redis, `km0-photos`	TLS; presigned URLs over HTTPS	Encrypted volume; SSE in MinIO	No
Orders (lines, amounts)	`km0_orders` (Cassandra)	TLS client-to-node and node-to-node	Encrypted volume + TDE	Yes (linked to `customer_id`)
Phone number and delivery address	`km0_orders`	TLS	Field encryption (envelope, DEK per customer, KEK in KMS)	Yes, sensitive
Card data	Not stored: gateway token in `payments`	TLS browser → gateway	Opaque token	PCI DSS delegated
Stock	`km0_inventory` (`inv-bcn`, `inv-vlc`)	TLS	Encrypted volume	No
Events (`orders.events`, `inventory.alerts`)	Kafka	TLS + HMAC of the envelope	Encrypted volume; limited retention	Yes, until pseudonymised
Positions (`delivery.positions`, `delivery.dashboard`)	Kafka, Flink	TLS + HMAC	Short retention (hours)	Yes (couriers): minimise and aggregate
Event lake	HDFS `/km0/events/...`	HTTPS / encrypted transfer	Encrypted volume; pseudonymised on write	Pseudonyms
Aggregates	`km0_analytics` (PostgreSQL)	TLS	Encrypted volume	Anonymous
PDF invoices	`km0-invoices` (MinIO)	Presigned HTTPS	SSE-S3 with MinIO's key + object lock	Yes
Backups	`km0-backups` (MinIO)	HTTPS	Encrypted by the backup process (AES-GCM, KMS key) before upload, in addition to SSE	Yes
etcd configuration	etcd	TLS between peers and clients	Encrypted volume; no secrets inside (06-04)	No

On server-side encryption in MinIO/S3 (SSE): the store encrypts each object on write and decrypts it when serving it. SSE-S3 (key managed by the store) protects against disks and volumes; SSE-KMS (key in the KMS, with per-access auditing) additionally protects against the store's administrators; SSE-C (the client sends the key on every request) leaves custody with the client. For km0-photos SSE-S3 is enough; for km0-invoices and km0-backups, SSE-KMS and, in the case of the backups, additional client-side encryption so that a compromise of MinIO does not expose them.

Common Mistakes and Tips

Encrypting without authenticating. Bare AES-CBC or AES-CTR allow the ciphertext to be manipulated. Always use an AEAD mode (GCM, ChaCha20-Poly1305) or encryption + HMAC (encrypt-then-MAC).
Reusing nonces. A fixed nonce, or one derived from something repeatable (the order id), destroys AES-GCM. os.urandom(12) per encryption, or a counter that is persisted.
Inventing cryptography. No home-made algorithms and no creative combinations of primitives. cryptography with its high-level API (AESGCM, Fernet) and standard protocols (TLS) are the limit of what a product team should touch.
verify=False in development that makes it to production. A TLS client that does not verify the certificate is one passive eavesdropper too many. Use the internal CA in development as well.
Certificate without the right SAN. "Hostname mismatch" is the most common TLS error; the SAN must contain every name by which the service is reached.
Storing the key next to the data (in the same database, in the same bucket, in the docker-compose.yml). Envelope encryption with the KEK outside, in the KMS.
Encrypting a field and then searching by it. It is not possible with a random nonce; plan a deterministic HMAC column if you need to search.
Comparing HMACs with ==. A normal comparison stops at the first differing byte and leaks information through timing; hmac.compare_digest, always.
Confusing pseudonymising with anonymising. Replacing the email with a hash is still personal data; only sufficient aggregation or generalisation anonymises.
Backups in the clear "because the bucket is private". Buckets are made public by mistake with astonishing frequency. Encrypt before uploading.
Tip: store with every encrypted piece of data the key identifier and the format version ({"v":1,"kek":"km0-orders-2026",...}). Without that, the first key rotation is a blind migration.
Tip: test recovery: an encrypted backup that has never been restored with the key from another environment is a backup that does not exist.

Exercises

Exercise 1: A tampered event

An unknown process inside the Kilometre Zero network publishes to orders.events an order.created event for P-2026-000126 with a line of 200 units of crianza-wine at €0.01. Explain what happens in inventory and analytics (a) with the design from 02-04 unchanged, (b) with TLS in Kafka but no HMAC, (c) with the HMAC from section 4. Then state what the HMAC protects that TLS does not, and vice versa, and why the HMAC does not stop the process from carrying on publishing junk events to the topic (which lesson solves that).

Exercise 2: Lucy exercises her right to erasure

Lucy asks for her data to be deleted. List where it is (use the table in section 13) and, for each place, whether it can be deleted directly, whether crypto-shredding is needed, or whether the data is no longer personal. Explain what would have had to be decided in section 10 for crypto-shredding to work, and what remains in km0_analytics afterwards.

Exercise 3: Rotating the KEK

The security team decides to rotate km0-orders-2026 to km0-orders-2027. Describe step by step, using the LocalKMS and FieldEncryptor classes, what needs to change so that (1) new data is encrypted with the new KEK, (2) old data can still be read, (3) old data is migrated to the new one without re-encrypting the phone numbers, and (4) the old one is retired. Which field of the envelope makes step 2 possible? How many bytes have to be re-encrypted per record in step 3?

Solutions

Exercise 1.

(a) The event arrives, inventory consumes it and reserves 200 units of crianza-wine from Roble Alto Winery's stock (or exhausts the stock and triggers inventory.alerts), analytics writes it to the lake and daily_sales books €2 of wine sales; nobody notices anything because the event is syntactically correct. (b) TLS encrypts the channel between each client and the broker and authenticates the broker, but not the producer: the unknown process opens its own valid TLS connection and publishes all the same; it only protects against someone eavesdropping on or modifying another client's traffic. (c) Both consumers compute HMAC(key, canonical envelope), it does not match the signature field (the attacker does not have the key) and they discard the event with an alert; the stock is not touched and the lake never receives it. The HMAC protects the integrity and authenticity of the message end to end, even against the broker, against something replaying from the lake or against an unauthorized producer; TLS protects the connection (confidentiality, and that the broker is the genuine one), which the HMAC does not (the event is still readable in the topic). The HMAC does not prevent publishing because Kafka does not require the producer to identify itself: that requires client authentication at the broker (SASL or client certificates, that is, mTLS and ACLs), which is the workload identity of 06-04.

Exercise 2.

identity (account, password hash): direct deletion. km0_orders: lines and amounts linked to customer_id can be deleted or dissociated (replacing customer_id with a placeholder), depending on what the tax obligation to keep invoices requires; encrypted phone number and address: crypto-shredding by destroying Lucy's DEK, which also invalidates the copies in replicas and in the backups in km0-backups, where direct deletion is impracticable. km0-invoices: invoices have a legal retention of years with object lock; they are not deleted, and keeping them is justified by legal obligation (and her contact details can be encrypted with the same DEK). Kafka orders.events: events with direct data that are within retention cannot be edited; if retention is short they expire on their own; if it is long, that is one more reason for events to carry the personal fields encrypted with the customer's DEK, or pseudonymised. HDFS lake: it contains only the HMAC pseudonym; it is still personal data for as long as identity keeps the key and the customer_id; once the account is deleted in identity, the pseudonym can no longer be linked and it can be argued that it becomes anonymised (to be reviewed with the data protection professional). km0_analytics: anonymous aggregates, untouched. The decision needed in section 10 is one DEK per customer (not per record), stored encrypted by the KEK in a central place: destroying that single DEK is the deletion.

Exercise 3.

(1) Add "km0-orders-2027": new_kek to the LocalKMS dictionary and set active_kek = "km0-orders-2027": encrypt_dek will use the new one for every new envelope. (2) Nothing else: decrypt_dek receives s["kek"] from the envelope and looks that KEK up in the dictionary, which still contains the 2026 one. The envelope's "kek" field is what makes it possible. (3) A migration process that, for each record, does dek = kms.decrypt_dek(s["kek"], s["dek"]) and s["kek"], s["dek"] = kms.encrypt_dek(dek), and saves the envelope; the encrypted phone number ("c") and its nonce are left alone. What gets re-encrypted is 32 bytes (the DEK) plus the 12 of the nonce and 16 of the tag of the DEK's envelope: about 60 bytes per record, compared with re-encrypting the whole field and, above all, without the migration process ever needing to see the phone numbers in the clear. (4) Once no envelope in the database, or in the backups you want to keep readable, has "kek": "km0-orders-2026", remove the entry from the dictionary (in Vault: disable the version). Any forgotten envelope will raise KeyError, which is exactly the crypto-shredding behaviour: that is why you check with a query before retiring it.

Conclusion

Protecting a piece of data means guaranteeing three different things, confidentiality, integrity and authenticity, and each has its tool: AES-GCM for encrypting bulk data with a unique nonce and an authentication tag; RSA and elliptic curves for agreeing on keys and signing, always in hybrid mode; hashes for fingerprints, HMAC for integrity with a shared secret, signatures for authorship that everyone can verify. TLS brings them all together for transit, with certificates, chain of trust, SAN and TLS 1.3, and Kilometre Zero now has it between orders and inventory with an internal CA, and pinned down in the configuration of PostgreSQL, Kafka, Redis, Cassandra, MinIO and HDFS. At rest, encryption stacks up in layers (volume, database, field) and envelope encryption with a KEK in a KMS makes it possible to rotate keys, revoke access and delete by destroying a key. The personal data of Anna, Mark and Lucy is minimised, pseudonymised before it enters the lake, aggregated for analytics, and encrypted with one DEK per customer that turns the right to erasure into a 32-byte operation. Card data, quite simply, never comes in. And all of it under the review of a compliance professional, because the GDPR and PCI DSS are a good deal more than cryptography.

There is one piece this lesson has treated as solved with a hand-copied .pub file and a home-grown login form: where identities come from. Customers sign up on the web, employees have corporate accounts, producers want to sign in with their Google account, the courier app needs a flow with no visible password, and orders needs a machine token to talk to inventory. Managing all of that service by service is unworkable. The next lesson centralises identity in a provider (Keycloak), connects employees through LDAP, and explains the protocols that make a token issued there valid across the whole platform: SAML, OAuth 2.0 and OpenID Connect.

Encryption and Data Protection

Contents

Three properties: confidentiality, integrity, authenticity

Symmetric encryption: AES-GCM, keys, nonces and tags

Asymmetric encryption and hybrid encryption

Hashes, HMAC and digital signatures

Summary table: which primitive for which problem

Data in transit: TLS

TLS in gRPC between `orders` and `inventory`

TLS in PostgreSQL, Kafka, Redis and MinIO

Data at rest: disk, database, column

Envelope encryption: Anna's phone number

Key management: rotation, environments and where keys do not belong

Personal data: minimisation, pseudonymisation, anonymisation and the right to be forgotten

Kilometre Zero protection map

Common Mistakes and Tips

Exercises

Exercise 1: A tampered event

Exercise 2: Lucy exercises her right to erasure

Exercise 3: Rotating the KEK

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

Encryption and Data Protection

Contents

Three properties: confidentiality, integrity, authenticity

Symmetric encryption: AES-GCM, keys, nonces and tags

Asymmetric encryption and hybrid encryption

Hashes, HMAC and digital signatures

Summary table: which primitive for which problem

Data in transit: TLS

TLS in gRPC between orders and inventory

TLS in PostgreSQL, Kafka, Redis and MinIO

Data at rest: disk, database, column

Envelope encryption: Anna's phone number

Key management: rotation, environments and where keys do not belong

Personal data: minimisation, pseudonymisation, anonymisation and the right to be forgotten

Kilometre Zero protection map

Common Mistakes and Tips

Exercises

Exercise 1: A tampered event

Exercise 2: Lucy exercises her right to erasure

Exercise 3: Rotating the KEK

Solutions

Conclusion

Distributed Architectures Course

Module 1: Introduction to Distributed Systems

Module 2: Communication in Distributed Systems

Module 3: Consistency and Replication

Module 4: Distributed Storage

Module 5: Distributed Computing

Module 6: Security in Distributed Systems

Module 7: Monitoring and Maintenance

Module 8: Case Studies and Applications

TLS in gRPC between `orders` and `inventory`