The previous lesson ended with two weak points. orders-service identifies itself to Keycloak with a client_secret that lives in a configuration file, and inventory accepts its token with no way of knowing whether the TLS connection reaching it comes from the orders container or from some random process that has copied that secret. And the km0_analytics password is still where 05-05 left it: in the AIRFLOW_CONN_KM0_ANALYTICS environment variable, visible to anyone who can run docker inspect or read the docker-compose.yml. Both are the same disease: the internal network is still being treated as a trusted place where simply "being inside" is enough to be believed, and the secrets that let you be inside are scattered across files, images and environments.

This lesson abandons the perimeter. First, the principle: zero trust, every call is authenticated, authorized and encrypted, no matter where it comes from. Then, the tool for identifying services: per-workload certificates and mTLS, with an internal PKI that issues and rotates them automatically, and the service mesh as a way of getting this without touching the code. Next, service-to-service authorization: what orders may call and what analytics may only read. And finally, secrets management: what a secret is, why it cannot live in the code, the image or the environment, and how HashiCorp Vault holds it, delivers it and, in the case of databases, generates it on demand with an expiry. The code leaves orders and inventory talking with certificates on both sides, inventory authorizing by service identity, and the km0_daily_sales DAG getting into km0_analytics with credentials that Vault creates and destroys on its behalf. The edge gateway, rate limiting and auditing are the next lesson.

1. From the perimeter to zero trust
2. Workload identity: per-service certificates
3. mTLS: authentication at both ends
4. Internal PKI: root CA, intermediate CA, issuance and rotation
5. Service mesh: mTLS without touching the code
6. Service-to-service authorization
7. Internal threats and countermeasures
8. Secrets management: what a secret is and where it does not belong
9. HashiCorp Vault: engines, authentication, leases and auditing
10. Injecting secrets into services
11. Kilometre Zero: PKI with OpenSSL and gRPC with mTLS
12. Kilometre Zero: Vault and dynamic credentials for km0_analytics
13. Common Mistakes and Tips
14. Exercises
15. Conclusion

The classic security model is the castle and moat: a firewall separates "outside" (hostile) from "inside" (trusted), and once inside, services talk to each other freely. Fallacy 4 from 01-04, "the network is secure", is precisely the belief that the moat is enough. And it is not, for reasons we have already seen pile up:

• The "inside" is huge and heterogeneous: dozens of services, hundreds of Spark and Airflow jobs, containers that are created and destroyed, cloud providers, a developer's laptop on the VPN.
• A single compromised component (a catalog with a vulnerable dependency, an image with a backdoor) gives lateral access to everything: hdfs dfs -rm from any container, SELECT * on km0_orders, publishing to orders.events.
• The attackers with the greatest impact are usually insiders, or outsiders who have already crossed the moat: leaked credentials, phishing an operator, a supplier with access.

Zero trust replaces "I trust you because you are inside" with "I trust nobody because of their position on the network": every request, including those between internal services, must prove who is making it (authentication), be evaluated against a policy (authorization), and travel encrypted. We already have the three pillars for users (06-01, 06-03) and for encryption (06-02); what is missing is for services to have their own verifiable identity, and for none of it to depend on scattered secrets.

For inventory to know that orders is talking to it, orders needs a workload identity: a credential tied to the service, not to a human, that is not a hand-copied password. The most robust form is an X.509 certificate with the service's identity in the SAN: orders presents its certificate, inventory verifies it against the internal CA and reads the name.

SPIFFE (Secure Production Identity Framework for Everyone) standardises this: every workload has a SPIFFE ID in the form of a URI, spiffe://km0.internal/orders, which goes in the certificate's SAN (an SVID, SPIFFE Verifiable Identity Document), and SPIRE is the reference implementation that issues and rotates those certificates by verifying on which node and with which attributes each process runs (attestation). Istio and other meshes use SPIFFE internally. In Kilometre Zero we will use a DNS SAN (orders.km0.internal) and a SPIFFE URI in the certificates, so that the verification code works with either.

What matters about a certificate as an identity, compared with a client_secret:

• The private key is generated where it is used and never travels: the CA only signs the public key. Stealing the configuration does not yield the key.
• It has a built-in short expiry (hours or days) and is rotated automatically: a stolen certificate expires before it can be exploited much.
• Verification does not require consulting anyone: a chain of signatures up to the CA, as in 06-02.
• It binds the identity to the connection, not to a token that can be forwarded: with mTLS, inventory knows that the other end of this socket is orders.

In the TLS of 06-02, only the server presented a certificate. mTLS (mutual TLS) adds that the client presents one too and the server verifies it. The handshake is the same with one more step:

What each end checks:

The last row is an important design decision: certificate revocation in TLS (CRL lists, OCSP queries) is slow and fragile; modern practice is to issue certificates lasting hours and rotate them, so that revoking simply means ceasing to renew.

mTLS lives alongside the tokens from 06-01; it does not replace them: the certificate says which service is making the call; the JWT in the metadata says on behalf of which user. inventory will see both: "orders (certificate) reserves stock for u-anna (token)".

A PKI (public key infrastructure) is the set of CAs, procedures and tools that issue, distribute and rotate certificates. The one from 06-02 (a km0-ca.key in a directory, signing by hand with OpenSSL) is fine for learning, but not for production, which requires:

• Offline root CA: its private key is only used to sign intermediate CAs, once a year, from an air-gapped machine; if an intermediate is compromised, that intermediate is revoked and another is issued without touching the trust stores (which contain only the root).
• Online intermediate CA: the one that signs service certificates every few hours. It has to be automated and protected (an HSM in demanding production environments, or Vault with its key sealed).
• Automatic issuance: the service (or an agent beside it) generates a key pair, sends a CSR with its identity, the PKI verifies that it is entitled to that identity (by its Kubernetes account, its AppRole role, its node) and signs. Nobody touches openssl by hand.
• Automatic rotation: the agent renews the certificate at two thirds of its lifetime, and the service reloads it without restarting (gRPC and Envoy support this).
• Tools: Vault PKI (Vault's pki engine: the intermediate CA lives in Vault and issues via the API with per-role policies), cert-manager (on Kubernetes: a Certificate resource and the controller issues and renews it, with Vault, your own CA or Let's Encrypt as the issuer), SPIRE, step-ca (smallstep). Section 11 builds the PKI by hand with OpenSSL to understand what these tools automate; section 12 configures Vault's pki engine so that Vault does the issuing.

Configuring mTLS in every service, in every language, with rotation, is repetitive, error-prone work. A service mesh takes it out of the code: next to each service runs a sidecar (an Envoy proxy) that intercepts all its inbound and outbound traffic; the service talks in the clear to its sidecar on localhost, and the sidecars talk to each other over mTLS, with certificates that the mesh's control plane issues and rotates (SPIFFE underneath). The mesh also adds authorization policies (section 6), metrics, traces (07-02) and resilience patterns (07-04) without orders knowing anything about it.

Istio (Envoy as a sidecar or in sidecar-less ambient mode, istiod control plane, PeerAuthentication and AuthorizationPolicy policies) and Linkerd (its own proxy written in Rust, lighter, mTLS by default) are the two main meshes; Consul Connect is HashiCorp's. All of them assume Kubernetes, whose deployment is the subject of 07-05: here it is enough to know that the mesh is the industrial way of getting what we will do by hand in section 11, and that the price is one proxy per instance (latency of ~1 ms per hop, memory) and a control plane to operate.

When to use mTLS in the code and when a mesh: with few services and a single language, in the code (grpcio does it well); from a dozen or so services upwards, or with several languages, or when you want uniform policies and observability, the mesh. Kilometre Zero, with six services in Python, does it in the code in this lesson and will evaluate the mesh when it moves to Kubernetes.

With mTLS, inventory knows that orders is calling it. Now it decides whether orders may do that. It is the RBAC of 06-01 applied to service identities, and the policy is expressed as a table which, in Kilometre Zero, lives in the service itself (and in the mesh, if there were one, as an AuthorizationPolicy):

Every empty cell is a call the server rejects with PERMISSION_DENIED even if the certificate is valid and so is the user's JWT. With this table, a compromised analytics can read stock and the catalogue, and nothing else: it does not reserve, it does not charge, it does not publish orders. This is least privilege for services, and its effect is measured in exercise 1.

Two nuances. First, the service identity and the user identity are combined: ReserveStock requires the caller to be orders and the JWT to carry the customer or operator role; orders cannot reserve without a user behind it except with its svc-orders machine token (06-03), which is only valid for ReleaseReservation in compensations. Second, the policy should live outside the code as soon as it grows (OPA from 06-01, or the mesh), but its semantics are always this table.

A secret is any piece of data whose knowledge grants access: database passwords, Keycloak client_secrets, the payment gateway's API keys, signing keys, certificate private keys, the field encryption KEK, MinIO access tokens, the event HMAC key. Kilometre Zero has already accumulated more than a dozen. And there are places where they must not be:

The analytics:analytics password in AIRFLOW_CONN_KM0_ANALYTICS is in three of those rows at once. What is needed is a system that holds the secrets encrypted, controls who can read each one, delivers each secret only to the authorized process and only for as long as it needs it, rotates without intervention and records every access. That is a secrets manager.

Vault is the reference secrets manager outside the public clouds (and inside them, when neutrality is wanted). Concepts:

• Sealing: Vault's data is encrypted with a master key that, on start-up, has to be reconstructed from several shares (Shamir) or with a cloud KMS (auto-unseal). A sealed Vault hands out nothing.
• Secrets engines, mounted on paths:
  • kv (version 2): static key-value pairs with versioning: kv/km0/orders/keycloak → {client_secret: ...}.
  • database: dynamic credentials. Vault has an administrator user in PostgreSQL and, when an authorized client asks, it runs CREATE ROLE "v-airflow-abc123" WITH PASSWORD '...' VALID UNTIL '...' with the defined GRANTs, returns the username and password, and destroys them when the lease expires. Each job has its own user, ephemeral and auditable.
  • pki: the intermediate CA from section 4: vault write pki_int/issue/km0-services common_name=orders.km0.internal ttl=24h returns a certificate and key.
  • transit: the KMS of the envelope encryption from 06-02: Vault encrypts and decrypts DEKs with keys that never leave it.
  • Others: AWS/GCP (temporary cloud credentials), SSH, RabbitMQ, Kubernetes.
• Authentication methods: how a client proves to Vault who it is: AppRole (a fixed role_id plus a single-use, short-lived secret_id delivered at start-up), Kubernetes (the pod's service account token, which Vault verifies against the API server: no prior secret), TLS certificates, OIDC/JWT (a Keycloak token: orders-service could authenticate to Vault with it), AWS/GCP IAM, and userpass/token for humans and development.
• Policies: HCL stating which paths each identity may read or write: path "kv/data/km0/orders/*" { capabilities = ["read"] }. Deny by default.
• Leases and renewal: every dynamic secret has a lease with a TTL; the client renews it for as long as it needs it, or Vault revokes it on expiry. Revoking a lease revokes the credential in the target system (the DROP ROLE in PostgreSQL).
• Rotation: for static secrets, kv keeps versions and the service reloads; for the PostgreSQL root password Vault uses, rotate-root changes it to one only Vault knows.
• Auditing: an audit device writes every request and response (with the secrets hashed) to a file or syslog: who asked for what, when, under which policy. It will feed the auditing in 06-05.

Alternatives: AWS Secrets Manager and Parameter Store, Google Secret Manager, Azure Key Vault (managed, integrated with each cloud's IAM, with scheduled rotation for RDS and the like); Kubernetes Secrets (base64 objects in etcd: useful as a mechanism for delivery to the pod, but they are not a manager: etcd must be encrypted at rest, there is no versioning and there are no dynamic credentials, and anyone with access to the namespace can read them; the usual pattern is for an operator, External Secrets or the Vault Agent Injector, to populate them from Vault); SOPS (Mozilla: encrypts YAML/JSON files with KMS or age keys so that they can be versioned; suitable for configuration, not for dynamic credentials); Doppler, Infisical, 1Password Secrets Automation.

Holding the secret is not enough; it has to reach the process in a way that does not make it reappear in the places from section 8:

The rule: prefer memory or tmpfs to disk, files to environment, and renewable leases to fixed values. For Airflow, which is not our own code, the pattern is a secrets backend: Airflow supports VaultBackend (apache-airflow-providers-hashicorp), with which PostgresHook(postgres_conn_id="km0_analytics") looks the connection up in Vault instead of in the environment variable; section 12 configures it and, in addition, uses hvac to request dynamic credentials.

We extend the PKI from 06-02 with a conceptually separate client CA (here, for simplicity, the same root signs both uses, but the client certificate carries clientAuth and a SPIFFE ID), and we issue certificates for orders and inventory:

Each service gets serverAuth and clientAuth because almost all of them are both (orders is a server for the web and a client of inventory). The inventory server now demands a client certificate:

The orders client presents its own:

And the interceptor that authorizes by service identity. gRPC exposes the client's certificate in context.auth_context(); from it we extract the SANs and compare them with the policy from section 6:

(For streaming methods such as WatchChanges, handler.unary_stream is wrapped in the same way; it is omitted for brevity.) A reservation now records both identities: in ReserveStock, calling_service() returns spiffe://km0.internal/orders and claims_of(context)["sub"] (06-01) returns u-anna. Test the rejection by issuing a certificate for catalog (./issue.sh catalog) and calling ReserveStock with it: PERMISSION_DENIED: spiffe://km0.internal/catalog may not invoke ReserveStock. And with no client certificate (ssl_channel_credentials with just the CA): the handshake fails before reaching any interceptor, with UNAVAILABLE: ... peer did not return a certificate.

In docker-compose.yml, each service mounts only its own key and certificate plus the CA; km0-ca.key is not mounted in any of them. So as not to rewrite 24-hour certificates by hand, the next section delegates issuance to Vault.

Initial configuration, once (in production this is a versioned script, run with an administration token that is revoked afterwards):

Note rotate-root: after that command, the changeme password stops working and nobody knows it except Vault. And secret_id_num_uses=1: the secret_id is handed to the container at start-up (by the orchestrator, or by an operator the first time), is exchanged for a Vault token and dies. What remains in the service's configuration is the role_id, which on its own is useless.

Every run of load_postgres enters PostgreSQL with a user that did not exist ten seconds earlier and will not exist ten seconds later; pg_stat_activity and the PostgreSQL logs show v-approle-airflow-write-..., different on every run, so that "who deleted the rows for the 19th?" has an exact answer in Vault's audit log.

And the task in dags/daily_sales.py (05-05) ends up like this, with no PostgresHook and no connection configured in the environment:

In Airflow's docker-compose.yml the line AIRFLOW_CONN_KM0_ANALYTICS: postgresql://analytics:analytics@... disappears and VAULT_ROLE_ID (not a secret) and the tmpfs mount of /run/secrets appear. For the connections Airflow manages on its own (WebHDFS, Spark), the airflow.providers.hashicorp.secrets.vault.VaultBackend backend is configured in airflow.cfg, and PostgresHook(postgres_conn_id=...) would look them up in kv/km0/airflow/connections/<id> transparently.

For orders, the same vault_client() reads kv/km0/orders/keycloak for the client_secret from 06-03 (which is finally no longer in the code) and requests its certificate from pki_int/issue/km0-services with common_name=orders.km0.internal and uri_sans=spiffe://km0.internal/orders, renewing it at 16 hours: the issue.sh from section 11 remains as a teaching reference.

• "It's internal, it doesn't need authentication". That is fallacy 4 in other words. Every call between services is authenticated (certificate), authorized (policy) and encrypted.
• mTLS with a valid certificate but without checking the name. The client having a certificate from km0-ca only says that it is someone on the platform; analytics must not be able to reserve stock. Extract the SAN and apply the policy.
• One-year certificates rotated by hand. They expire on a Sunday at 3:00 and nobody remembers. Short-lived certificates with automatic issuance and renewal (Vault PKI, cert-manager, SPIRE).
• Mounting the CA's key in the services "so they can generate their own certificate". Only the PKI has the CA's key; services send a CSR.
• Vault in dev mode in production, or with the root token in daily use. Dev mode is a classroom; the root token is revoked after the initial configuration.
• A secret in Vault and a copy "just in case" in the environment. The copy is the one that leaks. One place only.
• Leases that nobody renews or revokes. A long task dies halfway through because its credential expired; or thousands of zombie roles in PostgreSQL because nobody was revoking. Renew in long tasks, revoke on finishing, a reasonable maximum TTL.
• Kubernetes Secrets as a secrets manager. They are a delivery mechanism in base64; no etcd encryption, no per-access auditing, no dynamic secrets. Populate them from Vault or use the injector.
• Secrets in logs. A print(creds) for debugging ends up in Elasticsearch. Print the username and the lease_id, never the password.
• Tip: enable a secret scanner in CI (gitleaks, trufflehog) and a pre-commit hook: the cheapest moment to find a key in the code is before the commit.
• Tip: rehearse the "bad day": revoke all the leases of a role (vault lease revoke -prefix database/creds/airflow-write) and check that the DAG recovers on the next run by requesting new credentials. A secrets system that has never been rotated in production is one nobody knows how to rotate.
• Compliance warning. Access to databases holding personal data (km0_analytics contains aggregates, but km0_orders does not) and its auditing are subject to the GDPR; key custody procedures and Vault policies must be reviewed with a security professional.

A dependency of catalog includes malicious code that runs arbitrary commands inside the container. List what the attacker can do against inventory, orders, payments, Kafka, km0_orders and Vault (a) with the platform as it stood at the end of 06-03 and (b) with what has been put in place in this lesson. For (b), state exactly which piece (certificate, inventory policy, Vault policy, Kafka ACL) blocks each attempt, and what they can still do.

The key of km0 services CA (the intermediate in Vault) is suspected to be compromised. Describe the steps to replace it without stopping the platform, in order, and what happens to existing mTLS connections and to new ones at each step. Why is there no need to touch km0-ca.crt in any service? What would have happened if the services had trusted the intermediate's certificate directly instead of the root?

The Flink consumer of orders.events (05-04) needs the event HMAC key (06-02) to verify every envelope, and runs as a long-lived job (weeks) on a Flink cluster. Design how it obtains the key from Vault (authentication method, policy, injection mechanism), how it finds out about a key rotation without restarting the job, and how it verifies events signed with the previous key during the transition. Compare with the DAG's dynamic credentials: why is a 2-hour lease no good here?

Exercise 1.

(a) At the end of 06-03: catalog is on the internal network and has the CA for verifying servers, but inventory does not demand a client certificate; the attacker cannot forge a user token (they do not have Keycloak's private key), but they can call inventory with any token they capture or, if catalog has a client_secret of its own in the environment, obtain a svc-catalog token and try ReserveStock (it would depend solely on ROLES_BY_METHOD); they can publish to orders.events (Kafka without producer authentication: fake events, although the HMAC from 06-02 would get them rejected if it was already in place); they can connect to km0_orders if there is a connection string in catalog's environment or if Cassandra does not require authentication; they can read docker-compose.yml or their container's /proc/*/environ; Vault did not exist. (b) With this lesson: inventory demands mTLS, catalog has a certificate (spiffe://km0.internal/catalog) and the policy only allows it GetStock: ReserveStock, ReleaseReservation and AdjustStock fail with PERMISSION_DENIED (blocked by: SERVICE_PERMISSIONS in ServiceInterceptor); orders and payments apply the same table: catalog has no cell at all, everything denied; Kafka with certificate authentication and ACLs: catalog can only read inventory.alerts, not write to orders.events (blocked by: the broker's ACL); km0_orders: catalog has no credentials (there is no connection string in its environment; those of orders are dynamic and live in orders' memory); Vault: catalog can only authenticate with its role_id and an already consumed secret_id; even if it obtained a Vault token, the catalog policy only lets it read its own kv/km0/catalog/* and issue certificates in its own name (the km0-services role ought to be restricted per AppRole to specific allowed_domains, or a role per service should be used; it is an improvement worth applying). What they can do: everything catalog can: read the whole catalogue and the stock (GetStock), read inventory alerts, generate presigned URLs for km0-photos (with catalog's MinIO credentials, which should also be dynamic or minimally scoped), and serve altered content to the web. The blast radius has gone from "the whole platform" to "catalog's list of permissions", which is the definition of zero trust.

Exercise 2.

(1) Generate a new intermediate CA in Vault (pki_int2/intermediate/generate/internal), sign it with the offline root, install it with set-signed, and create the km0-services role in it. (2) Change the issuance path used by the services (or the agent) to pki_int2/issue/km0-services; as each service renews (at 16 h at most), it gets a certificate from the new intermediate, whose chain includes the new intermediate certificate. Existing connections keep working (TLS does not re-verify mid-session); new ones are verified against km0-ca.crt, which signs both intermediates, so both kinds of certificate are valid during the transition. (3) After 24 hours (the max_ttl), no certificate from the old intermediate exists any longer; the old intermediate is revoked at the root (it is published in the root's CRL, if one is used) and pki_int is removed. (4) If zero tolerance is wanted, the compromised intermediate can also be added to a denylist on the servers during the window. There is no need to touch km0-ca.crt because the services trust the root, and the root has not been compromised: any intermediate signed by it is acceptable, and the service certificate carries the full chain in the handshake. If they had trusted the intermediate directly, the trust store of every service would have to be updated before issuing with the new one, and with two intermediates at once during the transition: exactly the coordinated rollout that the root-intermediate hierarchy avoids.

Exercise 3.

Authentication: the Flink job runs on a cluster (YARN, Kubernetes or standalone); if it is Kubernetes, Vault's kubernetes method with the job's service account (no prior secret); if not, AppRole with a secret_id delivered by the orchestrator when launching the job. Policy: path "kv/data/km0/common/events" { capabilities = ["read"] } and nothing else (Flink needs neither PostgreSQL nor PKI, apart from its own certificate for Kafka, which would be pki_int/issue/km0-services). Injection: Vault Agent as a sidecar (or helper container) that renews the token, writes hmac_key to a tmpfs file and rewrites it on every version change of the secret in kv; the job reads the file in a RichFunction.open() and watches it (a thread that checks the mtime every minute, or the agent itself sending SIGHUP). Rotation: kv v2 keeps versions; rotation consists of writing the new key as version N+1 without deleting N, and the injected file contains both (hmac_key and hmac_key_previous, or a small JSON document with kid → key); producers sign with the new one and include a kid in the envelope; the consumer verifies with the key the kid indicates, accepting the previous one for the topic's retention period (7 days, for example) and rejecting it afterwards. Comparison: the DAG's dynamic credential is a temporary identity for a task lasting minutes; a 2-hour lease would force the Flink job to re-authenticate and reload constantly, and it makes no sense as a model: the HMAC key is a long-lived shared secret that changes through scheduled rotation, not a per-run credential. What is short-lived is the Vault token with which the agent reads the secret, and that one is indeed renewed every hour.

Zero trust is the definitive answer to fallacy 4: the network is not secure, so no hop relies on its position in it; every call is authenticated, authorized and encrypted. A service's identity is a short-lived certificate whose key never leaves the process, with a name (DNS or SPIFFE ID) in the SAN, and mTLS means both ends of every connection know who they are talking to; a PKI with an offline root, online intermediates and automatic issuance (Vault PKI, cert-manager, SPIRE) turns rotation into something that happens by itself, and a service mesh (Istio, Linkerd) provides it without touching the code when services multiply. On top of that identity, service-to-service authorization reduces the blast radius of any compromise to one row of a table. The secrets that remain (database passwords, API keys, HMAC keys) leave the code, the images and the environment variables to live in Vault: kv, database, pki and transit engines, AppRole or Kubernetes authentication, deny-by-default policies, leases with TTL and revocation, auditing of every access, and injection into memory or tmpfs. Kilometre Zero now has orders and inventory with mTLS and ServiceInterceptor authorizing by SPIFFE ID, Vault in docker-compose.yml, and the km0_daily_sales DAG getting into km0_analytics with a user that Vault creates for each run and destroys when it finishes; the password that 05-05 left in AIRFLOW_CONN_KM0_ANALYTICS no longer exists, and that of analytics_admin is known only to Vault.

All of this protects the inside. But the platform has an outside: the Internet, from which come the browsers of Anna, Mark and Lucy, the app of 140 couriers, and also the bots, the credential theft and the 40,000 requests per second of the first hour of Grape Harvest Week. Until now each service has done its own TLS termination, its own token verification and its own defence. The last lesson of the module builds the edge: a single gateway that terminates TLS, validates tokens, routes, rate-limits per client with a token bucket in Redis, and keeps an immutable record of who did what, closing the module with the complete table of who can do what in Kilometre Zero.