In 07-01 we left one assumption without a guarantee: that Ana's token, the X-User-* headers set by the gateway and the calls from orders-service to Catalog and Customers travel over a network where nobody listens or impersonates. A properly verified JWT is worthless if someone can read it in transit and reuse it, and an X-User-Roles: admin header is trustworthy only if nobody but the gateway can reach the service. This lesson secures the channel: which threats exist in transit, how the edge is encrypted with TLS and Let's Encrypt certificates managed by cert-manager, which options exist for encrypting and authenticating communication between services (mTLS per application or via service mesh, closing what 05-05 only named), how RabbitMQ and the databases are protected, and how the integrity of the payment provider's webhooks is guaranteed. It ends with TechCorp's decision for its first phase and a table that summarizes, channel by channel, threat, measure and where it is configured. Token verification belongs to 07-01, input validation to 07-03, and secrets, RBAC and NetworkPolicies in detail to 07-04.

Notice. The certificates, algorithms and configurations in this lesson are for teaching purposes and change over time. The real TLS configuration (versions, cipher suites, HSTS, mTLS) must be reviewed with a security professional and, in the payments area, with whoever is responsible for compliance (PCI DSS).

1. Threats in transit and the attack surface of a distributed system
2. TLS at the edge: certificates, chain of trust and versions
3. TLS on the Ingress with cert-manager and Let's Encrypt
4. HSTS, redirection and how to check the edge TLS
5. Internal TLS and mTLS between services: why and with which options
6. mTLS via service mesh: PeerAuthentication and AuthorizationPolicy
7. TechCorp's decision for phase 1
8. Securing RabbitMQ and the databases
9. Message and webhook integrity: HMAC, timestamp and nonce
10. Protecting internal headers, and gRPC with TLS
11. Summary table: channel, threat, measure and where it is configured

Anything that travels over a wire or a virtual network can suffer four things:

In the monolith there was a single external channel (browser → server) and one call to the database. In the distributed system, the surface multiplies: browser → Ingress → gateway → six services → PostgreSQL/MongoDB/RabbitMQ/Keycloak → external payment provider. The temptation is to split the world into perimeter (hostile, encrypted) and internal network (trusted, plain text). The zero trust principle says the opposite: no network is trusted just for being "inside"; every call is authenticated and encrypted as if it came from the Internet. TechCorp adopts it as a direction, and in section 7 decides how far it goes in the first phase.

TLS addresses the first three threats on a channel: the client verifies the server's identity with its certificate, both agree on session keys and from then on everything travels encrypted and with integrity. The minimum concepts:

• An X.509 certificate binds a name (api.techcorp.example) to a public key, and a certificate authority (CA) signs it. The browser trusts the certificate because it trusts the CA (or the intermediate CA signed by a root it already ships with): that is the chain of trust. A self-signed certificate has no chain: it works locally, never at the edge.
• Versions: TLS 1.3 is the current one (faster, no weak cipher suites); TLS 1.2 is kept for compatibility. TLS 1.0/1.1 and SSL are forbidden. In Kubernetes the ingress controller pins it (ssl-protocols: TLSv1.2 TLSv1.3 in the ingress-nginx ConfigMap).
• TLS termination: at TechCorp, external TLS terminates at the Ingress (05-02); from the Ingress to the gateway and from the gateway to the services, traffic goes over the cluster network. This is the usual practice: public certificates in a single place; the inside is protected with the measures of sections 5-7.

cert-manager is a Kubernetes operator that requests certificates from a CA, stores them in a Secret and renews them on its own before they expire. With Let's Encrypt (free, 90-day certificates, validation via HTTP-01 challenge) it is the standard solution. Installation and issuer:

And the gateway Ingress from 05-02, with the tls: line we left commented out and the annotation that activates cert-manager:

What happens underneath: cert-manager sees the annotation, creates a Certificate resource, generates a private key, solves the HTTP-01 challenge by publishing a temporary Ingress, receives the certificate and writes it to api-techcorp-tls; ingress-nginx reloads it. Renewal: 30 days before expiry it repeats the process without intervention. And since every automation fails at some point (Let's Encrypt quota exhausted, broken DNS), the CertificateExpiring alert from 06-05 (probe_ssl_earliest_cert_expiry with Blackbox Exporter, 14-day threshold) is the safety net: if it fires, renewal has been failing for 16 days and the platform/certificates runbook says what to look at (kubectl describe challenge -n techcorp).

The same resources serve auth.techcorp.example (Keycloak, 07-01) and shop.techcorp.example; with Traefik as the ingress controller (03-04) the annotation changes its name, but cert-manager is the same.

• HTTP→HTTPS redirection: ssl-redirect: "true" returns 308 to any http://. Necessary but insufficient: the first plain-text request can already be intercepted.
• HSTS (Strict-Transport-Security): the header that tells the browser "for N seconds, don't even try HTTP with this domain". It is enabled in the ingress-nginx ConfigMap (hsts: "true", hsts-max-age: "31536000", hsts-include-subdomains: "true") or from the gateway with helmet (07-03). Careful with preload: it is hard to revert.
• Checking:

External tools such as SSL Labs (Qualys) grade the full configuration; TechCorp requires an "A" before every audit.

Inside the cluster everything currently travels in the clear: gateway → orders-service:3002, Orders → catalog-service:3001, with Ana's JWT and the X-User-* headers in plain sight. Is that a problem? If an attacker manages to run a pod in the cluster (a compromised image, 07-04) or gains access to a node, they can eavesdrop on that traffic, impersonate a service and forge X-User-* to call a service directly, bypassing the gateway. The internal network is a barrier, not a guarantee.

mTLS (mutual TLS) adds client authentication to TLS: both ends present a certificate and each verifies the other's against an internal CA. The result is encryption + service identity on every connection: inventory-service knows cryptographically that the caller is orders-service, without trusting headers. The options:

To understand what the mesh does for us, the "by hand" version in Node.js. With an internal CA (ca.crt) and one certificate per service (tls.crt/tls.key, issued by cert-manager with an Issuer of type ca, mounted from a Secret at /etc/tls):

And the client (createHttpClient from 06-03) would need an https Agent with its key, cert and ca. It works, and it shows the price: every service manages certificate files, they must be rotated (cert-manager helps) and restarted or reloaded on rotation, the internal CA is a critical secret, and the CN becomes one more contract. With six services it is manageable; with twenty, it is not.

With Istio (05-05) the same result is two resources, without touching code or certificates: istiod issues a SPIFFE certificate per ServiceAccount (spiffe://cluster.local/ns/techcorp/sa/orders-service), rotates it every 24 h and the sidecars negotiate mTLS among themselves.

This closes what 05-05 only named: the identity comes from the certificate issued to the ServiceAccount (07-04 creates one per service precisely for this), and the policy reads like business language: "only Orders talks to Inventory, and Prometheus only reads metrics". An intruding pod without a sidecar or identity gets RBAC: access denied from Inventory's proxy before the request ever reaches Node. Linkerd has its equivalent (Server + AuthorizationPolicy/MeshTLSAuthentication), with mTLS enabled by default.

Marta and the Platform team apply the criterion from 05-05 (no mesh in the first phase) and decide, for internal communication:

Trigger to reevaluate: the audit of the Payments area planned for phase 2. If it requires mandatory mTLS between services (likely around the perimeter of payments-service), the route will be a mesh — Linkerd as first option because of its automatic mTLS, consistent with 05-05 — and not per-application certificates. Meanwhile, the lesson from section 5 serves to know what is being bought.

RabbitMQ. Three rules:

1. Encrypted channel: amqps:// (port 5671) with a server certificate (issued by cert-manager with the internal CA, or the Let's Encrypt one if the broker has a public name). The RABBITMQ_URL from 04-03 becomes amqps://orders:…@rabbitmq:5671/techcorp and the zod schema accepts amqps. With amqplib, connect(url, { ca: [fs.readFileSync('/etc/tls/ca.crt')] }).
2. One user per service with minimal permissions, never guest (which, besides, can only connect from localhost). RabbitMQ permissions are three regular expressions per vhost: configure (declare), write (publish / bind), read (consume). For orders, according to the contract from 03-02:

That is: orders configures only its own queues (and their .retry/.dlq from 06-03); writes to the techcorp.events exchange (publishing) and to its queues (binding them and re-queuing into .retry); and reads from its queues and from the two exchanges (RabbitMQ requires read on the exchange to bind a queue to it). It cannot consume from inventory.orders or publish directly into someone else's queue. A compromised orders cannot read Payments' events.

3. Passwords in Secrets (orders-rabbitmq), rotated (07-04), and the management UI (15672) not exposed outside the cluster, with its own administrator user.

Databases. The same logic in PostgreSQL and MongoDB:

• Mandatory TLS: ORDERS_DB_URL=postgres://svc_orders:…@postgres:5432/techcorp?sslmode=verify-full&sslrootcert=/etc/tls/ca.crt (require encrypts but does not verify the certificate; verify-full also prevents server impersonation); on the server, ssl = on and hostssl in pg_hba.conf to reject plain-text connections. In MongoDB, tls=true&tlsCAFile=… in the URL and net.tls.mode: requireTLS.
• svc_* users with minimal privileges per schema, exactly as in 02-04: svc_orders only has USAGE and SELECT/INSERT/UPDATE/DELETE on the orders schema, no CREATE (the migrations in the Job from 05-02 use a separate user, mig_orders, with more privileges and only during deployment), and svc_payments cannot read orders.* even if they share the instance.
• Databases not exposed outside the cluster or the VPC; on managed services, no public IP and the cluster network on the allowlist.

payments-service receives webhooks from the external payment provider (POST /v1/webhooks/payment-provider) telling it "charge ch_7f3a… for €79.70 has been confirmed". That route must be exposed to the Internet (through the gateway or its own Ingress), and TLS only guarantees the channel, not who is sending: anyone who knows the URL could send a fake payment.confirmed. Payment providers solve this with an HMAC signature: a shared secret (PAYMENT_PROVIDER_WEBHOOK_SECRET, in the payments-provider Secret), and on every webhook a header with t=<timestamp>,v1=<HMAC-SHA256(t + "." + body)>.

Three layers: the signature proves it was sent by whoever holds the secret and that the body did not change; the timestamp bounds the replay window; and the event identifier as a persisted nonce (received_webhooks) makes any resend harmless, just like processOnce in the consumers from 04-04. The same technique serves the webhooks TechCorp sends (to a partner, to an ERP): sign with a per-recipient secret, and the receiver verifies.

For internal events over RabbitMQ, each message is not signed: the channel is already amqps with an authenticated user and per-queue permissions (section 8), and idempotency by eventId (02-05) neutralizes duplicates and replays. Signing messages would make sense if the broker were shared with third parties.

Two brief closing points:

• Internal headers. In 07-01 the gateway already strips any incoming X-User-* before setting its own. The same must be done with X-Forwarded-For/X-Real-IP (the Ingress sets them; the gateway must not accept the client's for the rate limit from 03-04: app.set('trust proxy', 1) trusts only the Ingress hop) and with any header a service uses as "trusted". Rule: an internal header is only as trustworthy as the set of parties who can reach the port — hence the NetworkPolicies from 07-04 and, in due course, mTLS.
• gRPC (03-03): it uses HTTP/2 and is encrypted the same way. Server with grpc.ServerCredentials.createSsl(caCert, [{ private_key, cert_chain }], /* checkClientCertificate */ true) and client with grpc.credentials.createSsl(caCert, key, cert); with a mesh, the sidecar does it and the code uses createInsecure() toward localhost. Mention only: TechCorp does not expose gRPC in phase 1.

• Terminating TLS at the Ingress and believing "everything is encrypted". The inside travels in the clear; it must be a conscious decision (section 7), compensated with network controls and authentication.
• Certificates renewed by hand. Someone forgets, the site goes down on a Sunday. cert-manager + the CertificateExpiring alert; and test renewal with the Let's Encrypt staging issuer before exhausting the production quota.
• sslmode=require believing it verifies the certificate. It only encrypts; verify-full is what prevents impersonation of the database server.
• guest/guest in RabbitMQ or a single techcorp user with .* on all three permissions. A compromised service reads everything. One user per service with bounded regular expressions.
• Verifying the HMAC signature over the already parsed and re-serialized body: JSON.stringify(req.body) rarely reproduces the original bytes and the signature fails (or worse, gets loosened). Raw body with express.raw on that route.
• Comparing signatures with ===. It leaks information through timing; timingSafeEqual always.
• PeerAuthentication STRICT without a sidecar on every pod (or with Prometheus outside the mesh): everything stops talking (05-05). PERMISSIVE first.
• Tip: keep the commands from sections 3 and 4 in the platform/certificates runbook; and for every TLS Secret, who issues it and when it expires (kubectl get certificate -A).

Exercise 1. A Notifications developer proposes that, to avoid depending on cert-manager locally, all environments use sslmode=disable with PostgreSQL "because the database is inside the cluster". Explain which threats remain open and give a per-environment configuration consistent with the table from 04-03.

Exercise 2. Write the three RabbitMQ permissions (configure, write, read) for the inventory user, knowing that Inventory consumes from inventory.orders (events order.created and order.cancelled), publishes stock.reserved/stock.rejected/stock.released to techcorp.events, and has its .retry and .dlq queues. Explain what it cannot do with them.

Exercise 3. The payment provider resends the same webhook three times (retries due to timeout) and, in addition, an attacker captures one and resends it two hours later. Walk through the code from section 9 and say what happens in each of the four cases and what effect it has on the saga.

Solution 1. Without TLS, any process with access to the cluster network (compromised pod, node, sniffer on a misconfigured virtual switch) reads the svc_notifications credentials in the handshake and all the data (customer emails and names: personal data) in every query; and nothing stops a fake server from answering as postgres. Configuration consistent with the environments column from 04-03: local sslmode=disable (PostgreSQL in Docker Compose, no real data), staging and production sslmode=verify-full&sslrootcert=/etc/tls/ca.crt, with hostssl in pg_hba.conf so that the server rejects plain-text connections even if a service gets it wrong. The value goes in the notifications-db Secret, so the code does not change between environments; only the URL does.

Solution 2.

configure: it only declares its own queue and the auxiliary ones. write: it publishes to techcorp.events (the stock.* events) and to its queues (binding them, re-queuing into .retry). read: it consumes from its queues and may bind them to the two exchanges. It cannot: consume from orders.saga, payments.orders or anyone else's queue (it cannot even declare them in order to bind them); declare queues outside its prefix; or delete or redeclare the exchange (configure does not include it). A compromised inventory can publish fake stock.reserved events (which is why Orders validates the payload and correlates it with its pending reservation), but it reads neither payments nor orders.

Solution 3. (1) First webhook: valid signature, t inside the window, new event.id → it is processed, payment.confirmed is published, the saga moves forward. (2) Provider retries 2 and 3 (seconds later): valid signature and t, but recordWebhook returns isNew=false → 200 with no effects: the provider stops retrying and the saga sees no duplicates (even if it did, processOnce would discard them by eventId). (3) The attacker's resend two hours later: the signature is valid (it is the original message) but |Date.now() - t| > 5 min → 400 INVALID_WEBHOOK before looking at anything else; and if the attacker tried to "refresh" t, the signature would no longer match because t is part of what is signed. (4) If the attacker had also managed to wait less than 5 minutes: the event.id is already recorded → 200 with no effects. In no case is anything confirmed twice or a different order confirmed (the event.id and the orderId are inside the signed body).

TechCorp's communication no longer rests on faith in the internal network. The edge is encrypted with TLS 1.2/1.3 and Let's Encrypt certificates that cert-manager (ClusterIssuer letsencrypt-prod, cert-manager.io/cluster-issuer annotation, api-techcorp-tls Secret) issues and renews, with HSTS, 308 redirection and the CertificateExpiring alert as the safety net; we have seen what mTLS is and how much it costs to do it per application (https.createServer with requestCert), and how a mesh gives it away with PeerAuthentication STRICT and an AuthorizationPolicy that only lets orders-service talk to Inventory — closing what 05-05 only named; the phase 1 decision is TLS at the edge + private network + NetworkPolicies + client credentials + double JWT verification, with mTLS postponed until the Payments audit and Linkerd as first candidate; RabbitMQ goes over amqps:// with one user per service and per-queue permissions (orders only publishes to techcorp.events and consumes orders.saga/orders.customers), the databases with sslmode=verify-full and svc_* users; the payment provider's webhooks arrive signed with HMAC-SHA256, with a 5-minute window and received_webhooks against replay; and the gateway strips internal headers before propagating its own. With identity (07-01) and the channel settled, the next question is what each service does with what it receives: how it validates input, how it avoids injections, how it treats personal data and how it is audited. That is the topic of the next lesson: security practices.