We have closed two dimensions of Rutas Norte's security. With RBAC (08-01) we control who can do what against the API. With security contexts (08-02) and admission policies (08-03) we control what a container can do once it is running, and we got the cluster to enforce it on its own.
An entire dimension remains: what can talk to what. And here we already did important work in 04-06, when we put a deny-all NetworkPolicy in rutas-norte-pro and authorised each conversation one at a time. But we then pointed out three uncomfortable limits that we parked:
- NetworkPolicies work at L3/L4: they understand IPs and ports, not HTTP paths or methods.
bookings-apican reachbookings-postgres, yes, but the policy does not distinguish a legitimate query from a full dump of the customer table. - They log nothing. A denied connection attempt is completely invisible. If somebody is probing the internal network, we never find out.
- Inside the cluster, all traffic travels unencrypted once past the Ingress TLS (04-05). Whoever can observe the node's network sees the queries to
bookings-postgresin the clear, with names, ID numbers and phone numbers included.
And there is a fourth thing we have barely mentioned: outbound traffic. It is the natural route by which a customer database leaves.
This lesson does not repeat the NetworkPolicy syntax: it takes it as known and uses it. What we build here is the complete network strategy of a production platform, layer by layer.
Important warning. The network architecture design of a production platform must be reviewed by a security professional, who will assess the specific threat model. And since Rutas Norte's traffic carries customers' personal data —name, ID number, phone and email—, the decisions about encryption in transit, egress control and flow logging must also be known and approved by the compliance officer. This lesson's approach is exclusively defensive: understanding the paths by which information can leak in order to close them and detect them.
Contents
- The network defence layers
- Microsegmentation as a principle
- Egress traffic control
- The IPs versus domain names problem
- Encryption in transit inside the cluster
- What a service mesh is and what it costs
- Istio, Linkerd and Cilium compared
- Layer 7 authorization policies
- Protecting the perimeter
- Control plane security
- Traffic logging and visibility
- Zero trust applied to Rutas Norte
- Common mistakes and tips
- Exercises
- Conclusion
- The network defence layers
Network security is not one mechanism: it is a series of layers, each of which assumes the previous one can fail. This is the complete model we are going to build.
flowchart TB
I["Internet"] --> WAF["Layer 1: perimeter<br/>WAF + rate limiting<br/>+ DoS protection"]
WAF --> ING["Layer 2: Ingress<br/>TLS terminated (04-05)<br/>security headers"]
ING --> NP["Layer 3: microsegmentation<br/>deny-all NetworkPolicy + explicit<br/>permissions (04-06)"]
NP --> MESH["Layer 4: mTLS + L7 authorization<br/>per-workload identity<br/>(service mesh)"]
MESH --> APP["Rutas Norte workloads"]
APP --> EGR["Layer 5: egress control<br/>only the payment gateway"]
EGR --> EXT["pagos.proveedorexterno.example"]
APP -.-> OBS["Layer 6: visibility<br/>flow logging<br/>(Hubble)"]
CP["Layer 7: control plane<br/>apiserver, etcd, kubelet"] -.-> APP
style NP fill:#d5e8f9,stroke:#36c
style EGR fill:#f9d5d5,stroke:#c33
| Layer | What it mitigates | State at Rutas Norte |
|---|---|---|
| 1. Perimeter | Attacks from the internet, denial of service, abuse | To be built |
| 2. Ingress with TLS | Eavesdropping on traffic between client and platform | Done (04-05) |
| 3. Microsegmentation | Lateral movement after a pod is compromised | Done (04-06), to be reinforced |
| 4. mTLS and L7 authorization | Internal eavesdropping, service impersonation | To be decided |
| 5. Egress control | Exfiltration of customer data | To be built |
| 6. Visibility | Blindness during an incident | To be built |
| 7. Control plane | Total compromise of the cluster | To be reviewed |
The order matters: each layer is justified by what would happen if the previous one failed. If the WAF does not stop an attack, the Ingress still validates the certificate. If somebody manages to run code in web-store, microsegmentation stops them reaching bookings-postgres. If they did get there, mTLS would stop them passing themselves off as bookings-api. And if everything above failed, egress control would stop the data leaving the cluster.
- Microsegmentation as a principle
Microsegmentation means treating each workload as its own network segment, with explicit rules about what can talk to what. It is the opposite of the traditional "trusted internal network behind a firewall" model.
Why the perimeter model does not work in Kubernetes
In a classic network, the firewall separated "inside" from "outside", and inside everybody talked to everybody. In a Kubernetes cluster, by default, every pod can talk to every pod in every namespace. A curl from web-store reaches bookings-postgres with no obstacle.
Remember what we said in 02-06: a namespace is not a security boundary on its own. Without NetworkPolicies, rutas-norte-dev and rutas-norte-pro are on the same flat network.
The practical consequence: if somebody manages to run code in web-store —the most exposed component, because it faces the internet directly— they have network access to the whole platform. Microsegmentation turns that into "they have network access to bookings-api, and only to port 8080".
The three principles
Principle 1: deny by default in all three environments.
In 04-06 we applied deny-all in rutas-norte-pro. That is not enough. rutas-norte-dev and rutas-norte-pre need it too, for two reasons:
- If development and pre-production are open, a policy that works there can fail in production, and the failure is discovered at the worst moment.
- A compromised pod in
rutas-norte-devwith no network restrictions can reachrutas-norte-pro, because the cluster network is flat.
In 08-03 we solved this elegantly with a Kyverno policy that generates the deny-all in every new Rutas Norte namespace, with synchronize: true so that it comes back if somebody deletes it. It is the guarantee that no future namespace is born open.
Principle 2: policies per component, not per namespace.
It is tempting to write "everything in rutas-norte-pro can talk to everything in rutas-norte-pro". That is a mistake: it reproduces inside the namespace the very flat model we wanted to avoid.
| Approach | What it allows if web-store is compromised |
|---|---|
| Per namespace | Access to bookings-postgres, redis-cache, bookings-api and everything else |
| Per component | Only bookings-api:8080. Nothing else |
The difference is enormous and the cost is writing six policies instead of one.
Principle 3: periodic review of which conversations are still needed.
Network policies decay just like RBAC. A permission is added to debug something and it stays. A component is retired and its policy survives. A port is changed and the old one is left "just in case".
Rutas Norte's authorised communications matrix, as it stood in 04-06, must fit in one table and be reviewed every quarter:
| Source | Destination | Port | Justification | Last review |
|---|---|---|---|---|
| Ingress | web-store |
8080 | Public traffic | 2026-07 |
| Ingress | bookings-api |
8080 | Public API | 2026-07 |
web-store |
bookings-api |
8080 | Queries and bookings | 2026-07 |
bookings-api |
bookings-postgres |
5432 | Booking and customer data | 2026-07 |
bookings-api |
redis-cache |
6379 | Availability cache | 2026-07 |
bookings-api |
Internet (gateway) | 443 | Payments | 2026-07 |
notifications-worker |
bookings-postgres |
5432 | Reads pending bookings | 2026-07 |
notifications-worker |
Internet (SMTP) | 587 | Sending emails | 2026-07 |
occupancy-reports |
bookings-postgres |
5432 | Nightly aggregates | 2026-07 |
| Prometheus | All | 9090 | Metrics (07-03) | 2026-07 |
| All | CoreDNS | 53 | Name resolution | 2026-07 |
Each row must have a living reason. A useful question in the review: if I delete this rule right now, what breaks? If nobody knows, find out (in pre-production) or remove it.
A script to detect orphaned policies:
#!/usr/bin/env bash
# k8s/security/orphaned-policies.sh
# Detects NetworkPolicies whose podSelector matches no pod.
set -euo pipefail
NS="${1:-rutas-norte-pro}"
kubectl get networkpolicy -n "$NS" -o json | jq -r '
.items[] | select(.spec.podSelector.matchLabels != null)
| "\(.metadata.name)\t" +
([.spec.podSelector.matchLabels | to_entries[] | "\(.key)=\(.value)"] | join(","))' \
| while IFS=$'\t' read -r policy selector; do
n=$(kubectl get pods -n "$NS" -l "$selector" --no-headers 2>/dev/null | wc -l)
[[ "$n" -eq 0 ]] && echo "ORPHANED: $policy (selector: $selector) matches no pod"
doneThat policy outlived the component it protected. It is not dangerous in itself, but it is noise that makes review harder, and if tomorrow somebody deploys something with that label it inherits permissions nobody decided on.
- Egress traffic control
Here is, probably, the most serious gap left in Rutas Norte.
Why egress matters so much
Think about the sequence of an incident:
- Somebody finds a flaw in
bookings-apiand manages to run code. - The process has legitimate access to
bookings-postgres: it can read the customer table with names, ID numbers, phone numbers and emails. - Now it needs to get that data out of the cluster.
Step 3 is what turns a compromise into a data breach. With no egress control, step 3 is trivial: an HTTPS request to any server on the internet.
A rule to internalise: the ingress policy limits who gets in; the egress policy limits what can get out. Almost everybody writes the first and forgets the second. And the second is what decides whether an incident stays as "somebody ran code" or escalates to "the personal data of 200,000 customers was leaked".
Most NetworkPolicy guides only talk about Ingress. Check your policies: if policyTypes does not include Egress, outbound traffic is wide open.
The egress Rutas Norte needs
Let us audit which outbound flows are legitimate:
| Component | External destination | Port | Indispensable? |
|---|---|---|---|
bookings-api |
pagos.proveedorexterno.example |
443 | Yes: payments |
notifications-worker |
The provider's SMTP server | 587 | Yes: confirmation emails |
web-store |
None | — | It does not need to reach the internet |
bookings-postgres |
None | — | No |
redis-cache |
None | — | No |
occupancy-reports |
None | — | No |
Four of the six components do not need to reach the internet at all. That is the first thing to take advantage of: closing their egress completely is free and removes the exfiltration route for most of the platform.
The base egress policy
Recalling that in 04-06 we left deny-all with policyTypes: [Ingress, Egress], each component explicitly needs its own egress. The universal minimum is DNS:
# k8s/base/network/allow-dns.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: rutas-norte-pro
spec:
podSelector: {} # every pod
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53Without this, no pod resolves names and everything fails in a very confusing way: services cannot find each other even though the ingress policies are correct. It is the most frequent mistake when enabling Egress for the first time.
Egress to the payment gateway
bookings-api needs to reach pagos.proveedorexterno.example. This is where the problem of the next section appears, but let us first see the solution with ipBlock:
# k8s/base/network/bookings-api-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: bookings-api-egress
namespace: rutas-norte-pro
annotations:
security.rutasnorte.example/justification: >-
Egress to the external payment gateway. The IP ranges are published by the
provider in its documentation and are reviewed monthly by the scheduled
job "verify-gateway-ranges". The last verification is
documented in the network change log.
spec:
podSelector:
matchLabels:
app: bookings-api
policyTypes: [Egress]
egress:
# 1. DNS (indispensable)
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
# 2. Database and cache (internal traffic)
- to:
- podSelector:
matchLabels:
app: bookings-postgres
ports:
- { protocol: TCP, port: 5432 }
- to:
- podSelector:
matchLabels:
app: redis-cache
ports:
- { protocol: TCP, port: 6379 }
# 3. Payment gateway: ONLY these ranges, ONLY port 443
- to:
- ipBlock:
cidr: 203.0.113.0/24 # range published by the provider
- ipBlock:
cidr: 198.51.100.64/26 # secondary range
ports:
- { protocol: TCP, port: 443 }The important thing about this manifest is not what it allows, but what it does not allow: bookings-api cannot connect to any other address on the internet. If somebody manages to run code there and wants to send the customer table to a server of their own, the connection does not get out.
A note about ipBlock and private IPs: ipBlock refers to network IPs, and the cluster's pods have IPs too. An ipBlock: 0.0.0.0/0 would include all the cluster's internal traffic. When you want "the whole internet except the internal network", you have to use except:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8 # pod and service network
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- { protocol: TCP, port: 443 }This is "any public destination on 443", which is far laxer than the provider's range list and we do not recommend it for bookings-api. We include it because it is a pattern that comes up often and it is worth understanding why it is worse: it allows sending data to any server on the internet that speaks HTTPS.
The ambassador container
Remember from 06-04 that the connection with the payment gateway goes through an ambassador container inside the bookings-api pod. That has an important network consequence: since the containers in a pod share the network namespace, the policy applies to the whole pod, not to the container. The ambassador and the main container have exactly the same network permissions.
The ambassador brings other advantages —it centralises retries, timeout logic and the gateway credentials— but it is not a network security boundary. If we wanted only the ambassador to be able to talk to the gateway, it would have to be a separate pod with its own policy.
It is a good example of a common confusion: the ambassador pattern is an architecture pattern, not an isolation one.
- The IPs versus domain names problem
Here comes the structural limitation of NetworkPolicies.
Standard Kubernetes NetworkPolicies work with IP addresses and CIDR ranges. They know nothing about domain names. There is no
to: dnsName: pagos.proveedorexterno.examplefield.
And this is not an oversight: policies are translated into firewall rules in the node's kernel, which acts on IP packets. By the time the packet gets there, the name has already been resolved and no trace of it remains.
The three practical problems
Problem 1: IPs change. Cloud services rotate addresses. The payment provider's documentation may publish ranges today and change them in three months. If your ipBlock falls out of date, payments stop working and the diagnosis is especially thankless because nothing in the logs says "a NetworkPolicy blocked this".
Problem 2: a range is more than you want. 203.0.113.0/24 is 256 addresses. If the provider shares infrastructure, that range may include servers belonging to other customers of theirs. You have authorised egress to 256 destinations in order to reach one.
Problem 3: the reverse scenario is worse. If a legitimate service shares an IP with a generic storage service (something common in public clouds), authorising the first authorises the second, and there you really do have an exfiltration route.
The three options
| Option | How it works | Advantages | Drawbacks |
|---|---|---|---|
| Egress gateway | All outbound traffic goes through specific pods with a fixed IP, which apply the rules | A single point of control and logging; a stable source IP so the provider can filter on it | Another piece to maintain; a single point of failure |
| Proxy with an allow list | An HTTP(S) proxy the applications point to, with a list of authorised domains | Filters by domain name, SNI included; complete logging of destinations | The applications must be configured to use it; it adds latency |
| DNS-based policies (Cilium) | The CNI observes the DNS responses and programs the rules with the IPs they return | You write the name directly; it updates itself | Requires Cilium as the CNI |
Option A: an egress proxy with an allow list
It is the most portable option and the one that gives the best visibility. A Squid or a dedicated proxy in a pod, with a list of domains:
# k8s/base/network/egress-proxy-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: egress-proxy-config
namespace: rutas-norte-sistema
data:
allowed-domains.txt: |
# Payment gateway: ticket payments
pagos.proveedorexterno.example
# Outbound mail: booking confirmations
smtp.proveedorcorreo.example
# NOTHING ELSE. Any other destination is rejected and logged.The applications are configured with HTTPS_PROXY, and the NetworkPolicy only allows egress to the proxy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: bookings-api-egress-via-proxy
namespace: rutas-norte-pro
spec:
podSelector:
matchLabels:
app: bookings-api
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- { protocol: UDP, port: 53 }
# The only egress allowed: the proxy. Not one internet IP directly.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: rutas-norte-sistema
podSelector:
matchLabels:
app: egress-proxy
ports:
- { protocol: TCP, port: 3128 }The decisive advantage: the proxy logs every destination requested, allowed or denied. That partly solves the NetworkPolicies' lack of logging, at least for outbound traffic, which is the one that matters most to watch.
An honest drawback: an application that ignores the HTTPS_PROXY variables bypasses the proxy. That is why the NetworkPolicy is still necessary: it prevents direct egress, forcing traffic through the proxy even if the application does not cooperate. The two layers together work; neither one on its own.
Option B: DNS-based policies with Cilium
If the CNI is Cilium, the CiliumNetworkPolicy lets you write the name directly:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: bookings-api-egress-dns
namespace: rutas-norte-pro
spec:
endpointSelector:
matchLabels:
app: bookings-api
egress:
# DNS with inspection: Cilium observes the responses to program the rules
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: kube-system
k8s-app: kube-dns
toPorts:
- ports:
- { port: "53", protocol: UDP }
rules:
dns:
- matchPattern: "*"
# Egress by DOMAIN NAME, not by IP
- toFQDNs:
- matchName: "pagos.proveedorexterno.example"
toPorts:
- ports:
- { port: "443", protocol: TCP }How it works: Cilium intercepts the pod's DNS responses, sees which IP pagos.proveedorexterno.example returned and dynamically programs the rule for that IP, with the time to live the DNS record indicates. If tomorrow the provider changes IP, the policy keeps working without anyone touching anything.
It is by far the most elegant solution. Its cost is the dependency on the CNI: adopting Cilium is a platform decision that goes well beyond this one policy.
Recommendation for Rutas Norte
A two-stage approach:
- Now:
ipBlockwith the ranges published by the provider, plus a monthly scheduled job that verifies they are still valid and warns if they change. It is the solution that works with any CNI and requires no new pieces. - In the medium term: an egress proxy with an allow list, mainly for the destination logging. If at some point we migrate to Cilium (which would also bring the advantages of section 11), the policies are replaced with
toFQDNs.
And the check that must be automated in any case:
#!/usr/bin/env bash
# k8s/security/verify-egress.sh
# Confirms that the components which must not reach the internet, do not.
set -uo pipefail
NS=rutas-norte-pro
check_blocked() {
local workload="$1" target="$2"
local output
output=$(kubectl exec -n "$NS" "$workload" -- \
timeout 5 wget -q -O- --timeout=4 "$target" 2>&1 || echo "BLOCKED")
if [[ "$output" == *"BLOCKED"* ]]; then
echo "OK $workload cannot reach $target"
else
echo "FAIL $workload CAN reach $target <-- exfiltration route open"
fi
}
check_blocked deploy/web-store https://external-example.example
check_blocked statefulset/bookings-postgres https://external-example.example
check_blocked deploy/bookings-api https://external-example.exampleOK deploy/web-store cannot reach https://external-example.example
OK statefulset/bookings-postgres cannot reach https://external-example.example
OK deploy/bookings-api cannot reach https://external-example.exampleThis test must run in rutas-norte-pre on every deployment. It is the verification of the layer that prevents a personal data breach, and as such it should be in the file the compliance officer reviews.
- Encryption in transit inside the cluster
In 04-05 we put TLS on the Ingress with cert-manager: the traffic between the customer's browser and the platform is encrypted as far as https://www.rutasnorte.example. Perfect. But what happens after the Ingress?
What is left unencrypted
flowchart LR
C["Browser"] -->|"HTTPS<br/>encrypted (04-05)"| ING["Ingress"]
ING -->|"HTTP<br/>IN THE CLEAR"| TW["web-store"]
TW -->|"HTTP<br/>IN THE CLEAR"| API["bookings-api"]
API -->|"PostgreSQL protocol<br/>IN THE CLEAR"| PG[("bookings-postgres<br/>personal data")]
API -->|"RESP<br/>IN THE CLEAR"| R[("redis-cache")]
style PG fill:#f9d5d5,stroke:#c33
Everything to the right of the Ingress travels unencrypted over the cluster network. And through that stretch, in the clear, pass:
- The SQL queries with customers' names, ID numbers, phone numbers and emails.
- The API responses with the booking data.
- The database credentials, in the initial handshake of every connection.
Who can see that traffic?
This is the question that decides whether the effort is worth it:
| Scenario | Can it see the internal traffic? |
|---|---|
| An ordinary pod with strict NetworkPolicies | No (it cannot even connect) |
A pod with hostNetwork: true |
Yes: it sees all the node's traffic |
A pod with CAP_NET_RAW and hostNetwork |
Yes: it can capture packets |
| Somebody with node access (SSH, a privileged container) | Yes, everything |
| The cloud or data-centre provider | It depends on the contract and the infrastructure |
| An attacker who compromises the CNI | Yes |
Notice how the lessons connect: in 08-02 we forbade hostNetwork and CAP_NET_RAW, and in 08-03 we made the cluster reject them. That work is what makes unencrypted internal traffic an acceptable risk in many scenarios. Internal encryption is the defence for when that layer fails, or when regulation explicitly requires it.
The three internal encryption options
| Option | What it encrypts | Cost | Identity |
|---|---|---|---|
| TLS in the application | Only what the application implements | Code changes in every component | Certificates managed by hand |
| CNI encryption (WireGuard, IPsec) | All traffic between nodes | Low: one configuration option | Per node, not per workload |
| mTLS with a service mesh | All traffic between pods in the mesh | High: a whole infrastructure layer | Per workload |
Option 1: application-level TLS
It is what we already do with the payment gateway (the ambassador speaks HTTPS). For internal traffic it would mean configuring PostgreSQL with ssl = on, generating certificates, distributing them and configuring every client.
# Fragment: PostgreSQL with mandatory TLS
env:
- name: POSTGRES_INITDB_ARGS
value: "--auth-host=scram-sha-256"
volumeMounts:
- name: tls-certificates
mountPath: /var/lib/postgresql/certs
readOnly: trueAdvantage: it encrypts the stretch that matters most (the personal data) with little rollout. Drawback: it has to be done component by component, you have to manage certificate rotation and trust that each client actually verifies the certificate (many database clients, by default, do not).
For Rutas Norte, encrypting the bookings-postgres connection specifically is a high-value, moderate-cost measure, and probably the first step to take.
Option 2: CNI-level encryption
Several CNIs can automatically encrypt all traffic between nodes with WireGuard or IPsec.
# Cilium with WireGuard encryption (Helm values)
encryption:
enabled: true
type: wireguard
nodeEncryption: true# Calico with WireGuard
kubectl patch felixconfiguration default --type=merge \
-p '{"spec":{"wireguardEnabled":true}}'| For | Against | |
|---|---|---|
| Effort | Minimal: one option | — |
| Coverage | All traffic between nodes, no exceptions | Only between nodes: two pods on the same node are not encrypted |
| Performance | WireGuard is very efficient | Some CPU; with IPsec, more |
| Identity | — | It authenticates nodes, not workloads |
That last point is the essential limitation: CNI encryption protects against whoever observes the network between nodes, but it provides no per-service identity. web-store and bookings-api on the same node talk just as much in the clear, and neither can cryptographically prove who it is.
Even so, the value-to-effort ratio is excellent. If the CNI supports it, turn it on.
Option 3: mTLS with a service mesh
That is the next section, because it gives much more than encryption.
- What a service mesh is and what it costs
A service mesh is an infrastructure layer that takes care of service-to-service communication, moving it out of the application code.
How it works
In the classic model, a sidecar proxy (usually Envoy) is injected into each pod. All the traffic entering and leaving the pod goes through that proxy, which is what applies the encryption, the authorization and the metrics collection. The application never notices: it still does http://bookings-api:8080.
flowchart LR
subgraph P1["Pod web-store"]
A1["nginx"] <--> S1["sidecar proxy"]
end
subgraph P2["Pod bookings-api"]
S2["sidecar proxy"] <--> A2["Node.js"]
end
S1 <-->|"mTLS<br/>encryption + identity<br/>+ L7 authorization"| S2
CP["Control plane<br/>issues certificates<br/>distributes policies"] -.-> S1
CP -.-> S2
What it gives you
| Capability | What it solves | Can you have it without a mesh? |
|---|---|---|
| Cryptographic per-workload identity | Each service has a certificate proving who it is | Very hard by hand |
| Automatic mTLS | Encryption and mutual authentication without touching code | Hard, component by component |
| L7 authorization | "Only web-store may do POST /bookings" |
Not with NetworkPolicy |
| Retries and timeouts | Resilience without code | Yes, in each application |
| Circuit breakers | Isolating a degraded service | Yes, with libraries |
| Percentage-based canary deployments | Sending 5% of the traffic to the new version | Partially (11-04) |
| Communication observability | Latency and error metrics for every call | Partially (07-03) |
Per-workload identity is the crown jewel and deserves to be understood properly. Without a mesh, when bookings-postgres receives a connection from an IP, all it knows is that it comes from that IP. With the NetworkPolicy we know that IP corresponds to a pod with the app: bookings-api label... but labels are set by whoever creates the pod, and an IP can be spoofed.
With mTLS, bookings-postgres receives a certificate saying spiffe://cluster.local/ns/rutas-norte-pro/sa/bookings-api, signed by the cluster's certificate authority, which nobody else can forge. That is identity. And it fits perfectly with the ServiceAccounts of 03-06.
What it costs you
This is the part the vendor presentations skate over:
| Cost | Detail |
|---|---|
| Operational complexity | One more piece of critical infrastructure to upgrade, watch and debug |
| Resources | One sidecar per pod: between 50 and 100 MiB of memory and some CPU. With 50 pods, that is several GiB |
| Latency | Between 1 and 5 ms per hop. With several chained hops, it shows |
| Learning curve | New concepts: VirtualService, DestinationRule, AuthorizationPolicy, PeerAuthentication |
| Harder debugging | A 503 can come from the application or from the proxy. You have to learn to read Envoy's logs |
| Coupling on upgrades | Upgrading the mesh usually means restarting every pod |
| Odd cases | Jobs that finish while the sidecar stays alive; protocols the proxy does not understand well |
That last point has a direct example at Rutas Norte: the occupancy-reports CronJob finishes its work, but the sidecar keeps running and the Job is never marked as complete. There are solutions (native sidecar containers since 1.29, or calling the proxy's termination endpoint), but it is exactly the kind of unforeseen friction that shows up.
Does Rutas Norte need a service mesh?
Let us answer honestly, because it is an expensive decision.
Arguments in favour:
- It carries personal data between services and internal encryption is defensible in any serious risk analysis.
- L7 authorization would allow rules like "only
web-storemay doPOST /bookings", which NetworkPolicy cannot express. - Cryptographic identity eliminates service impersonation.
Arguments against:
- There are six components. A mesh shines with dozens or hundreds of microservices and complex communications.
- The call graph is almost linear:
web-store→bookings-api→bookings-postgres/redis-cache. There is no topology that justifies a routing layer. - The platform team is small. Adding a badly operated mesh can worsen security (expired certificates, misunderstood policies, deferred upgrades).
- A good part of the benefit can be had more cheaply: CNI encryption for transit, TLS in PostgreSQL for what is critical, and strict NetworkPolicies for segmentation.
Decision for Rutas Norte: no, not yet. The alternative plan:
- Enable the CNI's WireGuard encryption (one configuration option).
- Configure TLS on
bookings-postgres, which is where the personal data lives. - Keep the strict NetworkPolicies with egress control.
- Review the decision when any of these objective criteria is met: more than fifteen services, communication between several teams that do not coordinate, an explicit regulatory requirement for mTLS, or a real need for percentage-based canary deployments.
And a general recommendation that goes beyond this course:
A service mesh is an excellent answer to a problem you have to have first. Adopting it "because it is modern", without the complexity that justifies it, adds operational risk without adding net security. If you operate it badly, it is worse than not having it.
That said, you have to know it, because the day the platform grows it will be the right answer. Let us look at the options.
- Istio, Linkerd and Cilium compared
| Istio (sidecar) | Istio (ambient) | Linkerd | Cilium Service Mesh | |
|---|---|---|---|---|
| Architecture | An Envoy sidecar per pod | A per-node agent (ztunnel) + an optional L7 waypoint | Its own sidecar (linkerd2-proxy, in Rust) | eBPF in the kernel + Envoy only for L7 |
| Memory per pod | 50-100 MiB | Almost zero (no sidecar) | 10-20 MiB | Almost zero for L4 |
| Added latency | 2-5 ms | 1-3 ms | <1 ms | Minimal |
| Complexity | High | Medium | Low | Medium-high |
| Automatic mTLS | Yes | Yes | Yes | Yes |
| L7 authorization | Very complete | Yes (with a waypoint) | Basic | Yes |
| Advanced routing | The most complete | Complete | Sufficient | Good |
| Multi-cluster | Very mature | Yes | Yes | Yes |
| Requires a specific CNI | No | No | No | Yes: Cilium |
| Community and ecosystem | The largest | Growing | Solid | Growing |
| Learning curve | Steep | Medium | Gentle | Medium |
| Governance | CNCF (graduated) | CNCF | CNCF (graduated) | CNCF (graduated) |
Selection criteria
Istio in sidecar mode if you need the most complete feature set: sophisticated routing, complex multi-cluster, API gateway integration, WebAssembly extensions. It is the option with the most capability and also the most expensive to operate. You need dedicated people.
Istio in ambient mode is the evolution that tackles the main criticism of the sidecar model: the per-pod cost. Instead of a proxy in every pod, there is a per-node agent (ztunnel) providing mTLS and L4, and an L7 proxy (waypoint) is deployed only where layer 7 authorization is genuinely needed. It reduces consumption enormously and removes the problem of Jobs that never finish. If we had to choose Istio today, it would be this mode.
Linkerd if you want mTLS and observability with minimum complexity. Its Rust proxy is remarkably lightweight and its philosophy is to do few things and do them well. For a platform like Rutas Norte, if a mesh were ever needed, this would be the first candidate: it covers what we need (mTLS, identity, basic authorization) without Istio's surface area.
Cilium Service Mesh if you already use Cilium as your CNI. By integrating the mesh into the CNI you avoid a whole layer: mTLS and L4 policies are applied in eBPF, inside the kernel, with no proxy. Envoy is only deployed for the L7 policies. It is the most efficient option, and it also brings the domain-name policies from section 4 and the Hubble visibility from section 11. The trade-off is that it ties the mesh decision to the CNI decision.
- Layer 7 authorization policies
This is the specific gap NetworkPolicy cannot fill, and it is worth seeing with a real example.
The L3/L4 limit
Our current policy says: web-store may connect to bookings-api on port 8080. That is all it can express. As soon as the connection is open, web-store can do:
| Request | Should it be able to? | Does the NetworkPolicy prevent it? |
|---|---|---|
GET /availability |
Yes | — |
POST /bookings |
Yes | — |
GET /admin/customers |
No | No |
DELETE /bookings/12345 |
No | No |
GET /metrics |
Debatable | No |
An L7 policy can tell the difference. Let us see how it would be expressed with Istio, taking the example from the statement: only bookings-api may do POST /bookings.
# Example L7 policy with Istio (only if a mesh were adopted)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: bookings-api-authorization
namespace: rutas-norte-pro
spec:
selector:
matchLabels:
app: bookings-api
action: ALLOW
rules:
# Rule 1: web-store may check availability and create bookings
- from:
- source:
# CRYPTOGRAPHIC IDENTITY, not a label or an IP
principals:
- "cluster.local/ns/rutas-norte-pro/sa/web-store"
to:
- operation:
methods: ["GET"]
paths: ["/availability", "/routes", "/health", "/ready"]
- operation:
methods: ["POST"]
paths: ["/bookings"]
# Rule 2: only the notifications worker queries pending bookings
- from:
- source:
principals:
- "cluster.local/ns/rutas-norte-pro/sa/notifications-worker"
to:
- operation:
methods: ["GET"]
paths: ["/bookings/pending"]
# Rule 3: Prometheus may only read the metrics
- from:
- source:
principals:
- "cluster.local/ns/monitoring/sa/prometheus"
to:
- operation:
methods: ["GET"]
paths: ["/metrics"]With action: ALLOW, anything that matches no rule is denied. It is the same default-deny principle as the NetworkPolicies, now applied to HTTP methods and paths.
Look at the key line:
That is neither a label nor an IP: it is the SPIFFE identity the mesh verifies cryptographically in the mTLS handshake. A pod cannot claim that identity without holding the corresponding private key. And note that it is based on the ServiceAccount, closing the circle with 03-06 and 08-01: the ServiceAccount goes from being only the identity before the API to also being the identity before the other services.
The complete comparison
| NetworkPolicy (04-06) | AuthorizationPolicy (mesh) | |
|---|---|---|
| Level | L3/L4: IP and port | L7: method, path, headers |
| Identity | Pod labels (spoofable) | Cryptographic certificate |
| Encryption | No | Yes (mTLS) |
| Logging of denials | No | Yes |
| Requires installing | Nothing (with a compatible CNI) | A complete mesh |
| Operational cost | Low | High |
The two are complementary, not alternatives. The NetworkPolicy is the first line and the cheapest; the L7 policy is the fine tuning. If you had to keep only one, keep the NetworkPolicy: without it, the L7 policy protects a service anyone can still connect to by other routes.
The alternative without a mesh
Without a mesh, L7 authorization is done by the application itself. bookings-api can validate in its own code who is calling it:
# The application validates a service token injected as a Secret
env:
- name: EXPECTED_SERVICE_TOKEN
valueFrom:
secretKeyRef:
name: internal-tokens
key: web-storeIt works and for six services it is perfectly reasonable, but it has real drawbacks: it must be implemented in every service, the tokens have to be rotated by hand, and if somebody reads the Secret (RBAC, 08-01) they can impersonate the service. The mesh's mTLS solves all three automatically. It is exactly the kind of cost that grows with the number of services and that, at some point, justifies the mesh.
- Protecting the perimeter
Everything above protects the inside. Now, the front door.
Minimum exposure: why there is no NodePort in production
In 04-02 we looked at the Service types. Let us go over their security implication:
| Type | Exposure | In production? |
|---|---|---|
ClusterIP |
Only inside the cluster | Yes: the default for everything |
NodePort |
A port (30000-32767) on every node | No |
LoadBalancer |
A provider load balancer | Only for the Ingress |
| Ingress | HTTP/HTTPS by name and path | Yes: the only entry point |
Why NodePort is no good in production:
- It opens the port on every node, including those not running the pod.
- It bypasses the Ingress, and with it the TLS (04-05), the WAF and the rate limiting.
- The port is in a high, unconventional range, hard to protect with a conventional firewall.
- It has no name: anyone who reaches the node's IP reaches the service.
And remember from 08-02 that hostPort is worse still, because there is not even a Service in front.
The Rutas Norte rule: a single entry point, the Ingress load balancer. Everything else is ClusterIP.
# Audit: is there any exposed service that should not be?
kubectl get svc -A -o json | jq -r '
.items[] | select(.spec.type == "NodePort" or .spec.type == "LoadBalancer")
| "\(.spec.type)\t\(.metadata.namespace)/\(.metadata.name)\t" +
([.spec.ports[] | "\(.port)->\(.nodePort // "-")"] | join(","))'A single LoadBalancer, the Ingress one. Any other line in that output is a question that has to be answered.
WAF and rate limiting
A web application firewall (WAF) inspects the HTTP requests and blocks known attack patterns: SQL injection, cross-site scripting, directory traversal. With ingress-nginx you can enable ModSecurity with the OWASP rule set:
# k8s/base/ingress/waf-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
enable-modsecurity: "true"
enable-owasp-modsecurity-crs: "true"
modsecurity-snippet: |
SecRuleEngine On
SecRequestBodyAccess On
SecRequestBodyLimit 5242880
SecAuditEngine RelevantOnly
SecAuditLogParts ABIJDEFHZ
# Paranoia level 1: few false positives. Raise it carefully.
SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"A serious warning about WAFs: a badly tuned WAF blocks legitimate requests. If web-store can no longer create bookings because a WAF rule considers a name with an apostrophe suspicious, you have created an incident. The correct procedure is identical to the PSA one in 08-03:
SecRuleEngine DetectionOnly: log but do not block.- Analyse the false positives over weeks of real traffic.
- Tune the problematic rules.
- Only then,
SecRuleEngine On.
And the rate limiting, via an Ingress annotation:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rutas-norte
namespace: rutas-norte-pro
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production # from 04-05
# Rate limiting
nginx.ingress.kubernetes.io/limit-rps: "20"
nginx.ingress.kubernetes.io/limit-connections: "10"
nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
# Maximum body size: avoids exhausting memory
nginx.ingress.kubernetes.io/proxy-body-size: "2m"
# Timeouts: avoids connections that hang
nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
# Security headers
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-Frame-Options: DENY";
more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains";
spec:
ingressClassName: nginx
tls:
- hosts: [www.rutasnorte.example]
secretName: rutasnorte-tls
rules:
- host: www.rutasnorte.example
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web-store
port: { number: 80 }
- path: /api
pathType: Prefix
backend:
service:
name: bookings-api
port: { number: 8080 }The headers, one by one:
| Header | What it mitigates |
|---|---|
X-Content-Type-Options: nosniff |
The browser guessing the content type and executing something as a script |
X-Frame-Options: DENY |
The page being embedded in another site's frame (clickjacking) |
Referrer-Policy |
Internal URLs leaking when navigating to external sites |
Permissions-Policy |
Unrequested access to camera, microphone or location |
Strict-Transport-Security |
A client reconnecting over unencrypted HTTP |
One detail about HSTS: max-age=31536000 is 365 days, and browsers remember it. If one day your TLS stops working, customers who have already visited the site will not be able to fall back to HTTP. That is what you want from a security standpoint, but you have to be aware of it before enabling it.
Denial of service protection
| Attack level | Where it is mitigated |
|---|---|
| Volumetric (saturating the bandwidth) | Outside the cluster: the cloud provider or a CDN |
| Connection exhaustion | The load balancer and limit-connections |
| Expensive requests | limit-rps, and in the application: pagination and timeouts |
| Internal resource exhaustion | resources.limits (03-04) and autoscaling (module 9) |
An important point that gets forgotten: autoscaling is not a defence against denial of service, it is a way of paying for it. If bookings-api scales to 50 replicas under an attack, you have turned an outage into an invoice. The HPA's upper limit (09-01) is also a security and cost control.
- Control plane security
Everything above protects the workloads. The control plane is the highest-value target: whoever controls it controls the whole cluster.
Access to the apiserver
| Control | How |
|---|---|
| Do not expose it to the internet | A private endpoint, or an allow list of IPs |
| Strong authentication | OIDC with a second factor, never shared certificates |
| Disable anonymous access | --anonymous-auth=false |
| Audit log | An audit policy (08-06) |
| Rate limiting | --max-requests-inflight, API priority and fairness |
A private IP is a good sign. If you saw a public IP with no source restriction, that is a first-order security finding.
On managed Kubernetes (10-06), all three major providers allow restricting access to the endpoint. It is one of the first things to configure on a new cluster.
etcd
etcd stores all the cluster's state, Secrets included. As we said in 03-02, if etcd is not encrypted, the Secrets are in the clear on disk.
Encryption at rest:
# /etc/kubernetes/encryption/config.yaml (on the control plane nodes)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
# An external KMS: the master key lives outside the cluster. Preferable.
- kms:
apiVersion: v2
name: kms-rutasnorte
endpoint: unix:///var/run/kmsplugin/socket.sock
cachesize: 1000
- identity: {} # unencrypted: needed in order to read the old dataAn important detail: the order of the providers matters. The first is used to write; all of them are used to read. Having identity: {} at the end allows reading the objects written before encryption was enabled. And after enabling it, they have to be rewritten:
Without that step, the old Secrets stay in the clear in etcd forever.
Other etcd controls:
| Control | Why |
|---|---|
| Mutual TLS between etcd and the apiserver | Nobody else should be able to talk to etcd |
| Restricted access to port 2379 | Only from the control plane |
| Encrypted backups | An unencrypted etcd backup is a backup of every Secret |
| Custody of the backups | With the same protection as the customer database |
That point about the backups is easy to overlook and very serious. In 05-06 we talked about Velero and personal data; an etcd backup also contains every credential in the cluster. It must be encrypted, stored with restricted and logged access, and its restore tested periodically.
The kubelet
Every node runs a kubelet with an API on port 10250. Unprotected, it allows listing pods, reading logs and executing commands in any container on that node.
| Setting | Correct value | What it prevents |
|---|---|---|
--anonymous-auth |
false |
Requests with no credentials |
--authorization-mode |
Webhook |
Any valid identity being able to do everything |
--read-only-port |
0 |
Port 10255, unauthenticated, exposes pod metadata |
--protect-kernel-defaults |
true |
The kubelet modifying kernel parameters |
A check:
# It must answer 401: the kubelet API does not accept anonymous requests
kubectl get --raw /api/v1/nodes/node-pro-01/proxy/pods -v6 2>&1 | grep -i "response"And a NetworkPolicy or firewall rule stopping pods from reaching the nodes' port 10250 is a very worthwhile defence, because it closes a known escalation route.
Remember from 08-01 that the get nodes/proxy permission is critical precisely because of this: it allows talking to the kubelet and, through it, executing in any pod on the node.
Isolating the control plane nodes
The control plane nodes must not run application workloads. Kubernetes achieves this with a taint (06-05):
That taint stops ordinary pods being scheduled there. The reason is direct: a pod on a control plane node is one flaw away from the cluster's certificates and the etcd database.
And you have to verify that nobody has tolerated it out of convenience:
kubectl get pods -A -o json | jq -r '
.items[]
| select(.spec.tolerations[]? |
.key == "node-role.kubernetes.io/control-plane" or
(.operator == "Exists" and (.key // "") == ""))
| "\(.metadata.namespace)/\(.metadata.name)"'Only system components that legitimately run on every node. If a rutas-norte-pro pod appeared here, you would have to investigate why that toleration was added. Beware the universal toleration (operator: Exists with no key): it tolerates every taint, the control plane one included, and it often gets added without anyone noticing.
- Traffic logging and visibility
We come back to the second limit we pointed out in 04-06: NetworkPolicies log nothing.
What that means exactly
When a NetworkPolicy denies a connection, the packet is dropped in the kernel. There is no Kubernetes event, no log, no metric. From the source pod, the connection simply times out.
The consequences are two, and both are bad:
For operating: debugging "why does my service not connect" is painful. It could be DNS, a policy, a misconfigured Service or the application. Nothing tells you which.
For security: if somebody compromises a pod and starts probing the internal network, they will generate dozens of denied connections and nobody will notice. The clearest signal of lateral movement is precisely the one that is invisible.
Hubble: flow observability with Cilium
Hubble, Cilium's observability component, solves this by taking advantage of the fact that Cilium already inspects every packet in eBPF.
helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
--set hubble.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.enabled=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,httpV2}"Watching flows in real time:
TIMESTAMP SOURCE DESTINATION TYPE VERDICT
Aug 6 09:14:22.481 rutas-norte-pro/web-store-x2 rutas-norte-pro/bookings-api-a1:8080 to-endpoint FORWARDED
Aug 6 09:14:22.503 rutas-norte-pro/bookings-api-a1 rutas-norte-pro/bookings-postgres-0:5432 to-endpoint FORWARDED
Aug 6 09:14:23.117 rutas-norte-pro/bookings-api-a1 rutas-norte-pro/redis-cache-0:6379 to-endpoint FORWARDEDAnd what really matters, the denials:
TIMESTAMP SOURCE DESTINATION TYPE VERDICT REASON
Aug 6 03:47:11.229 rutas-norte-pro/web-store-x2k4 rutas-norte-pro/bookings-postgres-0:5432 to-endpoint DROPPED Policy denied
Aug 6 03:47:11.884 rutas-norte-pro/web-store-x2k4 rutas-norte-pro/redis-cache-0:6379 to-endpoint DROPPED Policy denied
Aug 6 03:47:12.401 rutas-norte-pro/web-store-x2k4 203.0.113.44:443 to-stack DROPPED Policy denied
Aug 6 03:47:13.055 rutas-norte-pro/web-store-x2k4 198.51.100.7:22 to-stack DROPPED Policy deniedRead it carefully, because that output tells a complete story. web-store —the most exposed component— has tried, within five seconds, to: connect to the customer database, connect to the cache, reach the internet over HTTPS and reach the internet over SSH. web-store does none of that in normal operation. It is exactly the pattern of somebody exploring what they can reach from a compromised pod.
The policies did their job: everything DROPPED. But without Hubble, this would have gone completely unnoticed.
Questions you can answer after an incident
| Question | Query |
|---|---|
| What did this pod try to reach? | hubble observe --pod <name> --last 1000 |
| Did anybody reach the internet from production? | hubble observe --namespace rutas-norte-pro --to-fqdn "*" |
| When did the anomalous attempts start? | hubble observe --verdict DROPPED --since 24h |
| Who talked to the database? | hubble observe --to-pod rutas-norte-pro/bookings-postgres-0 |
| Which HTTP paths were requested? | hubble observe --protocol http --http-path "/admin*" |
Turning it into alerts
Hubble exposes Prometheus metrics, so it integrates directly with module 7:
# k8s/base/monitoring/network-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: network-security-alerts
namespace: monitoring
labels:
app.kubernetes.io/part-of: rutas-norte
spec:
groups:
- name: network-security
rules:
- alert: AnomalousDeniedConnections
expr: |
sum by (source_pod, source_namespace) (
rate(hubble_drop_total{
source_namespace="rutas-norte-pro",
reason="POLICY_DENIED"
}[5m])
) > 0.5
for: 5m
labels:
severity: warning
team: platform
annotations:
summary: >-
Pod {{ $labels.source_pod }} is generating denied connections in a
sustained way
description: >-
More than 0.5 denied connections per second for 5 minutes.
It may be a deployment with a misconfigured policy, or a compromised
pod probing the network. Investigate with:
hubble observe --pod {{ $labels.source_namespace }}/{{ $labels.source_pod }} --verdict DROPPED
runbook_url: https://wiki.rutasnorte.example/runbooks/denied-connections
- alert: UnexpectedEgressFromProduction
expr: |
sum by (source_pod) (
rate(hubble_flows_processed_total{
source_namespace="rutas-norte-pro",
destination_namespace="",
verdict="FORWARDED"
}[5m])
) > 0
for: 2m
labels:
severity: critical
team: platform
annotations:
summary: >-
Outbound internet traffic from {{ $labels.source_pod }}
description: >-
Only bookings-api (payment gateway) and notifications-worker (SMTP)
should reach the internet. Any other source is a possible route for
customer data exfiltration. Escalate immediately.That second alert is one of the most valuable on the whole platform: it warns of a possible personal data exfiltration in progress. The route to Alertmanager and its routing are those of module 7; the only new thing is the metric source.
Alternatives without Cilium
| Tool | What it brings |
|---|---|
| The provider's flow logs (VPC Flow Logs) | Flows at the virtual network level, with no pod identity |
| Calico Enterprise | Flow logging with Kubernetes context (a commercial product) |
| Falco with network rules | Unexpected connections observed from the system calls (08-06) |
| The egress proxy log | External destinations requested, with the domain name |
For Rutas Norte, with no Cilium today, the practical combination is: the egress proxy log for external traffic (which is the critical one) and Falco for anomalous connections, which we will see in the next lesson.
- Zero trust applied to Rutas Norte
Zero trust is a model that can be summed up in one phrase: never trust, always verify. Nothing is considered trusted because of where it sits on the network.
The principles and their translation
| Principle | At Rutas Norte |
|---|---|
| Nothing is trusted because of its location | A pod in rutas-norte-pro has no privileged access just for being there |
| Verify identity on every request | ServiceAccounts (03-06) and RBAC (08-01); mTLS if there were a mesh |
| Least privilege | Tight RBAC and per-component network policies |
| Assume compromise | Microsegmentation and egress control limit the damage |
| Encrypt in transit | TLS on the Ingress, WireGuard in the CNI, TLS in PostgreSQL |
| Log and verify everything | Hubble, apiserver auditing (08-06), Falco |
The idea of assuming compromise is the one that changes the design most. It is not about "how do I stop them getting in", but "when they get in, what can they do and how quickly do I find out?".
The complete defence table
| Layer | Mechanism | Attack it mitigates | State |
|---|---|---|---|
| Perimeter | WAF + OWASP CRS | SQL injection, XSS, directory traversal | To be rolled out |
| Perimeter | Rate limiting | Abuse, brute force, application-level denial of service | To be rolled out |
| Perimeter | Ingress only, no NodePort | Accidental exposure of internal services | Done |
| Transport | TLS with cert-manager | Eavesdropping on client-platform traffic | Done (04-05) |
| Transport | Security headers | Clickjacking, downgrade to HTTP, URL leakage | To be rolled out |
| Transport | WireGuard in the CNI | Eavesdropping on traffic between nodes | Recommended |
| Transport | TLS in PostgreSQL | Eavesdropping on personal data in transit | Recommended |
| Segmentation | deny-all in all three environments |
Lateral movement after a compromise | Done (04-06) + Kyverno (08-03) |
| Segmentation | Per-component policies | The reach of lateral movement | Done (04-06) |
| Segmentation | Egress control with ipBlock |
Exfiltration of customer data | Priority |
| Identity | Dedicated ServiceAccounts | Process impersonation before the API | Done (03-06) |
| Identity | mTLS with a mesh | Service-to-service impersonation | Deferred, with criteria |
| Authorization | Least-privilege RBAC | Abuse of legitimate credentials | Done (08-01) |
| Authorization | L7 policies | Access to unauthorised API paths | Deferred |
| Hardening | Restrictive securityContext |
Container escape to the node | Done (08-02) |
| Hardening | PSA + Kyverno | Somebody deploying something insecure | Done (08-03) |
| Control plane | Private apiserver, encrypted etcd | Total compromise of the cluster | To be reviewed |
| Control plane | Authenticated kubelet, no read-only port | Execution in pods via the kubelet | To be reviewed |
| Visibility | Hubble or the proxy log | Blindness to internal probing and exfiltration | Priority |
| Visibility | Alerts on denials | Late detection of an incident | To be rolled out |
The three priorities that come out of this table, in order: egress control, traffic visibility and control plane review. All three attack the same scenario: somebody is already inside.
Common Mistakes and Tips
Writing only Ingress policies. If policyTypes does not include Egress, egress is open and with it the exfiltration route. It is the most serious and most frequent mistake in this lesson.
Enabling Egress and forgetting DNS. Everything stops working in a very confusing way: names do not resolve and it looks like a Service problem. The rule towards kube-dns on port 53, UDP and TCP, is the first one to write.
Using ipBlock: 0.0.0.0/0 with no except. It includes the cluster's internal network, so the policy is far more permissive than it looks.
Trusting that an external provider's ipBlock ranges do not change. They change. Schedule a monthly verification or use a proxy with a domain list.
Believing that the ambassador container isolates the network. It shares the network namespace with the rest of the pod. It is an architecture pattern, not a security boundary.
Writing policies per namespace instead of per component. It reproduces the flat network inside the namespace. The cost of doing it properly is five more policies.
Leaving rutas-norte-dev and rutas-norte-pre without policies. The cluster network is flat between namespaces, and a policy only tested in production is tested at the worst moment.
Adopting a service mesh without needing one. It is critical infrastructure: badly operated, it worsens security. Define objective criteria for revisiting the decision.
Enabling a WAF straight into blocking mode. It will block legitimate requests and you will have an incident. DetectionOnly first, weeks of analysis, and then blocking.
Using NodePort in production "because it is quicker to configure". It bypasses the TLS, the WAF and the rate limiting, and it opens the port on every node.
Forgetting that autoscaling does not defend against denial of service. It turns an outage into an invoice. Put an upper limit on the HPA.
Encrypting etcd and not rewriting the existing Secrets. The old ones stay in the clear. kubectl get secrets -A -o json | kubectl replace -f -.
Not encrypting the etcd backups. An etcd backup contains every Secret in the cluster.
Universal tolerations. An operator: Exists with no key tolerates every taint, the control plane one included, and it usually gets added without anyone noticing.
Golden tip: draw the diagram of what talks to what and check that the policies reproduce it exactly, not one arrow more. Then ask the key question: if they compromise this pod, where can they reach and how can they get the data out? If the answer to the second part is not "nowhere", there is outstanding work on egress.
Exercises
Exercise 1: close the egress of the components that do not need it
Four of Rutas Norte's six components do not need to reach the internet: web-store, bookings-postgres, redis-cache and occupancy-reports. Write a single NetworkPolicy that allows them only what is indispensable inside the cluster and closes their internet egress completely.
Bear in mind that:
- All of them need DNS.
bookings-postgresandoccupancy-reportsmust be able to talk to each other (the CronJob queries the database).web-storemust be able to reachbookings-api.- All of them must be scrapeable by Prometheus.
Include the command that would verify that egress is effectively closed.
Exercise 2: decide on the service mesh
Rutas Norte's management proposes deploying Istio "because it is what the big companies use". Prepare a one-page technical response that includes:
- What concrete problems it would solve on the current platform.
- What it would cost, quantified.
- What cheaper alternatives cover part of the benefit.
- Objective, measurable criteria for revisiting the decision.
- A clear recommendation.
Exercise 3: investigate an incident using the network flows
At 03:47 the AnomalousDeniedConnections alert fires. You query Hubble and get:
TIMESTAMP SOURCE DESTINATION TYPE VERDICT REASON
Aug 6 03:47:09.112 rutas-norte-pro/notif-worker-k9x2 rutas-norte-pro/bookings-api-a1:8080 to-endpoint FORWARDED
Aug 6 03:47:11.229 rutas-norte-pro/notif-worker-k9x2 rutas-norte-pro/bookings-postgres-0:5432 to-endpoint FORWARDED
Aug 6 03:47:11.884 rutas-norte-pro/notif-worker-k9x2 rutas-norte-pro/redis-cache-0:6379 to-endpoint DROPPED Policy denied
Aug 6 03:47:12.401 rutas-norte-pro/notif-worker-k9x2 192.0.2.55:443 to-stack DROPPED Policy denied
Aug 6 03:47:12.902 rutas-norte-pro/notif-worker-k9x2 192.0.2.55:8443 to-stack DROPPED Policy denied
Aug 6 03:47:13.455 rutas-norte-pro/notif-worker-k9x2 192.0.2.55:53 to-stack DROPPED Policy denied
Aug 6 03:47:20.001 rutas-norte-pro/notif-worker-k9x2 rutas-norte-pro/bookings-postgres-0:5432 to-endpoint FORWARDED- What is happening? Justify your reading line by line.
- Which defence layers worked and which did not?
- List the immediate actions, in priority order.
- What would have happened without egress control? And without Hubble?
Solutions
Solution 1
# k8s/base/network/closed-egress.yaml
# Egress policy for the components that must NOT reach the internet.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: no-internet-egress
namespace: rutas-norte-pro
labels:
app.kubernetes.io/part-of: rutas-norte
annotations:
security.rutasnorte.example/justification: >-
web-store, bookings-postgres, redis-cache and occupancy-reports have no
need whatsoever to reach the internet. Closing their egress removes the
personal data exfiltration route for four of the six components.
spec:
podSelector:
matchExpressions:
- key: app
operator: In
values:
- web-store
- bookings-postgres
- redis-cache
- occupancy-reports
policyTypes: [Egress]
egress:
# 1. DNS: indispensable for all of them
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
# 2. web-store -> bookings-api
# (this rule's podSelector filters the destination, not the source;
# the other components simply do not use this permission)
- to:
- podSelector:
matchLabels:
app: bookings-api
ports:
- { protocol: TCP, port: 8080 }
# 3. occupancy-reports -> bookings-postgres
- to:
- podSelector:
matchLabels:
app: bookings-postgres
ports:
- { protocol: TCP, port: 5432 }
# THERE IS NO ipBlock rule: internet egress is closed.And the ingress policy so that Prometheus can scrape metrics (it is Ingress, complementary):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-prometheus
namespace: rutas-norte-pro
spec:
podSelector: {}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- { protocol: TCP, port: 9090 }An important observation about the first policy: by using a podSelector with matchExpressions that groups four components, they all share the same egress rules. That means redis-cache technically could connect to bookings-api:8080, even though it does not.
It is a conscious trade-off between concision and precision. The strictly correct version would be one policy per component, and that is what we would recommend in production, following principle 2 of section 2. This grouped version is acceptable because the exercise's stated objective —closing internet egress— is met all the same, and none of the extra internal permissions granted gives access to personal data. Documenting that reasoning is as important as the YAML.
Verification:
#!/usr/bin/env bash
NS=rutas-norte-pro
for target in deploy/web-store statefulset/bookings-postgres statefulset/redis-cache; do
echo -n "$target -> internet: "
if kubectl exec -n "$NS" "$target" -- \
timeout 5 sh -c 'wget -q -O- --timeout=4 https://external-example.example' \
>/dev/null 2>&1; then
echo "REACHES <-- FAIL"
else
echo "blocked (correct)"
fi
echo -n "$target -> DNS: "
kubectl exec -n "$NS" "$target" -- \
timeout 5 nslookup bookings-api >/dev/null 2>&1 \
&& echo "resolves (correct)" || echo "DOES NOT RESOLVE <-- FAIL"
donedeploy/web-store -> internet: blocked (correct)
deploy/web-store -> DNS: resolves (correct)
statefulset/bookings-postgres -> internet: blocked (correct)
statefulset/bookings-postgres -> DNS: resolves (correct)
statefulset/redis-cache -> internet: blocked (correct)
statefulset/redis-cache -> DNS: resolves (correct)Checking DNS is as important as checking the block: an egress policy that blocks the internet and DNS breaks the platform without it being obvious why.
Solution 2
Technical report: adopting a service mesh on the Rutas Norte platform
1. What problems it would solve
| Real problem | Does it solve it? | Is it urgent today? |
|---|---|---|
| Unencrypted internal traffic, personal data included | Yes, automatic mTLS | Yes, it is a real risk |
| A service cannot cryptographically verify who is calling it | Yes, SPIFFE identity | Moderate: there are 6 services, all internal and controlled |
| There is no per-HTTP-path authorization | Yes, AuthorizationPolicy |
Low: the call graph is almost linear |
| Retries and timeouts duplicated in every service | Yes | Low: it is already implemented |
| Lack of inter-service latency observability | Partially | Low: module 7 already covers the essentials |
| Percentage-based canary deployments | Yes | Low: today it is done by replicas (11-04) |
Of six problems, one is urgent (encryption) and five are desirable improvements with no urgency.
2. Quantified cost
| Item | Estimate |
|---|---|
| Sidecar memory | 6 components × ~15 pods × 80 MiB ≈ 1.2 GiB of additional memory |
| Sidecar CPU | ~0.05 cores per pod ≈ 0.75 cores |
| Added latency | 2-5 ms per hop; on the web-store→bookings-api→postgres path, up to 10 ms per request |
| Initial rollout | 3-4 weeks of one platform engineer |
| Ongoing operation | ~1 day a month: upgrades, certificates, diagnosis |
| Team training | 2 weeks for 4 people |
| Risk of incidents | Medium-high in the first months: it is the most common cause of "the service returns a 503 and I do not know why" |
With Istio in ambient mode or with Linkerd, the memory and the latency would drop substantially. The operational complexity and the risk of the first months would not.
3. Cheaper alternatives
| Alternative | Covers | Cost |
|---|---|---|
| WireGuard in the CNI | Encryption of all traffic between nodes | One configuration option; hours of work |
TLS on bookings-postgres |
Encryption of the stretch with personal data, even within the same node | 1-2 days |
| Strict NetworkPolicies (already done) | L3/L4 segmentation | Done |
| Service tokens in the application | Basic authentication between services | 2-3 days per service |
| Hubble or the egress proxy log | Flow visibility | 1 week |
The first two together cover 80% of the security benefit for less than 5% of the cost.
4. Objective criteria for revisiting the decision
The evaluation will be reopened when any of these is met:
- The platform exceeds 15 services that communicate with each other.
- More than three teams deploy to the cluster without daily coordination.
- An explicit regulatory or contractual requirement for inter-service mTLS appears (for example, a certification from a corporate customer).
- Percentage-based canary deployments are needed that cannot be approximated with replicas.
- Routing between several clusters is needed (11-05).
- The platform team reaches 5 people or more, with capacity to dedicate one to the mesh.
5. Recommendation
Do not adopt a service mesh now. Instead, in this order:
- Enable WireGuard in the CNI (this week).
- Configure TLS on
bookings-postgres(this quarter). - Roll out flow visibility (this quarter).
- Review the decision every six months against the criteria above.
If it is adopted in the future, the first candidate would be Linkerd for its benefit-to-complexity ratio, or Istio in ambient mode if by then advanced routing is needed.
Final note: this recommendation is an architecture assessment based on the current size and topology. It must be validated by the organisation's security professional, especially regarding the encryption of traffic carrying personal data, where the compliance officer has the final word on what level of protection is required.
Solution 3
1. What is happening, line by line
| Time | Flow | Reading |
|---|---|---|
| 03:47:09 | notif-worker → bookings-api:8080 FORWARDED |
Anomalous. The worker reads from the database, it does not call the API. The policy allows it (probably an over-broad permission), but it is not its normal behaviour |
| 03:47:11 | notif-worker → postgres:5432 FORWARDED |
Legitimate on its own: the worker queries pending bookings |
| 03:47:11 | notif-worker → redis-cache:6379 DROPPED |
Anomalous. The worker does not use the cache. This is probing |
| 03:47:12 | notif-worker → 192.0.2.55:443 DROPPED |
Very serious. An attempt to reach an external IP that is neither the payment gateway nor the SMTP server |
| 03:47:12 | notif-worker → 192.0.2.55:8443 DROPPED |
Retry on another port: it is looking for a way out |
| 03:47:13 | notif-worker → 192.0.2.55:53 DROPPED |
Retry on the DNS port: a classic technique for getting through permissive firewalls |
| 03:47:20 | notif-worker → postgres:5432 FORWARDED |
The most worrying part: after every egress attempt failed, it goes back to the database |
The complete sequence —four seconds, three different ports towards the same external IP, probing of internal services, and back to the database— corresponds to no legitimate behaviour of notifications-worker. The signature is that of a compromised process looking for a way to get information out, and the last line suggests it is still accessing data.
An important timing detail: it is 03:47. The occupancy-reports CronJob runs at 03:00. Any connection will have to be ruled out.
2. Layers that worked and that did not
| Layer | Result |
|---|---|
Egress control (Egress) |
Worked. The three attempts to reach the internet were blocked. This is the layer that prevented the data breach |
| Microsegmentation | Partly worked. It blocked Redis, but it allowed reaching bookings-api |
| Visibility (Hubble) | Worked. Without it, this incident would be invisible |
| Alerting | Worked. The alert fired and that is why we are investigating |
| Preventing the initial compromise | Failed. Something allowed code to run in the worker |
| The worker's network policy | Too permissive. It should not be able to reach bookings-api |
| L7 authorization | Absent. With access to bookings-api:8080, it can call any path |
| Internal encryption | Absent. The traffic to postgres goes in the clear |
3. Immediate actions, by priority
| # | Action | Why |
|---|---|---|
| 1 | Isolate the pod without destroying it: change its app label so that the Services and the policies stop applying to it, leaving it running |
It cuts the access while preserving the memory and the state for analysis. Deleting it destroys the evidence |
| 2 | Apply a policy that denies it everything, the database included | It stops the ongoing access to personal data |
| 3 | Rotate the bookings-postgres credentials and review the database's pg_stat and logs |
The process had legitimate access; we need to know what it queried |
| 4 | Query the apiserver audit log (08-06): did it use the ServiceAccount? did it read Secrets? | Determine the real scope |
| 5 | Review the last 72 hours of flows for that pod and for every pod in the namespace | Establish when it started and whether other pods are affected |
| 6 | Notify the compliance officer | There are indications of unauthorised access to personal data. The notification deadlines start running from the moment the fact is known |
| 7 | Scan the worker's image for known vulnerabilities (08-06) | Find the entry vector |
| 8 | Harden the worker's policy: remove the access to bookings-api |
Fix the over-broad permission |
| 9 | Review the rest of the policies looking for equally broad permissions | The same mistake may be elsewhere |
The commands for the first two steps:
# 1. Isolate: remove the label that makes it a target of policies and Services
kubectl label pod -n rutas-norte-pro notif-worker-k9x2 app-
kubectl label pod -n rutas-norte-pro notif-worker-k9x2 status=quarantined
# 2. Total denial for the quarantined pod
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine
namespace: rutas-norte-pro
spec:
podSelector:
matchLabels:
status: quarantined
policyTypes: [Ingress, Egress]
# No rules: absolutely everything is denied
YAMLBy removing the app label, the ReplicaSet notices it is a replica short and creates a fresh, clean pod: the service recovers on its own while the compromised pod is isolated for analysis. It is a technique worth rehearsing before you need it.
4. The counterfactual scenarios
Without egress control: the three connections to 192.0.2.55 would have been FORWARDED. The process had legitimate access to bookings-postgres, that is, to the customer table with names, ID numbers, phone numbers and emails. This would have been a consummated personal data breach, with an obligation to notify the supervisory authority and the individuals affected. The egress policy —those lines of YAML almost nobody writes— is the only thing that separated a contained incident from a data breach.
Without Hubble: there would have been no alert. Denied connections are silent: with no flow logging, the compromised pod would have carried on querying the database indefinitely, and sooner or later it would probably have found a way out (a changed configuration, a new permission, a badly reviewed policy). The incident would have been discovered weeks or months later, or never.
This exercise sums up the whole lesson: prevention failed —it always fails somewhere in the end—, containment worked, and detection allowed a reaction. All three are necessary, and the one most often neglected is the third.
Conclusion
We have built Rutas Norte's complete network strategy, layer by layer:
- Microsegmentation starts from denying by default in all three environments, with policies per component rather than per namespace, and with a periodic review of which conversations are still needed.
- Egress traffic control is the layer that decides whether a compromise turns into a data breach. Four of Rutas Norte's six components do not need to reach the internet at all, and closing their egress is the highest-value, lowest-cost measure in the whole lesson.
- NetworkPolicies work with IPs, not domain names. The ways out are the egress gateway, the proxy with an allow list of domains (which also logs the destinations) or Cilium's
toFQDNspolicies. - Inside the cluster, past the Ingress TLS, everything travels in the clear. The options are TLS in the application (a priority for
bookings-postgres), CNI-level encryption with WireGuard (cheap and effective between nodes) or mTLS with a service mesh. - A service mesh gives cryptographic per-workload identity, automatic mTLS and L7 authorization, in exchange for operational complexity, resources and latency. Rutas Norte does not need one today, and we have set objective criteria for revisiting that decision. Istio (sidecar or ambient), Linkerd and Cilium Service Mesh cover different profiles.
- At the perimeter: a single entry point (no NodePort), a WAF adopted gradually, rate limiting and security headers.
- The control plane deserves its own review: an apiserver that is not exposed, etcd encrypted and with the Secrets rewritten, encrypted backups, an authenticated kubelet with no read-only port, and control nodes isolated with taints.
- What NetworkPolicy lacks is logging. Hubble, or an egress proxy's log, turns internal probing and exfiltration attempts into something visible and alertable.
- Zero trust boils down to assuming compromise: not "how do I stop them getting in", but "when they get in, what can they do and how quickly do I find out".
With this, Rutas Norte's network is segmented, with controlled egress and visibility of what happens. But notice something: everything we have done in this module protects the cluster at runtime. We take for granted that the images we run are the ones we think they are.
And that assumption is fragile. registry.rutasnorte.example/bookings-api:2.7.1 is a tag, and a tag can be reassigned: the image pulled yesterday may not be today's. The base image it was built on may contain software nobody has reviewed. The dependencies installed during the build come from public repositories. And nothing, absolutely nothing, stops somebody with registry access today from pushing a modified image under the same name and having the cluster run it without complaint, with all our securityContext settings perfectly applied to a container that does something different from what we think.
The next lesson, 08-05, Image Security, walks the whole supply chain: where an image can be poisoned, how to build minimal images, why the digest is the only genuinely reproducible reference, how to sign with Cosign and —most importantly— how to make the cluster reject any unsigned image, with the admission policy that guarantees it.
Kubernetes Course
Module 1: Introduction to Kubernetes
- What Is Kubernetes?
- Kubernetes Architecture
- Key Concepts and Terminology
- Setting Up a Kubernetes Cluster
- The Kubernetes CLI: kubectl
- Objects, YAML Manifests and the Declarative Model
- The Course Project: the Rutas Norte Platform
Module 2: Core Kubernetes Components
- Pods
- ReplicaSets
- Deployments
- Updates, Rollbacks and Deployment Strategies
- Services
- Namespaces
- Labels, Selectors and Annotations
Module 3: Configuration and Secret Management
- ConfigMaps
- Secrets
- Environment Variables
- Resource Quotas and Limits
- LimitRanges and Quality of Service (QoS) Classes
- ServiceAccounts and API Access from Pods
Module 4: Networking in Kubernetes
- Cluster Networking
- Service Types
- Internal DNS and Service Discovery
- Ingress Controllers
- TLS and Certificate Management with cert-manager
- Network Policies
Module 5: Storage in Kubernetes
- Volumes
- Persistent Volumes
- Persistent Volume Claims
- Storage Classes
- Dynamic Provisioning, Expansion and Snapshots
- Backup and Restore of Persistent Data
Module 6: Advanced Kubernetes Concepts
- StatefulSets
- DaemonSets
- Jobs and CronJobs
- Init Containers, Sidecars and Multi-Container Patterns
- Scheduling: Affinity, Taints and Tolerations
- Custom Resource Definitions (CRDs)
- Operators and the Controller Pattern
Module 7: Monitoring and Logging
- Health Checks and Probes
- Metrics Server and kubectl top
- Monitoring with Prometheus
- Visualization and Alerting with Grafana and Alertmanager
- Centralized Logging with Elasticsearch, Fluentd and Kibana (EFK)
- Application Debugging and Cluster Events
Module 8: Kubernetes Security
- Role-Based Access Control (RBAC)
- Security Contexts and Container Hardening
- Pod Security Policies and Pod Security Standards
- Network Security
- Image Security
- Auditing, Scanning and Vulnerability Management
Module 9: Scaling and Performance
- Horizontal Pod Autoscaling
- Vertical Pod Autoscaling
- Cluster Autoscaling
- Event-Driven and Custom-Metric Scaling with KEDA
- High Availability: PodDisruptionBudgets and Topology
- Performance Tuning
Module 10: Kubernetes Ecosystem and Tooling
- Minikube and Local Environments with kind
- Kubeadm
- Helm
- Kustomize
- GitOps with Argo CD and Flux
- Managed Kubernetes: EKS, AKS and GKE
Module 11: Case Studies and Real-World Applications
- Deploying a Web Application
- Running Stateful Applications
- CI/CD with Kubernetes
- Deployment Strategies: Blue-Green and Canary
- Multi-Cluster Management
- Production Operations: Incidents, Runbooks and Costs
