We have closed two dimensions of Rutas Norte's security. With RBAC (08-01) we control who can do what against the API. With security contexts (08-02) and admission policies (08-03) we control what a container can do once it is running, and we got the cluster to enforce it on its own.

An entire dimension remains: what can talk to what. And here we already did important work in 04-06, when we put a deny-all NetworkPolicy in rutas-norte-pro and authorised each conversation one at a time. But we then pointed out three uncomfortable limits that we parked:

  1. NetworkPolicies work at L3/L4: they understand IPs and ports, not HTTP paths or methods. bookings-api can reach bookings-postgres, yes, but the policy does not distinguish a legitimate query from a full dump of the customer table.
  2. They log nothing. A denied connection attempt is completely invisible. If somebody is probing the internal network, we never find out.
  3. Inside the cluster, all traffic travels unencrypted once past the Ingress TLS (04-05). Whoever can observe the node's network sees the queries to bookings-postgres in the clear, with names, ID numbers and phone numbers included.

And there is a fourth thing we have barely mentioned: outbound traffic. It is the natural route by which a customer database leaves.

This lesson does not repeat the NetworkPolicy syntax: it takes it as known and uses it. What we build here is the complete network strategy of a production platform, layer by layer.

Important warning. The network architecture design of a production platform must be reviewed by a security professional, who will assess the specific threat model. And since Rutas Norte's traffic carries customers' personal data —name, ID number, phone and email—, the decisions about encryption in transit, egress control and flow logging must also be known and approved by the compliance officer. This lesson's approach is exclusively defensive: understanding the paths by which information can leak in order to close them and detect them.

Contents

  1. The network defence layers
  2. Microsegmentation as a principle
  3. Egress traffic control
  4. The IPs versus domain names problem
  5. Encryption in transit inside the cluster
  6. What a service mesh is and what it costs
  7. Istio, Linkerd and Cilium compared
  8. Layer 7 authorization policies
  9. Protecting the perimeter
  10. Control plane security
  11. Traffic logging and visibility
  12. Zero trust applied to Rutas Norte
  13. Common mistakes and tips
  14. Exercises
  15. Conclusion

  1. The network defence layers

Network security is not one mechanism: it is a series of layers, each of which assumes the previous one can fail. This is the complete model we are going to build.

flowchart TB
    I["Internet"] --> WAF["Layer 1: perimeter<br/>WAF + rate limiting<br/>+ DoS protection"]
    WAF --> ING["Layer 2: Ingress<br/>TLS terminated (04-05)<br/>security headers"]
    ING --> NP["Layer 3: microsegmentation<br/>deny-all NetworkPolicy + explicit<br/>permissions (04-06)"]
    NP --> MESH["Layer 4: mTLS + L7 authorization<br/>per-workload identity<br/>(service mesh)"]
    MESH --> APP["Rutas Norte workloads"]
    APP --> EGR["Layer 5: egress control<br/>only the payment gateway"]
    EGR --> EXT["pagos.proveedorexterno.example"]

    APP -.-> OBS["Layer 6: visibility<br/>flow logging<br/>(Hubble)"]
    CP["Layer 7: control plane<br/>apiserver, etcd, kubelet"] -.-> APP

    style NP fill:#d5e8f9,stroke:#36c
    style EGR fill:#f9d5d5,stroke:#c33
Layer What it mitigates State at Rutas Norte
1. Perimeter Attacks from the internet, denial of service, abuse To be built
2. Ingress with TLS Eavesdropping on traffic between client and platform Done (04-05)
3. Microsegmentation Lateral movement after a pod is compromised Done (04-06), to be reinforced
4. mTLS and L7 authorization Internal eavesdropping, service impersonation To be decided
5. Egress control Exfiltration of customer data To be built
6. Visibility Blindness during an incident To be built
7. Control plane Total compromise of the cluster To be reviewed

The order matters: each layer is justified by what would happen if the previous one failed. If the WAF does not stop an attack, the Ingress still validates the certificate. If somebody manages to run code in web-store, microsegmentation stops them reaching bookings-postgres. If they did get there, mTLS would stop them passing themselves off as bookings-api. And if everything above failed, egress control would stop the data leaving the cluster.

  1. Microsegmentation as a principle

Microsegmentation means treating each workload as its own network segment, with explicit rules about what can talk to what. It is the opposite of the traditional "trusted internal network behind a firewall" model.

Why the perimeter model does not work in Kubernetes

In a classic network, the firewall separated "inside" from "outside", and inside everybody talked to everybody. In a Kubernetes cluster, by default, every pod can talk to every pod in every namespace. A curl from web-store reaches bookings-postgres with no obstacle.

Remember what we said in 02-06: a namespace is not a security boundary on its own. Without NetworkPolicies, rutas-norte-dev and rutas-norte-pro are on the same flat network.

The practical consequence: if somebody manages to run code in web-store —the most exposed component, because it faces the internet directly— they have network access to the whole platform. Microsegmentation turns that into "they have network access to bookings-api, and only to port 8080".

The three principles

Principle 1: deny by default in all three environments.

In 04-06 we applied deny-all in rutas-norte-pro. That is not enough. rutas-norte-dev and rutas-norte-pre need it too, for two reasons:

  • If development and pre-production are open, a policy that works there can fail in production, and the failure is discovered at the worst moment.
  • A compromised pod in rutas-norte-dev with no network restrictions can reach rutas-norte-pro, because the cluster network is flat.

In 08-03 we solved this elegantly with a Kyverno policy that generates the deny-all in every new Rutas Norte namespace, with synchronize: true so that it comes back if somebody deletes it. It is the guarantee that no future namespace is born open.

Principle 2: policies per component, not per namespace.

It is tempting to write "everything in rutas-norte-pro can talk to everything in rutas-norte-pro". That is a mistake: it reproduces inside the namespace the very flat model we wanted to avoid.

Approach What it allows if web-store is compromised
Per namespace Access to bookings-postgres, redis-cache, bookings-api and everything else
Per component Only bookings-api:8080. Nothing else

The difference is enormous and the cost is writing six policies instead of one.

Principle 3: periodic review of which conversations are still needed.

Network policies decay just like RBAC. A permission is added to debug something and it stays. A component is retired and its policy survives. A port is changed and the old one is left "just in case".

Rutas Norte's authorised communications matrix, as it stood in 04-06, must fit in one table and be reviewed every quarter:

Source Destination Port Justification Last review
Ingress web-store 8080 Public traffic 2026-07
Ingress bookings-api 8080 Public API 2026-07
web-store bookings-api 8080 Queries and bookings 2026-07
bookings-api bookings-postgres 5432 Booking and customer data 2026-07
bookings-api redis-cache 6379 Availability cache 2026-07
bookings-api Internet (gateway) 443 Payments 2026-07
notifications-worker bookings-postgres 5432 Reads pending bookings 2026-07
notifications-worker Internet (SMTP) 587 Sending emails 2026-07
occupancy-reports bookings-postgres 5432 Nightly aggregates 2026-07
Prometheus All 9090 Metrics (07-03) 2026-07
All CoreDNS 53 Name resolution 2026-07

Each row must have a living reason. A useful question in the review: if I delete this rule right now, what breaks? If nobody knows, find out (in pre-production) or remove it.

A script to detect orphaned policies:

#!/usr/bin/env bash
# k8s/security/orphaned-policies.sh
# Detects NetworkPolicies whose podSelector matches no pod.
set -euo pipefail
NS="${1:-rutas-norte-pro}"

kubectl get networkpolicy -n "$NS" -o json | jq -r '
  .items[] | select(.spec.podSelector.matchLabels != null)
  | "\(.metadata.name)\t" +
    ([.spec.podSelector.matchLabels | to_entries[] | "\(.key)=\(.value)"] | join(","))' \
| while IFS=$'\t' read -r policy selector; do
    n=$(kubectl get pods -n "$NS" -l "$selector" --no-headers 2>/dev/null | wc -l)
    [[ "$n" -eq 0 ]] && echo "ORPHANED: $policy (selector: $selector) matches no pod"
  done
ORPHANED: allow-legacy-exporter (selector: app=legacy-exporter) matches no pod

That policy outlived the component it protected. It is not dangerous in itself, but it is noise that makes review harder, and if tomorrow somebody deploys something with that label it inherits permissions nobody decided on.

  1. Egress traffic control

Here is, probably, the most serious gap left in Rutas Norte.

Why egress matters so much

Think about the sequence of an incident:

  1. Somebody finds a flaw in bookings-api and manages to run code.
  2. The process has legitimate access to bookings-postgres: it can read the customer table with names, ID numbers, phone numbers and emails.
  3. Now it needs to get that data out of the cluster.

Step 3 is what turns a compromise into a data breach. With no egress control, step 3 is trivial: an HTTPS request to any server on the internet.

A rule to internalise: the ingress policy limits who gets in; the egress policy limits what can get out. Almost everybody writes the first and forgets the second. And the second is what decides whether an incident stays as "somebody ran code" or escalates to "the personal data of 200,000 customers was leaked".

Most NetworkPolicy guides only talk about Ingress. Check your policies: if policyTypes does not include Egress, outbound traffic is wide open.

The egress Rutas Norte needs

Let us audit which outbound flows are legitimate:

Component External destination Port Indispensable?
bookings-api pagos.proveedorexterno.example 443 Yes: payments
notifications-worker The provider's SMTP server 587 Yes: confirmation emails
web-store None It does not need to reach the internet
bookings-postgres None No
redis-cache None No
occupancy-reports None No

Four of the six components do not need to reach the internet at all. That is the first thing to take advantage of: closing their egress completely is free and removes the exfiltration route for most of the platform.

The base egress policy

Recalling that in 04-06 we left deny-all with policyTypes: [Ingress, Egress], each component explicitly needs its own egress. The universal minimum is DNS:

# k8s/base/network/allow-dns.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: rutas-norte-pro
spec:
  podSelector: {}                 # every pod
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Without this, no pod resolves names and everything fails in a very confusing way: services cannot find each other even though the ingress policies are correct. It is the most frequent mistake when enabling Egress for the first time.

Egress to the payment gateway

bookings-api needs to reach pagos.proveedorexterno.example. This is where the problem of the next section appears, but let us first see the solution with ipBlock:

# k8s/base/network/bookings-api-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bookings-api-egress
  namespace: rutas-norte-pro
  annotations:
    security.rutasnorte.example/justification: >-
      Egress to the external payment gateway. The IP ranges are published by the
      provider in its documentation and are reviewed monthly by the scheduled
      job "verify-gateway-ranges". The last verification is
      documented in the network change log.
spec:
  podSelector:
    matchLabels:
      app: bookings-api
  policyTypes: [Egress]
  egress:
    # 1. DNS (indispensable)
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    # 2. Database and cache (internal traffic)
    - to:
        - podSelector:
            matchLabels:
              app: bookings-postgres
      ports:
        - { protocol: TCP, port: 5432 }
    - to:
        - podSelector:
            matchLabels:
              app: redis-cache
      ports:
        - { protocol: TCP, port: 6379 }
    # 3. Payment gateway: ONLY these ranges, ONLY port 443
    - to:
        - ipBlock:
            cidr: 203.0.113.0/24        # range published by the provider
        - ipBlock:
            cidr: 198.51.100.64/26      # secondary range
      ports:
        - { protocol: TCP, port: 443 }

The important thing about this manifest is not what it allows, but what it does not allow: bookings-api cannot connect to any other address on the internet. If somebody manages to run code there and wants to send the customer table to a server of their own, the connection does not get out.

A note about ipBlock and private IPs: ipBlock refers to network IPs, and the cluster's pods have IPs too. An ipBlock: 0.0.0.0/0 would include all the cluster's internal traffic. When you want "the whole internet except the internal network", you have to use except:

    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8        # pod and service network
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - { protocol: TCP, port: 443 }

This is "any public destination on 443", which is far laxer than the provider's range list and we do not recommend it for bookings-api. We include it because it is a pattern that comes up often and it is worth understanding why it is worse: it allows sending data to any server on the internet that speaks HTTPS.

The ambassador container

Remember from 06-04 that the connection with the payment gateway goes through an ambassador container inside the bookings-api pod. That has an important network consequence: since the containers in a pod share the network namespace, the policy applies to the whole pod, not to the container. The ambassador and the main container have exactly the same network permissions.

The ambassador brings other advantages —it centralises retries, timeout logic and the gateway credentials— but it is not a network security boundary. If we wanted only the ambassador to be able to talk to the gateway, it would have to be a separate pod with its own policy.

It is a good example of a common confusion: the ambassador pattern is an architecture pattern, not an isolation one.

  1. The IPs versus domain names problem

Here comes the structural limitation of NetworkPolicies.

Standard Kubernetes NetworkPolicies work with IP addresses and CIDR ranges. They know nothing about domain names. There is no to: dnsName: pagos.proveedorexterno.example field.

And this is not an oversight: policies are translated into firewall rules in the node's kernel, which acts on IP packets. By the time the packet gets there, the name has already been resolved and no trace of it remains.

The three practical problems

Problem 1: IPs change. Cloud services rotate addresses. The payment provider's documentation may publish ranges today and change them in three months. If your ipBlock falls out of date, payments stop working and the diagnosis is especially thankless because nothing in the logs says "a NetworkPolicy blocked this".

Problem 2: a range is more than you want. 203.0.113.0/24 is 256 addresses. If the provider shares infrastructure, that range may include servers belonging to other customers of theirs. You have authorised egress to 256 destinations in order to reach one.

Problem 3: the reverse scenario is worse. If a legitimate service shares an IP with a generic storage service (something common in public clouds), authorising the first authorises the second, and there you really do have an exfiltration route.

The three options

Option How it works Advantages Drawbacks
Egress gateway All outbound traffic goes through specific pods with a fixed IP, which apply the rules A single point of control and logging; a stable source IP so the provider can filter on it Another piece to maintain; a single point of failure
Proxy with an allow list An HTTP(S) proxy the applications point to, with a list of authorised domains Filters by domain name, SNI included; complete logging of destinations The applications must be configured to use it; it adds latency
DNS-based policies (Cilium) The CNI observes the DNS responses and programs the rules with the IPs they return You write the name directly; it updates itself Requires Cilium as the CNI

Option A: an egress proxy with an allow list

It is the most portable option and the one that gives the best visibility. A Squid or a dedicated proxy in a pod, with a list of domains:

# k8s/base/network/egress-proxy-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: egress-proxy-config
  namespace: rutas-norte-sistema
data:
  allowed-domains.txt: |
    # Payment gateway: ticket payments
    pagos.proveedorexterno.example
    # Outbound mail: booking confirmations
    smtp.proveedorcorreo.example
    # NOTHING ELSE. Any other destination is rejected and logged.

The applications are configured with HTTPS_PROXY, and the NetworkPolicy only allows egress to the proxy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: bookings-api-egress-via-proxy
  namespace: rutas-norte-pro
spec:
  podSelector:
    matchLabels:
      app: bookings-api
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
    # The only egress allowed: the proxy. Not one internet IP directly.
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: rutas-norte-sistema
          podSelector:
            matchLabels:
              app: egress-proxy
      ports:
        - { protocol: TCP, port: 3128 }

The decisive advantage: the proxy logs every destination requested, allowed or denied. That partly solves the NetworkPolicies' lack of logging, at least for outbound traffic, which is the one that matters most to watch.

An honest drawback: an application that ignores the HTTPS_PROXY variables bypasses the proxy. That is why the NetworkPolicy is still necessary: it prevents direct egress, forcing traffic through the proxy even if the application does not cooperate. The two layers together work; neither one on its own.

Option B: DNS-based policies with Cilium

If the CNI is Cilium, the CiliumNetworkPolicy lets you write the name directly:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: bookings-api-egress-dns
  namespace: rutas-norte-pro
spec:
  endpointSelector:
    matchLabels:
      app: bookings-api
  egress:
    # DNS with inspection: Cilium observes the responses to program the rules
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - { port: "53", protocol: UDP }
          rules:
            dns:
              - matchPattern: "*"
    # Egress by DOMAIN NAME, not by IP
    - toFQDNs:
        - matchName: "pagos.proveedorexterno.example"
      toPorts:
        - ports:
            - { port: "443", protocol: TCP }

How it works: Cilium intercepts the pod's DNS responses, sees which IP pagos.proveedorexterno.example returned and dynamically programs the rule for that IP, with the time to live the DNS record indicates. If tomorrow the provider changes IP, the policy keeps working without anyone touching anything.

It is by far the most elegant solution. Its cost is the dependency on the CNI: adopting Cilium is a platform decision that goes well beyond this one policy.

Recommendation for Rutas Norte

A two-stage approach:

  1. Now: ipBlock with the ranges published by the provider, plus a monthly scheduled job that verifies they are still valid and warns if they change. It is the solution that works with any CNI and requires no new pieces.
  2. In the medium term: an egress proxy with an allow list, mainly for the destination logging. If at some point we migrate to Cilium (which would also bring the advantages of section 11), the policies are replaced with toFQDNs.

And the check that must be automated in any case:

#!/usr/bin/env bash
# k8s/security/verify-egress.sh
# Confirms that the components which must not reach the internet, do not.
set -uo pipefail
NS=rutas-norte-pro

check_blocked() {
  local workload="$1" target="$2"
  local output
  output=$(kubectl exec -n "$NS" "$workload" -- \
    timeout 5 wget -q -O- --timeout=4 "$target" 2>&1 || echo "BLOCKED")
  if [[ "$output" == *"BLOCKED"* ]]; then
    echo "OK    $workload cannot reach $target"
  else
    echo "FAIL  $workload CAN reach $target <-- exfiltration route open"
  fi
}

check_blocked deploy/web-store               https://external-example.example
check_blocked statefulset/bookings-postgres  https://external-example.example
check_blocked deploy/bookings-api            https://external-example.example
OK    deploy/web-store cannot reach https://external-example.example
OK    statefulset/bookings-postgres cannot reach https://external-example.example
OK    deploy/bookings-api cannot reach https://external-example.example

This test must run in rutas-norte-pre on every deployment. It is the verification of the layer that prevents a personal data breach, and as such it should be in the file the compliance officer reviews.

  1. Encryption in transit inside the cluster

In 04-05 we put TLS on the Ingress with cert-manager: the traffic between the customer's browser and the platform is encrypted as far as https://www.rutasnorte.example. Perfect. But what happens after the Ingress?

What is left unencrypted

flowchart LR
    C["Browser"] -->|"HTTPS<br/>encrypted (04-05)"| ING["Ingress"]
    ING -->|"HTTP<br/>IN THE CLEAR"| TW["web-store"]
    TW -->|"HTTP<br/>IN THE CLEAR"| API["bookings-api"]
    API -->|"PostgreSQL protocol<br/>IN THE CLEAR"| PG[("bookings-postgres<br/>personal data")]
    API -->|"RESP<br/>IN THE CLEAR"| R[("redis-cache")]
    style PG fill:#f9d5d5,stroke:#c33

Everything to the right of the Ingress travels unencrypted over the cluster network. And through that stretch, in the clear, pass:

  • The SQL queries with customers' names, ID numbers, phone numbers and emails.
  • The API responses with the booking data.
  • The database credentials, in the initial handshake of every connection.

Who can see that traffic?

This is the question that decides whether the effort is worth it:

Scenario Can it see the internal traffic?
An ordinary pod with strict NetworkPolicies No (it cannot even connect)
A pod with hostNetwork: true Yes: it sees all the node's traffic
A pod with CAP_NET_RAW and hostNetwork Yes: it can capture packets
Somebody with node access (SSH, a privileged container) Yes, everything
The cloud or data-centre provider It depends on the contract and the infrastructure
An attacker who compromises the CNI Yes

Notice how the lessons connect: in 08-02 we forbade hostNetwork and CAP_NET_RAW, and in 08-03 we made the cluster reject them. That work is what makes unencrypted internal traffic an acceptable risk in many scenarios. Internal encryption is the defence for when that layer fails, or when regulation explicitly requires it.

The three internal encryption options

Option What it encrypts Cost Identity
TLS in the application Only what the application implements Code changes in every component Certificates managed by hand
CNI encryption (WireGuard, IPsec) All traffic between nodes Low: one configuration option Per node, not per workload
mTLS with a service mesh All traffic between pods in the mesh High: a whole infrastructure layer Per workload

Option 1: application-level TLS

It is what we already do with the payment gateway (the ambassador speaks HTTPS). For internal traffic it would mean configuring PostgreSQL with ssl = on, generating certificates, distributing them and configuring every client.

# Fragment: PostgreSQL with mandatory TLS
env:
  - name: POSTGRES_INITDB_ARGS
    value: "--auth-host=scram-sha-256"
volumeMounts:
  - name: tls-certificates
    mountPath: /var/lib/postgresql/certs
    readOnly: true

Advantage: it encrypts the stretch that matters most (the personal data) with little rollout. Drawback: it has to be done component by component, you have to manage certificate rotation and trust that each client actually verifies the certificate (many database clients, by default, do not).

For Rutas Norte, encrypting the bookings-postgres connection specifically is a high-value, moderate-cost measure, and probably the first step to take.

Option 2: CNI-level encryption

Several CNIs can automatically encrypt all traffic between nodes with WireGuard or IPsec.

# Cilium with WireGuard encryption (Helm values)
encryption:
  enabled: true
  type: wireguard
  nodeEncryption: true
# Calico with WireGuard
kubectl patch felixconfiguration default --type=merge \
  -p '{"spec":{"wireguardEnabled":true}}'
For Against
Effort Minimal: one option
Coverage All traffic between nodes, no exceptions Only between nodes: two pods on the same node are not encrypted
Performance WireGuard is very efficient Some CPU; with IPsec, more
Identity It authenticates nodes, not workloads

That last point is the essential limitation: CNI encryption protects against whoever observes the network between nodes, but it provides no per-service identity. web-store and bookings-api on the same node talk just as much in the clear, and neither can cryptographically prove who it is.

Even so, the value-to-effort ratio is excellent. If the CNI supports it, turn it on.

Option 3: mTLS with a service mesh

That is the next section, because it gives much more than encryption.

  1. What a service mesh is and what it costs

A service mesh is an infrastructure layer that takes care of service-to-service communication, moving it out of the application code.

How it works

In the classic model, a sidecar proxy (usually Envoy) is injected into each pod. All the traffic entering and leaving the pod goes through that proxy, which is what applies the encryption, the authorization and the metrics collection. The application never notices: it still does http://bookings-api:8080.

flowchart LR
    subgraph P1["Pod web-store"]
        A1["nginx"] <--> S1["sidecar proxy"]
    end
    subgraph P2["Pod bookings-api"]
        S2["sidecar proxy"] <--> A2["Node.js"]
    end
    S1 <-->|"mTLS<br/>encryption + identity<br/>+ L7 authorization"| S2
    CP["Control plane<br/>issues certificates<br/>distributes policies"] -.-> S1
    CP -.-> S2

What it gives you

Capability What it solves Can you have it without a mesh?
Cryptographic per-workload identity Each service has a certificate proving who it is Very hard by hand
Automatic mTLS Encryption and mutual authentication without touching code Hard, component by component
L7 authorization "Only web-store may do POST /bookings" Not with NetworkPolicy
Retries and timeouts Resilience without code Yes, in each application
Circuit breakers Isolating a degraded service Yes, with libraries
Percentage-based canary deployments Sending 5% of the traffic to the new version Partially (11-04)
Communication observability Latency and error metrics for every call Partially (07-03)

Per-workload identity is the crown jewel and deserves to be understood properly. Without a mesh, when bookings-postgres receives a connection from an IP, all it knows is that it comes from that IP. With the NetworkPolicy we know that IP corresponds to a pod with the app: bookings-api label... but labels are set by whoever creates the pod, and an IP can be spoofed.

With mTLS, bookings-postgres receives a certificate saying spiffe://cluster.local/ns/rutas-norte-pro/sa/bookings-api, signed by the cluster's certificate authority, which nobody else can forge. That is identity. And it fits perfectly with the ServiceAccounts of 03-06.

What it costs you

This is the part the vendor presentations skate over:

Cost Detail
Operational complexity One more piece of critical infrastructure to upgrade, watch and debug
Resources One sidecar per pod: between 50 and 100 MiB of memory and some CPU. With 50 pods, that is several GiB
Latency Between 1 and 5 ms per hop. With several chained hops, it shows
Learning curve New concepts: VirtualService, DestinationRule, AuthorizationPolicy, PeerAuthentication
Harder debugging A 503 can come from the application or from the proxy. You have to learn to read Envoy's logs
Coupling on upgrades Upgrading the mesh usually means restarting every pod
Odd cases Jobs that finish while the sidecar stays alive; protocols the proxy does not understand well

That last point has a direct example at Rutas Norte: the occupancy-reports CronJob finishes its work, but the sidecar keeps running and the Job is never marked as complete. There are solutions (native sidecar containers since 1.29, or calling the proxy's termination endpoint), but it is exactly the kind of unforeseen friction that shows up.

Does Rutas Norte need a service mesh?

Let us answer honestly, because it is an expensive decision.

Arguments in favour:

  • It carries personal data between services and internal encryption is defensible in any serious risk analysis.
  • L7 authorization would allow rules like "only web-store may do POST /bookings", which NetworkPolicy cannot express.
  • Cryptographic identity eliminates service impersonation.

Arguments against:

  • There are six components. A mesh shines with dozens or hundreds of microservices and complex communications.
  • The call graph is almost linear: web-storebookings-apibookings-postgres/redis-cache. There is no topology that justifies a routing layer.
  • The platform team is small. Adding a badly operated mesh can worsen security (expired certificates, misunderstood policies, deferred upgrades).
  • A good part of the benefit can be had more cheaply: CNI encryption for transit, TLS in PostgreSQL for what is critical, and strict NetworkPolicies for segmentation.

Decision for Rutas Norte: no, not yet. The alternative plan:

  1. Enable the CNI's WireGuard encryption (one configuration option).
  2. Configure TLS on bookings-postgres, which is where the personal data lives.
  3. Keep the strict NetworkPolicies with egress control.
  4. Review the decision when any of these objective criteria is met: more than fifteen services, communication between several teams that do not coordinate, an explicit regulatory requirement for mTLS, or a real need for percentage-based canary deployments.

And a general recommendation that goes beyond this course:

A service mesh is an excellent answer to a problem you have to have first. Adopting it "because it is modern", without the complexity that justifies it, adds operational risk without adding net security. If you operate it badly, it is worse than not having it.

That said, you have to know it, because the day the platform grows it will be the right answer. Let us look at the options.

  1. Istio, Linkerd and Cilium compared

Istio (sidecar) Istio (ambient) Linkerd Cilium Service Mesh
Architecture An Envoy sidecar per pod A per-node agent (ztunnel) + an optional L7 waypoint Its own sidecar (linkerd2-proxy, in Rust) eBPF in the kernel + Envoy only for L7
Memory per pod 50-100 MiB Almost zero (no sidecar) 10-20 MiB Almost zero for L4
Added latency 2-5 ms 1-3 ms <1 ms Minimal
Complexity High Medium Low Medium-high
Automatic mTLS Yes Yes Yes Yes
L7 authorization Very complete Yes (with a waypoint) Basic Yes
Advanced routing The most complete Complete Sufficient Good
Multi-cluster Very mature Yes Yes Yes
Requires a specific CNI No No No Yes: Cilium
Community and ecosystem The largest Growing Solid Growing
Learning curve Steep Medium Gentle Medium
Governance CNCF (graduated) CNCF CNCF (graduated) CNCF (graduated)

Selection criteria

Istio in sidecar mode if you need the most complete feature set: sophisticated routing, complex multi-cluster, API gateway integration, WebAssembly extensions. It is the option with the most capability and also the most expensive to operate. You need dedicated people.

Istio in ambient mode is the evolution that tackles the main criticism of the sidecar model: the per-pod cost. Instead of a proxy in every pod, there is a per-node agent (ztunnel) providing mTLS and L4, and an L7 proxy (waypoint) is deployed only where layer 7 authorization is genuinely needed. It reduces consumption enormously and removes the problem of Jobs that never finish. If we had to choose Istio today, it would be this mode.

Linkerd if you want mTLS and observability with minimum complexity. Its Rust proxy is remarkably lightweight and its philosophy is to do few things and do them well. For a platform like Rutas Norte, if a mesh were ever needed, this would be the first candidate: it covers what we need (mTLS, identity, basic authorization) without Istio's surface area.

Cilium Service Mesh if you already use Cilium as your CNI. By integrating the mesh into the CNI you avoid a whole layer: mTLS and L4 policies are applied in eBPF, inside the kernel, with no proxy. Envoy is only deployed for the L7 policies. It is the most efficient option, and it also brings the domain-name policies from section 4 and the Hubble visibility from section 11. The trade-off is that it ties the mesh decision to the CNI decision.

  1. Layer 7 authorization policies

This is the specific gap NetworkPolicy cannot fill, and it is worth seeing with a real example.

The L3/L4 limit

Our current policy says: web-store may connect to bookings-api on port 8080. That is all it can express. As soon as the connection is open, web-store can do:

Request Should it be able to? Does the NetworkPolicy prevent it?
GET /availability Yes
POST /bookings Yes
GET /admin/customers No No
DELETE /bookings/12345 No No
GET /metrics Debatable No

An L7 policy can tell the difference. Let us see how it would be expressed with Istio, taking the example from the statement: only bookings-api may do POST /bookings.

# Example L7 policy with Istio (only if a mesh were adopted)
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: bookings-api-authorization
  namespace: rutas-norte-pro
spec:
  selector:
    matchLabels:
      app: bookings-api
  action: ALLOW
  rules:
    # Rule 1: web-store may check availability and create bookings
    - from:
        - source:
            # CRYPTOGRAPHIC IDENTITY, not a label or an IP
            principals:
              - "cluster.local/ns/rutas-norte-pro/sa/web-store"
      to:
        - operation:
            methods: ["GET"]
            paths: ["/availability", "/routes", "/health", "/ready"]
        - operation:
            methods: ["POST"]
            paths: ["/bookings"]

    # Rule 2: only the notifications worker queries pending bookings
    - from:
        - source:
            principals:
              - "cluster.local/ns/rutas-norte-pro/sa/notifications-worker"
      to:
        - operation:
            methods: ["GET"]
            paths: ["/bookings/pending"]

    # Rule 3: Prometheus may only read the metrics
    - from:
        - source:
            principals:
              - "cluster.local/ns/monitoring/sa/prometheus"
      to:
        - operation:
            methods: ["GET"]
            paths: ["/metrics"]

With action: ALLOW, anything that matches no rule is denied. It is the same default-deny principle as the NetworkPolicies, now applied to HTTP methods and paths.

Look at the key line:

principals:
  - "cluster.local/ns/rutas-norte-pro/sa/web-store"

That is neither a label nor an IP: it is the SPIFFE identity the mesh verifies cryptographically in the mTLS handshake. A pod cannot claim that identity without holding the corresponding private key. And note that it is based on the ServiceAccount, closing the circle with 03-06 and 08-01: the ServiceAccount goes from being only the identity before the API to also being the identity before the other services.

The complete comparison

NetworkPolicy (04-06) AuthorizationPolicy (mesh)
Level L3/L4: IP and port L7: method, path, headers
Identity Pod labels (spoofable) Cryptographic certificate
Encryption No Yes (mTLS)
Logging of denials No Yes
Requires installing Nothing (with a compatible CNI) A complete mesh
Operational cost Low High

The two are complementary, not alternatives. The NetworkPolicy is the first line and the cheapest; the L7 policy is the fine tuning. If you had to keep only one, keep the NetworkPolicy: without it, the L7 policy protects a service anyone can still connect to by other routes.

The alternative without a mesh

Without a mesh, L7 authorization is done by the application itself. bookings-api can validate in its own code who is calling it:

# The application validates a service token injected as a Secret
env:
  - name: EXPECTED_SERVICE_TOKEN
    valueFrom:
      secretKeyRef:
        name: internal-tokens
        key: web-store

It works and for six services it is perfectly reasonable, but it has real drawbacks: it must be implemented in every service, the tokens have to be rotated by hand, and if somebody reads the Secret (RBAC, 08-01) they can impersonate the service. The mesh's mTLS solves all three automatically. It is exactly the kind of cost that grows with the number of services and that, at some point, justifies the mesh.

  1. Protecting the perimeter

Everything above protects the inside. Now, the front door.

Minimum exposure: why there is no NodePort in production

In 04-02 we looked at the Service types. Let us go over their security implication:

Type Exposure In production?
ClusterIP Only inside the cluster Yes: the default for everything
NodePort A port (30000-32767) on every node No
LoadBalancer A provider load balancer Only for the Ingress
Ingress HTTP/HTTPS by name and path Yes: the only entry point

Why NodePort is no good in production:

  • It opens the port on every node, including those not running the pod.
  • It bypasses the Ingress, and with it the TLS (04-05), the WAF and the rate limiting.
  • The port is in a high, unconventional range, hard to protect with a conventional firewall.
  • It has no name: anyone who reaches the node's IP reaches the service.

And remember from 08-02 that hostPort is worse still, because there is not even a Service in front.

The Rutas Norte rule: a single entry point, the Ingress load balancer. Everything else is ClusterIP.

# Audit: is there any exposed service that should not be?
kubectl get svc -A -o json | jq -r '
  .items[] | select(.spec.type == "NodePort" or .spec.type == "LoadBalancer")
  | "\(.spec.type)\t\(.metadata.namespace)/\(.metadata.name)\t" +
    ([.spec.ports[] | "\(.port)->\(.nodePort // "-")"] | join(","))'
LoadBalancer	ingress-nginx/ingress-nginx-controller	80->31023,443->30987

A single LoadBalancer, the Ingress one. Any other line in that output is a question that has to be answered.

WAF and rate limiting

A web application firewall (WAF) inspects the HTTP requests and blocks known attack patterns: SQL injection, cross-site scripting, directory traversal. With ingress-nginx you can enable ModSecurity with the OWASP rule set:

# k8s/base/ingress/waf-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"
  modsecurity-snippet: |
    SecRuleEngine On
    SecRequestBodyAccess On
    SecRequestBodyLimit 5242880
    SecAuditEngine RelevantOnly
    SecAuditLogParts ABIJDEFHZ
    # Paranoia level 1: few false positives. Raise it carefully.
    SecAction "id:900000,phase:1,nolog,pass,t:none,setvar:tx.paranoia_level=1"

A serious warning about WAFs: a badly tuned WAF blocks legitimate requests. If web-store can no longer create bookings because a WAF rule considers a name with an apostrophe suspicious, you have created an incident. The correct procedure is identical to the PSA one in 08-03:

  1. SecRuleEngine DetectionOnly: log but do not block.
  2. Analyse the false positives over weeks of real traffic.
  3. Tune the problematic rules.
  4. Only then, SecRuleEngine On.

And the rate limiting, via an Ingress annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rutas-norte
  namespace: rutas-norte-pro
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production   # from 04-05
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "20"
    nginx.ingress.kubernetes.io/limit-connections: "10"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"
    # Maximum body size: avoids exhausting memory
    nginx.ingress.kubernetes.io/proxy-body-size: "2m"
    # Timeouts: avoids connections that hang
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    # Security headers
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Content-Type-Options: nosniff";
      more_set_headers "X-Frame-Options: DENY";
      more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
      more_set_headers "Permissions-Policy: geolocation=(), microphone=(), camera=()";
      more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains";
spec:
  ingressClassName: nginx
  tls:
    - hosts: [www.rutasnorte.example]
      secretName: rutasnorte-tls
  rules:
    - host: www.rutasnorte.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-store
                port: { number: 80 }
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: bookings-api
                port: { number: 8080 }

The headers, one by one:

Header What it mitigates
X-Content-Type-Options: nosniff The browser guessing the content type and executing something as a script
X-Frame-Options: DENY The page being embedded in another site's frame (clickjacking)
Referrer-Policy Internal URLs leaking when navigating to external sites
Permissions-Policy Unrequested access to camera, microphone or location
Strict-Transport-Security A client reconnecting over unencrypted HTTP

One detail about HSTS: max-age=31536000 is 365 days, and browsers remember it. If one day your TLS stops working, customers who have already visited the site will not be able to fall back to HTTP. That is what you want from a security standpoint, but you have to be aware of it before enabling it.

Denial of service protection

Attack level Where it is mitigated
Volumetric (saturating the bandwidth) Outside the cluster: the cloud provider or a CDN
Connection exhaustion The load balancer and limit-connections
Expensive requests limit-rps, and in the application: pagination and timeouts
Internal resource exhaustion resources.limits (03-04) and autoscaling (module 9)

An important point that gets forgotten: autoscaling is not a defence against denial of service, it is a way of paying for it. If bookings-api scales to 50 replicas under an attack, you have turned an outage into an invoice. The HPA's upper limit (09-01) is also a security and cost control.

  1. Control plane security

Everything above protects the workloads. The control plane is the highest-value target: whoever controls it controls the whole cluster.

Access to the apiserver

Control How
Do not expose it to the internet A private endpoint, or an allow list of IPs
Strong authentication OIDC with a second factor, never shared certificates
Disable anonymous access --anonymous-auth=false
Audit log An audit policy (08-06)
Rate limiting --max-requests-inflight, API priority and fairness
# Check whether the apiserver is publicly exposed
kubectl cluster-info
Kubernetes control plane is running at https://10.0.4.11:6443

A private IP is a good sign. If you saw a public IP with no source restriction, that is a first-order security finding.

On managed Kubernetes (10-06), all three major providers allow restricting access to the endpoint. It is one of the first things to configure on a new cluster.

etcd

etcd stores all the cluster's state, Secrets included. As we said in 03-02, if etcd is not encrypted, the Secrets are in the clear on disk.

Encryption at rest:

# /etc/kubernetes/encryption/config.yaml (on the control plane nodes)
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      # An external KMS: the master key lives outside the cluster. Preferable.
      - kms:
          apiVersion: v2
          name: kms-rutasnorte
          endpoint: unix:///var/run/kmsplugin/socket.sock
          cachesize: 1000
      - identity: {}     # unencrypted: needed in order to read the old data

An important detail: the order of the providers matters. The first is used to write; all of them are used to read. Having identity: {} at the end allows reading the objects written before encryption was enabled. And after enabling it, they have to be rewritten:

kubectl get secrets -A -o json | kubectl replace -f -

Without that step, the old Secrets stay in the clear in etcd forever.

Other etcd controls:

Control Why
Mutual TLS between etcd and the apiserver Nobody else should be able to talk to etcd
Restricted access to port 2379 Only from the control plane
Encrypted backups An unencrypted etcd backup is a backup of every Secret
Custody of the backups With the same protection as the customer database

That point about the backups is easy to overlook and very serious. In 05-06 we talked about Velero and personal data; an etcd backup also contains every credential in the cluster. It must be encrypted, stored with restricted and logged access, and its restore tested periodically.

The kubelet

Every node runs a kubelet with an API on port 10250. Unprotected, it allows listing pods, reading logs and executing commands in any container on that node.

Setting Correct value What it prevents
--anonymous-auth false Requests with no credentials
--authorization-mode Webhook Any valid identity being able to do everything
--read-only-port 0 Port 10255, unauthenticated, exposes pod metadata
--protect-kernel-defaults true The kubelet modifying kernel parameters

A check:

# It must answer 401: the kubelet API does not accept anonymous requests
kubectl get --raw /api/v1/nodes/node-pro-01/proxy/pods -v6 2>&1 | grep -i "response"

And a NetworkPolicy or firewall rule stopping pods from reaching the nodes' port 10250 is a very worthwhile defence, because it closes a known escalation route.

Remember from 08-01 that the get nodes/proxy permission is critical precisely because of this: it allows talking to the kubelet and, through it, executing in any pod on the node.

Isolating the control plane nodes

The control plane nodes must not run application workloads. Kubernetes achieves this with a taint (06-05):

kubectl describe node control-node-01 | grep -A2 Taints
Taints:  node-role.kubernetes.io/control-plane:NoSchedule

That taint stops ordinary pods being scheduled there. The reason is direct: a pod on a control plane node is one flaw away from the cluster's certificates and the etcd database.

And you have to verify that nobody has tolerated it out of convenience:

kubectl get pods -A -o json | jq -r '
  .items[]
  | select(.spec.tolerations[]? |
      .key == "node-role.kubernetes.io/control-plane" or
      (.operator == "Exists" and (.key // "") == ""))
  | "\(.metadata.namespace)/\(.metadata.name)"'
kube-system/kube-proxy-hd8k2
kube-system/cilium-p2m4x

Only system components that legitimately run on every node. If a rutas-norte-pro pod appeared here, you would have to investigate why that toleration was added. Beware the universal toleration (operator: Exists with no key): it tolerates every taint, the control plane one included, and it often gets added without anyone noticing.

  1. Traffic logging and visibility

We come back to the second limit we pointed out in 04-06: NetworkPolicies log nothing.

What that means exactly

When a NetworkPolicy denies a connection, the packet is dropped in the kernel. There is no Kubernetes event, no log, no metric. From the source pod, the connection simply times out.

The consequences are two, and both are bad:

For operating: debugging "why does my service not connect" is painful. It could be DNS, a policy, a misconfigured Service or the application. Nothing tells you which.

For security: if somebody compromises a pod and starts probing the internal network, they will generate dozens of denied connections and nobody will notice. The clearest signal of lateral movement is precisely the one that is invisible.

Hubble: flow observability with Cilium

Hubble, Cilium's observability component, solves this by taking advantage of the fact that Cilium already inspects every packet in eBPF.

helm upgrade cilium cilium/cilium --namespace kube-system --reuse-values \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,httpV2}"

Watching flows in real time:

hubble observe --namespace rutas-norte-pro --follow
TIMESTAMP             SOURCE                          DESTINATION                    TYPE          VERDICT
Aug  6 09:14:22.481   rutas-norte-pro/web-store-x2    rutas-norte-pro/bookings-api-a1:8080   to-endpoint   FORWARDED
Aug  6 09:14:22.503   rutas-norte-pro/bookings-api-a1 rutas-norte-pro/bookings-postgres-0:5432  to-endpoint   FORWARDED
Aug  6 09:14:23.117   rutas-norte-pro/bookings-api-a1 rutas-norte-pro/redis-cache-0:6379      to-endpoint   FORWARDED

And what really matters, the denials:

hubble observe --namespace rutas-norte-pro --verdict DROPPED --last 100
TIMESTAMP             SOURCE                            DESTINATION                              TYPE          VERDICT   REASON
Aug  6 03:47:11.229   rutas-norte-pro/web-store-x2k4    rutas-norte-pro/bookings-postgres-0:5432 to-endpoint   DROPPED   Policy denied
Aug  6 03:47:11.884   rutas-norte-pro/web-store-x2k4    rutas-norte-pro/redis-cache-0:6379       to-endpoint   DROPPED   Policy denied
Aug  6 03:47:12.401   rutas-norte-pro/web-store-x2k4    203.0.113.44:443                         to-stack     DROPPED   Policy denied
Aug  6 03:47:13.055   rutas-norte-pro/web-store-x2k4    198.51.100.7:22                          to-stack     DROPPED   Policy denied

Read it carefully, because that output tells a complete story. web-store —the most exposed component— has tried, within five seconds, to: connect to the customer database, connect to the cache, reach the internet over HTTPS and reach the internet over SSH. web-store does none of that in normal operation. It is exactly the pattern of somebody exploring what they can reach from a compromised pod.

The policies did their job: everything DROPPED. But without Hubble, this would have gone completely unnoticed.

Questions you can answer after an incident

Question Query
What did this pod try to reach? hubble observe --pod <name> --last 1000
Did anybody reach the internet from production? hubble observe --namespace rutas-norte-pro --to-fqdn "*"
When did the anomalous attempts start? hubble observe --verdict DROPPED --since 24h
Who talked to the database? hubble observe --to-pod rutas-norte-pro/bookings-postgres-0
Which HTTP paths were requested? hubble observe --protocol http --http-path "/admin*"

Turning it into alerts

Hubble exposes Prometheus metrics, so it integrates directly with module 7:

# k8s/base/monitoring/network-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: network-security-alerts
  namespace: monitoring
  labels:
    app.kubernetes.io/part-of: rutas-norte
spec:
  groups:
    - name: network-security
      rules:
        - alert: AnomalousDeniedConnections
          expr: |
            sum by (source_pod, source_namespace) (
              rate(hubble_drop_total{
                source_namespace="rutas-norte-pro",
                reason="POLICY_DENIED"
              }[5m])
            ) > 0.5
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: >-
              Pod {{ $labels.source_pod }} is generating denied connections in a
              sustained way
            description: >-
              More than 0.5 denied connections per second for 5 minutes.
              It may be a deployment with a misconfigured policy, or a compromised
              pod probing the network. Investigate with:
              hubble observe --pod {{ $labels.source_namespace }}/{{ $labels.source_pod }} --verdict DROPPED
            runbook_url: https://wiki.rutasnorte.example/runbooks/denied-connections

        - alert: UnexpectedEgressFromProduction
          expr: |
            sum by (source_pod) (
              rate(hubble_flows_processed_total{
                source_namespace="rutas-norte-pro",
                destination_namespace="",
                verdict="FORWARDED"
              }[5m])
            ) > 0
          for: 2m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: >-
              Outbound internet traffic from {{ $labels.source_pod }}
            description: >-
              Only bookings-api (payment gateway) and notifications-worker (SMTP)
              should reach the internet. Any other source is a possible route for
              customer data exfiltration. Escalate immediately.

That second alert is one of the most valuable on the whole platform: it warns of a possible personal data exfiltration in progress. The route to Alertmanager and its routing are those of module 7; the only new thing is the metric source.

Alternatives without Cilium

Tool What it brings
The provider's flow logs (VPC Flow Logs) Flows at the virtual network level, with no pod identity
Calico Enterprise Flow logging with Kubernetes context (a commercial product)
Falco with network rules Unexpected connections observed from the system calls (08-06)
The egress proxy log External destinations requested, with the domain name

For Rutas Norte, with no Cilium today, the practical combination is: the egress proxy log for external traffic (which is the critical one) and Falco for anomalous connections, which we will see in the next lesson.

  1. Zero trust applied to Rutas Norte

Zero trust is a model that can be summed up in one phrase: never trust, always verify. Nothing is considered trusted because of where it sits on the network.

The principles and their translation

Principle At Rutas Norte
Nothing is trusted because of its location A pod in rutas-norte-pro has no privileged access just for being there
Verify identity on every request ServiceAccounts (03-06) and RBAC (08-01); mTLS if there were a mesh
Least privilege Tight RBAC and per-component network policies
Assume compromise Microsegmentation and egress control limit the damage
Encrypt in transit TLS on the Ingress, WireGuard in the CNI, TLS in PostgreSQL
Log and verify everything Hubble, apiserver auditing (08-06), Falco

The idea of assuming compromise is the one that changes the design most. It is not about "how do I stop them getting in", but "when they get in, what can they do and how quickly do I find out?".

The complete defence table

Layer Mechanism Attack it mitigates State
Perimeter WAF + OWASP CRS SQL injection, XSS, directory traversal To be rolled out
Perimeter Rate limiting Abuse, brute force, application-level denial of service To be rolled out
Perimeter Ingress only, no NodePort Accidental exposure of internal services Done
Transport TLS with cert-manager Eavesdropping on client-platform traffic Done (04-05)
Transport Security headers Clickjacking, downgrade to HTTP, URL leakage To be rolled out
Transport WireGuard in the CNI Eavesdropping on traffic between nodes Recommended
Transport TLS in PostgreSQL Eavesdropping on personal data in transit Recommended
Segmentation deny-all in all three environments Lateral movement after a compromise Done (04-06) + Kyverno (08-03)
Segmentation Per-component policies The reach of lateral movement Done (04-06)
Segmentation Egress control with ipBlock Exfiltration of customer data Priority
Identity Dedicated ServiceAccounts Process impersonation before the API Done (03-06)
Identity mTLS with a mesh Service-to-service impersonation Deferred, with criteria
Authorization Least-privilege RBAC Abuse of legitimate credentials Done (08-01)
Authorization L7 policies Access to unauthorised API paths Deferred
Hardening Restrictive securityContext Container escape to the node Done (08-02)
Hardening PSA + Kyverno Somebody deploying something insecure Done (08-03)
Control plane Private apiserver, encrypted etcd Total compromise of the cluster To be reviewed
Control plane Authenticated kubelet, no read-only port Execution in pods via the kubelet To be reviewed
Visibility Hubble or the proxy log Blindness to internal probing and exfiltration Priority
Visibility Alerts on denials Late detection of an incident To be rolled out

The three priorities that come out of this table, in order: egress control, traffic visibility and control plane review. All three attack the same scenario: somebody is already inside.

Common Mistakes and Tips

Writing only Ingress policies. If policyTypes does not include Egress, egress is open and with it the exfiltration route. It is the most serious and most frequent mistake in this lesson.

Enabling Egress and forgetting DNS. Everything stops working in a very confusing way: names do not resolve and it looks like a Service problem. The rule towards kube-dns on port 53, UDP and TCP, is the first one to write.

Using ipBlock: 0.0.0.0/0 with no except. It includes the cluster's internal network, so the policy is far more permissive than it looks.

Trusting that an external provider's ipBlock ranges do not change. They change. Schedule a monthly verification or use a proxy with a domain list.

Believing that the ambassador container isolates the network. It shares the network namespace with the rest of the pod. It is an architecture pattern, not a security boundary.

Writing policies per namespace instead of per component. It reproduces the flat network inside the namespace. The cost of doing it properly is five more policies.

Leaving rutas-norte-dev and rutas-norte-pre without policies. The cluster network is flat between namespaces, and a policy only tested in production is tested at the worst moment.

Adopting a service mesh without needing one. It is critical infrastructure: badly operated, it worsens security. Define objective criteria for revisiting the decision.

Enabling a WAF straight into blocking mode. It will block legitimate requests and you will have an incident. DetectionOnly first, weeks of analysis, and then blocking.

Using NodePort in production "because it is quicker to configure". It bypasses the TLS, the WAF and the rate limiting, and it opens the port on every node.

Forgetting that autoscaling does not defend against denial of service. It turns an outage into an invoice. Put an upper limit on the HPA.

Encrypting etcd and not rewriting the existing Secrets. The old ones stay in the clear. kubectl get secrets -A -o json | kubectl replace -f -.

Not encrypting the etcd backups. An etcd backup contains every Secret in the cluster.

Universal tolerations. An operator: Exists with no key tolerates every taint, the control plane one included, and it usually gets added without anyone noticing.

Golden tip: draw the diagram of what talks to what and check that the policies reproduce it exactly, not one arrow more. Then ask the key question: if they compromise this pod, where can they reach and how can they get the data out? If the answer to the second part is not "nowhere", there is outstanding work on egress.

Exercises

Exercise 1: close the egress of the components that do not need it

Four of Rutas Norte's six components do not need to reach the internet: web-store, bookings-postgres, redis-cache and occupancy-reports. Write a single NetworkPolicy that allows them only what is indispensable inside the cluster and closes their internet egress completely.

Bear in mind that:

  • All of them need DNS.
  • bookings-postgres and occupancy-reports must be able to talk to each other (the CronJob queries the database).
  • web-store must be able to reach bookings-api.
  • All of them must be scrapeable by Prometheus.

Include the command that would verify that egress is effectively closed.

Exercise 2: decide on the service mesh

Rutas Norte's management proposes deploying Istio "because it is what the big companies use". Prepare a one-page technical response that includes:

  1. What concrete problems it would solve on the current platform.
  2. What it would cost, quantified.
  3. What cheaper alternatives cover part of the benefit.
  4. Objective, measurable criteria for revisiting the decision.
  5. A clear recommendation.

Exercise 3: investigate an incident using the network flows

At 03:47 the AnomalousDeniedConnections alert fires. You query Hubble and get:

TIMESTAMP             SOURCE                              DESTINATION                                TYPE         VERDICT   REASON
Aug  6 03:47:09.112   rutas-norte-pro/notif-worker-k9x2   rutas-norte-pro/bookings-api-a1:8080       to-endpoint  FORWARDED
Aug  6 03:47:11.229   rutas-norte-pro/notif-worker-k9x2   rutas-norte-pro/bookings-postgres-0:5432   to-endpoint  FORWARDED
Aug  6 03:47:11.884   rutas-norte-pro/notif-worker-k9x2   rutas-norte-pro/redis-cache-0:6379         to-endpoint  DROPPED   Policy denied
Aug  6 03:47:12.401   rutas-norte-pro/notif-worker-k9x2   192.0.2.55:443                             to-stack     DROPPED   Policy denied
Aug  6 03:47:12.902   rutas-norte-pro/notif-worker-k9x2   192.0.2.55:8443                            to-stack     DROPPED   Policy denied
Aug  6 03:47:13.455   rutas-norte-pro/notif-worker-k9x2   192.0.2.55:53                              to-stack     DROPPED   Policy denied
Aug  6 03:47:20.001   rutas-norte-pro/notif-worker-k9x2   rutas-norte-pro/bookings-postgres-0:5432   to-endpoint  FORWARDED
  1. What is happening? Justify your reading line by line.
  2. Which defence layers worked and which did not?
  3. List the immediate actions, in priority order.
  4. What would have happened without egress control? And without Hubble?

Solutions

Solution 1

# k8s/base/network/closed-egress.yaml
# Egress policy for the components that must NOT reach the internet.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: no-internet-egress
  namespace: rutas-norte-pro
  labels:
    app.kubernetes.io/part-of: rutas-norte
  annotations:
    security.rutasnorte.example/justification: >-
      web-store, bookings-postgres, redis-cache and occupancy-reports have no
      need whatsoever to reach the internet. Closing their egress removes the
      personal data exfiltration route for four of the six components.
spec:
  podSelector:
    matchExpressions:
      - key: app
        operator: In
        values:
          - web-store
          - bookings-postgres
          - redis-cache
          - occupancy-reports
  policyTypes: [Egress]
  egress:
    # 1. DNS: indispensable for all of them
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

    # 2. web-store -> bookings-api
    #    (this rule's podSelector filters the destination, not the source;
    #     the other components simply do not use this permission)
    - to:
        - podSelector:
            matchLabels:
              app: bookings-api
      ports:
        - { protocol: TCP, port: 8080 }

    # 3. occupancy-reports -> bookings-postgres
    - to:
        - podSelector:
            matchLabels:
              app: bookings-postgres
      ports:
        - { protocol: TCP, port: 5432 }

    # THERE IS NO ipBlock rule: internet egress is closed.

And the ingress policy so that Prometheus can scrape metrics (it is Ingress, complementary):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus
  namespace: rutas-norte-pro
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
      ports:
        - { protocol: TCP, port: 9090 }

An important observation about the first policy: by using a podSelector with matchExpressions that groups four components, they all share the same egress rules. That means redis-cache technically could connect to bookings-api:8080, even though it does not.

It is a conscious trade-off between concision and precision. The strictly correct version would be one policy per component, and that is what we would recommend in production, following principle 2 of section 2. This grouped version is acceptable because the exercise's stated objective —closing internet egress— is met all the same, and none of the extra internal permissions granted gives access to personal data. Documenting that reasoning is as important as the YAML.

Verification:

#!/usr/bin/env bash
NS=rutas-norte-pro
for target in deploy/web-store statefulset/bookings-postgres statefulset/redis-cache; do
  echo -n "$target -> internet: "
  if kubectl exec -n "$NS" "$target" -- \
       timeout 5 sh -c 'wget -q -O- --timeout=4 https://external-example.example' \
       >/dev/null 2>&1; then
    echo "REACHES <-- FAIL"
  else
    echo "blocked (correct)"
  fi
  echo -n "$target -> DNS: "
  kubectl exec -n "$NS" "$target" -- \
    timeout 5 nslookup bookings-api >/dev/null 2>&1 \
    && echo "resolves (correct)" || echo "DOES NOT RESOLVE <-- FAIL"
done
deploy/web-store -> internet: blocked (correct)
deploy/web-store -> DNS: resolves (correct)
statefulset/bookings-postgres -> internet: blocked (correct)
statefulset/bookings-postgres -> DNS: resolves (correct)
statefulset/redis-cache -> internet: blocked (correct)
statefulset/redis-cache -> DNS: resolves (correct)

Checking DNS is as important as checking the block: an egress policy that blocks the internet and DNS breaks the platform without it being obvious why.

Solution 2

Technical report: adopting a service mesh on the Rutas Norte platform

1. What problems it would solve

Real problem Does it solve it? Is it urgent today?
Unencrypted internal traffic, personal data included Yes, automatic mTLS Yes, it is a real risk
A service cannot cryptographically verify who is calling it Yes, SPIFFE identity Moderate: there are 6 services, all internal and controlled
There is no per-HTTP-path authorization Yes, AuthorizationPolicy Low: the call graph is almost linear
Retries and timeouts duplicated in every service Yes Low: it is already implemented
Lack of inter-service latency observability Partially Low: module 7 already covers the essentials
Percentage-based canary deployments Yes Low: today it is done by replicas (11-04)

Of six problems, one is urgent (encryption) and five are desirable improvements with no urgency.

2. Quantified cost

Item Estimate
Sidecar memory 6 components × ~15 pods × 80 MiB ≈ 1.2 GiB of additional memory
Sidecar CPU ~0.05 cores per pod ≈ 0.75 cores
Added latency 2-5 ms per hop; on the web-storebookings-apipostgres path, up to 10 ms per request
Initial rollout 3-4 weeks of one platform engineer
Ongoing operation ~1 day a month: upgrades, certificates, diagnosis
Team training 2 weeks for 4 people
Risk of incidents Medium-high in the first months: it is the most common cause of "the service returns a 503 and I do not know why"

With Istio in ambient mode or with Linkerd, the memory and the latency would drop substantially. The operational complexity and the risk of the first months would not.

3. Cheaper alternatives

Alternative Covers Cost
WireGuard in the CNI Encryption of all traffic between nodes One configuration option; hours of work
TLS on bookings-postgres Encryption of the stretch with personal data, even within the same node 1-2 days
Strict NetworkPolicies (already done) L3/L4 segmentation Done
Service tokens in the application Basic authentication between services 2-3 days per service
Hubble or the egress proxy log Flow visibility 1 week

The first two together cover 80% of the security benefit for less than 5% of the cost.

4. Objective criteria for revisiting the decision

The evaluation will be reopened when any of these is met:

  • The platform exceeds 15 services that communicate with each other.
  • More than three teams deploy to the cluster without daily coordination.
  • An explicit regulatory or contractual requirement for inter-service mTLS appears (for example, a certification from a corporate customer).
  • Percentage-based canary deployments are needed that cannot be approximated with replicas.
  • Routing between several clusters is needed (11-05).
  • The platform team reaches 5 people or more, with capacity to dedicate one to the mesh.

5. Recommendation

Do not adopt a service mesh now. Instead, in this order:

  1. Enable WireGuard in the CNI (this week).
  2. Configure TLS on bookings-postgres (this quarter).
  3. Roll out flow visibility (this quarter).
  4. Review the decision every six months against the criteria above.

If it is adopted in the future, the first candidate would be Linkerd for its benefit-to-complexity ratio, or Istio in ambient mode if by then advanced routing is needed.

Final note: this recommendation is an architecture assessment based on the current size and topology. It must be validated by the organisation's security professional, especially regarding the encryption of traffic carrying personal data, where the compliance officer has the final word on what level of protection is required.

Solution 3

1. What is happening, line by line

Time Flow Reading
03:47:09 notif-workerbookings-api:8080 FORWARDED Anomalous. The worker reads from the database, it does not call the API. The policy allows it (probably an over-broad permission), but it is not its normal behaviour
03:47:11 notif-workerpostgres:5432 FORWARDED Legitimate on its own: the worker queries pending bookings
03:47:11 notif-workerredis-cache:6379 DROPPED Anomalous. The worker does not use the cache. This is probing
03:47:12 notif-worker192.0.2.55:443 DROPPED Very serious. An attempt to reach an external IP that is neither the payment gateway nor the SMTP server
03:47:12 notif-worker192.0.2.55:8443 DROPPED Retry on another port: it is looking for a way out
03:47:13 notif-worker192.0.2.55:53 DROPPED Retry on the DNS port: a classic technique for getting through permissive firewalls
03:47:20 notif-workerpostgres:5432 FORWARDED The most worrying part: after every egress attempt failed, it goes back to the database

The complete sequence —four seconds, three different ports towards the same external IP, probing of internal services, and back to the database— corresponds to no legitimate behaviour of notifications-worker. The signature is that of a compromised process looking for a way to get information out, and the last line suggests it is still accessing data.

An important timing detail: it is 03:47. The occupancy-reports CronJob runs at 03:00. Any connection will have to be ruled out.

2. Layers that worked and that did not

Layer Result
Egress control (Egress) Worked. The three attempts to reach the internet were blocked. This is the layer that prevented the data breach
Microsegmentation Partly worked. It blocked Redis, but it allowed reaching bookings-api
Visibility (Hubble) Worked. Without it, this incident would be invisible
Alerting Worked. The alert fired and that is why we are investigating
Preventing the initial compromise Failed. Something allowed code to run in the worker
The worker's network policy Too permissive. It should not be able to reach bookings-api
L7 authorization Absent. With access to bookings-api:8080, it can call any path
Internal encryption Absent. The traffic to postgres goes in the clear

3. Immediate actions, by priority

# Action Why
1 Isolate the pod without destroying it: change its app label so that the Services and the policies stop applying to it, leaving it running It cuts the access while preserving the memory and the state for analysis. Deleting it destroys the evidence
2 Apply a policy that denies it everything, the database included It stops the ongoing access to personal data
3 Rotate the bookings-postgres credentials and review the database's pg_stat and logs The process had legitimate access; we need to know what it queried
4 Query the apiserver audit log (08-06): did it use the ServiceAccount? did it read Secrets? Determine the real scope
5 Review the last 72 hours of flows for that pod and for every pod in the namespace Establish when it started and whether other pods are affected
6 Notify the compliance officer There are indications of unauthorised access to personal data. The notification deadlines start running from the moment the fact is known
7 Scan the worker's image for known vulnerabilities (08-06) Find the entry vector
8 Harden the worker's policy: remove the access to bookings-api Fix the over-broad permission
9 Review the rest of the policies looking for equally broad permissions The same mistake may be elsewhere

The commands for the first two steps:

# 1. Isolate: remove the label that makes it a target of policies and Services
kubectl label pod -n rutas-norte-pro notif-worker-k9x2 app-
kubectl label pod -n rutas-norte-pro notif-worker-k9x2 status=quarantined

# 2. Total denial for the quarantined pod
kubectl apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
  namespace: rutas-norte-pro
spec:
  podSelector:
    matchLabels:
      status: quarantined
  policyTypes: [Ingress, Egress]
  # No rules: absolutely everything is denied
YAML

By removing the app label, the ReplicaSet notices it is a replica short and creates a fresh, clean pod: the service recovers on its own while the compromised pod is isolated for analysis. It is a technique worth rehearsing before you need it.

4. The counterfactual scenarios

Without egress control: the three connections to 192.0.2.55 would have been FORWARDED. The process had legitimate access to bookings-postgres, that is, to the customer table with names, ID numbers, phone numbers and emails. This would have been a consummated personal data breach, with an obligation to notify the supervisory authority and the individuals affected. The egress policy —those lines of YAML almost nobody writes— is the only thing that separated a contained incident from a data breach.

Without Hubble: there would have been no alert. Denied connections are silent: with no flow logging, the compromised pod would have carried on querying the database indefinitely, and sooner or later it would probably have found a way out (a changed configuration, a new permission, a badly reviewed policy). The incident would have been discovered weeks or months later, or never.

This exercise sums up the whole lesson: prevention failed —it always fails somewhere in the end—, containment worked, and detection allowed a reaction. All three are necessary, and the one most often neglected is the third.

Conclusion

We have built Rutas Norte's complete network strategy, layer by layer:

  • Microsegmentation starts from denying by default in all three environments, with policies per component rather than per namespace, and with a periodic review of which conversations are still needed.
  • Egress traffic control is the layer that decides whether a compromise turns into a data breach. Four of Rutas Norte's six components do not need to reach the internet at all, and closing their egress is the highest-value, lowest-cost measure in the whole lesson.
  • NetworkPolicies work with IPs, not domain names. The ways out are the egress gateway, the proxy with an allow list of domains (which also logs the destinations) or Cilium's toFQDNs policies.
  • Inside the cluster, past the Ingress TLS, everything travels in the clear. The options are TLS in the application (a priority for bookings-postgres), CNI-level encryption with WireGuard (cheap and effective between nodes) or mTLS with a service mesh.
  • A service mesh gives cryptographic per-workload identity, automatic mTLS and L7 authorization, in exchange for operational complexity, resources and latency. Rutas Norte does not need one today, and we have set objective criteria for revisiting that decision. Istio (sidecar or ambient), Linkerd and Cilium Service Mesh cover different profiles.
  • At the perimeter: a single entry point (no NodePort), a WAF adopted gradually, rate limiting and security headers.
  • The control plane deserves its own review: an apiserver that is not exposed, etcd encrypted and with the Secrets rewritten, encrypted backups, an authenticated kubelet with no read-only port, and control nodes isolated with taints.
  • What NetworkPolicy lacks is logging. Hubble, or an egress proxy's log, turns internal probing and exfiltration attempts into something visible and alertable.
  • Zero trust boils down to assuming compromise: not "how do I stop them getting in", but "when they get in, what can they do and how quickly do I find out".

With this, Rutas Norte's network is segmented, with controlled egress and visibility of what happens. But notice something: everything we have done in this module protects the cluster at runtime. We take for granted that the images we run are the ones we think they are.

And that assumption is fragile. registry.rutasnorte.example/bookings-api:2.7.1 is a tag, and a tag can be reassigned: the image pulled yesterday may not be today's. The base image it was built on may contain software nobody has reviewed. The dependencies installed during the build come from public repositories. And nothing, absolutely nothing, stops somebody with registry access today from pushing a modified image under the same name and having the cluster run it without complaint, with all our securityContext settings perfectly applied to a container that does something different from what we think.

The next lesson, 08-05, Image Security, walks the whole supply chain: where an image can be poisoned, how to build minimal images, why the digest is the only genuinely reproducible reference, how to sign with Cosign and —most importantly— how to make the cluster reject any unsigned image, with the admission policy that guarantees it.

Kubernetes Course

Module 1: Introduction to Kubernetes

Module 2: Core Kubernetes Components

Module 3: Configuration and Secret Management

Module 4: Networking in Kubernetes

Module 5: Storage in Kubernetes

Module 6: Advanced Kubernetes Concepts

Module 7: Monitoring and Logging

Module 8: Kubernetes Security

Module 9: Scaling and Performance

Module 10: Kubernetes Ecosystem and Tooling

Module 11: Case Studies and Real-World Applications

Module 12: Preparing for Kubernetes Certification

© Copyright 2026. All rights reserved