We closed module 3 by pointing out that the Rutas Norte network is completely flat: any pod in rutas-norte-pro can open a connection to bookings-postgres:5432, and no client on the internet can reach www.rutasnorte.example yet. Before fitting doors (the network policies of 04-06) and windows (the Ingress of 04-04), you need to understand how the house is built. This is the lesson that explains what really sits behind that 10.244.x.x you see in kubectl get pods -o wide, who creates that network interface, why a Service IP appears on no network card of any machine, and how a packet leaving bookings-api ends up arriving at bookings-postgres. It is the most "plumbing" lesson of the course, and also the one that stops everything else from looking like magic.

1. The Kubernetes network model and its four rules
2. Why that model simplifies everything else
3. The four communication planes
4. CNI: the interface that plugs the pod into the network
5. The plugins: Flannel, Calico, Cilium and Weave Net
6. Overlay (VXLAN), native routing (BGP) and eBPF
7. The cluster's address ranges
8. How a ClusterIP is really implemented: kube-proxy
9. The full journey of a packet in Rutas Norte
10. Hands-on verification in minikube

Kubernetes does not implement the network. What it does is impose a contract: it defines a model that any network implementation must satisfy, and lets others satisfy it however they like. The contract has four rules.

The fourth rule looks redundant, but it is the one that explicitly outlaws the classic Docker model. In a docker run -p 8080:80, the container believes it is 172.17.0.4:80 while the rest of the world sees it as 192.168.1.10:8080. That mismatch breaks any protocol that advertises its own address: clustered databases, discovery systems, service registries. Kubernetes eliminates it at the root.

Notice what the contract does not say: nothing about how packets are routed, whether there is encapsulation, which IP range is used or whether there is a firewall. All of that is left to the implementation.

The model boils down to a single phrase: "one IP per pod, flat network". Its consequences are enormous.

• Ports stop being a scarce resource. In a port-mapping world, deploying three replicas of bookings-api on the same node forces you to assign 3000, 3001 and 3002, and to have somebody keep track. With one IP per pod, all three listen on 3000. The manifest does not change depending on where the pod lands.
• Applications do not need to know they are in Kubernetes. An application that works on a virtual machine works in a pod: it opens its port and connects to host:port. This is what made it possible to migrate so much software without rewriting it.
• Discovery becomes trivial. Since there is no NAT, the IP that the EndpointSlice of 02-05 points to is directly connectable. Without that guarantee, every Service would have to translate addresses and ports per node.
• Pod migration is transparent. A pod dies and another is born with a different IP, but the mechanics of connecting are identical.

The price is that by default there is no isolation whatsoever. Rule 2 says "every pod can talk to every pod", and that is exactly what it means: the most insignificant pod in rutas-norte-pro can try to connect to bookings-postgres. The NetworkPolicies of 04-06 exist precisely to trim back that universal permission.

When somebody says "the Kubernetes network" they are usually mixing up four different problems, with different solutions.

A very common diagnostic mistake is attacking the wrong plane. If bookings-api cannot reach bookings-postgres by pod IP, the problem belongs to the CNI. If it reaches it by pod IP but not by the Service name, the problem belongs to kube-proxy or to the DNS of 04-03. Separating the planes saves hours.

CNI (Container Network Interface) is a CNCF specification, very short and deliberately boring: it defines how a container runtime asks an external program to connect or disconnect a container from a network. It is not a product: it is a contract between two parties.

The call chain when a pod is created is this:

The points worth fixing in your mind:

• The plugin is invoked by the kubelet through the runtime (containerd, in our cluster). The apiserver never touches the network.
• The plugin is installed as a binary in /opt/cni/bin and configured with a JSON file in /etc/cni/net.d. That is why plugins are almost always deployed as a DaemonSet (06-02): one pod per node that copies the binary and the configuration file at startup and then keeps its agent running.
• The operations are ADD, DEL, CHECK and VERSION. When the pod is deleted, DEL is invoked and the IP returns to the pool.
• IPAM (IP Address Management) is a subcomponent: it decides which specific IP is assigned within the range allotted to the node.

What the plugin does in a typical ADD, translated into Linux commands you already know:

1. It creates a veth pair (two virtual interfaces joined like a cable).
2. It leaves one end in the host's network namespace (with a name such as veth3a7f2c1@if3) and moves the other inside the pod's namespace, where it is called eth0.
3. It asks IPAM for an IP and assigns it to eth0.
4. It adds the pod's default route, pointing at the bridge or at the node's gateway.
5. On the host, it attaches its end to the bridge (cni0, docker0, cbr0) or creates the corresponding route.
6. It returns the assigned IP to the runtime, which ends up in the pod's status.podIP.

If the CNI fails or is not installed, you will see the classic symptom:

Pods stuck forever in ContainerCreating with sandbox errors are nearly always a node networking problem, not a problem with your manifest.

Choosing a plugin is one of the longest-lasting decisions in a cluster: changing it with production traffic running is a delicate operation. These are the four most widely used historically.

The practical consequence for Rutas Norte, and it deserves underlining: if the cluster uses Flannel, the NetworkPolicies you write in 04-06 will be applied without error and will do absolutely nothing. The object is created, kubectl get networkpolicy lists it, and the traffic keeps flowing. It is the most dangerous silent failure in this part of the course, which is why we will verify it explicitly.

The managed clusters of 10-06 come with their own plugin (Amazon VPC CNI, Azure CNI, the GKE one), which usually assigns pods IPs from the real cloud network. That makes a pod directly routable from other machines in the VPC, at the cost of consuming addresses from the corporate space.

The problem they all solve is the same: pod 10.244.1.5 is on node A and wants to talk to 10.244.2.4, which is on node B. The physical network between nodes knows nothing about 10.244.0.0/16: if you hand it that packet as it is, it drops it.

A virtual network is built on top of the physical one. The source node puts the entire original packet inside a UDP packet addressed to the real IP of the destination node (port 8472 in VXLAN), which decapsulates it and delivers it to the pod.

• Decisive advantage: it works over any network, without touching routers or asking anything of the networking team. That is why it is the default mode in so many installations.
• Cost: the header adds about 50 bytes, which reduces the effective MTU (typically from 1500 to 1450) and causes fragmentation if something ignores it; there is extra work encapsulating and decapsulating every packet; and the traffic is opaque to the firewalls and analysers of the physical network, which only see UDP between nodes.

Nothing is encapsulated. Each node advertises over BGP to the routers of the physical network: "the range 10.244.2.0/24 is reachable through me". The physical network learns the routes and delivers pod packets directly.

• Advantage: no overhead, full MTU, pod traffic is visible and filterable with the existing network tools.
• Requirement: the network must allow it. In your own data centre with routers that speak BGP, it is ideal. In many clouds or in closed corporate networks it is not possible and you fall back to encapsulation.

eBPF makes it possible to load verified programs inside the Linux kernel and hook them to specific points of the network stack. Cilium uses it to make routing, load-balancing and policy decisions before the packet travels through the whole iptables machinery.

• It can replace kube-proxy entirely, removing the iptables chains we will see in section 8.
• It scales far better: where iptables degrades with thousands of Services, eBPF uses constant-cost hash tables.
• It enables HTTP-aware policies (by path and method) and workload identity.
• In exchange, it demands a reasonably recent kernel and raises the bar of knowledge needed to debug it.

Three address spaces coexist in a cluster, and they must never be confused.

The global cluster CIDR is shared out: the kube-controller-manager, with --allocate-node-cidrs, assigns each node a slice (a /24 by default, that is 254 usable pods per node) and writes it into spec.podCIDR of the Node object. The CNI plugin's IPAM hands out IPs within that slice. This guarantees that two nodes never assign the same IP.

Where they are configured:

Three warnings that save incidents:

1. They cannot be changed on the fly. Changing the serviceCIDR of a live cluster is not a supported operation; you plan it before creating the cluster.
2. They must not overlap with the corporate network. If your company uses 10.96.0.0/16 for servers, the Rutas Norte pods will not be able to talk to them: the node will believe those IPs are cluster Services.
3. The /24 per node is a real limit. A large node with 300 pods runs out of addresses even if it has CPU and memory to spare.

Here is the revelation of the lesson. When in 02-05 you created the bookings-postgres Service with clusterIP: 10.96.140.22, that address was not assigned to any interface on any machine. It is a fiction. It does not answer ping. There is no process listening on it.

What does exist are destination-rewriting rules installed on every node by kube-proxy, a DaemonSet in the kube-system namespace that watches the API and, every time a Service or an EndpointSlice changes, updates those rules.

For each Service, kube-proxy creates a KUBE-SERVICES chain that matches on destination IP and port, jumps to a KUBE-SVC-XXXX chain belonging to the Service, and from there spreads across the KUBE-SEP-YYYY chains, one per endpoint, each with its own DNAT rule.

Three important facts come out of this:

• Balancing is random per connection, not per request. A long TCP connection (a PostgreSQL session, a WebSocket) stays stuck to one pod until it closes. That is why an API that multiplexes requests over a few connections can spread load badly.
• The probability is cascaded: with 3 endpoints, the rules carry 1/3, then 1/2, then the remainder. That comes out uniform.
• The cost is linear: iptables evaluates rules in order. With tens of thousands of Services, every update forces the reprogramming of huge tables and synchronisation latencies appear. That is the reason IPVS and eBPF exist.

IPVS is the Linux kernel's L4 load balancer, with hash tables instead of lists of rules.

In IPVS mode a kube-ipvs0 interface does appear on the node with the Service IPs assigned to it, but it is a dummy interface: it exists so the kernel accepts the packets, not to answer them.

The mental rule to take away: a Service is a rule, not a machine. If a ClusterIP does not answer, do not look for a crashed process: look for empty endpoints (a selector that does not match, as you saw in 02-05) or a kube-proxy that is not synchronising.

Let us follow a real connection: pod bookings-api-7d9f5c8b4-x2kp9 (10.244.1.5, node 1) opens a connection to bookings-postgres:5432 (ClusterIP 10.96.140.22), whose only endpoint is 10.244.2.4 on node 2. The cluster uses VXLAN.

The details that matter:

• Name resolution happens first and is independent of everything else (04-03).
• The DNAT happens on the source node, before routing. Node 2 never sees the Service IP.
• bookings-postgres sees 10.244.1.5 as the source, the real IP of the client pod, not the node's. This is rule 2 (no source NAT) and it is what allows the NetworkPolicies of 04-06 to identify the sender by its labels. Careful: this changes when traffic comes in from outside with externalTrafficPolicy: Cluster (04-02).
• conntrack is indispensable: the kernel remembers the translation so it can undo it in the reply. A full conntrack table causes connections that are lost at random under load, a classic symptom during the Rutas Norte bank-holiday peaks.

Everything above can be touched with your own hands on the rutas-norte profile.

It is a very handy trick: the error message reveals the exact range to you.

The pod IPs are in 10.244.0.x (the podCIDR of the only node) and the Service one is in 10.96.x.x. Different ranges, different natures.

Not a trace of 10.96.140.22. Now the rules:

The comment includes <namespace>/<service>: it is the quickest way to locate the rules of a specific Service.

The nicolaka/netshoot image ships dig, curl, nc, tcpdump, traceroute and ipvsadm. It is the Swiss army knife for debugging networks in Kubernetes.

Inside the pod:

If step 2 works and step 3 does not, the CNI is fine and the problem is the Service or DNS. If step 2 does not work either, it is the CNI or a network policy. That fork covers 80% of network diagnosis in Kubernetes.

• Confusing the three ranges. Seeing 10.96.x.x and hunting for the node that pod is on is a classic. Rule of thumb: 10.244.x.x is a pod (it exists), 10.96.x.x is a Service (it is a rule), 192.168.x.x is a node.
• Pinging a ClusterIP. It does not answer and that is normal: kube-proxy's rules only match TCP/UDP on the declared port, not ICMP. A failed ping to a Service proves nothing. Use nc -zv or curl.
• Overlapping the podCIDR or the serviceCIDR with the corporate network. It causes baffling failures when talking to external services, such as the payment gateway. Check it before creating the cluster, not afterwards.
• Ignoring the MTU with VXLAN. Connections that open fine but hang when transferring large blocks (a PostgreSQL restore, a 2 MB JSON response) usually mean a badly tuned MTU. Diagnosis: ping -M do -s 1400 <ip>.
• Choosing Flannel and writing NetworkPolicies. They apply without error and filter nothing. Before trusting a policy, verify that the CNI implements it with a real test.
• Blaming the CNI for everything. Before that, check the planes in order: localhost inside the pod, pod IP, ClusterIP, DNS name. The first one that fails points at the guilty layer.
• Tip: keep an alias for the debugging pod. alias kshoot='kubectl run netshoot-$RANDOM --rm -it --restart=Never --image=nicolaka/netshoot -- bash'. With --rm you leave no rubbish behind in the cluster.
• Tip: in production, watch the conntrack metrics (nf_conntrack_count against nf_conntrack_max). During the Rutas Norte bank-holiday peaks, a full table shows up as intermittent connection errors with no pod down at all.

On the rutas-norte profile, document: the node's podCIDR, the serviceCIDR, the node's IP, and the IPs of the pods and Services in rutas-norte-pro. Classify each address into its range and explain which ones exist physically and which do not.

Pick the redis-cache Service. Prove with three checks that its IP exists on no interface, that iptables rules do exist for it, and that you can nevertheless connect to it. Explain who does the translation and on which node it happens.

A colleague says that bookings-api "cannot see the database". Design a sequence of checks, from the lowest layer to the highest, that lets you decide whether the problem is the CNI, the Service, DNS or the application itself. Run it with netshoot and note what output you would expect in each failure case.

Exercise 1

Expected classification:

Exercise 2

The translation (DNAT) is done by the kernel of the node where the client pod runs, using the rules kube-proxy programmed from the EndpointSlice. The packet leaves the node already carrying the real IP of the destination pod.

Exercise 3

Sequence from the lowest to the highest level, with the conclusion for each failure:

Every rung that works rules out a whole layer. The first one that fails names the culprit.

You now know what lies beneath the Kubernetes network. The model rests on four mandatory rules —one IP per pod, pod-to-pod communication without NAT, node agents reaching their own pods, and the IP a pod sees for itself being the one everybody else sees— and it is that uniformity that lets applications remain unaware they are in a cluster and lets ports stop being a resource to administer. You can tell apart the four planes of communication and you know which one to diagnose first when something fails.

You have seen that Kubernetes delegates the implementation to a CNI plugin, which the kubelet invokes through the runtime when creating each pod, and which does a very specific job: create a veth pair, ask IPAM for an IP within the node's podCIDR and programme the routes. You know the real differences between Flannel, Calico, Cilium and Weave Net, and in particular that Flannel does not implement NetworkPolicy, a detail that will shape lesson 04-06. You understand the trade-off between encapsulating with VXLAN (works anywhere, costs 50 bytes and some MTU) and routing natively with BGP (free, but requires cooperation from the physical network), and why eBPF is displacing both approaches in demanding clusters.

And above all, you have dismantled the most useful fiction in Kubernetes: the ClusterIP does not exist. It is on no interface, it does not answer ping and there is no process behind it. There are only iptables rules or IPVS entries that kube-proxy keeps synchronised with the EndpointSlices, doing DNAT on the source node and undoing it in the reply thanks to conntrack. You have followed a packet from bookings-api to bookings-postgres from start to finish and you have verified it on your own minikube.

With the plumbing understood, it is time to go up a floor. In 04-02 we will see that ClusterIP is only one of the four Service types, that each one is built on top of the previous one, and how NodePort, LoadBalancer and ExternalName start opening the platform to the outside: the first step towards letting a customer who wants to buy a coach ticket finally reach web-store.