The previous lesson deliberately left a loose end: ExternalName works because CoreDNS returns a CNAME, headless services work because CoreDNS publishes one A record per pod, and bookings-api finds bookings-postgres because somebody resolves that name. We have spent three modules writing bookings-postgres:5432 in configuration without ever explaining who translates that into an IP address. This lesson opens the box: what CoreDNS is, how it is configured, what names it generates for each kind of object, what is really inside a pod's /etc/resolv.conf, why the ndots: 5 setting can multiply DNS queries by five when bookings-api calls the payment gateway, and how to diagnose the three DNS failures you will run into again and again.

1. Why discovery is done over DNS
2. CoreDNS: what it is and where it lives
3. The Corefile: the cluster DNS configuration
4. The naming scheme and the short forms
5. Records for Services, headless Services and pods
6. SRV records and named ports
7. The pod's /etc/resolv.conf and the effect of ndots: 5
8. dnsPolicy and dnsConfig
9. The environment variables inherited from Docker
10. Diagnosis: the three typical failures
11. Scale: NodeLocal DNSCache and CoreDNS replicas

The alternative would be configuring IP addresses, and it does not work: pod IPs are ephemeral (02-05). A rollout, an eviction or a node restart changes them. We could pin the Service IP, which is stable for as long as the Service exists, but that does not hold up either:

With DNS, bookings-api has had bookings-postgres written in its ConfigMap since 03-01, and that name is valid in all three environments, in a brand-new cluster and after recreating the Service: it is the only stable reference that exists in Kubernetes. On top of that, DNS is universal: it needs no library, no SDK, and no awareness that you are in Kubernetes.

CoreDNS is a DNS server written in Go, modular through plugins, and it has been the default DNS of Kubernetes since version 1.13 (it replaced kube-dns). It runs as a normal Deployment in the kube-system namespace, exposed by a Service called kube-dns (the name was kept for compatibility).

Three observations that clear up a lot of doubts:

• CoreDNS is a workload like any other. It can crash, run out of resources, be evicted. When CoreDNS suffers, the whole platform looks broken at once, and that global symptom is what should make you look here.
• The IP 10.96.0.10 is the tenth address of the serviceCIDR by convention, and it is the one the kubelet injects into every pod.
• Port 9153 exposes Prometheus metrics (07-03): query rate, latency, errors. It is one of the first things worth monitoring.

Notice the apparent circularity: pods reach CoreDNS through a ClusterIP, which needs kube-proxy but does not need DNS. There is no chicken-and-egg problem because the IP 10.96.0.10 is injected literally into the resolv.conf.

CoreDNS is configured with a file called Corefile, which lives in a ConfigMap and is mounted into the pod. You can read it as it is:

Read this as a chain of plugins evaluated in order for every query arriving on port 53.

A practical Rutas Norte case: the office has an internal DNS at 10.20.30.5 that resolves *.interno.rutasnorte.example, where the legacy billing database lives. A specific block is added with kubectl edit configmap coredns -n kube-system:

The reload plugin picks it up on its own. Careful: this is cluster-wide configuration; a syntax error leaves CoreDNS in CrashLoopBackOff and the whole platform with it. Check the log after every change.

The canonical name of a Service always has this structure:

The short forms work thanks to the search list in resolv.conf (section 7). From a pod in rutas-norte-pro:

That last case deserves emphasis. A pod in rutas-norte-dev can resolve and connect to bookings-postgres.rutas-norte-pro. Separation by namespace is organisational, not a security boundary. Only the NetworkPolicies of 04-06 really prevent it.

Practical rule for Rutas Norte:

• Within the same environment, use the short name: bookings-postgres, redis-cache. The same ConfigMap serves dev, pre and pro unchanged.
• To cross namespaces, always use the full form with .svc.cluster.local. It is explicit and, as we will see, faster.

A single answer: the virtual IP. Balancing between pods is done afterwards by kube-proxy, as you saw in 04-01. DNS plays no part in the distribution.

Three A records, one per ready pod (ready: true in the EndpointSlice). Here DNS does take part in discovery, and it is the client that chooses. The loadbalance plugin shuffles the order in each answer so that clients which only look at the first record still spread out somewhat.

A pod that is not ready disappears from this list, unless the Service has publishNotReadyAddresses: true (needed so that database replicas can discover each other during startup, something we will see in 06-01).

With pods insecure in the Corefile, every pod has a name derived from its IP with the dots replaced by hyphens: 10-244-0-14.rutas-norte-pro.pod.cluster.local resolves to 10.244.0.14. Note pod instead of svc. Its use is very limited: nobody types that by hand. Its real importance arrives with StatefulSets, where each pod gets a stable, readable name inside a headless Service:

That is the network identity that makes a clustered database possible, and it arrives in 06-01.

SRV records publish the port as well as the address. They are generated for every named port of a Service, with the format _<port>._<protocol>.<service>.<ns>.svc.cluster.local.

Bringing back the multi-port bookings-api from 04-02, with ports http (3000) and metrics (9090):

The format is priority weight port target. A client that knows how to read SRV discovers the port without having it configured. In a headless Service, an SRV returns one line per pod, with each one's port:

Few applications query SRV; most configure the port explicitly. But it is the mechanism used by several operators and discovery libraries, and it is the concrete reason we insist on always naming ports.

The kubelet writes this file inside every container:

Three lines, three decisions:

• nameserver 10.96.0.10: the ClusterIP of kube-dns. It is what makes the short forms resolve.
• search: the suffixes the resolver will try. The first is the pod's own namespace: that is why a bare bookings-postgres works inside rutas-norte-pro.
• options ndots:5: here is the trap.

ndots:5 tells the resolver: "if the name you are asked for has fewer than 5 dots, try it first with each search suffix, and only at the end as it is".

Count the dots in pagos.proveedorexterno.example: two. Fewer than five. So the resolver does this:

Four queries, three useless. And worse: each one is made for A and for AAAA, so in practice that is 8 queries where 2 would have done. The first three also travel to CoreDNS, which queries the API and returns NXDOMAIN.

Now transfer this to Rutas Norte on an August bank holiday: bookings-api calls the payment gateway on every purchase. With 2,000 purchases per minute and no resolution cache in the application, that is 16,000 queries per minute where there should be 4,000. CoreDNS saturates, resolution latency rises, and the whole cluster starts to crawl for a reason nobody connects with DNS.

A name ending in a dot is an absolute FQDN: the resolver adds no suffixes to it.

One query. Done.

Warning: if you lower ndots to 1, short names such as redis-cache (zero dots) still work, but bookings-postgres.rutas-norte-pro (one dot) would stop resolving through search and would need the FQDN. Lowering it to 2 is a common and safe compromise.

dnsPolicy is a field in the pod spec that decides which resolv.conf it receives.

The ClusterFirstWithHostNet case is a classic trap: a pod with hostNetwork: true and dnsPolicy: ClusterFirst (the default value) receives the node's resolv.conf and therefore cannot resolve any Service. If you turn on host networking, you have to change the policy too.

dnsConfig lets you tune the resulting file:

With dnsPolicy: None you must supply at least one nameserver in dnsConfig, or the pod will not start.

Kubernetes injects into every container environment variables with the address and port of every Service that existed in its namespace at the moment the pod was created. It is a leftover from Docker container linking.

They look convenient and must never be used. Four reasons:

1. Only the Services created BEFORE the pod exist. If you deploy bookings-api and then redis-cache, the variable is not there. And creation order is not something you control reliably.
2. They are frozen when the pod starts. If the ClusterIP changes (the Service is recreated), the variable keeps the old value and the pod points at a dead IP until it restarts.
3. They pollute the environment. With 50 Services in the namespace that is hundreds of variables per container. There are documented cases of startup failures from exceeding the environment size.
4. They break portability. That BOOKINGS_POSTGRES_SERVICE_HOST means nothing outside Kubernetes.

Always use the DNS name, which does not depend on ordering, resolves fresh and works the same anywhere. The variable noise is removed with enableServiceLinks: false in the pod spec, recommended in every Rutas Norte Deployment.

The working pod, as in 04-01:

Basic checks inside:

Cause: CoreDNS is down, saturated or unreachable. It is a global failure, and its signature is that every component fails at once.

The usual suspects: CoreDNS pods in CrashLoopBackOff from a badly edited Corefile; the kube-dns Service with no endpoints; CoreDNS OOMKilled for lack of memory under load; or —and this will be very relevant in 04-06— an egress NetworkPolicy blocking port 53. That last case is so frequent that we will devote a whole section to it.

Cause: the pod's resolv.conf is not the cluster one. It is nearly always dnsPolicy: Default set by accident, or hostNetwork: true without ClusterFirstWithHostNet. Check it:

A second possible cause: the name is wrong. A typo, a Service in another namespace, or a Service that simply does not exist.

DNS is not the problem. You have already proved that resolution works; the failure is in the Service (empty endpoints, 02-05), in the CNI or in a network policy. This is the most misread case: DNS gets the blame because "it does not connect by name", when nslookup has already answered correctly.

The slowness variant can indeed be DNS: if resolution takes seconds, look at ndots and at external domains without a trailing dot (section 7), or look for saturation in the CoreDNS metrics.

In high-load clusters DNS becomes a bottleneck. Two standard measures:

CoreDNS replicas. The default is 2 (1 in minikube). It scales with the size of the cluster, watching its metrics before it starts to hurt: kubectl scale deployment coredns -n kube-system --replicas=3. There is a cluster-proportional-autoscaler, which adjusts the replicas according to the number of nodes and cores, and it is what the managed distributions use.

NodeLocal DNSCache. A DaemonSet that puts a DNS cache on every node. Pods query that local cache (on a link-local IP, typically 169.254.20.10) instead of crossing the network to CoreDNS.

For Rutas Norte, with its bank-holiday peaks, it is the measure to consider alongside the autoscaling of 09-01.

• Blaming DNS when nslookup answers correctly. If it resolves, DNS has done its job. Carry on to the Service and its endpoints.
• Editing the Corefile without checking the log. A syntax error takes CoreDNS down and the whole cluster with it. After every kubectl edit configmap coredns, look at kubectl logs -n kube-system -l k8s-app=kube-dns.
• External domains without a trailing dot. They multiply queries by four (or eight). In Rutas Norte, the payment gateway URL must carry the dot.
• Using hostNetwork: true without dnsPolicy: ClusterFirstWithHostNet. The pod loses the ability to resolve Services and the error is baffling.
• Using the *_SERVICE_HOST variables. They depend on creation order and get frozen. Use DNS names and switch them off with enableServiceLinks: false.
• Caching resolutions forever in the application. Some runtimes (the JVM with its historical default configuration) cache with no expiry and keep using a dead IP. Tune the resolver TTL of your language.
• Assuming the namespace isolates. A pod in dev resolves and connects to bookings-postgres.rutas-norte-pro. Only 04-06 cuts that off.
• Tip: whenever in doubt, run dig +search and not plain dig. Without +search the suffixes are not applied and you will be testing something different from what your application does.
• Tip: watch coredns_dns_request_duration_seconds and the NXDOMAIN rate. A spike in NXDOMAIN usually means somebody has deployed an application with external names lacking a trailing dot.

From an ephemeral pod in rutas-norte-pro, resolve bookings-postgres in its four forms (short, with namespace, with .svc, full) and check that they all give the same IP. Then resolve bookings-postgres.rutas-norte-dev and explain what the result proves.

From an ephemeral pod, count how many queries resolving pagos.proveedorexterno.example triggers without a trailing dot and with one. Work out the saving for 2,000 purchases per minute and propose two ways of fixing it in the bookings-api manifest.

Three test pods are deployed: A with hostNetwork: true and the default dnsPolicy; B a normal one but querying a non-existent Service; C a normal one, with the right Service but whose Deployment is scaled to zero. Predict the symptom of each and verify it.

Exercise 1

The first four are the same IP: the short forms are completed with the search list. The fifth proves that the development Service resolves from production, with a different IP: namespaces do not isolate the network. And if you also try nc -zv bookings-postgres.rutas-norte-dev 5432, it will connect.

Exercise 2

Without the dot: 4 attempts (3 NXDOMAIN + 1 hit), times 2 counting AAAA = 8 queries. With the dot: 2. A 75% saving.

At 2,000 purchases per minute: 16,000 queries/min against 4,000. Twelve thousand useless queries every minute hitting CoreDNS.

Two fixes in the manifest:

A is surgical and affects nothing else; B is useful when you cannot touch the URLs, but remember that it breaks short forms containing a dot.

Exercise 3

Case C is the most valuable lesson of the three: resolving is not connecting.

You now know who translates bookings-postgres into an IP address. CoreDNS is an ordinary Deployment in kube-system, exposed by the kube-dns Service at 10.96.0.10, configured by a Corefile whose kubernetes plugin watches Services and EndpointSlices in order to answer the cluster.local domain, and whose forward plugin sends everything else to the node's resolver. The fact that it is a run-of-the-mill workload has an operational consequence: it can crash, saturate or run out of memory, and when that happens the whole platform looks broken at once.

You have a firm grip on the naming scheme <service>.<namespace>.svc.cluster.local and you know that the short forms work thanks to the search list in resolv.conf, not by magic. You know the records each object generates: one A with the ClusterIP for a normal Service, one A per ready pod for a headless one, pod names derived from the IP, and SRV records that publish the port of named ports. And you understand what resolution does not do: it does not distribute load in a normal Service (that is kube-proxy) and it does not guarantee connectivity.

The section that will save you the most is ndots: 5. An external domain with fewer than five dots triggers four resolution attempts, eight queries counting AAAA, and during the Rutas Norte bank-holiday peaks that is thousands of useless queries per minute against CoreDNS. The trailing dot in pagos.proveedorexterno.example. brings it down to one. You know how to adjust dnsPolicy and dnsConfig, you have seen why hostNetwork: true without ClusterFirstWithHostNet leaves the pod with no access to cluster DNS, and why the *_SERVICE_HOST variables inherited from Docker must never be used —they depend on creation order and get frozen— to the point of switching them off with enableServiceLinks: false. You also have the three typical failures with their unmistakable signatures, and on the horizon NodeLocal DNSCache and CoreDNS scaling for when the load bites.

With the network understood (04-01), the Service types mastered (04-02) and discovery solved, there are no excuses left: it is time to open the platform to the world. In 04-04 we will deploy an Ingress controller and finally publish www.rutasnorte.example against web-store and api.rutasnorte.example against bookings-api, with a single entry point, routed by domain and by path. Rutas Norte's customers will be able, for the first time in the course, to buy a ticket.