In the previous four lessons we have written, without justifying it, addresses like http://catalog-service:3001, inventory-service:50051 or amqp://rabbitmq:5672. They have worked as stable names in the examples, but in a real system each of those names hides a problem: behind "catalog-service" there will be two replicas on a Tuesday morning and eight on Black Friday, each in a container with an IP that is assigned at startup and destroyed at death. Which of those IPs does Orders call when it needs GET /products?ids=? Who knows which ones are alive right now? How is the work spread among them? This lesson answers those three questions.

We will see why addresses cannot be hard-wired (the fallacies from 01-02 come back), service registration (who records that an instance exists: the instance itself or a third party), client-side versus server-side discovery, the classic tools (Consul, Eureka) with a real registration snippet including a health check, the Kubernetes-native DNS-based discovery that TechCorp will adopt, the algorithms and layers of load balancing and where it happens in each model, health checks (liveness versus readiness, with both endpoints in Express) as part of a service's contract, and TechCorp's decision: Kubernetes DNS + stable *-service names as configuration. Actually deploying to Kubernetes belongs to 05-02, the service mesh to 05-05 and resilience (what to do when the chosen instance fails) to 06-03.

1. The problem: ephemeral instances with dynamic IPs
2. Service registration: who records that an instance exists
3. Client-side discovery
4. Server-side discovery
5. Comparison of the two models
6. Tools: Consul and Eureka
7. Kubernetes-native discovery: Service, Endpoints and DNS
8. Load balancing: algorithms and layers
9. Where balancing happens in each model
10. Health checks: liveness versus readiness
11. TechCorp's decision

In the monolith, the web called techcorp-shop:3000 and that name pointed to a fixed machine with a fixed IP. With microservices on containers, the situation changes in three ways:

• Many instances per service. Catalog has N identical replicas and N changes with the load (the ×20 peaks from 01-05) or with a deployment (05-04).
• Ephemeral addresses. Each container gets an IP from the internal range at startup (10.244.3.17) and that IP disappears with it. Restarting means changing address.
• Partial failures. A replica may be alive but without a connection to its database, or starting up and not yet able to serve, or about to shut down.

Hard-wiring addresses (CATALOG_URL=http://10.244.3.17:3001) fails because of the fallacies of distributed computing from 01-02: "topology doesn't change" (it changes every minute), "the network is reliable" (it is not), "there is one administrator" (Kubernetes reschedules containers without asking). And a list of IPs in configuration goes stale before it is even deployed.

We need three things: a place where which instances exist and where is recorded (service registry), a mechanism for a caller to find a healthy instance at call time (discovery), and a way to spread calls across the available instances (load balancing).

The registry is a database of "service → list of instances (IP, port, status)". There are two ways to fill it:

Consul and Eureka are traditionally used in self-registration mode (with clients in the service) or with a local agent; Kubernetes is pure third-party: the orchestrator itself knows which containers exist because it created them.

In client-side discovery, it is the caller who queries the registry, gets the list of healthy instances, picks one (applying a balancing algorithm) and calls it directly.

Consequences: the client needs a library that talks to the registry and balances (Netflix Ribbon with Eureka was the classic example); that library must exist for every language in the system; balancing can be very smart (the client knows its own latencies); and there is no extra network hop. But every service carries infrastructure logic inside, and updating that logic means redeploying every service.

In server-side discovery, the caller always calls one stable address (a load balancer, a virtual DNS name, a proxy) and it is that intermediate piece that queries the registry and forwards the request to a healthy instance.

Consequences: the client is trivial (a fetch to a fixed name, like the ones from 03-01); the discovery logic lives in the platform and is updated without touching services; it works the same for Node.js, Go or Java. In exchange, there is one more piece on the path (which must be highly available) and, in some cases, an extra network hop.

The industry trend is clear: with orchestrators and service meshes, server-side discovery has won because it takes infrastructure out of the business code. It is consistent with "smart endpoints, dumb pipes" from 02-01: the service thinks about orders, not about IPs.

Although TechCorp will use the Kubernetes mechanism, it is worth knowing the two classic tools because they appear in many existing systems and because they explain where the concepts come from.

• Netflix Eureka. A REST registry created by Netflix for AWS. Instances register with POST /eureka/apps/{app} and send a heartbeat every 30 s with PUT; if three heartbeats are missed, it is marked as down. Clients download the full registry and cache it. Very tied to the Spring Cloud (Java) ecosystem; clients exist for Node.js, but choosing it today outside Java is rare.
• HashiCorp Consul. Registry + key-value store + health checks + DNS. Each node runs a local agent; services register against their agent (via HTTP API or file), and Consul runs the health checks (HTTP, TCP, script) from the agent. It exposes the registry via API and via DNS (catalog-service.service.consul resolves to the healthy instances), which is already a form of server-side discovery. It is the usual option outside Kubernetes (virtual machines, Nomad) and remains relevant.

Snippet with the Node.js consul client registering catalog-service with an HTTP health check:

Notice everything this code adds to the Catalog service that has nothing to do with products: instance ids, IPs, heartbeats, deregistrations. That is the cost of self-registration and client-side discovery. With Kubernetes, none of this exists in the service's code.

Kubernetes ships with registration, discovery and balancing built in. Three objects are enough to understand it (the full deployment YAML belongs to 05-02; here only the mechanism):

• Pod. The unit that runs a container (one Catalog replica). It has an ephemeral cluster IP (10.244.3.17).
• Service. An object with a stable name (catalog-service) and a stable virtual IP (ClusterIP, e.g. 10.96.12.5) that selects pods by labels (app: catalog-service). It lives on even as pods die and are born.
• Endpoints (or EndpointSlice in current versions). The list, maintained automatically by Kubernetes, of the IPs of the pods the Service selects and that are ready (readiness, section 10). It is the registry, filled by a third party (the orchestrator): third-party registration without writing a single line.

The flow when Orders does fetch('http://catalog-service:3001/products?ids=...'):

1. The Orders pod resolves the name catalog-service in CoreDNS, the cluster's internal DNS. The full name is catalog-service.<namespace>.svc.cluster.local, but from the same namespace the short name is enough.
2. CoreDNS returns the Service's ClusterIP, which never changes as long as the Service exists.
3. Orders opens a connection to that virtual IP and port 3001.
4. On each node, kube-proxy maintains network rules (iptables or IPVS) that intercept traffic to the ClusterIPs.
5. Those rules pick one of the pods from the Endpoints list and rewrite the destination to its real IP. If a pod stops being ready, it leaves Endpoints and stops receiving traffic within seconds.

What matters to the developer: the Orders code knows nothing about this. It calls a fixed name and port, exactly the CATALOG_URL from 03-01. Registration, health and balancing belong to the platform. It is server-side discovery and third-party registration, for free.

Snippet of the Catalog Service (just to see where the name and port come from; the Deployment that creates the pods belongs to 05-02):

A nuance for RabbitMQ and other long-lived connections: kube-proxy balancing happens when the TCP connection is established. An AMQP connection kept open for hours stays with the same pod; for HTTP with keep-alive something similar happens (all the requests on one connection go to the same pod). For HTTP it works well because connections get renewed; to truly spread per request you need layer 7 balancing (section 8), which is one of the things the service mesh from 05-05 provides.

Algorithms for spreading across N instances:

Layers at which balancing happens:

kube-proxy is layer 4: it spreads connections, not requests. The gateway from 03-04 is layer 7: it routes by path and header. A service mesh puts a layer 7 proxy (Envoy) next to every pod and provides per-request balancing, retries and metrics between internal services without touching the code: that is why it appears in 05-05 as the natural evolution.

For internal calls (Orders → Catalog) TechCorp starts with kube-proxy: enough for the current volume and at no cost. When it needs per-request balancing, declarative retries or per-route metrics between services, it will adopt Istio.

None of the above is any use if the registry does not know which instances are healthy. A container with a live process is not the same as a service able to serve. That is why there are two different checks, and confusing them causes incidents:

Example of both endpoints in Express (Catalog, with MongoDB and RabbitMQ as dependencies):

And its use in Catalog:

Response of GET /health/ready with MongoDB down:

With status code 503. Kubernetes (05-02) will call these endpoints periodically (livenessProbe, readinessProbe), Traefik already used them in the healthCheck from 03-04, and Consul in the check from section 6. That is why we say health checks are part of a service's contract: just as POST /orders responds 202, every TechCorp service responds on /health/live and /health/ready with the semantics of the table; the Platform team takes it for granted when configuring the platform. And two more nuances: /health/* does not require authentication (the infrastructure calls it) but is not exposed through the gateway; and it must not do heavy work (a SELECT 1 query, a ping), because it runs every few seconds for every replica.

• Discovery: Kubernetes-native. One Service per microservice with a stable name catalog-service, orders-service, payments-service, customers-service, notifications-service, inventory-service, bff-mobile, plus rabbitmq and the databases. Third-party registration (Kubernetes maintains the Endpoints), server-side discovery (DNS + kube-proxy). No service carries registration code.
• Balancing: the Service's (kube-proxy, layer 4) between internal services; layer 7 at the gateway/Ingress (Traefik) for external traffic; service mesh as the next step when needed (05-05).
• Stable names as configuration: each service receives the addresses of its dependencies through environment variables whose value is the Service's DNS name. In 04-03 we will see how they are managed (ConfigMaps, .env files); the contract from today is this:

• Health as a contract: /health/live and /health/ready in every service, with the semantics from section 10.
• In local development (Docker Compose, 05-01) the same names work because Compose also provides DNS by service name: the code does not change between the laptop and the cluster, only the value of the variables if need be.

• Hard-wiring IPs in configuration "only in development". They end up in production. Names from day one.
• Caching DNS resolution forever in the client. Node.js does not cache DNS by default, but some libraries or connection pools do keep the IP; if the Service is recreated with another ClusterIP, the client keeps pointing at the old one. Renew connections and do not pin IPs.
• Liveness that checks the database. The DB goes down → every pod "not alive" → Kubernetes restarts them in a loop → when the DB comes back, nobody is up. Liveness only looks at the process.
• Readiness that checks nothing. The pod enters balancing before connecting to RabbitMQ and fails the first requests of every deployment.
• Not switching to "not ready" on shutdown. The pod receives requests until the last millisecond and cuts them off halfway. SIGTERM → shuttingDown = true → wait a few seconds → close.
• Expensive health checks. A SELECT COUNT(*) FROM orders every 5 s per replica is made-up load. SELECT 1 or ping.
• Exposing /health through the gateway. It is internal information and one more attack surface. Inside the cluster only.
• Trusting that kube-proxy spreads per request. It spreads connections. With keep-alive, a very active client may always load the same pod. If the distribution matters, layer 7 (mesh).
• Adding a Consul client "just in case" when you are already on Kubernetes. It duplicates the registry and fills the service with infrastructure code.

Exercise 1. The Orders service has three dependencies: PostgreSQL (orders), RabbitMQ and the Catalog service (synchronous, via CATALOG_URL). Decide, justifying each case, which ones should be part of Orders' /health/ready and which should not, and write the resulting call to createHealthRoutes. Hint: think about what would happen to the Orders replicas if Catalog went down and it were in the check.

Exercise 2. Explain step by step what happens in Kubernetes when a Catalog replica starts failing its readinessProbe because MongoDB is slow, and what the difference would be if the liveness failed instead of the readiness. Indicate at which moment Orders stops receiving errors and why the Orders code does not change in either case.

Exercise 3. The Platform team proposes that the Notifications replicas use consistent hashing by customerId so that all the emails of the same customer are sent by the same replica and thus reuse one SMTP connection per customer. Notifications consumes events from RabbitMQ (03-02), it does not receive HTTP. Reason about whether hash balancing makes sense here, which real RabbitMQ (or Kubernetes) mechanism would or would not allow it, and which alternative you would recommend.

Solution 1.

• Orders' PostgreSQL: yes. Without its database, Orders cannot serve any request (neither create nor read orders). Better to take it out of balancing than to return 500 for everything.
• RabbitMQ: yes, with a nuance. Orders publishes through the outbox (02-05): POST /orders saves order and event in PostgreSQL and the relay publishes afterwards. Strictly, it could accept orders without RabbitMQ; but it also consumes orders.saga and without it the saga does not advance. TechCorp includes it: an accepted order that cannot advance is a worse experience than a brief 503. (Arguing the opposite is valid if accepting orders is prioritized.)
• Catalog: no. It is a dependency on another service. If Catalog went down and were in the readiness, all the Orders replicas would leave balancing at once and GET /orders/{id} (which does not need Catalog) would stop working: one failure would become two. Orders must stay ready and respond 503 DEPENDENCY_UNAVAILABLE only on the operations that need Catalog (03-01). Rule: readiness includes what this service needs to work at all, not other services (that is cascading failure, the subject of 06-03).

Solution 2.

Readiness failing: (1) the node's kubelet calls GET /health/ready every N seconds and receives 503 (the MongoDB check exceeds the 1 s timeout); (2) after failureThreshold consecutive failures (3 by default), Kubernetes marks the pod as not ready; (3) the Endpoints controller removes that pod's IP from the Endpoints of catalog-service; (4) kube-proxy updates the rules on every node and new connections go only to the ready replicas; (5) the pod stays alive, keeps trying; when MongoDB recovers and /health/ready returns 200 (successThreshold, 1 by default), it goes back into Endpoints. Orders stops seeing errors at step 4, a few seconds after the first failure, and before that it only saw errors on the fraction of requests kube-proxy sent to that pod.

Liveness failing: Kubernetes kills and restarts the container. If the problem is a slow MongoDB, the restart fixes nothing and, if the liveness checked MongoDB, every replica would restart in a loop: that is why liveness does not touch dependencies.

The Orders code does not change because it calls http://catalog-service:3001, a stable name and ClusterIP; who is behind it and whether it is ready is managed by the platform. All Orders sees is fewer errors.

Solution 3.

Hash balancing is a concept for incoming calls (HTTP/gRPC): the one deciding is a balancer that sees the request and picks the instance. Notifications does not receive calls: its replicas compete for messages from the notifications.orders queue (03-02), and RabbitMQ delivers each message to whichever replica has room in its prefetch; kube-proxy does not intervene (AMQP connections go from the replica to the broker, not the other way around) and it cannot know anything about customerId. Therefore, consistent hashing by customerId does not apply with the current topology.

Alternatives: (a) RabbitMQ has the consistent-hash exchange plugin, which spreads across several queues by hash of the routing key or of a header; you would have to create one queue per replica and lose the symmetry of "one queue per service"; complex and fragile when scaling. (b) Kafka would solve this naturally (partition by key), but changing broker for an SMTP optimization does not pay off. (c) The recommendation: do not do it. A pool of SMTP connections in each replica (reused across customers) gives the same saving without affinity, and the ordering between a customer's emails is better guaranteed by the event (which carries everything needed) than by the topology.

The problem was clear: ephemeral instances with dynamic IPs and a caller that needs to find a healthy one. The solution has three pieces: a registry (filled by the instance itself with heartbeats, or by a third party such as the orchestrator), a discovery mechanism (in the client, with a library, or on the server, with a stable address the platform resolves) and balancing (round robin, weighted, least connections, hash; at layer 4 per connection or at layer 7 per request). We have looked at Consul and Eureka as a reference and understood the Kubernetes-native mechanism: a Service with a stable name and IP, Endpoints as the automatic registry, CoreDNS to resolve catalog-service and kube-proxy to spread; and we have fixed the health checks (/health/live for "I am alive", /health/ready for "I can serve") as part of the contract of every TechCorp service. The decision: Kubernetes DNS, Service balancing, stable *-service names as configuration values (CATALOG_URL=http://catalog-service:3001), and zero infrastructure code in the services.

With this, the services know how to talk (REST, events, gRPC/GraphQL when the time comes), have a door (the gateway) and find each other. The last question of the module remains, and perhaps the one that prevents the most incidents in the long run: when the Catalog team changes the shape of GET /products or the Orders team adds a field to order.created, how do they do it without breaking anyone? Which changes are compatible and which are not, how a REST API, an event, a .proto or a GraphQL schema is versioned, how a version is retired and how the contract is agreed before writing code. That is the subject of the next lesson: API contracts and versioning.