The previous lesson left the Kilometre Zero context map with an arrow that ended in Kafka: delivery publishes on delivery.positions, Flink computes over that topic and writes to delivery.dashboard (05-04), and that is where the journey stopped. But Anna does not read Kafka. Anna has her order's tracking screen open on her phone, on the bus, with patchy coverage, and she wants to watch van-3 move across the map without tapping "refresh". Jordan Hall has the operators' dashboard open in a desktop browser with 140 couriers at once. The van app sends its position every 5 seconds over a mobile network that drops in every tunnel. And Montblanc Dairy wants a notice on its dashboard as soon as aged-cheese falls below the threshold in Girona. These are all last-mile problems: between the platform (services, Kafka, databases, all in a data centre with a reliable network) and the people and devices outside it, on poor networks, with thousands or tens of thousands of simultaneous connections, and with no way of running a Kafka consumer. This lesson develops what 02-01 only introduced: MQTT for devices and WebSockets for browsers, together with the alternatives (polling, long polling, Server-Sent Events), and joins them into an end-to-end pub/sub architecture with problems of its own: fan-out across instances, authenticating long-lived connections, ordering and duplicates on the client, pressure from slow clients and scaling the number of connections. The cloud (08-03) and edge processing (08-04) are left out.

1. What "real time" means here and the four Kilometre Zero cases
2. The last mile versus messaging between services
3. Last-mile techniques: polling, long polling, SSE and WebSockets
4. MQTT for devices
5. End-to-end pub/sub architecture
6. Fan-out across many instances, presence and subscriptions
7. Authentication, ordering, deduplication, backpressure and scaling
8. Code: from the MQTT broker to the browser
9. SSE as an alternative for stock alerts
10. When to use each technique
11. Common Mistakes and Tips
12. Exercises
13. Conclusion

"Real time" has a strict meaning in control engineering: a hard real-time system is one that guarantees a response arrives before a deadline, and if it does not, the result is a failure (the airbag, an engine controller). None of that applies here. In web systems, "real time" means latency as perceived by people: information arriving fast enough that whoever is looking at it feels they are seeing what is happening, without having to ask for it. That threshold lies between tenths of a second and a few seconds, and it varies from case to case. It is worth pinning down per case, because the design changes with it.

And a fifth, on the source side: the van app publishes its position every 5 seconds, from a mobile network that loses coverage, with a battery to look after, and it must keep working when coverage returns without losing what matters. Its requirements are the opposite of Anna's: few bytes, a single persistent connection, tolerance for long disconnections.

The messaging of Module 2 (RabbitMQ, Kafka) solves communication between services: long-lived processes, on the same network, with library clients that maintain connections, offsets and consumer groups. The last mile is different in every respect:

That is why Kafka is never exposed directly to a browser: the protocol does not allow it, nor does the security model, nor the number of connections. Between Kafka and the browser there has to be a component that speaks the client's protocol, maintains its connections and delivers to each one only what belongs to it. That component is the real-time server we will build as part of delivery.

They all start from one limitation: HTTP is request-response and the server cannot speak first. The four techniques are ways of getting around that limitation, each at a cost.

The client asks every n seconds: GET /api/v1/delivery/orders/P-2026-000124/position. It is trivial, cacheable at the gateway, compatible with everything, and it is exactly what the monolith did in symptom 4 of 01-06, with the well-known result: with 5,000 customers asking every 3 seconds, that is 1,700 requests per second, most of them to receive "nothing new", each with its TLS handshake if the connection is not reused (02-01) and its 300 bytes of headers. The average latency is half the interval.

The client asks, but the server does not answer until there is something new (or until a timeout of, say, 30 s), and the client asks again immediately. It reduces empty requests and brings latency down to almost zero, but it keeps one HTTP request open per client (a blocked worker on synchronous servers), suffers with proxies that cut idle connections, and every message still costs a full request. It was the dominant technique before WebSockets, and it remains valid as a fallback.

The client makes a GET with Accept: text/event-stream and the server leaves the response open indefinitely, writing events in a simple text format (event:, data:, id:, separated by a blank line). It is ordinary HTTP (it passes through proxies and gateways, and runs over multiplexed HTTP/2), the browser implements it with EventSource, which reconnects on its own and sends the Last-Event-ID header with the last id received so that the server can resume. Its limit: it is unidirectional (server → client; if the client wants to send, it uses ordinary HTTP requests) and text only.

What 02-01 introduced: an HTTP request with Upgrade: websocket which, if the server accepts (101 Switching Protocols), turns the TCP connection into a bidirectional, persistent, frame-based channel (binary or text), with no HTTP headers per message (2 to 14 bytes of overhead per frame). It is worth understanding the four parts of the protocol that affect the design:

• Handshake: the client sends Sec-WebSocket-Key; the server answers with a Sec-WebSocket-Accept derived from it. It is the only moment when there are HTTP headers, and therefore the only one at which Kong can apply its plugins (06-05): the JWT is verified at the handshake, not per message.
• Frames: each message travels in one or more frames with an opcode (text, binary, close, ping, pong). Client frames are masked (XOR with a random key) for a security reason to do with old proxies, not for encryption; encryption is TLS (wss://).
• Ping/pong: control frames that either side sends to check the other is still there. Without them, a phone that loses coverage leaves an "open" connection on the server for the minutes it takes TCP to notice (02-01). The Kilometre Zero server pings every 20 s and closes if there is no pong within 10 s.
• Reconnection: the protocol does not define it. When the connection drops, it is the client that reconnects, with exponential backoff and jitter (the same ones as in 07-04, for the same reason: 5,000 clients reconnecting at once after a deployment is a stampede), and it is the client that has to tell the server "this is the last thing I saw" to recover what was missed. All of that is application code.

Lesson 02-01 introduced MQTT as a pub/sub protocol for resource-constrained devices on poor networks. Here we design it for the Kilometre Zero fleet, relying on the seven features that make it suitable:

1. Central broker. Devices do not know each other or the consumers: they publish to the broker (Mosquitto to begin with; EMQX or HiveMQ for hundreds of thousands of connections) and the broker distributes. The van does not know Kafka exists.
2. Hierarchical topics. km0/delivery/van-3/position, km0/delivery/van-3/status, km0/delivery/van-3/commands. Wildcards allow subscribing at different levels: + for one level (km0/delivery/+/position: every position), # for everything below (km0/delivery/van-3/#). The hierarchy is also the basis for the ACLs: van-3 may only publish under its own prefix.
3. QoS per message. QoS 0 (at most once): fire and forget; suitable for the position every 5 s, because the next one replaces it. QoS 1 (at least once): the broker acknowledges with PUBACK and the client resends if it does not arrive; it may duplicate. QoS 2 (exactly once): a four-message exchange; expensive and rarely necessary. Kilometre Zero publishes positions with QoS 1 rather than 0 for a reason explained in section 7: on reconnecting after a tunnel, it wants the last known position to arrive for sure, even if duplicated (the consumer deduplicates by sequence number).
4. Retained messages. When publishing with retain=True, the broker keeps the last message on the topic and delivers it immediately to anyone who subscribes later. This is what lets a freshly opened dashboard see the last position of every van without waiting 5 s.
5. Last will. On connecting, the client registers a message that the broker will publish on its behalf if the connection is lost without a clean DISCONNECT: km0/delivery/van-3/status = disconnected. It is the failure detection of 07-03, done by the broker.
6. Persistent sessions. With clean_session=False (MQTT 3.1.1) or session expiry (MQTT 5), the broker remembers the client's subscriptions and queues the QoS 1/2 messages that arrive for it while it is disconnected, delivering them when it returns. For the commands the operator sends to the van ("new stop added") this is essential: the tunnel must not lose them.
7. MQTT over WebSockets. A browser cannot open arbitrary TCP sockets, but it can open WebSockets; brokers expose a WebSocket port (9001 on Mosquitto) over which they speak MQTT inside WebSocket frames. This is the route by which the operators' dashboard could subscribe directly to the broker; the Kilometre Zero design does not do so (section 5), but it is useful for diagnostic tools.

With the pieces above, the journey of a position from the van to Anna's map is this:

Every hop has a reason:

• The van speaks MQTT, not HTTP or Kafka. A cheap persistent connection, with reconnection and QoS handled by the library, and a broker that authenticates it and confines it to its topic prefix.
• The MQTT → Kafka bridge exists because the rest of the platform consumes Kafka: Flink (05-04), Cassandra for the history, analytics. It translates the hierarchical topic into a Kafka topic with key = courier id, which preserves per-van ordering (02-04) and lets Flink window by key. Some brokers (EMQX) ship this bridge built in; with Mosquitto, a small one is written.
• delivery consumes Kafka and fans out over WebSockets. It is the only component that knows the clients: it knows that Anna is subscribed to order P-2026-000124, that the order is being carried by van-3, and therefore that every van-3 position must reach Anna's connection. That knowledge (the subscription table) is what no other piece has.
• Redis pub/sub between instances solves the problem of the next section.

Note what is not done: the browser does not connect to the MQTT broker (though it could, over WebSockets) because that would mean giving every customer MQTT credentials, managing per-order ACLs on the broker and giving up the enrichment delivery performs (Anna should not receive the raw van-3 position along with its other twelve orders, but "your order is 4 stops and 12 minutes away").

delivery runs with several replicas on Kubernetes (07-05). Anna is connected to instance 1; Mark, who is waiting for another delivery from the same van-3, to instance 2. When the position arrives via Kafka, who receives it? If the two instances form a Kafka consumer group, each partition is read by one of them (02-04): the position reaches instance 1, which sends it to Anna, and Mark sees nothing. If each instance consumes the whole topic with a different group, each one receives every position and distributes it to its own clients: it works, but every instance processes the full 28 positions per second and consumes the entire topic, which with dozens of instances and delivery.dashboard included starts to weigh, and it does not solve messages that originate in one instance (the chat: Anna types on instance 1 and the dairy is on instance 2).

Sticky sessions: the load balancer (Kong, or the Ingress) always sends the same client to the same instance (by cookie or by IP hash), and each message is routed to the right instance. It requires someone to know which instance each client is on (a client → instance map in Redis) and the instances to talk to each other. It is fragile: when scaling or when an instance dies the map is invalidated, and the IP hash fails with mobile operators' NAT (thousands of clients behind the same IP).

Inter-instance bus (Kilometre Zero's option): each instance subscribes to a Redis pub/sub channel (or a Kafka topic with one group per instance) for every topic in which it has at least one interested client. Whoever has something to broadcast (the Kafka consumer on any instance, or the chat handler) publishes it on the bus, and every instance with subscribers receives it and distributes it to its connections. Nobody needs to know where anyone is; the instances are interchangeable and scale without coordination.

Redis pub/sub is fire-and-forget: if an instance is disconnected from the bus at that instant, the message is lost, and there is no history. For positions that does not matter (the next one arrives in 5 s and the client can ask for the last known one on connecting). For the chat it does: messages are persisted first (in Cassandra or PostgreSQL) and the bus only notifies; on reconnecting, the client asks the API for "everything since id X". When the volume or the guarantees demand more, the bus becomes Kafka with a consumer group per instance, or Redis Streams.

The subscription table lives in memory in each instance ({topic: {connections}}): it is local, fast and lost with the instance, which is the right thing because the connections are lost too. Presence (who is connected right now, useful for "Martha is online" in the chat or so that the operator knows which customers are watching) is stored in Redis with a TTL (presence:u-anna → instance-1, renewed with every ping): if the instance dies, the key expires on its own. And resolving which van Anna's order corresponds to is done at subscription time, by querying the delivery model, and done again if a reassignment event arrives.

The WebSocket connection lasts minutes; the JWT from 06-01 expires after 15. Three decisions:

• The JWT is verified at the handshake (Kong does it with the jwt plugin as with any route; delivery verifies it again, defence in depth). It is passed as a query parameter (wss://.../ws?token=...) or, better, as the first message after connecting, because URLs end up in logs (07-02) and the token must not.
• Authorization per subscription: every {"action": "subscribe", "order": "P-2026-000124"} message is authorized against the claims (sub = u-anna must be the order's customer; a roles: ["operator"] may subscribe to a market). There is no authorization for every message sent by the server: it was done at subscription time.
• Expiry: when the token expires, the server does not cut the connection abruptly (the map would freeze mid-journey); it sends {"type": "renew"} and the client sends a new {"action": "token", "jwt": "..."}. If it does not do so within 60 s, the connection is closed with code 4401.

QoS 1 may duplicate; Kafka with a key orders per partition, but a bridge retry may repeat; the client's reconnection may cause it to receive the retained "last position" and the new position at the same time. The solution is that of 01-05: every position carries a per-van sequence number (seq, a monotonic counter the app increments on every send and which survives restarts because it is persisted locally) in addition to timestamp_ms. The client keeps last_seq per van and discards anything that is not strictly greater. There is no need to sort: an old position adds nothing, so it is thrown away. The server does the same before broadcasting, so as not to spend bandwidth on duplicates.

A browser on 3G does not consume 28 messages per second (Jordan's dashboard would receive all of them). If the server writes without control, the send buffer for that connection grows without limit and ends up exhausting the instance's memory, taking every other client down with it: the same problem as in 05-04, on the last mile. Three defences, from least to most aggressive: coalescing (for each van only the latest position matters: if the client has three pending, one is sent), a bounded queue per connection (for example 100 messages; if it fills up, the oldest are discarded and counted in a km0_ws_drops_total metric), and closing (if the queue stays full for more than 30 s, the client cannot keep up: it is closed with code 1013 Try again later and the client reconnects with backoff, perhaps asking for a lower frequency).

An idle WebSocket connection costs little: a file descriptor, some 20-50 KB between kernel and application with asyncio. A delivery instance with 2 GB can sustain on the order of 20,000 to 50,000 connections, provided the operating system allows it: ulimit -n (descriptors per process, 1,024 by default; in the container it is raised to 65,536 or more), net.core.somaxconn and net.ipv4.ip_local_port_range on the node, and the load balancer in front (Kong, the Ingress), which also keeps one connection per client and has its own limits. What does cost is the traffic: 50,000 clients receiving 1 message/s is 50,000 writes per second per instance; there the limit is the CPU spent serialising and the bandwidth, and it is scaled by adding instances (the HPA from 07-05 with the km0_ws_connections metric instead of CPU). Deployments are the delicate moment: a rolling update cuts every connection on the instance being retired, so the graceful shutdown (07-05) sends a 1001 Going away close staggered over 30 s and relies on the client's reconnection with jitter.

The passwd file is generated with mosquitto_passwd -c passwd van-3 (one password per courier, issued when the device is registered and kept in Vault, 06-04). With the ACL, a compromised van-3 app cannot publish fake positions for van-7 or read its commands: the broker rejects it before it reaches the platform.

Details that matter: clean_session=False with a fixed client_id makes the broker keep the commands subscription and queue QoS 1 commands during the tunnel; keepalive=30 is what brings the last will to life (with no PINGREQ for 1.5 × keepalive, the broker publishes disconnected); and the seq counter is persisted before publishing, so that an app restart does not reuse a number (which would make the client discard new positions as old). The library queuing while there is no network means that, after a 3-minute tunnel, 36 old positions will arrive all at once: delivery will process them in seq order and only the last one will reach Anna, thanks to coalescing.

The event_id is built deterministically as courier-seq instead of a random UUID: that way, if the broker resends a PUBLISH (QoS 1) or the bridge restarts and reprocesses it, the event in Kafka carries the same id and the idempotent consumers from 02-05 discard it. It is a small detail that eliminates a whole class of duplicates.

And the Kafka consumer that feeds the bus (one task on each instance, all in the same consumer group, so that each partition is read by one of them):

The browser does not expose ping/pong to JavaScript (it handles it underneath when the server sends a ping), so detecting a "zombie" connection from the client relies on the server: if no message arrives within 60 s, the client can close and reconnect. lastSeq on the client is the last line of defence against duplicates: even though the server already filters, on reconnecting to another instance the "last known" position is received, and it may be one already seen.

The alerts from inventory.alerts to producers are the opposite case to the map: few messages, server → client only, and a dashboard that may stay open for hours. WebSockets would be over-engineering; SSE fits exactly, with reconnection for free and Last-Event-ID so that no alerts are lost during a disconnection.

A consumer of inventory.alerts (the one that in 05-04 wrote the Flink alerts) appends each alert to the alerts:<producer> stream with XADD and a MAXLEN of a few hundred entries. The difference from the map is that here it does matter not to lose any: hence Redis Streams (with history and monotonic ids) rather than pub/sub.

• Exposing Kafka or the MQTT broker to the browser. Wrong protocol, wrong security model, wrong number of connections. Always a last-mile server that authenticates, authorizes per subscription and enriches.
• Passing the JWT in the WebSocket URL. It ends up in Kong's logs and the server's (07-02). First message after connecting, and renewal without disconnecting.
• One Kafka consumer group per WebSocket instance with no bus. Either each instance receives only a share and its clients are left without data, or each consumes everything and it does not scale. Inter-instance bus (Redis pub/sub or Kafka).
• Writing to the socket without a bounded queue. A client on 3G exhausts the instance's memory and takes everyone down. A queue per connection, coalescing, a drops metric and a 1013 close.
• Reconnection without backoff or jitter. A delivery deployment makes thousands of clients reconnect in the same second. Exponential with a cap and jitter, and a staggered graceful shutdown on the server.
• QoS 2 "to be safe". Four messages per position, a heavy session on the broker, and the consumer has to deduplicate anyway for other reasons. QoS 1 and seq.
• Forgetting the last will and the keepalive. Without them, a courier with no coverage appears "connected" for the minutes it takes TCP to find out. keepalive=30 and a last will on status.
• Tip: measure km0_ws_connections, km0_ws_drops_total, the lag of the delivery-ws group and the end-to-end latency (the van's timestamp_ms against the time of receipt in the browser, with the clock corrected): that is the real SLO of "real time".
• Tip: when designing, start from the table in section 1. Acceptable latency, volume and direction decide the technique before any preference for WebSockets does.

Design the chat between Anna and Montblanc Dairy on top of this lesson's infrastructure: (a) which transport each end uses and why; (b) what is persisted, where, and in which order relative to the notification on the bus; (c) what the client does on reconnecting after 2 minutes without network so as neither to lose nor to duplicate messages; (d) which bus topic and which authorization rule ws_server.py applies.

van-3 goes through a 3-minute tunnel. Describe, step by step and naming the mechanisms, what happens in: the app (van_mqtt.py), the broker (last will, retained, session), the bridge, delivery.positions, Flink (the session window from 05-04), ws_server.py and Anna's screen. How many messages does Anna receive on leaving the tunnel, and why?

During a campaign there are 140 vans sending every 5 s and 40 operators with the dashboard open, each subscribed to the whole market (35 vans on average). (a) How many messages per second does each operator receive, and how many do the ws_server instances write in total? (b) If an operator is on a connection that only admits 2 messages/s, what happens to the queue of 100, and with which mechanism is it solved without closing the connection? (c) Propose a change in the Kafka consumer or in the server that reduces the traffic to the dashboard to 1 message/s per operator while keeping the useful information.

Exercise 1.

(a) Both ends are browsers (Anna on her phone, the dairy on its dashboard): WebSockets on both, because the chat is bidirectional and low-latency, and because Anna already has the tracking connection open: the same channel is reused with another message type ({"action": "chat", "conversation": "P-2026-000124", "text": "..."}).

(b) Each message is persisted first in orders or in a messages context (a messages_by_conversation table in Cassandra with partition key = conversation and clustering by a monotonic id, for example a TimeUUID or a per-conversation seq assigned by the server), and only then published on the conversation:P-2026-000124 bus topic. If it were published before persisting and the process died in between, the other end would see a message that does not exist.

(c) On reconnecting, the client sends {"action": "subscribe", "conversation": "...", "since": <last id seen>}; the server replies with the persisted messages after that id (a query by partition key, cheap) and then subscribes to the bus. The client deduplicates by message id (it may receive one both through the recovery and through the bus if it arrived at exactly that moment). It is the same Last-Event-ID of SSE, done by hand.

(d) Topic conversation:<order>; authorization: claims["sub"] is the order's customer, or claims["producer_id"] is the producer of some line of the order (a query to the orders model at subscription time, not per message).

Exercise 2.

1. App: it loses the PINGREQ; paho detects the drop and enters reconnection with backoff (1 s → 60 s). The position loop carries on: it increments seq, persists it and calls publish with QoS 1; paho queues in memory (~36 messages in 3 minutes).
2. Broker: after 45 s without keepalive (1.5 × 30) it declares the connection dead and publishes the last will km0/delivery/van-3/status = disconnected (retained). The persistent session keeps the commands subscription and queues any QoS 1 command the operator sends.
3. Bridge: it receives the disconnected status and publishes it to Kafka (as a status.updated event); it receives no positions.
4. delivery.positions: no van-3 events for 3 minutes.
5. Flink: the session window with a 3-minute gap closes (according to the watermark) and emits "courier with no signal" to delivery.dashboard; Jordan's dashboard shows it.
6. ws_server: it sends nothing to Anna; her connection stays alive (ping/pong with the server, which does have network). The map shows the last position and, if the client implements it, "last signal 2 min ago".
7. Leaving the tunnel: paho reconnects (the session was present), receives the queued commands and drains its queue: 36 QoS 1 PUBLISH messages in a burst, in seq order. The broker updates the retained message with the last one. The bridge publishes the 36 to Kafka with deterministic event_ids. Flink opens a new session ("it is back"). The ws_server consumer publishes 36 messages on the bus; in Anna's Connection, enqueue accepts each one because seq is increasing, but since they arrive within milliseconds and the queue has a capacity of 100, they are all queued: Anna would receive 36 messages in a burst, and the map would "jump" through the tunnel. For her to receive only the last one, coalescing per van would be needed (keeping only the latest position of each van in the queue), which the code in section 8 does not implement and which exercise 3(c) introduces. Without it there is no error, only wasted traffic.

Exercise 3.

(a) 140 vans / 5 s = 28 positions/s in total. Each operator, with 35 vans: 7 messages/s. Total writes: 40 × 7 = 280 messages/s across all the instances (plus those of the customers with tracking open). That is little: a single instance sustains it; the problem is not volume but robustness.

(b) 7/s come in and 2/s go out: the queue grows by 5/s and fills up in 20 s. From then on, enqueue discards the oldest for every new one (km0_ws_drops_total rises by 5/s) and the operator sees positions up to 100/7 ≈ 14 s late, but the connection stays up. It is solved without closing through coalescing: if there is already a pending position for the same van in the queue, the new one replaces it instead of being appended. With 35 vans, the queue never exceeds 35 entries and the operator always receives the most recent position of each one, with bounded lag.

(c) Two options. On the server: a sender per connection which, instead of draining the queue message by message, every second groups what is pending into a single {"type": "positions", "items": [...]} message with the latest position of each van (coalescing + batching: 1 message/s, 35 positions inside). In the Kafka consumer: instead of publishing every position on market:<m>, keep a market:<m>:positions hash in Redis (HSET van-3 <json>) and publish one "tick" per second; the server, on receiving the tick, reads the hash and sends the complete state. The first is simpler and leaves the bus unchanged; the second decouples the dashboard's frequency from the vans' and is more appropriate if the dashboard grows to hundreds of operators.

The last mile has rules of its own because its clients are browsers, phones and devices on networks that fail, in numbers no internal service ever reaches, and without the trust or the libraries of a Kafka consumer. "Real time" here is perceived latency, fixed per case: seconds for Anna's map, one or two for Jordan's dashboard, tens for the dairy's alerts, under one for the chat. With that table in front of you, the technique chooses itself: polling and long polling as fallbacks, SSE for unidirectional traffic that must not be lost (with Redis Streams and Last-Event-ID), WebSockets for what is bidirectional and high-volume, and MQTT for devices, with hierarchical topics that double as ACLs, QoS 1 with deduplication by sequence number, retained messages, a last will and persistent sessions that survive the tunnel. The end-to-end architecture chains the van, Mosquitto, the bridge to delivery.positions, Flink and the delivery WebSocket server, which is the only one that knows who wants what; fan-out across its instances is solved with a bus (Redis pub/sub) rather than sticky sessions, the JWT is verified at the handshake and renewed without disconnecting, ordering and duplicates are handled with the seq from 01-05 on server and client, pressure from slow clients with bounded queues and coalescing, and scaling with interchangeable instances, tuned operating-system limits and staggered shutdowns.

Everything built so far, from the MQTT broker to the Kubernetes cluster, runs on machines that somebody has bought, installed and maintains. The next lesson changes that premise: what happens when the infrastructure becomes an API from a cloud provider, which managed services replace each piece we have assembled by hand, how it is all described with Terraform, and what it costs per month. It is the lesson on Cloud Applications.