The previous lesson ended with a roadmap and a promise: the first step towards splitting the Kilometre Zero monolith is getting two processes to talk to each other reliably. Before choosing a high-level mechanism (RPC, gRPC, message queues), we need to understand what all of them rest on: network protocols. Everything we will see in this module (and a good part of what we will see in the rest of the course) is bytes travelling over TCP or UDP, almost always wrapped in HTTP. Unless you understand what each layer guarantees and what it does not, it is impossible to reason about timeouts, duplicates, message ordering or performance.

In this lesson we will build the first, deliberately primitive version of the inventory service as a TCP server that answers a text command; we will send van-3 positions over UDP and watch what gets lost; and we will make an HTTP request to the monolith to see what lies underneath a curl. With that hands-on material we will compare TCP, UDP, HTTP (in its 1.1, 2 and 3 versions) and MQTT, and settle on criteria for choosing between them.

1. What a protocol is and why protocols are organised in layers
2. The TCP/IP stack
3. TCP: the reliable connection and its price
4. UDP: speed without guarantees
5. HTTP/1.1, HTTP/2 and HTTP/3
6. Lightweight protocols: MQTT and WebSockets
7. Hands-on: inventory over TCP, telemetry over UDP and an HTTP request to the monolith
8. How to choose: comparison table
9. Common mistakes and tips
10. Exercises
11. Conclusion

A protocol is an agreement between two parties on three things: the format of the messages (which bytes mean what), the exchange rules (who speaks first, how to reply, what happens when there is an error) and the meaning of each message. Without a prior agreement no communication is possible: when orders sends bytes to inventory, both must have agreed beforehand that STOCK aged-cheese\n is a query and that OK 5\n is its reply.

Network protocols are organised in layers because the problem is too big to solve in one go. Each layer solves one specific problem and offers a service to the layer above, hiding the details of the layers below:

• The link layer moves frames between two physically connected machines (Ethernet, Wi-Fi).
• The network layer (IP) moves packets between machines on different networks, hopping through routers. It guarantees nothing: packets can be lost, duplicated or arrive out of order.
• The transport layer (TCP, UDP) delivers data to a specific process (identified by a port) and, in the case of TCP, adds reliability and ordering.
• The application layer (HTTP, MQTT, DNS, the gRPC protocol...) defines what the messages mean for a particular use.

This separation is the reason you can write an HTTP server without knowing anything about Ethernet, and it is also the reason the fallacies from lesson 01-04 are so dangerous: each layer hides the problems of the one below, but does not eliminate them. TCP hides the fact that IP loses packets by retrying, and that retry shows up as unpredictable latency (fallacy 2).

The TCP/IP model has four layers (the more academic OSI model has seven; in practice people use the terminology of both, so it is worth knowing how they map onto each other).

When orders sends "STOCK aged-cheese" to inventory, the message travels down the stack on the sender (TCP headers are added, then IP, then Ethernet: encapsulation) and up the stack on the receiver, where each layer strips off its own header. It helps to visualise it:

One practical detail that is often forgotten: a port identifies a process, not a logical service. When inventory has several replicas (inv-bcn, inv-vlc), each one will listen on its own IP and port, and something on top (DNS, a load balancer, Kubernetes service discovery, lesson 07-05) will be needed so that orders knows which one to talk to.

TCP (Transmission Control Protocol) is the transport protocol used by HTTP, gRPC, AMQP, the PostgreSQL protocol and almost everything else. It offers four guarantees on top of IP, which offers none:

1. Connection. Before any data is sent, a bidirectional channel is established between two endpoints (IP:port ↔ IP:port). Data belongs to that connection, and the connection is closed explicitly.
2. Reliability. Every byte sent has a sequence number; the receiver acknowledges (ACK) what it has received; anything unacknowledged is retransmitted. From the application's point of view, either the byte arrives or the connection ends with an error.
3. Ordering. Bytes are delivered to the application in the same order in which they were sent, even if the IP packets arrive out of order.
4. Flow and congestion control. The receiver advertises how much it can accept (the receive window), and the sender slows down when it detects losses, interpreting them as network congestion.

TCP delivers a byte stream, not messages. If orders makes two consecutive send() calls with "STOCK aged-cheese\n" and "STOCK pink-tomato\n", the receiver may get them in a single recv(), or in three chunks of arbitrary size. The application is responsible for framing the messages (with a terminator such as \n, with a length prefix, or with a self-describing format). This is the most frequent source of bugs in socket programming and the reason application protocols exist: HTTP uses headers and Content-Length; gRPC uses a 5-byte length prefix per message.

Opening a TCP connection requires an exchange of three messages (SYN, SYN-ACK, ACK): one full round trip (RTT) before the first useful byte can be sent. If TLS is involved as well (lesson 06-02), one or two more RTTs are added. Between two containers in the same data centre the RTT is ~0.5 ms and the cost is negligible; between a courier's app on 4G in Lleida and the server, the RTT can be 80-100 ms, and opening a connection for every message is an enormous waste. This is where the importance of persistent connections (keep-alive) and connection pools comes from: open few connections and reuse them.

The ordering guarantee comes at a price. If packet number 7 is lost on a TCP connection, TCP holds back packets 8, 9 and 10 (which did arrive) until the retransmission of 7 gets through, because it must deliver the bytes in order. This phenomenon is called head-of-line blocking (HOL): one loss delays everything queued behind it, even if it is unrelated. When several logical requests share a connection (as HTTP/2 does), a loss in Anna's request delays Mark's request. We will come back to this when we discuss HTTP/3.

Never immediately. If the inventory process dies cleanly, the operating system closes the connection and the client gets a fast error. But if the whole server powers off or a router stops forwarding, no closing packet ever arrives: the client keeps waiting until its retransmissions hit a limit (minutes, by default) or until TCP's keepalive mechanism (disabled by default, and with intervals measured in hours) notices. This is the technical reason why lesson 01-04 insisted that no remote call should ever be made without an application-level timeout: TCP is not going to do it for you.

UDP (User Datagram Protocol) is the bare minimum you can put on top of IP: it adds source and destination ports and a checksum, and nothing else. There is no connection, no acknowledgements, no retransmission, no ordering, no flow control. Each sendto() is an independent datagram that either arrives whole or does not arrive at all.

So what is it good for? For the cases where TCP's guarantees get in the way:

• When stale data is worthless. The van-3 app sends its GPS position every second. If the 10:00:03 position is lost, retransmitting it at 10:00:05 adds nothing: the 10:00:04 one will already have arrived. With TCP, that retransmission would also block the later positions (HOL). With UDP, it is simply lost and the next one replaces it.
• When latency matters more than completeness. Voice, live video, games, telemetry.
• When there are huge numbers of small messages and the cost of maintaining connections would be prohibitive (this is why DNS uses UDP).
• When you want to build your own transport protocol on top, as QUIC does.

At Kilometre Zero, position telemetry is the clear-cut case: thousands of couriers during a campaign sending one datagram per second, where losing one in a hundred is irrelevant. What you would never send over UDP is an order or a payment: there, every message matters. One important nuance: an application that uses UDP usually ends up adding some of TCP's guarantees by hand (a sequence number to detect reordering, for example), and we will see that in the code.

HTTP is the dominant application protocol, not only for web pages but also as the transport for REST APIs, for gRPC and, increasingly, for almost anything (because it gets through firewalls and proxies without trouble). It is worth understanding its three versions because the differences have direct architectural consequences.

A text protocol over TCP. A request is a line with the method and path, some headers, a blank line and an optional body; the response is a status line, headers and body. This is what you will see in section 7 when we make the request to the monolith. Its limitations:

• One request in flight per connection. The client sends a request and must wait for the complete response before sending the next one on that connection (pipelining exists in the specification, but it never worked well in practice). Browsers compensate by opening 6 connections per domain; API clients, with pools.
• Repeated, plain-text headers. Every request sends Host, User-Agent, Accept, cookies... uncompressed. In an API that makes thousands of small calls, the headers can weigh more than the bodies.
• Keep-alive (persistent connections) is the default behaviour and mitigates the cost of the handshake, but not the problem of requests being serialised one after another.

Same semantics (methods, status codes, headers), different format: binary and with multiplexing. A single TCP connection carries multiple simultaneous, independent streams: Anna's stock request and Mark's travel interleaved on the same connection, without waiting for each other. It also compresses headers (HPACK) and allows streams to be prioritised. It is the foundation of gRPC (lesson 02-03) precisely because of multiplexing and because streams allow streaming in both directions.

Its weak point is the one we already know: since all the streams share one TCP connection, TCP's head-of-line blocking affects all of them. On lossy networks (mobile), HTTP/2 can perform worse than HTTP/1.1 with several connections.

HTTP/3 solves that problem by changing the transport: instead of TCP it uses QUIC, a protocol built on UDP that reimplements reliability, ordering and congestion control, but per stream, not per connection. If a packet from Anna's stream is lost, only Anna's stream is delayed. QUIC also integrates TLS 1.3 into the handshake (a secure connection in 1 RTT, or 0 RTT on resumption) and survives IP changes (the van that moves from Wi-Fi to 4G keeps its connection). It is deployed in the major browsers and CDNs; for service-to-service communication inside a data centre its advantage is smaller, because there is hardly any loss there.

There are scenarios where HTTP is too heavy or its request-response model does not fit.

MQTT is a publish/subscribe protocol designed for resource-constrained devices and unreliable networks (its origins lie in satellite telemetry for oil pipelines). A central broker receives messages published to hierarchical topics (km0/delivery/van-3/position) and forwards them to whoever has subscribed (with wildcards: km0/delivery/+/position). Its headers take up 2 bytes, it keeps a single persistent TCP connection (so a message pays no handshake), it defines three quality of service levels (QoS 0: at most once; QoS 1: at least once; QoS 2: exactly once, with a four-message exchange) and it includes the concept of a last will, a message the broker publishes if the device disconnects abruptly. For the couriers' app it is a serious alternative to raw UDP: it pays a few more bytes in exchange for getting through any mobile network and for having a broker that distributes the positions. Here we only introduce it; the design of a real-time messaging system with MQTT, and the propagation all the way to the customer's map, is developed in lesson 08-02.

WebSockets turns an HTTP connection into a persistent bidirectional channel between browser and server, so that the server can send without the client asking. It is what Anna's tracking screen will use to watch van-3 move without reloading. It is also developed in lesson 08-02; for now it is enough to know that it exists, that it runs over TCP and that it starts out as a normal HTTP request with the Upgrade: websocket header.

Let's get our hands on all three using Python's standard library. The goal is not to write production code (that is what the following lessons are for), but to see at first hand what each protocol does. All the code goes in km0/services/, which has been empty until now.

The first version of the inventory service extracted from the monolith: a server that accepts connections and answers the STOCK <product> command. We define a minimal application protocol: UTF-8 text messages terminated by \n; replies are OK <units> or ERR <reason>.

The key parts explained:

• socket.AF_INET, socket.SOCK_STREAM means "IPv4, TCP". With SOCK_DGRAM it would be UDP.
• bind associates the socket with an address and port; listen puts it in server mode; accept blocks until a client completes the handshake and returns a new socket dedicated to that connection. The original socket keeps listening.
• Each connection is served in its own thread, because recv blocks: if we served clients one after another, the second would have to wait until the first disconnected. In lesson 01-02 we saw the alternative with asyncio; here we use threads for simplicity.
• read_line is the most important part of the whole file. It accumulates bytes until it sees the terminator. If you remove it and trust that "one recv = one message", the server will work in your local tests and fail in production when the network splits a message or merges two.
• sendall (rather than send) guarantees that all the bytes are sent; send may send only part of them and return how many it sent.

Expected output:

Notice two things. First: the connection is opened once and used for all three queries (hand-made keep-alive); opening it for each query would triple the handshakes. Second: the client's buffer keeps any bytes left over from a recv, in case part of the next reply came along with it. Try stopping the server with Ctrl+C while the client is waiting: you will see an immediate ConnectionError because the operating system closed the connection. Now try blocking the port with a firewall or unplugging the network: the client will see nothing until the 2-second timeout fires. That is the difference between a clean crash failure and an omission (lesson 01-02).

Now for the courier's app. It sends one JSON datagram per second with its position and a sequence number. To simulate a mobile network, the sender "loses" 10% of the datagrams before sending them.

And the receiver in delivery, which detects gaps in the sequence and discards positions that arrive late:

Key points:

• There is no listen, accept or connect. recvfrom returns one complete datagram and the address of whoever sent it. Unlike TCP, here messages do have boundaries: there is no need for read_line.
• The sequence number is something we add: UDP does not provide it. With it we detect loss and reordering, and decide what to do about them (ignore the loss and discard anything late, which is the right policy for positions). We are rebuilding part of TCP, the part we care about, without paying for the part we do not.
• The van-3 datagram is about 90 bytes long. Over TCP+HTTP/1.1, with headers, it would exceed 300, and every new send from a closed connection would add a handshake. Over MQTT (section 6) it would be about 100 bytes on an already open connection: a middle ground that we will see in 08-02.

Finally, let's talk to the monolith we left running in 01-06, but looking at what curl hides.

What travels over the TCP connection to port 8000 is exactly this text (you can see it with curl -v http://localhost:8000/health or with nc localhost 8000, typing it in by hand):

And the response:

Look at Content-Length: it is how HTTP/1.1 solves the problem of framing messages on top of TCP's byte stream (the same one we solved with \n in 7.1). And notice that, compared with our text protocol, HTTP adds headers that here take up more space than the useful data: that is the price of universal interoperability. For an internal call between orders and inventory that is repeated 1,200 times per second (lesson 01-03), that price adds up, and it is one of the reasons why lesson 02-03 will use gRPC over HTTP/2 with binary bodies.

Selection criteria, in order of importance:

1. Does every message matter? If so, rule out UDP and MQTT QoS 0.
2. Is it request-response or a stream of events? Request-response → HTTP/gRPC (02-02, 02-03). A stream to many interested parties → messaging (02-04) or MQTT.
3. Who is on the other end? A browser or a third party → HTTP. One of our own services → gRPC. A battery-powered device on 4G → MQTT.
4. How many messages per second, and how big? From thousands per second with small bodies upwards, the overhead of text-based HTTP/1.1 becomes noticeable; binary HTTP/2 or MQTT reduce it.
5. Only then look at raw performance. It is almost never the deciding factor and almost always the first one to be argued about.

• Assuming that one recv returns one message. This is the number one mistake with TCP. It works locally and fails in production. Always frame your messages (terminator, length prefix or self-describing format) or use an application protocol that does it for you.
• Trusting TCP to detect that the server has died. It only detects it if the process closes the connection. Faced with a powered-off server or a severed network, your client will wait for minutes unless it has an application-level timeout.
• Opening a connection per request. Each handshake costs one RTT (more with TLS). Reuse connections: pools in HTTP clients, persistent channels in gRPC, one connection per process in MQTT.
• Using UDP "because it's faster" for data that matters. A lost order is not faster, it is a lost order. UDP is for data that expires.
• Adding all of TCP's reliability to UDP by hand. If you end up implementing retransmissions, ordering and flow control, use TCP (or QUIC). UDP only pays off if you need fewer guarantees.
• Ignoring the size of HTTP headers. In very frequent internal calls, measuring bytes per request turns up surprises. HTTP/2 and binary bodies exist for this reason.
• Tip: learn to use curl -v, nc (netcat) and tcpdump/Wireshark. Seeing the actual bytes on the wire settles in minutes arguments that last for hours in the abstract.
• Tip: document the application protocol even if it is plain text and three lines long, as we did with STOCK/OK/ERR. The day another team writes a client, they will thank you; and the day you want to change it, you will know what breaks.

Add the RESERVE <product> <quantity> command to the TCP server: it should deduct stock if there is enough and reply OK <remaining>, or ERR insufficient stock otherwise. Bear in mind that several threads may reserve the same product at the same time (remember Anna and Mark's last cheese, lesson 01-05). Then, from the client, launch 4 reservations of 2 units of aged-cheese from 4 simultaneous threads and check that the stock never goes negative.

Modify tcp_client.py so that it sends the three STOCK commands in one go with a single sendall (joined with \n) and then reads the three replies. Check whether the server processes them correctly when they arrive together and explain why, then work out what would happen if read_line had been written as return conn.recv(1024).decode().strip().

For each Kilometre Zero flow, say which transport/application protocol you would use (raw TCP, UDP, HTTP/1.1, HTTP/2, MQTT) and justify it using the criteria from section 8:

1. Lucy's app queries the Montblanc Dairy catalogue.
2. orders queries inventory for stock 1,200 times per second during a campaign.
3. van-3 sends its position every second while crossing areas with patchy coverage between Lleida and Tarragona.
4. A producer uploads 40 photos of their products from the office.
5. A battery-powered temperature sensor in the Roble Alto Winery cold store reports once a minute.

Solution 1:

The key is that the check and the deduction happen inside the same with lock. If you checked outside the lock and deducted inside it, two threads could both see "5 left" and both deduct 2... and then a third and a fourth as well: the stock would end up at -3. With 4 concurrent reservations of 2 units against 5, exactly two will succeed (3 remaining, then 1) and two will receive ERR insufficient stock. Test client:

Note that, within a single process, the lock solves the problem; once there are inv-bcn and inv-vlc replicas, no in-memory lock will help: that is the problem tackled in Module 3.

Solution 2:

It works because the server's read_line accumulates up to the first \n and... there is a catch here: as written, read_line checks data.endswith(b"\n"), and if the three commands arrive in a single recv, data ends in \n and contains three lines; process will see "STOCK aged-cheese\nSTOCK pink-tomato\nSTOCK ham" and reply ERR unknown command just once, leaving the client waiting for two replies that will never come (until the timeout). The correct version of the server must keep a per-connection buffer and extract lines with partition, exactly as the client does. It is a perfect example of how "it works in my tests" does not mean "it handles the byte stream properly". With the naive version conn.recv(1024).decode().strip(), moreover, a command that arrived split across two recv calls would produce two ERR unknown command errors and no result.

Solution 3:

1. HTTP (1.1 or 2, depending on the client): request-response, external client (mobile app or browser), interoperability and HTTP caching. With a CDN in front, HTTP/3 improves the mobile experience.
2. HTTP/2 (gRPC): every message matters (rules out UDP), synchronous request-response (rules out messaging), one of our own services, high volume with small messages (multiplexing and the binary format reduce overhead). Lesson 02-03.
3. MQTT (or UDP if we owned the network): the data expires (an old position is worthless), lossy mobile network, battery-powered device. MQTT with QoS 0 gives practically the efficiency of UDP, but keeps a connection that gets through the mobile operators' networks and reconnects on its own; and the broker distributes the positions to whoever needs them (08-02).
4. HTTP: every photo matters (no UDP), it is a request-response transfer, and it comes from a browser; HTTP/2 allows all 40 to be uploaded in parallel over a single connection.
5. MQTT with QoS 1: constrained device, possibly unstable network, small and infrequent messages where delivery does matter (an out-of-range temperature is an alert). The last will would warn us if the sensor disconnects.

This lesson has gone down to the foundations on which everything else rests. We have seen that a protocol is an agreement on format, rules and meaning, and that the TCP/IP stack splits the problem into layers that hide, but do not eliminate, the failures of the layer below. TCP gives us connection, reliability, ordering and flow control in exchange for the handshake, head-of-line blocking and failure detection that is never immediate; UDP takes all of that away, which makes it ideal for van-3's positions, where stale data is worthless. HTTP/1.1 is text and one request per connection; HTTP/2 multiplexes streams over a binary connection and is the foundation of gRPC; HTTP/3 relies on QUIC over UDP to eliminate head-of-line blocking. MQTT and WebSockets have been introduced and deferred to lesson 08-02. With the code we have confirmed that TCP is a byte stream that requires message framing, that a client without a timeout hangs when a server vanishes without closing, and that HTTP headers can weigh more than the data.

The inventory TCP server with its STOCK/OK/ERR protocol works, but it is awkward: every new operation means inventing a format, writing a parser at both ends and translating errors by hand. What we would really like is to write inventory.get_stock("aged-cheese") in the orders code and have someone else take care of turning it into bytes, sending it, waiting for the reply and returning it as a Python value. That idea, with its advantages and its pitfalls, is the subject of the next lesson: RPC and RMI.