Module 5 ended with a confession: in everything built so far, any process could talk to any service and read any data. orders calls inventory.ReserveStock without saying who it is; the Flink consumer reads orders.events without identifying itself; and the original monolith kept Anna's session in the memory of the only process that existed, which stopped working the moment there were two instances behind a load balancer. The two questions no system can dodge are who is who (authentication) and who can do what (authorization), and in a distributed system they come with an added difficulty: the answer has to travel with every request across services that share no memory, no process and sometimes not even a development team.
In this lesson we will separate the two concepts precisely, see how to store passwords so that a database leak is not a catastrophe, compare stateful sessions (cookie plus Redis) with stateless tokens, take a JSON Web Token apart piece by piece, and study how that identity propagates from the web to orders and from orders to inventory. Then we will build authorization: roles, attributes and relationships, and the principle of least privilege. The code leaves Kilometre Zero with an identity service that issues RS256 tokens, and with inventory rejecting any call that does not arrive signed and authorized. Deliberately left out are how login is delegated to an identity provider (OAuth 2.0, OpenID Connect, SSO: lesson 06-03), how services identify themselves to each other with certificates (06-04) and how validation is centralised at the edge (06-05).
Contents
- Authentication and authorization: two different questions
- Why an in-memory session does not survive distribution
- Authentication factors and password storage
- Stateful sessions versus stateless tokens
- JSON Web Tokens: anatomy, signature and claims
- Expiry, refresh tokens and revocation
- Classic JWT mistakes
- Propagating identity between services
- Authorization models: RBAC, ABAC, ReBAC
- Where the policy lives: centralised or in each service
- Kilometre Zero:
identity, gRPC interceptor and ABAC check - Common Mistakes and Tips
- Exercises
- Conclusion
- Authentication and authorization: two different questions
They are constantly confused because they usually happen back to back and because both get abbreviated to "auth". It is worth pinning them down:
| Authentication (AuthN) | Authorization (AuthZ) | |
|---|---|---|
| Question | Are you who you say you are? | Are you allowed to do what you are asking? |
| Input | Credentials: password, code, certificate, token | An already verified identity + action + resource |
| Output | An identity (subject) and, often, attributes (roles, ids) | A decision: allow or deny |
| When | Once per session or per token | On every operation |
| Typical failure | 401 Unauthorized (badly named: it means "not authenticated") | 403 Forbidden |
| In Kilometre Zero | Anna proves she is [email protected] with her password |
Anna can view P-2026-000123 because it is hers; she cannot edit aged-cheese |
A common design mistake is to solve the first and forget the second: the application checks that there is a valid user and from then on everything is allowed. In a marketplace that means any registered customer could, by changing an id in the URL, view Mark's orders or change Roble Alto Winery's prices. Authentication identifies; authorization protects.
- Why an in-memory session does not survive distribution
The monolith from 01-01 did what every web framework does by default: at login, it stored {"user": "anna", "roles": ["customer"]} in an in-memory dictionary, generated a random identifier and returned it in a cookie. On every subsequent request, it looked the cookie up in the dictionary. Perfect with one process. And in 01-06 the platform came to have:
- Several instances of the same service behind a load balancer: Anna's second request may land on an instance that never saw her log in. The emergency fix, sticky sessions (the load balancer always sends Anna to the same instance), breaks load distribution and loses the sessions when that instance dies or is redeployed.
- Several services:
ordersreceives a request coming from the web, but it needs to know it is Anna's in order to query her orders and nobody else's, and then it callsinventory, which should also know on whose behalf it is acting. - Processes without a browser: the courier app, the Airflow DAG, a Kafka consumer. They have no cookies and do not "log in" in the human sense.
There are two families of solutions, and both are used:
flowchart LR
subgraph S["Stateful session"]
C1[Cookie: opaque id] --> R[(Shared Redis<br/>session:abc123 → anna, roles)]
R --> S1[Instance 1]
R --> S2[Instance 2]
end
subgraph T["Stateless token"]
C2[Signed token:<br/>sub=anna, roles, exp] --> V1[Instance 1<br/>verifies signature]
C2 --> V2[Instance 2<br/>verifies signature]
C2 --> V3[inventory<br/>verifies signature]
end
In the first, the identity lives in a shared store and the cookie is just a reference. In the second, the identity travels inside the token itself, protected by a signature that any service can verify without asking anyone. Before comparing them, we need to settle the previous step: how we verify Anna's password the first time.
- Authentication factors and password storage
Authentication factors fall into three families: something you know (password, PIN), something you have (a phone with a TOTP app, a FIDO2 key, a card), something you are (fingerprint, face). Multi-factor authentication (MFA) combines two different families; two passwords are not two factors. Rolling it out on the platform, together with account recovery, is a topic for 06-03; here we focus on the factor every application ends up managing: the password.
The rule: never reversible encryption, always a salted hash
A password must not be recoverable, not even by the administrator. If it is encrypted with AES and a key, whoever holds the key (or steals it along with the database) gets every password in the clear, and users reuse those passwords for their bank and their email. The right approach is to store a hash: a one-way function. But not just any hash:
| Method | Problem |
|---|---|
| MD5 / SHA-1 / SHA-256 of the password | Extremely fast: a GPU tries billions per second; precomputed tables (rainbow tables) reverse common hashes instantly |
| Salted SHA-256 | Defeats precomputed tables, but it is still fast: a per-user dictionary attack remains feasible |
| bcrypt, scrypt, Argon2 (password hashes) | Designed to be slow and tunable (cost in time and, for Argon2/scrypt, in memory), with a built-in random salt |
The salt is a random value, different for each user, that is concatenated with the password before hashing and stored next to the result. Two users with the password summer2026 will have different hashes, and the attacker cannot amortise one computation across users. The cost factor makes verifying a password deliberately take tens of milliseconds: irrelevant for a login, devastating for someone attempting billions.
Argon2id is the current recommendation (winner of the Password Hashing Competition, GPU-resistant thanks to its memory usage); bcrypt is still perfectly acceptable and more common in existing systems. In Kilometre Zero we will use argon2-cffi:
# km0/services/identity/passwords.py
# pip install argon2-cffi
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
# Parameters: 3 iterations, 64 MiB of memory, 4 threads. Tune them so that
# 'hash' takes ~100 ms on your production hardware: that is the cost you pay
# once per login and the attacker pays billions of times.
ph = PasswordHasher(time_cost=3, memory_cost=65536, parallelism=4)
def register(password: str) -> str:
"""Returns the string that is stored in the database. It includes the
algorithm, parameters, salt and hash: everything needed to verify."""
return ph.hash(password)
def verify(stored_hash: str, password: str) -> bool:
try:
ph.verify(stored_hash, password)
except VerifyMismatchError:
return False
return True
def needs_rehash(stored_hash: str) -> bool:
"""True if the hash was made with weaker parameters than the current
ones: on the next successful login, it is recomputed and stored."""
return ph.check_needs_rehash(stored_hash)
if __name__ == "__main__":
h = register("summer2026")
print(h)
# $argon2id$v=19$m=65536,t=3,p=4$UmFuZG9tU2FsdFZhbHVl$...hash...
print(verify(h, "summer2026"), verify(h, "Summer2026")) # True FalseLook at the resulting string: $argon2id$v=19$m=65536,t=3,p=4$<salt>$<hash>. It carries the algorithm and the parameters inside, so that three years from now, when you raise memory_cost, the old hashes will still verify and needs_rehash will tell you which ones to update at the next login. With bcrypt the code is analogous: bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12)) and bcrypt.checkpw(...), with the 72-byte password limit that is worth knowing about.
Two design details that are often forgotten: the login error message must be identical whether the user does not exist or the password is wrong (otherwise accounts can be enumerated), and so must the response time, which is achieved by verifying against a dummy hash when the user does not exist.
- Stateful sessions versus stateless tokens
Stateful session: cookie + Redis
Lesson 04-05 left Redis as the catalogue cache; it is also the classic session store. The flow is:
- Anna sends her username and password to
identity; they are verified with Argon2. identitygenerates a random 256-bit id (secrets.token_urlsafe(32)), storessession:<id> → {"sub": "u-anna", "roles": ["customer"]}in Redis with a TTL, and returns the id in anHttpOnly; Secure; SameSite=Laxcookie.- Every instance of the web, on every request, reads the cookie, runs
GET session:<id>against Redis and obtains the identity.
# Fragment: shared session store (the web has N instances)
import json, secrets, redis
r = redis.Redis(host="redis", decode_responses=True)
TTL_S = 8 * 3600
def create_session(sub: str, roles: list[str]) -> str:
sid = secrets.token_urlsafe(32)
r.set(f"session:{sid}", json.dumps({"sub": sub, "roles": roles}), ex=TTL_S)
return sid
def load_session(sid: str) -> dict | None:
v = r.get(f"session:{sid}")
if v is None:
return None
r.expire(f"session:{sid}", TTL_S) # sliding session: every use renews it
return json.loads(v)
def close_session(sid: str) -> None:
r.delete(f"session:{sid}") # immediate, trivial revocationAdvantages: revocation is a DEL; the client sees nothing of the identity (the id is opaque); the session can hold whatever you like without worrying about size. Drawbacks: every request costs a round trip to Redis, Redis is now a critical point (if it goes down, nobody is authenticated), and above all it does not propagate: when the web calls orders and inventory, those services do not have the cookie, nor should they be querying the web's session Redis.
Stateless token
The alternative is for identity to issue a signed document containing the identity and its attributes: a token. Whoever receives it verifies the signature and trusts the content without consulting anyone. It is the same principle as MinIO's presigned URLs (04-03), which we called a capability: a portable, self-contained permission with an expiry, verified cryptographically.
| Criterion | Session (cookie + Redis) | Token (JWT) |
|---|---|---|
| Server-side state | Yes (Redis) | No |
| Cost per request | One GET to Redis |
Verifying a signature (microseconds with HS256, ~tens of µs with RS256) |
| Revocation | Immediate (DEL) |
Hard: the token is valid until exp (section 6) |
| Propagation between services | Not natural | Natural: the token travels with every call |
| What the client sees | An opaque id | The content (signed, not encrypted) |
| Size | A ~50-byte cookie | Hundreds of bytes up to 1-2 KB per request |
| Typical use | Traditional monolithic web | APIs, microservices, mobile apps, machine-to-machine |
In practice many platforms combine the two: the web keeps a cookie-based session for the browser (safer against XSS, as we will see) and, when calling internal services, exchanges that session for a short-lived token that does propagate. Kilometre Zero will use tokens from the courier app and between services, and the decision for the web is discussed in section 7.
- JSON Web Tokens: anatomy, signature and claims
A JWT (RFC 7519) is a string with three dot-separated parts, each in Base64URL:
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImttMC0yMDI2LTA5In0 . eyJpc3MiOiJodHRwczovL2lkZW50aXR5LmttMCIsInN1YiI6InUtYW5uYSIsImF1ZCI6WyJvcmRlcnMiLCJpbnZlbnRvcnkiXSwiZXhwIjoxNzczNjc0MDAwLCJpYXQiOjE3NzM2NzMxMDAsImp0aSI6IjdmNC4uLiIsInJvbGVzIjpbImN1c3RvbWVyIl19 . Xk9v...signature...
- Header:
{"alg": "RS256", "typ": "JWT", "kid": "km0-2026-09"}. The signing algorithm and, optionally, the key identifier (kid), so the verifier knows which key to use when several are in rotation. - Payload: the claims, a JSON object with statements about the subject.
- Signature: computed over
base64url(header) + "." + base64url(payload)with the stated algorithm. If anyone changes a single letter of the payload, the signature no longer matches.
Important: the payload is encoded, not encrypted. Anyone holding the token can read it with base64 -d. Never put in a JWT anything the bearer should not see (a password, another user's phone number). An encrypted variant exists (JWE), rarely used in APIs; data encryption is the subject of 06-02.
Standard claims
| Claim | Name | Meaning | In Kilometre Zero |
|---|---|---|---|
iss |
issuer | Who issued the token | https://identity.km0 |
sub |
subject | Identifier of the subject (stable, not an email that may change) | u-anna, prod-montblanc-dairy, svc-orders |
aud |
audience | Who the token is for; the recipient must check that it is on the list | ["orders", "inventory"] |
exp |
expiration | Unix time after which it is invalid | iat + 15 min |
iat |
issued at | When it was issued | — |
nbf |
not before | Not valid before this instant | Rare; useful with unsynchronised clocks |
jti |
JWT id | Unique identifier of the token | Lets you revoke a specific one (section 6) and correlate audit records (06-05) |
On top of these come the application's own private claims. Kilometre Zero will use roles (a list) and, for producers, producer_id (a slug), and for couriers courier_id. The rule: few claims, stable ones, and never those that change every minute (the balance, the basket), because the token is a snapshot taken at the time of issue.
HS256 versus RS256/ES256: why services verify with a public key
| Algorithm | Type | Key for signing | Key for verifying | Consequence |
|---|---|---|---|---|
| HS256 | HMAC with SHA-256 (symmetric) | Shared secret | The same secret | Every service that verifies can also issue tokens |
| RS256 | RSA + SHA-256 (asymmetric) | Private key (only identity) |
Public key (anyone) | Services verify without being able to forge; signature of ~256 bytes |
| ES256 | ECDSA P-256 (asymmetric) | Private key | Public key | Like RS256, with shorter signatures (64 bytes) and fast verification; less support in older libraries |
| EdDSA | Ed25519 | Private key | Public key | The most modern and fastest; growing support |
With HS256, inventory, orders, delivery and analytics would need the secret in order to verify, and a compromise of any of them (or of any of their developers with access to the configuration) would make it possible to forge an operator token. With RS256, only identity holds the private key; the rest load the public one, which can be published without risk (in 06-03 we will see the standard for doing so, JWKS). In a distributed system with more than one verifier, asymmetric signing is not an option: it is the only sensible choice.
- Expiry, refresh tokens and revocation
The price of having no state is that an issued token cannot be withdrawn: it is valid until exp, even if the user logs out or loses a role. The strategies, which can be combined:
- Short expiry: access tokens of 5 to 15 minutes. This limits the damage window of a stolen token and the latency with which role changes take effect.
- Refresh token: a second, long-lived token (days or weeks), stateful (stored in
identity, in PostgreSQL or Redis) and single-use, whose only purpose is to request a new access token. Logging out or revoking a user means deleting their refresh tokens: within 15 minutes at most, all their access tokens will have expired. Rotation (every use of the refresh token issues a new one and invalidates the previous one) detects theft: if the old one shows up, someone copied it, and the whole family is invalidated. - Denylist in Redis by
jti: for the cases where 15 minutes is too long (urgently blocking a compromised account), the service checksSISMEMBER jwt:revoked <jti>before accepting the token, with a TTL equal to the token's remaining lifetime. It reintroduces a round trip to Redis per request, but only a small structure is queried, and it is the only way to get immediate revocation with tokens. - Signing key rotation: publish the new public key with a new
kid, start signing with it, and retire the old one once every token signed with it has expired. Verifiers must accept several keys at once, chosen bykid.
sequenceDiagram
participant A as Anna's app
participant I as identity
participant O as orders
A->>I: POST /login (email, password)
I->>I: Argon2 verify
I-->>A: access (15 min, RS256) + refresh (30 days, opaque, in DB)
A->>O: GET /orders Authorization: Bearer <access>
O->>O: verify signature with public key, exp, aud
O-->>A: 200 orders of u-anna
Note over A: 15 min go by: access expired
A->>O: GET /orders Bearer <access>
O-->>A: 401 (exp)
A->>I: POST /token/refresh (refresh)
I->>I: exists and unused? rotate
I-->>A: new access + new refresh
- Classic JWT mistakes
The JWT specification is flexible, and that flexibility has produced some famous vulnerabilities. Get to know them so you do not repeat them:
| Mistake | What happens | Defence |
|---|---|---|
Accepting alg: none |
The spec allows "unsigned" tokens; older libraries accepted them and an attacker could write whatever payload they liked | Always pass the list algorithms=["RS256"] to the verification; never read the algorithm from the token itself |
| Algorithm confusion (RS256 → HS256) | The attacker signs with HS256 using the (well-known) public key as the secret; a library that accepts both algorithms with the same key validates it | The same: a closed list of algorithms, and typed key objects (PyJWT ≥ 2 prevents it) |
Not validating aud |
A token issued for the courier app is reused against payments |
Every service checks that its own name is in aud |
Not validating exp / iss |
Eternal tokens, or tokens from another issuer using the same library | Full verification by default; explicit expected iss |
| Weak secrets in HS256 | With HS256 and secret123, the signature is brute-forced in minutes |
RS256/ES256; if HS256, a secret of ≥ 256 random bits |
JWT in localStorage |
Any injected script (XSS) reads localStorage and steals the token; with an HttpOnly cookie it cannot |
On the web, an HttpOnly; Secure; SameSite cookie (even if it contains a JWT), or a stateful session; the JWT in the SPA's memory with refresh via cookie is the usual compromise |
| Sensitive data in the payload | It is readable by the bearer and ends up in logs and proxies | Identifiers and roles only |
| Tokens lasting hours or days with no refresh | A huge damage window; role changes that take a day | 5-15 minutes + refresh with rotation |
| Verifying "by hand" in each service | One forgets aud, another forgets exp |
A shared library (km0/services/common/auth.py) and the gateway as a first barrier (06-05) |
In Kilometre Zero: the web will keep the access token only in the SPA's memory and the refresh token in an HttpOnly cookie; the courier app, in the operating system's secure storage; the services will never persist it.
- Propagating identity between services
When Anna confirms order P-2026-000126, the request crosses several hops: web → orders → inventory (reserve stock) → payments. Who is acting at each hop?
- In HTTP, the token travels in the
Authorization: Bearer <jwt>header. - In gRPC, in the call's metadata (02-03 introduced it, and hinted that it would be used for credentials): the
authorizationkey with the sameBearer <jwt>value. gRPC compresses headers with HPACK over HTTP/2, so the cost of repeating them on every call is small.
There are two propagation models:
| Model | Description | Advantages | Drawbacks |
|---|---|---|---|
| Forwarded user token (on-behalf-of) | orders forwards Anna's very same JWT to inventory |
inventory knows the reservation is Anna's; end-to-end auditing; least privilege is preserved |
The aud must include every service in the chain; a compromised intermediate service holds Anna's token for 15 min |
| Service identity | orders calls inventory with its own token (sub: svc-orders), and passes the user id as data |
Simple service tokens; does not expose Anna's | inventory has to trust that orders is telling the truth about the user: only valid if orders is authenticated as a service (06-04) |
Kilometre Zero will use the first for calls that carry out a user's action (orders → inventory.ReserveStock carries Anna's token), and the second for autonomous processes (Airflow reading the lake, Flink consuming orders.events), which in 06-03 will obtain machine tokens and in 06-04 will additionally identify themselves by certificate. The same metadata mechanism will be used in 07-02 to propagate the trace identifier.
- Authorization models: RBAC, ABAC, ReBAC
With the identity verified comes the second question. The main models, from least to most expressive:
RBAC: role-based access control
Each subject is assigned roles, and each role is assigned permissions (an action on a type of resource). Kilometre Zero defines four:
| Role | Who | Permissions (excerpt) |
|---|---|---|
customer |
Anna, Mark, Lucy | orders:create, orders:read (their own), catalog:read |
producer |
La Vega Farm, Montblanc Dairy, Roble Alto Winery | catalog:read, products:edit (their own), inventory:adjust (their own), orders:read (those containing their products) |
courier |
The 140 of van-3 and the rest |
delivery:position, orders:read (those assigned to them), orders:mark_delivered |
operator |
Kilometre Zero employees | All of the above + orders:cancel, producers:approve, campaigns:create |
RBAC is simple, auditable ("who can approve producers? the operators") and sufficient for type-level permissions. Its limit is in the brackets in the table: "their own", "those assigned to them". A role does not know who owns which cheese.
ABAC: attribute-based access control
The decision is made by evaluating attributes of the subject (role, producer_id), of the resource (owner, status), of the action and of the context (time, IP, market). The rule RBAC could not express: "a subject with the producer role may products:edit a product if product.producer_id == subject.producer_id". With ABAC, Montblanc Dairy edits aged-cheese and fresh-cheese and cannot touch crianza-wine. It is more expressive and harder to audit: to answer "who can edit aged-cheese?" you have to evaluate rules, not read a table.
ReBAC: relationship-based access control
It generalises ABAC by modelling relationships as a graph (user:anna is owner of order:P-2026-000123; user:mark is member of producer:roble-alto-winery, which is owner of product:crianza-wine) and answering "is there a path that allows this action?". It is the model behind Google Zanzibar and systems such as OpenFGA or SpiceDB, and it shines when there are hierarchies and shared permissions (a producer with several employees, a folder with subfolders). For Kilometre Zero, RBAC plus a handful of ABAC rules is enough; ReBAC would be for when producers have teams with delegated permissions.
Least privilege
It cuts across all three: every subject (person or service) has exactly the permissions it needs for its task and no more. analytics reads the lake but cannot write to km0_orders; delivery updates positions but cannot cancel orders; a customer support operator sees orders but cannot create campaigns. And, by default, deny: whatever is not explicitly allowed is forbidden. Every extra permission is attack surface and, when a service or an account is compromised, the blast radius is exactly its list of permissions.
- Where the policy lives: centralised or in each service
Once the rules have been decided, where do they run?
| Option | How | Advantages | Drawbacks |
|---|---|---|---|
| In each service (code) | Decorators and checks in inventory, orders... |
No dependencies; zero latency; full domain context | Duplicated, inconsistent rules; changing "who can cancel" is a deployment across several services |
| Centralised policy (policy engine) | Each service asks an engine with the tuple (subject, action, resource, context); the engine evaluates rules written in a declarative language | A single source of truth; changes without redeploying; auditing of the rules | One more component; the latency of a call (mitigated by running the engine as a local sidecar); the rules need the resource's attributes |
The best-known representative of the second approach is Open Policy Agent (OPA) with its Rego language: services send an input JSON and OPA answers allow: true/false. A rule for the Montblanc Dairy case would be written roughly like this:
package km0.products
default allow := false
allow if {
"operator" in input.subject.roles
}
allow if {
input.action == "edit"
"producer" in input.subject.roles
input.resource.producer_id == input.subject.producer_id
}The usual pragmatic decision, and Kilometre Zero's in this lesson: coarse-grained RBAC at the edge and in each service (does it have the role required for this method?) through a shared library, and fine-grained ABAC inside the service that owns the data (is this product theirs?), where the attributes are. If the rules grow or need changing without redeploying, they are externalised to OPA without changing the structure: the decorator simply asks the engine instead.
- Kilometre Zero:
identity, gRPC interceptor and ABAC check
identity, gRPC interceptor and ABAC checkWe are going to build three pieces: the token issuer, the verification in inventory as a gRPC interceptor, and the ABAC rule for products.
Signing keys
# Once, in development. In production: generated and held by the secrets manager (06-04).
mkdir -p km0/services/identity/keys
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
-out km0/services/identity/keys/km0-2026-09.pem
openssl pkey -in km0/services/identity/keys/km0-2026-09.pem -pubout \
-out km0/services/identity/keys/km0-2026-09.pubThe .pem (private) is mounted only by the identity container; the .pub is distributed to every service (in 06-03 it will be replaced by a JWKS endpoint).
services/identity/tokens.py
# km0/services/identity/tokens.py
# pip install "PyJWT[crypto]"
import time, uuid
import jwt # PyJWT
ISS = "https://identity.km0"
KID = "km0-2026-09"
ACCESS_TTL_S = 15 * 60
with open("services/identity/keys/km0-2026-09.pem", "rb") as f:
PRIVATE_KEY = f.read() # only 'identity' has this file
def issue_access(sub: str, roles: list[str], aud: list[str], **extra) -> str:
"""Issues a 15-minute RS256 JWT. 'extra' allows private claims
such as producer_id or courier_id."""
now = int(time.time())
claims = {
"iss": ISS, "sub": sub, "aud": aud,
"iat": now, "exp": now + ACCESS_TTL_S,
"jti": str(uuid.uuid4()),
"roles": roles, **extra,
}
return jwt.encode(claims, PRIVATE_KEY, algorithm="RS256",
headers={"kid": KID})
if __name__ == "__main__":
print(issue_access("u-anna", ["customer"], ["orders", "inventory"]))
print(issue_access("prod-montblanc-dairy", ["producer"],
["catalog", "inventory"],
producer_id="montblanc-dairy"))And the verification, which does not live in identity but in a shared library that every service imports:
# km0/services/common/auth.py
import jwt
from jwt import PyJWKClient # will be used in 06-03; here we load the public key from a file
ISS = "https://identity.km0"
PUBLIC_KEYS = {}
with open("services/common/keys/km0-2026-09.pub", "rb") as f:
PUBLIC_KEYS["km0-2026-09"] = f.read()
class InvalidToken(Exception):
pass
def verify(token: str, audience: str) -> dict:
"""Returns the claims if the token is valid for this service.
Each check below corresponds to a mistake from section 7."""
try:
kid = jwt.get_unverified_header(token).get("kid")
key = PUBLIC_KEYS[kid] # rotation by kid
return jwt.decode(
token, key,
algorithms=["RS256"], # closed list: never 'none', never HS256
audience=audience, # aud must contain this service
issuer=ISS, # only our issuer
options={"require": ["exp", "iat", "sub", "jti"]},
leeway=30, # 30 s of clock tolerance (01-05)
)
except (KeyError, jwt.PyJWTError) as e:
raise InvalidToken(str(e))leeway=30 deserves a comment: the clocks of identity and inventory are not perfectly synchronised (01-05), and a token issued 200 ms ago may appear to be "from the future" (iat later than the local clock) or to have "expired" half a second early. Thirty seconds of tolerance avoids phantom errors without opening a significant window.
gRPC interceptor in inventory
gRPC lets you register interceptors on the server: code that runs before every method, with access to the metadata. It is the natural place to authenticate, so that ReserveStock and the other methods from 02-03 do not have to repeat it:
# km0/services/inventory/auth_interceptor.py
import grpc
from services.common.auth import verify, InvalidToken
# Coarse-grained RBAC per method: which roles may invoke each RPC of the inventory.proto contract
ROLES_BY_METHOD = {
"/km0.inventory.v1.Inventory/GetStock": {"customer", "producer", "courier", "operator"},
"/km0.inventory.v1.Inventory/ReserveStock": {"customer", "operator"},
"/km0.inventory.v1.Inventory/AdjustStock": {"producer", "operator"},
"/km0.inventory.v1.Inventory/WatchChanges": {"operator"},
}
def _abort(code, details):
def handler(request, context):
context.abort(code, details)
return grpc.unary_unary_rpc_method_handler(handler)
class AuthInterceptor(grpc.ServerInterceptor):
def intercept_service(self, continuation, handler_call_details):
method = handler_call_details.method
meta = dict(handler_call_details.invocation_metadata)
auth = meta.get("authorization", "")
if not auth.startswith("Bearer "):
return _abort(grpc.StatusCode.UNAUTHENTICATED, "missing token")
try:
claims = verify(auth[len("Bearer "):], audience="inventory")
except InvalidToken as e:
return _abort(grpc.StatusCode.UNAUTHENTICATED, f"invalid token: {e}")
allowed = ROLES_BY_METHOD.get(method, set())
if not allowed & set(claims.get("roles", [])):
return _abort(grpc.StatusCode.PERMISSION_DENIED,
f"{claims['sub']} may not invoke {method}")
# We make the claims available to the method through an internal metadata entry.
# (grpc-python offers no shared 'request context'; the km0 convention
# is for the servicer to read them back with 'claims_of(context)'.)
_CURRENT_CLAIMS.set(claims)
return continuation(handler_call_details)
import contextvars
_CURRENT_CLAIMS = contextvars.ContextVar("km0_claims")
def claims_of(context) -> dict:
return _CURRENT_CLAIMS.get()Note the distinction between codes: UNAUTHENTICATED (I do not know who you are: the equivalent of 401) versus PERMISSION_DENIED (I know who you are and you may not: 403). Clients react differently to each: the first triggers a token refresh; the second does not.
In the server from 02-03 only the start-up changes:
# km0/services/inventory/server.py (fragment)
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),
interceptors=[AuthInterceptor()])The requires_role decorator for the other services
For the HTTP services (the orders API, catalog with FastAPI or Flask), the equivalent of the interceptor is a decorator:
# km0/services/common/auth.py (continued)
from functools import wraps
from flask import request, g, abort
def requires_role(*allowed_roles, audience):
def decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
auth = request.headers.get("Authorization", "")
if not auth.startswith("Bearer "):
abort(401)
try:
g.claims = verify(auth[7:], audience)
except InvalidToken:
abort(401)
if not set(allowed_roles) & set(g.claims.get("roles", [])):
abort(403)
return fn(*args, **kwargs)
return wrapper
return decorator
# Usage in services/orders/api.py:
# @app.get("/orders/<order_id>")
# @requires_role("customer", "operator", audience="orders")
# def view_order(order_id):
# order = repository.load(order_id)
# if "operator" not in g.claims["roles"] and order.customer_id != g.claims["sub"]:
# abort(403) # ABAC: only the order's owner (or an operator) sees it
# return order.to_json()That abort(403) in the last few lines is the check that was missing in section 1: without it, Mark would read Anna's P-2026-000123 by changing the URL.
The ABAC rule: "Montblanc Dairy only modifies its own products"
In inventory, the AdjustStock method has already passed the RBAC filter (producer or operator role). The fine-grained rule is applied inside, where the resource's attribute is:
# km0/services/inventory/server.py (fragment of the servicer)
from services.inventory.auth_interceptor import claims_of
def AdjustStock(self, request, context):
claims = claims_of(context)
product = self._repo.load(request.product) # e.g. 'aged-cheese'
if product is None:
context.abort(grpc.StatusCode.NOT_FOUND, "unknown product")
is_operator = "operator" in claims["roles"]
is_their_product = claims.get("producer_id") == product.producer_id
if not (is_operator or is_their_product):
context.abort(grpc.StatusCode.PERMISSION_DENIED,
f"{claims['sub']} does not own {request.product}")
new_stock = self._repo.adjust(request.product, request.market, request.delta)
return inventory_pb2.AdjustStockResponse(stock=new_stock)Test from the client:
# Test client (host): two tokens, two outcomes
from services.identity.tokens import issue_access
import grpc, inventory_pb2, inventory_pb2_grpc
channel = grpc.insecure_channel("localhost:50051") # TLS arrives in 06-02, mTLS in 06-04
stub = inventory_pb2_grpc.InventoryStub(channel)
t_dairy = issue_access("prod-montblanc-dairy", ["producer"], ["inventory"],
producer_id="montblanc-dairy")
t_winery = issue_access("prod-roble-alto-winery", ["producer"], ["inventory"],
producer_id="roble-alto-winery")
def adjust(token, product, delta):
try:
r = stub.AdjustStock(inventory_pb2.AdjustStockRequest(
product=product, market="girona", delta=delta),
metadata=(("authorization", f"Bearer {token}"),), timeout=2)
print("OK ", product, "->", r.stock)
except grpc.RpcError as e:
print("FAIL", product, e.code().name, e.details())
adjust(t_dairy, "aged-cheese", +20) # OK aged-cheese -> 45
adjust(t_winery, "aged-cheese", -45) # FAIL aged-cheese PERMISSION_DENIED prod-roble-alto-winery does not own...
adjust("junk", "aged-cheese", +1) # FAIL aged-cheese UNAUTHENTICATED invalid token: Not enough segmentsWith this, inventory is no longer a service whose stock anyone could empty (fallacy 4 from 01-04). It still does not know whether whoever is talking to it really is orders (that is service identity, 06-04) and it still talks in the clear over the network (06-02), but it no longer executes anything without knowing on whose behalf.
Common Mistakes and Tips
- Confusing 401 and 403. 401/
UNAUTHENTICATEDis "I do not know who you are" and should lead the client to authenticate (or to refresh). 403/PERMISSION_DENIEDis "I know who you are, and no". Mixing them up breaks clients' retry logic and confuses support. - Storing passwords with SHA-256 "because it's secure". SHA-256 is secure as a cryptographic hash and dreadful as a password hash: it is fast. Argon2id or bcrypt, always.
- Encrypted passwords "in case the user forgets them". If you can email users their password, you hold it in the clear, or as good as. The right flow is to reset, never to recover.
- Roles in the token and roles in the database that contradict each other. The token is a snapshot: if you remove a role, it takes until
expto disappear. Design with that latency in mind (short tokens) or revoke byjti. audas a single string in a token that crosses several services: each one will reject the token because it is not its audience.audaccepts a list, and the on-behalf-of token must name them all.- Issuing tokens from several places. A single authority (
identity) signs; everything else verifies. If a service needs to "act as" someone, it requests a token; it does not make one up. - Trusting an
X-User-Idthat arrives in a header. Without a signature, anyone can write it. Only the identity that comes in a verified token (or in a certificate, 06-04) can be trusted. - Authorizing only at the edge. The gateway from 06-05 will be a convenient first barrier, but every service must keep verifying: a call arriving from another internal service never went through the edge.
- Tip: centralise verification in a library (
services/common/auth.py) with tests that include the attacks from section 7 (alg: none, HS256 with the public key, wrongaud, pastexp). One test per known vulnerability is worth more than any manual review. - Tip: log every deny decision with
sub,jti, method and reason; in 06-05 those records will be the basis for auditing and anomaly detection. - Compliance warning. The processing of credentials and customer identifiers is subject to the GDPR and, where payments are involved, to PCI DSS. What is described here is technical design; a real implementation must be reviewed by a security and regulatory compliance professional.
Exercises
Exercise 1: A stolen token
An attacker obtains, through a script injected into the web, Anna's access token at 10:00:00 (exp = 10:15:00). List what they can do with it against orders and inventory under this lesson's design, until when, and which three measures from sections 6 and 7 reduce or eliminate the damage. Then state which mechanism would let an operator cut it off at 10:04 as soon as Anna raises the alarm, and what that mechanism costs on each request.
Exercise 2: RBAC and ABAC for delivery
The 140 couriers of van-3 use the app, which calls the delivery service with the methods PublishPosition(courier_id, lat, lon), ListDeliveries(courier_id) and MarkDelivered(order_id). An operator can see any position and any delivery. Define (a) the ROLES_BY_METHOD table for the delivery interceptor, (b) which private claim a courier's token needs, (c) the ABAC check for each method in pseudocode, and (d) what happens if a courier calls MarkDelivered("P-2026-000124") and that order is assigned to another courier.
Exercise 3: HS256 in a single service
A colleague suggests: "For catalog, which is only called by the web, let's use HS256 with a secret in its configuration; it's simpler and faster than RS256". Argue in which cases they would be right and why, in Kilometre Zero, it is ruled out all the same. Include in your answer what would happen the day analytics wanted to call catalog.
Solutions
Exercise 1.
With Anna's token (sub: u-anna, roles: [customer], aud: [orders, inventory]) the attacker can do everything Anna can: list and read her orders, create orders in her name (up to the payment step, which in a correct design requires the payment gateway), check stock and reserve stock in her name. They cannot edit products or see Mark's orders: RBAC and ABAC still apply to the token, which carries only the customer role and Anna's sub. They can do so until 10:15:00 (plus the 30 s of leeway), and they cannot renew it if the refresh token is in an HttpOnly cookie that the script cannot read. Measures: (1) short expiry, which is what limits the damage to 15 minutes; (2) not keeping the token in localStorage (keeping it in the SPA's memory and the refresh token in an HttpOnly cookie) greatly reduces the likelihood of the theft and eliminates renewal; (3) refresh token rotation, so that if they somehow did get hold of the refresh token, its use by both the attacker and Anna triggers a detection and invalidates the family. To cut it off at 10:04: a denylist by jti in Redis (SADD jwt:revoked <jti> with a TTL until exp) checked by orders and inventory before accepting a token; the cost is one SISMEMBER (a round trip to Redis, sub-millisecond) per request, and an operational dependency on Redis for authentication, which is precisely what stateless tokens set out to avoid; that is why it is enabled only for sensitive services, or reduced to a local cache of the revoked set refreshed every few seconds.
Exercise 2.
(a) PublishPosition: {courier}; ListDeliveries: {courier, operator}; MarkDelivered: {courier, operator}. An operator does not publish positions (they do not drive). (b) courier_id (for example van-3-017), issued by identity when the app logs in. (c) PublishPosition: request.courier_id == claims.courier_id, otherwise PERMISSION_DENIED (better still: ignore the argument and always use the one from the token, removing any possibility of impersonation). ListDeliveries: if operator is in the roles, any id; otherwise, request.courier_id == claims.courier_id. MarkDelivered: load the order; if operator, allow; otherwise, order.assigned_courier == claims.courier_id. (d) The interceptor lets the call through (the courier role may invoke the method); the servicer loads P-2026-000124, sees that it is assigned to another courier and aborts with PERMISSION_DENIED and a message that does not reveal who it is assigned to; the denial is logged with sub, jti and order for the audit trail.
Exercise 3.
They would be right if catalog were the only verifier and the only issuer: a standalone application that issues and verifies its own tokens with a secret of 256 random bits is correct with HS256, and HMAC verification is faster than RSA. It is ruled out because in Kilometre Zero the issuer is identity, not catalog: to verify with HS256, catalog would have to know identity's secret, and with it could issue operator tokens valid for orders, inventory and payments; the blast radius of compromising catalog (the most exposed service, the one serving photos and listings to the Internet) would become the whole platform. There is no real gain in simplicity either: the shared library already verifies RS256 and the public key is distributed without any special care. The day analytics wanted to call catalog with HS256, it would have to be given the secret as well (one more service able to issue), or two secrets and two verification paths would have to be maintained; with RS256, it is enough for the analytics token to include catalog in aud, and catalog changes nothing.
Conclusion
Authenticating means establishing who the subject is; authorizing means deciding, on every operation, whether they may do what they are asking. In a distributed system identity has to travel: an in-memory session dies with the second instance, a session in Redis solves the web but does not propagate to the services, and a signed token (JWT) carries the identity with it and makes it verifiable by anyone holding the public key. That is why the signature must be asymmetric (RS256/ES256): a single authority issues, everyone verifies, and no verifier can forge. A token is a snapshot with an expiry: access tokens lasting minutes, stateful refresh tokens with rotation, a denylist by jti for emergencies, and strict verification of alg, aud, iss and exp so as not to repeat the known mistakes. Passwords, where they exist, are stored with Argon2id or bcrypt, with a salt and a cost, never reversibly. Authorization is built on top of the verified identity: RBAC for type-level permissions (customer, producer, courier, operator), ABAC for ownership ("their products", "their orders"), ReBAC when relationships get complicated, with least privilege and deny by default, applied coarsely at the edge and in each service and finely where the data lives, with OPA as the externalisation path if the rules grow. Kilometre Zero now has services/identity/tokens.py, a services/common/auth.py library, an AuthInterceptor in inventory and its first ABAC rule: Montblanc Dairy only touches its own cheeses.
But Anna's token still travels in the clear between orders and inventory, just like her phone number, her address and the events in orders.events, which anyone with access to the internal network could read or alter. Knowing who is who is not much use if the conversation can be overheard. The next lesson deals with encryption and data protection: in transit, with TLS in gRPC and in every data store; at rest, with field and volume encryption; and in its personal dimension, with pseudonymisation and the right to be forgotten.
Distributed Architectures Course
Module 1: Introduction to Distributed Systems
- Basic Concepts of Distributed Systems
- Distributed System Models
- Advantages and Challenges of Distributed Systems
- The Fallacies of Distributed Computing
- Time, Clocks and Event Ordering
- From Monolith to Distributed Platform: the Kilometre Zero Case
Module 2: Communication in Distributed Systems
- Communication Protocols
- RPC and RMI
- gRPC and Data Serialization
- Messaging and Message Queues
- Asynchronous Communication Patterns
Module 3: Consistency and Replication
- Consistency Models
- The CAP Theorem and PACELC
- Consensus Algorithms
- Data Replication
- Distributed Transactions and Sagas
Module 4: Distributed Storage
- Data Partitioning and Consistent Hashing
- Distributed File Systems
- Object Storage
- Distributed Databases
- Distributed Caches
Module 5: Distributed Computing
- Distributed Computing Models
- MapReduce and Hadoop
- Spark and In-Memory Computing
- Stream Processing
- Job Scheduling and Data Pipelines
Module 6: Security in Distributed Systems
- Authentication and Authorization
- Encryption and Data Protection
- Identity Management
- Service-to-Service Security: mTLS and Secrets Management
- API Gateways, Rate Limiting and Auditing
Module 7: Monitoring and Maintenance
- Monitoring Distributed Systems
- Centralized Logs and Distributed Tracing
- Failure Management and Recovery
- Resilience Patterns: Timeouts, Retries and Circuit Breakers
- Automation and Orchestration
- Testing Distributed Systems and Chaos Engineering
