The previous four lessons have secured the inside of Kilometre Zero: verifiable tokens, encrypted channels, centralised identities, mTLS between services and secrets in Vault. But the platform has an outside. From the Internet come the browsers of Anna, Mark and Lucy, the app of the 140 couriers of van-3, the producers' dashboards, and also the bots that scrape prices, the scripts that try stolen passwords, and the 40,000 requests per second of the first hour of Grape Harvest Week. Until now, each exposed service (catalog, orders, delivery) did its own TLS termination, its own token verification, its own defence against abuse, and none of them kept a reliable record of who did what.
This lesson builds the edge: an API gateway as the single point of entry that terminates TLS, validates the tokens from 06-01 and 06-03, routes to each service, versions the APIs and curbs abuse; rate limiting with its algorithms and a distributed implementation in Redis; the additional protections at the edge (schema validation, sizes, timeouts, WAF); and auditing: what to record, how to make it immutable with hash chaining and object-locked storage, and how to answer "who looked at Anna's order?". It ends with the idea of security as a process (STRIDE threat modelling applied to orders, vulnerability management) and closes the module with the complete map of who can do what in Kilometre Zero. Metrics monitoring, operational logs and the internal resilience patterns are Module 7.
Contents
- The edge of the platform: gateway versus load balancer
- What an API gateway does
- Options: Kong, Envoy Gateway, NGINX, managed gateways; BFF
- Rate limiting: why and against what
- Algorithms: token bucket, leaky bucket, fixed window, sliding window
- Distributed token bucket in Redis with Lua
- Headers, limits by identity and quotas
- Additional protection at the edge
- Auditing: what to record and why it is not just another log
- Immutable audit logs: hash chaining and object lock
- Auditing access to personal data and anomaly detection
- Security as a process: STRIDE, vulnerabilities and dependencies
- Kilometre Zero: Kong in
docker-compose.yml,rate_limiter.pyandaudit.py - Common Mistakes and Tips
- Exercises
- Conclusion
- The edge of the platform: gateway versus load balancer
A load balancer (L4 or L7) spreads connections across instances of the same service and, at most, terminates TLS. It does not know what a JWT is, cannot tell Anna from a bot, and cannot say "this route goes to orders and this other one to catalog v2". In the monolith that was enough; with six exposed services, each one would have to repeat the same dozen cross-cutting responsibilities, and clients would have to know six addresses.
An API gateway is a specialised reverse proxy placed in front of all the services that concentrates those responsibilities:
flowchart LR
subgraph Internet
N[Anna's browser]
R[Courier app]
B[Bots / abuse]
end
N & R & B -->|HTTPS| G[API Gateway<br/>TLS · JWT · routes · rate limit<br/>CORS · schemas · auditing]
G -->|mTLS| C[catalog]
G -->|mTLS| O[orders]
G -->|mTLS| D[delivery]
G -.->|audit.events| K[(Kafka)]
G -.->|counters| RD[(Redis)]
O -->|mTLS| I[inventory]
style B stroke-dasharray: 5 5
Nothing from outside reaches a service without going through the gateway; the services only accept mTLS connections from the gateway and from other services (06-04). And the gateway does not replace each service's own security: it is the first barrier, not the only one. inventory still verifies the token (06-01) because the call arriving from orders did not go through the edge.
- What an API gateway does
| Responsibility | What it means | Without a gateway |
|---|---|---|
| TLS termination | A single place with the public certificate for api.km0.example, renewed by Let's Encrypt/ACME; inwards, mTLS with the internal CA |
One public certificate per service |
| Authentication | Verifies the JWT (RS256 against Keycloak's JWKS, 06-03) and rejects anything invalid at the edge; optionally, introspection of opaque tokens | Every service does it, and invalid requests consume internal resources |
| Routing | /api/catalog/* → catalog, /api/orders/* → orders; by path, method, header, version |
Clients know six hosts |
| Transformation | Adding headers (X-Request-Id, verified identity), stripping internal headers from responses, adapting formats |
Repeated |
| API versioning | /api/v1/orders and /api/v2/orders to different services or routes; gradual retirement |
Breakage for old clients |
| CORS | Access-Control-* headers so that the browser on km0.example can call api.km0.example |
Per-service configuration, often * through carelessness |
| Abuse protection | Rate limiting, quotas, maximum size, timeouts, IP blocking | The service is overwhelmed before it can defend itself |
| Observability | A single point through which all external traffic passes: metrics (07-01), traces (07-02), auditing | Fragmented |
| Response caching | For public catalogue GETs, complementing Redis (04-05) | — |
What it should not do: business logic, complex aggregation of several services (that is the role of the BFF, section 3), or fine-grained authorization (the ABAC of "their own products" needs data only the service has).
- Options: Kong, Envoy Gateway, NGINX, managed gateways; BFF
| Option | Nature | Configuration | Strengths | Things to bear in mind |
|---|---|---|---|---|
| Kong Gateway | Proxy on top of NGINX/OpenResty with plugins (Lua, Go, Python) | Declarative (YAML, DB-less mode) or administration API | Plugin ecosystem (JWT, OIDC, rate limiting, auditing), mature, complete open-source version | Advanced plugins in the commercial edition |
| Envoy Gateway / Envoy | High-performance L7 proxy (C++), the basis of Istio | Kubernetes Gateway API or xDS | Performance, filters (JWT, ext_authz for OPA, global rate limit), the same Envoy as the mesh | Verbose configuration outside Kubernetes |
| NGINX / OpenResty | Web server and proxy | nginx.conf |
Ubiquitous, fast, simple for routing and terminating TLS | JWT and distributed rate limiting require commercial modules or your own Lua |
| Traefik, KrakenD, Tyk, APISIX | Open-source gateways | Varied | Traefik: automatic container discovery; KrakenD: aggregation; APISIX: performance | — |
| Managed (AWS API Gateway, Google Apigee/Cloud Endpoints, Azure API Management) | Cloud service | Console/IaC | Nothing to operate; integration with IAM, WAF, quotas and usage-based billing | Cost per request, vendor lock-in (08-03) |
BFF (Backend for Frontend): when each type of client (web, mobile app, producer dashboard) needs different aggregations (the app's home screen combines the catalogue, orders in progress and the courier's position), you place one service per client that composes those responses, behind the gateway. The gateway remains cross-cutting; the BFF is specific. Kilometre Zero, in this lesson, uses Kong with no BFF; the courier app's BFF will appear in the final project (08-05).
- Rate limiting: why and against what
Rate limiting bounds how many requests a client can make in an interval. It protects against:
- Deliberate abuse: brute force against
/login(thousands of passwords per minute), scraping the whole catalogue every minute, denial of service with valid requests. - Client bugs: an app with a retry loop without backoff (07-04) which, after a failure, fires a thousand requests per second from every device.
- Overload during legitimate spikes: Grape Harvest Week fills the web; without a limit,
ordersis overwhelmed, every request fails, and nobody buys. With a limit, 95% buy and 5% get a "try again in a few seconds". - Fairness: so that an integrator consuming the API does not monopolise
catalogat everyone else's expense. - Cost: if there are calls behind it to the payment gateway or to a maps provider billed by usage.
It is applied at the edge (per client, token or IP, before touching a service) and sometimes inside as well (a service protecting itself from another, which in 07-04 will be called a bulkhead). And it differs from a quota: the rate limit is short-term (100 per minute: smoothing), the quota is long-term (10,000 per day: a contract).
- Algorithms: token bucket, leaky bucket, fixed window, sliding window
| Algorithm | How it works | Bursts | Memory per key | Accuracy | Typical use |
|---|---|---|---|---|---|
| Fixed window | One counter per key and per interval (10:04 → 37); it resets when the minute changes |
Allows double the limit at the window boundary (100 at 10:04:59 + 100 at 10:05:00) | 1 integer | Low | Daily quotas, where the window boundary does not matter |
| Sliding window (log) | Stores the timestamp of every request; counts those in the last 60 s | Exact, no boundary effect | One entry per request (expensive) | High | Low, strict limits (login) |
| Sliding window (weighted counter) | Counter for the current window + counter for the previous one weighted by the fraction elapsed | A good approximation, without the boundary spike | 2 integers | Medium-high | The most widely used in gateways (Kong, Cloudflare) |
| Token bucket | A bucket of capacity C refilled at r tokens/s; each request consumes one; no tokens, rejection |
Allows bursts of up to C and then the sustained rate r |
2 values (tokens, last refill) | High | APIs: tolerates natural bursts (loading a page fires 20 requests) |
| Leaky bucket | A queue of capacity C drained at r/s; the request gets in if there is room and is served at a constant rate |
Smooths: the output is always r, bursts wait or are dropped |
Queue | High | Shaping traffic towards a service that does not tolerate spikes (the payment gateway) |
Token bucket and leaky bucket are duals: the former limits admission while allowing bursts, the latter limits output by eliminating them. For a public API, token bucket is the usual choice: Anna loads the home page (20 requests in one second, which fit in the bucket) and then browses at a low rate that the refill trickle covers with room to spare; a bot doing 50 per second drains the bucket in an instant and is held to r.
flowchart LR
R[Refill: r tokens/s] --> B[(Bucket<br/>capacity C)]
P[Request] -->|token available?| B
B -->|yes: consumes 1| OK[200 → service]
B -->|no| KO[429 Too Many Requests<br/>Retry-After]
- Distributed token bucket in Redis with Lua
The gateway has several instances; the bucket's state must be shared, and the operation "read tokens, refill according to the time elapsed, decide, write" must be atomic, or two instances would grant the same last token. Redis (04-05) solves both: the state lives there, and a Lua script runs atomically on the server.
# km0/services/edge/rate_limiter.py
import time, redis
# The script runs in its entirety inside Redis with no other command interleaved:
# reading, refilling, deciding and writing are a single operation.
_LUA_TOKEN_BUCKET = """
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2]) -- tokens per second
local now = tonumber(ARGV[3]) -- seconds with decimals (passed in by the client so that
-- the script is deterministic and replicable)
local cost = tonumber(ARGV[4]) -- tokens this request consumes (normally 1)
local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts = tonumber(state[2])
if tokens == nil then tokens = capacity; ts = now end -- first time: full bucket
-- Refill in proportion to the time elapsed, without exceeding the capacity
tokens = math.min(capacity, tokens + (now - ts) * rate)
local allowed = 0
if tokens >= cost then
tokens = tokens - cost
allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now)
-- The bucket refills by itself in capacity/rate seconds: after that, the key is surplus
redis.call('EXPIRE', key, math.ceil(capacity / rate) + 1)
-- Seconds until there will be 'cost' tokens (for Retry-After); 0 if allowed
local wait = 0
if allowed == 0 then wait = math.ceil((cost - tokens) / rate) end
return {allowed, math.floor(tokens), wait}
"""
class RateLimiter:
def __init__(self, r: redis.Redis, capacity: int, rate_per_s: float, prefix="rl"):
self._r = r
self._script = r.register_script(_LUA_TOKEN_BUCKET) # loaded once (EVALSHA afterwards)
self.capacity, self.rate, self.prefix = capacity, rate_per_s, prefix
def allow(self, identity: str, cost: int = 1) -> tuple[bool, int, int]:
"""Returns (allowed, tokens_remaining, seconds_to_wait)."""
allowed, remaining, wait = self._script(
keys=[f"{self.prefix}:{identity}"],
args=[self.capacity, self.rate, time.time(), cost])
return bool(allowed), int(remaining), int(wait)
if __name__ == "__main__":
r = redis.Redis(host="localhost", port=6379)
lim = RateLimiter(r, capacity=20, rate_per_s=5) # burst of 20, then 5/s sustained
ok = ko = 0
for i in range(60): # 60 requests as fast as possible
allowed, remaining, wait = lim.allow("u-anna")
ok += allowed; ko += not allowed
print(f"allowed={ok} rejected={ko}") # allowed=20 rejected=40 (approx.)
time.sleep(2)
print(lim.allow("u-anna")) # (True, 9, 0): 2 s × 5/s = 10 tokens refilledDetails that matter:
time.time()is passed in by the client, notredis.call('TIME'): Lua scripts must be deterministic so that Redis replication can reproduce them; and clock skew between gateway instances (01-05) only introduces an error of milliseconds in the refill, irrelevant here.- One key per identity (
rl:u-anna) withEXPIRE: inactive clients take up no memory. costlets expensive operations (a text search of the catalogue) consume more tokens than a simple GET.- In Redis Cluster (04-05), each key is routed to its node by hash slot; the script only touches one key, so it works unchanged.
- Cost: one round trip to Redis per request at the edge (~0.3 ms on the same network). For extreme spikes, approximate local limiting is done in each gateway instance (an in-memory bucket with
C/n) and synchronised with Redis asynchronously; that is what thelocal/redis/clusterpolicies of Kong's rate-limiting plugin choose between.
As middleware in a Flask service (for orders, which on top of the gateway's protection wants a stricter limit of its own on POST /orders):
# km0/services/orders/api.py (fragment)
from flask import Flask, request, g, jsonify
from services.edge.rate_limiter import RateLimiter
import redis
app = Flask(__name__)
create_limiter = RateLimiter(redis.Redis(host="redis"), capacity=5, rate_per_s=0.1, prefix="rl:create")
@app.before_request
def limit_creation():
if request.method == "POST" and request.path == "/orders":
subject = getattr(g, "claims", {}).get("sub") or request.remote_addr # per user; otherwise, per IP
allowed, remaining, wait = create_limiter.allow(subject)
if not allowed:
resp = jsonify(error="too many requests", retry_in_s=wait)
resp.status_code = 429
resp.headers["Retry-After"] = str(wait)
resp.headers["RateLimit-Limit"] = "5"
resp.headers["RateLimit-Remaining"] = "0"
resp.headers["RateLimit-Reset"] = str(wait)
return resp
g.rl_remaining = remaining
@app.after_request
def rl_headers(resp):
if hasattr(g, "rl_remaining"):
resp.headers["RateLimit-Limit"] = "5"
resp.headers["RateLimit-Remaining"] = str(g.rl_remaining)
return respFive orders in one go and then one every ten seconds: nobody creates legitimate orders any faster, and a script attempting hundreds of purchases with stolen cards stops at the fifth.
- Headers, limits by identity and quotas
429 Too Many Requestsis the status code;Retry-After(seconds or a date) tells the client when to come back, and a well-written client respects it (07-04 will cover client-side retries with backoff).RateLimit-Limit,RateLimit-Remaining,RateLimit-Reset(an IETF draft, already in common use; Kong and others useX-RateLimit-*variants) report on every response, not just on the 429, so that the client can regulate itself before hitting the wall.- Which identity to limit by, from best to worst: by the token's
sub(Anna, orsvc-integrator-x) → by OAuthclient_id(the whole courier app as one, useful for protecting against a bug in the app) → by API key → by IP (the only thing possible before authentication, on/login; imprecise because of NAT: a whole office shares one IP; and avoidable with proxies). Combine them:/loginby IP and by target account; the rest bysub. - Different limits per route and role:
GET /api/cataloggenerous (200/min) and cacheable;POST /api/ordersstrict; operators with higher limits; integrators with contractual quotas (100,000/day) counted with a daily fixed window in addition to the token bucket. - What to return when Redis does not respond: fail open (allow everything, and alert) is usually the right call at the edge of a marketplace (better to sell without a limit for a few minutes than not to sell); fail closed on
/loginand on payment operations.
- Additional protection at the edge
| Protection | What it does | In Kilometre Zero |
|---|---|---|
| Schema validation (OpenAPI) | The gateway (or the service) rejects bodies that do not meet the contract: types, required fields, ranges, lengths | contracts/orders.openapi.yaml; an order with quantity: -5 or a 10,000-character product dies at the edge |
| Maximum body size | Rejects requests larger than N KB/MB before reading them | 64 KB for the API; photos go by presigned URL to MinIO (04-03), not through the gateway |
| Timeouts at the edge | Maximum connection, read and total time towards each service | 5 s to catalog, 10 s to orders; prevents slow connections from tying up the gateway (slowloris). Internal timeouts, retries and circuit breakers are 07-04 |
| Limit on concurrent connections per client and per service | Complements the rate limit (slow requests, many at once) | 50 per IP |
| WAF (web application firewall: ModSecurity with the OWASP Core Rule Set, Cloudflare, AWS WAF) | Rules against attack patterns: SQL injection, XSS, scanner paths, malicious agents | In front of the gateway, managed by the CDN/cloud; only mentioned here: tuning it is a craft of its own |
| Security headers on responses | Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options |
Added by the gateway to every response |
| Blocking by reputation / geography | Lists of IPs, countries, ASNs | Blocking ranges with repeated abuse |
| Bot protection | Challenges, browser fingerprints | A CDN service; only mentioned |
Input validation is the most important and the least practised: most application vulnerabilities start with an input nobody validated. The OpenAPI schema, versioned in contracts/ alongside the .proto files from 02-03, is to public HTTP what protobuf is to the inside: the contract as the source of truth, and the gateway as its enforcer.
- Auditing: what to record and why it is not just another log
An audit log answers, months later and before an auditor, a judge or a customer, the question: who did what, when, from where and with what result? It is not the same as operational logs (07-02), even though technically both are timestamped lines:
| Operational log (07-02) | Audit log | |
|---|---|---|
| Purpose | Debugging, understanding the system's behaviour | Accountability, investigating incidents, meeting obligations |
| Content | Whatever the programmer thought useful: stack traces, timings, variables | A fixed schema: subject, action, resource, result, context |
| Volume | High; sampled and discarded | Every relevant event, no sampling |
| Retention | Days or weeks | Years (depending on the obligation: tax, GDPR, PCI DSS) |
| Mutability | Rotated, deleted, edited without drama | Immutable: nobody, not even an administrator, can alter or delete it |
| Access | The whole team | Restricted: security, compliance; access itself audited |
| Example | WARN orders: retry 2/3 to inventory (deadline 1.8s) |
2026-09-15T10:04:12Z sub=u-mark jti=7f4… action=orders.read resource=P-2026-000123 result=DENIED ip=… |
What to record (every event, with a fixed schema):
- Who: the JWT's
subandjti(to correlate with the specific token),client_id, roles at the time; for services, the SPIFFE ID (06-04). - What: action (
orders.create,orders.read,products.edit,login.failed,token.revoked,secret.read) and resource (P-2026-000123,aged-cheese,kv/km0/orders/keycloak). - When: the gateway's/service's timestamp in UTC (01-05 explains why the source of the clock is recorded too).
- From where: source IP,
User-Agent, intermediate service,X-Request-Id. - Result:
ALLOWED/DENIED/ERROR, with the reason for the denial. - Never: passwords, complete tokens, personal data that is not essential (the order
P-2026-000123, yes; Anna's phone number, no).
Which events: every authentication (success and failure), every authorization denial, every access to personal data (section 11), every administrative operation (role changes, producer approvals, gateway configuration changes), every access to secrets (Vault already does this), and sensitive business operations (cancellations, refunds, stock adjustments).
- Immutable audit logs: hash chaining and object lock
An audit log that the attacker (or an administrator) can edit is worthless: a competent intruder's first action is to cover their tracks. Three layers of immutability:
- Append-only at the source: the service publishes each event to the Kafka topic
audit.events(02-04) and does not write it anywhere it could edit. Kafka is an append-only log; with ACLs (06-04), services only have write permission on that topic, not read or administration. - Hash chaining: each record includes the hash of the previous record. Modifying or removing one breaks the chain from that point on, and a verifier detects it. It is the structure of a blockchain with none of the rest: one
prev_hashper record and, periodically, an anchor (the hash of the day's last record, signed and published somewhere external: a different topic, a timestamping notary, a third party) so that not even rewriting the whole chain from the start goes unnoticed. - Object-locked storage: a dedicated consumer writes the events in batches (an hour, a day) to the
km0-auditbucket in MinIO with object lock in compliance mode (04-03): neither the bucket owner nor the administrator can delete or overwrite the object until the retention expires (5 years, for example). Together with SSE-KMS (06-02) and a read-only access lease for the security team.
flowchart LR
S[Service / gateway] -->|signed event, prev_hash| K[Kafka: audit.events<br/>ACL: write-only]
K --> C[Audit consumer<br/>verifies chain]
C -->|hourly batches| M[(MinIO km0-audit<br/>object lock 5 years, SSE-KMS)]
C -->|signed daily anchor| A[External anchor]
C --> Q[(Query index<br/>read-only, audited access)]
- Auditing access to personal data and anomaly detection
The GDPR requires being able to demonstrate who accessed a person's data. "Who looked at Anna's order?" must be answerable with a query against the audit index:
sub action resource result timestamp remote u-anna orders.read P-2026-000123 ALLOWED 2026-09-14T18:02:11Z web-km0 / 83.x.x.x mhill orders.read P-2026-000123 ALLOWED 2026-09-15T09:41:03Z operator-dashboard / office u-mark orders.read P-2026-000123 DENIED 2026-09-15T10:04:12Z web-km0 / 90.x.x.x svc-delivery orders.read P-2026-000123 ALLOWED 2026-09-15T11:20:45Z spiffe://km0.internal/delivery
Martha (an operator) read the order at 9:41: legitimate if there was an open support ticket (and the record should carry the ticket_id as context); Mark tried to read it and was denied (the ABAC from 06-01 at work); delivery read it to assign a courier. Correlation with the identity is done by sub and, when it matters to tell sessions apart, by jti: if Anna's token was stolen (exercise 1 of 06-01), every access with that jti from a different IP is the attacker's footprint.
On top of that record you build basic anomaly detection, without machine learning: rules over time windows that produce alerts (07-01 will route them into the general alerting system):
| Rule | A sign of |
|---|---|
> 10 login.failed for the same account in 5 min, or > 100 from the same IP |
Brute force, credential stuffing |
A sub with the operator role reads > 200 different orders in an hour |
Exfiltration, improper curiosity |
A jti used from two countries within 10 minutes |
Stolen token |
Repeated DENIED from the same sub on other users' resources |
Id enumeration (P-2026-000123, 124, 125...) |
| Access to Vault outside deployment hours, or from a new identity | A compromised service |
| A producer adjusts the stock of > 50 products in a minute | A compromised producer account |
- Security as a process: STRIDE, vulnerabilities and dependencies
None of the pieces in this module is final: vulnerabilities are discovered, services change, routes are added. Security is a process, with three minimum practices:
Lightweight threat modelling. Before designing or changing a service, sit down for half an hour with its diagram and ask, for every flow and component, what could go wrong. STRIDE is the classic checklist. Applied to orders:
| Threat | Meaning | Example in orders |
Countermeasure | Lesson |
|---|---|---|---|---|
| Spoofing | Impersonating an identity | A process passes itself off as payments to mark P-2026-000125 as paid |
mTLS + service-to-service policy; verified JWT for users | 06-01, 06-04 |
| Tampering | Altering data | Changing an order's amount in transit; modifying an event in orders.events |
TLS; HMAC of events; schema validation; the amount is recomputed on the server | 06-02, 06-05 |
| Repudiation | Denying having done something | Anna says she never placed the order; an operator denies having cancelled | Immutable auditing with sub/jti; signing of critical actions |
06-05 |
| Information disclosure | Revealing information | Mark reads Anna's order; the phone number shows up in a log; an error displays a stack trace with the connection string | ABAC; field encryption; pseudonymisation; generic error messages; never secrets in logs | 06-01, 06-02, 06-04 |
| Denial of service | Preventing service | 40,000 fake orders per second during Grape Harvest Week; 100 MB bodies | Rate limiting; maximum size; timeouts; quotas; (07-04 for internal resilience) | 06-05 |
| Elevation of privilege | Gaining more permissions | A customer adds operator to their token; an injection in a search parameter runs SQL |
RS256 signature and strict verification; parameterised queries; input validation; least privilege in the DB (dynamic credentials with only the necessary GRANTs) |
06-01, 06-04, 06-05 |
Vulnerability and dependency management. Kilometre Zero's own code is a fraction of what it runs: grpcio, PyJWT, cryptography, redis, Kong, Keycloak, Vault, Kafka, PostgreSQL, Docker base images. Each of them publishes vulnerabilities (CVEs) regularly. A minimum process: pin versions (requirements.txt with hashes, images by digest), a dependency scanner in CI (pip-audit, Dependabot/Renovate for updates, Trivy or Grype for images), a patching window defined by severity (critical: 48 h), and an inventory of what runs where (SBOM). And review: code review for changes to authentication and authorization, review of the gateway and Vault configuration, and periodic penetration tests by someone outside the team.
- Kilometre Zero: Kong in
docker-compose.yml, rate_limiter.py and audit.py
docker-compose.yml, rate_limiter.py and audit.pyKong as the gateway, in declarative mode
Kong in DB-less mode reads all its configuration from a YAML file: routes, services, plugins and consumers. It can be versioned and reviewed, like the Keycloak realm.
# km0/docker-compose.yml (fragment)
kong:
image: kong:3.8
environment:
KONG_DATABASE: "off"
KONG_DECLARATIVE_CONFIG: /kong/kong.yaml
KONG_PROXY_LISTEN: "0.0.0.0:8443 ssl"
KONG_SSL_CERT: /certs/api.km0.example.crt # public certificate (ACME in production)
KONG_SSL_CERT_KEY: /certs/api.km0.example.key
KONG_ADMIN_LISTEN: "127.0.0.1:8001" # the administration API is NEVER exposed
KONG_LUA_SSL_TRUSTED_CERTIFICATE: /certs/km0-ca.crt # to verify the services (internal TLS)
KONG_CLIENT_SSL: "on" # mTLS towards the services (06-04)
KONG_CLIENT_SSL_CERT: /certs/kong.crt
KONG_CLIENT_SSL_CERT_KEY: /certs/kong.key
volumes:
- ./edge/kong.yaml:/kong/kong.yaml:ro
- ./certs:/certs:ro
ports: [ "8443:8443" ]
depends_on: [ redis, catalog, orders ]# km0/edge/kong.yaml
_format_version: "3.0"
services:
- name: catalog
url: https://catalog:8000
connect_timeout: 2000
read_timeout: 5000
routes:
- name: catalog-v1
paths: [ "/api/v1/catalog" ]
strip_path: true
plugins:
- name: rate-limiting
config: { minute: 200, policy: redis, redis_host: redis, limit_by: consumer,
fault_tolerant: true, hide_client_headers: false } # fail open
- name: request-size-limiting
config: { allowed_payload_size: 64, size_unit: kilobytes }
- name: orders
url: https://orders:8000
connect_timeout: 2000
read_timeout: 10000
routes:
- name: orders-v1
paths: [ "/api/v1/orders" ]
strip_path: true
plugins:
- name: jwt # rejects at the edge anything not signed by Keycloak
config:
key_claim_name: iss # looks the consumer up by the 'iss' claim
claims_to_verify: [ exp ]
maximum_expiration: 900 # tokens longer than 15 min: rejected
- name: rate-limiting
config: { minute: 60, policy: redis, redis_host: redis, limit_by: consumer,
fault_tolerant: false } # fail closed: orders is sensitive
- name: request-size-limiting
config: { allowed_payload_size: 64, size_unit: kilobytes }
consumers:
- username: keycloak-km0
jwt_secrets:
- key: "http://localhost:8080/realms/km0" # = the tokens' 'iss' claim
algorithm: RS256
rsa_public_key: | # the realm's public key (or the openid-connect plugin with JWKS)
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA...
-----END PUBLIC KEY-----
plugins: # global
- name: cors
config: { origins: [ "https://km0.example" ], credentials: true, max_age: 3600 }
- name: correlation-id
config: { header_name: X-Request-Id, generator: uuid, echo_downstream: true }
- name: response-transformer
config:
add:
headers:
- "Strict-Transport-Security: max-age=31536000; includeSubDomains"
- "X-Content-Type-Options: nosniff"
remove:
headers: [ "Server", "X-Powered-By" ]
- name: http-log # every request → audit consumer (next subsection)
config: { http_endpoint: "http://audit:9000/edge", timeout: 1000, queue_size: 100 }With this, https://api.km0.example/api/v1/catalog/products/aged-cheese reaches catalog without a token (public catalogue) but limited to 200 per minute; https://api.km0.example/api/v1/orders/P-2026-000123 without Authorization gets a 401 from Kong itself without touching orders, and with a valid token it reaches orders with X-Request-Id, which orders propagates to inventory in the gRPC metadata (07-02 will use it for traces). The jwt plugin in the open-source edition verifies the signature and exp; verification of aud and of roles is still done by requires_role in orders (06-01), and to discover keys via JWKS and validate aud at the edge you would use the openid-connect plugin or Envoy's JWT filter, which do both. The principle does not change: the edge filters out the obvious; the service decides.
services/edge/audit.py
The audit log with hash chaining, used both by the services (a record function) and by the consumer that archives to MinIO:
# km0/services/edge/audit.py
import hashlib, json, time, uuid, threading
from datetime import datetime, timezone
from kafka import KafkaProducer # 02-04
import boto3 # 04-03
TOPIC = "audit.events"
class AuditLog:
"""Emits hash-chained audit events to Kafka. One emitter per process;
the chain is per emitter ('source' field), and the consumer verifies each chain."""
def __init__(self, source: str, producer: KafkaProducer):
self.source, self._p = source, producer
self._prev = "0" * 64 # genesis of this chain
self._lock = threading.Lock()
@staticmethod
def _hash(ev: dict) -> str:
canonical = json.dumps({k: v for k, v in ev.items() if k != "hash"},
sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(canonical).hexdigest()
def record(self, sub: str, action: str, resource: str, result: str,
jti: str | None = None, remote: str | None = None, context: dict | None = None):
ev = {
"id": str(uuid.uuid4()),
"timestamp": datetime.now(timezone.utc).isoformat(timespec="milliseconds"),
"source": self.source, # 'orders', 'kong', 'spiffe://.../inventory'
"sub": sub, "jti": jti, "action": action, "resource": resource,
"result": result, "remote": remote,
"context": context or {}, # ticket_id, request_id... NEVER personal data or tokens
}
with self._lock: # the chain demands order: one event at a time
ev["prev_hash"] = self._prev
ev["hash"] = self._hash(ev)
self._prev = ev["hash"]
# Key = source: same source → same partition → order preserved (02-04)
self._p.send(TOPIC, key=self.source.encode(), value=json.dumps(ev).encode())
return ev["id"]
def verify_chain(events: list[dict]) -> tuple[bool, int | None]:
"""Walks through events from a single source in order; returns (ok, index of the first failure)."""
prev = "0" * 64
for i, ev in enumerate(events):
if ev["prev_hash"] != prev or AuditLog._hash(ev) != ev["hash"]:
return False, i
prev = ev["hash"]
return True, None
class AuditArchiver:
"""Consumer: accumulates events by source and hour, verifies the chain and writes to
km0-audit with object lock (5-year retention, COMPLIANCE mode)."""
def __init__(self, s3, bucket="km0-audit"):
self._s3, self._bucket = s3, bucket
self._batches: dict[tuple[str, str], list[dict]] = {}
def accumulate(self, ev: dict):
hour = ev["timestamp"][:13] # '2026-09-15T10'
self._batches.setdefault((ev["source"], hour), []).append(ev)
def close_batches_before(self, current_hour: str):
for (source, hour), evs in list(self._batches.items()):
if hour >= current_hour:
continue
ok, idx = verify_chain(evs)
key = f"{source.replace('/', '_')}/{hour[:10]}/{hour[11:13]}.jsonl"
body = "\n".join(json.dumps(e) for e in evs).encode()
self._s3.put_object(
Bucket=self._bucket, Key=key, Body=body,
ObjectLockMode="COMPLIANCE",
ObjectLockRetainUntilDate=datetime.fromtimestamp(time.time() + 5 * 365 * 86400, timezone.utc),
ServerSideEncryption="aws:kms", # SSE-KMS (06-02)
Metadata={"chain_ok": str(ok), "first_failure": str(idx),
"last_hash": evs[-1]["hash"]})
if not ok:
print(f"[audit] ALERT: broken chain in {source} {hour} position {idx}")
del self._batches[(source, hour)]Usage in orders, on the route that 06-01 protected with ABAC:
# km0/services/orders/api.py (fragment)
audit = AuditLog("orders", KafkaProducer(bootstrap_servers="kafka:19092", acks="all"))
@app.get("/orders/<order_id>")
@requires_role("customer", "operator", audience="orders")
def view_order(order_id):
order = repository.load(order_id)
allowed = order is not None and ("operator" in g.claims["roles"] or order.customer_id == g.claims["sub"])
audit.record(sub=g.claims["sub"], jti=g.claims.get("jti"), action="orders.read",
resource=order_id, result="ALLOWED" if allowed else "DENIED",
remote=request.headers.get("X-Forwarded-For", request.remote_addr),
context={"request_id": request.headers.get("X-Request-Id"),
"ticket_id": request.headers.get("X-Ticket-Id")})
if not allowed:
abort(403 if order is not None else 404)
return order.to_json()And the docker-compose.yml gains the topic and the bucket, created with object lock from the outset (it cannot be enabled afterwards):
docker compose exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic audit.events --partitions 6 --config retention.ms=-1 --config cleanup.policy=delete
mc mb --with-lock local/km0-audit
mc retention set --default COMPLIANCE 1825d local/km0-auditMark's attempt to read P-2026-000123 now produces a DENIED event in audit.events, chained to the previous one from orders, archived in km0-audit/orders/2026-09-15/10.jsonl with five years' retention, and visible in the index for the query in section 11. If anyone alters the file (they cannot, because of the object lock) or tries to rewrite the topic, verify_chain detects it.
Common Mistakes and Tips
- Exposing the gateway's administration API.
KONG_ADMIN_LISTENon0.0.0.0means handing the edge's configuration to the Internet. Only onlocalhostor on a management network with mTLS. - Trusting that "the gateway already validates". Internal calls do not go through it. Every service verifies the token and the service identity.
- Rate limiting by IP alone. A corporate NAT blocks a hundred legitimate customers and an attacker with a thousand proxies never notices. By identity when there is one; by IP only before authentication.
- No
Retry-Afterand noRateLimit-*headers. Clients cannot regulate themselves and retry in a loop, aggravating the very problem you were trying to avoid. - An in-memory counter in a gateway with several instances. Each instance lets the full limit through. State in Redis with an atomic script.
- Auditing in the same log as operational output, with the same retention. After 14 days the evidence has been deleted. Its own schema, its own channel, its own retention.
- Auditing that can be edited by whoever writes it. Write-only in Kafka (ACL), hash chaining, object lock. If the administrator can delete it, it is not auditing.
- Personal data or tokens in audit events. The audit log is itself a treasure trove for an attacker; identifiers, not contents; the
jti, not the token. - Validating input only in the browser. The browser belongs to the user; the attacker does not use it. The schema is enforced at the edge and in the service.
- Dependencies neither pinned nor scanned. The most likely vulnerability in Kilometre Zero is not in its own code, but in a library running a two-year-old version.
- Tip: keep
kong.yaml, the Keycloak realm, the Vault policies and the OpenAPI schema in the repository and review them as code: a change fromminute: 60tominute: 600000must go through review. - Tip: rehearse the query "who accessed X's data?" before an auditor asks it; if it takes more than a few minutes to answer, the audit index is not well designed.
- Compliance warning. The retention of audit logs, their content (which includes identifiers of people and IP addresses, themselves personal data), access to them and the response to access or erasure requests are regulated by the GDPR; PCI DSS imposes specific audit requirements on any component that touches payments. Everything described here must be reviewed with a security and compliance professional.
Exercises
Exercise 1: The first hour of Grape Harvest Week
At 10:00 the campaign opens. In the first 60 seconds 2,400,000 requests reach the gateway: 1,900,000 GET /api/v1/catalog/* from 60,000 different clients, 300,000 POST /api/v1/orders from 40,000 clients, and 200,000 requests from 50 IPs doing GET /api/v1/catalog/products/crianza-wine in a loop. With the kong.yaml configuration and the create_limiter in orders (capacity 5, 0.1/s), work out roughly how many requests from each group reach the services and how many get a 429, and what a legitimate client loading the home page (18 requests to the catalogue) sees. What would you change for the third group, and why does the per-consumer limit not stop them?
Exercise 2: A curious operator
An operator is suspected of having been looking up customers' orders for no reason. Describe (a) what query you would run against the audit index and which fields you would use, (b) how you would prove that the records have not been altered by the operator themselves (who has access to the Kafka console and to MinIO as an administrator), (c) which anomaly rule would have raised the alert sooner, and (d) which two things in the design of record mean that the query does not in turn expose the customers' data.
Exercise 3: STRIDE for delivery
Apply the STRIDE table to the delivery service (it receives positions from 140 couriers through the app, publishes to delivery.positions, serves the real-time dashboard from delivery.dashboard to the operators, and shows Anna the approximate position of her courier). Give one concrete example per letter and the countermeasure with its lesson. Point out the threat you consider the most serious and why.
Solutions
Exercise 1.
Group 1 (catalogue, 60,000 clients, ~32 requests each in the minute): the limit is 200/min per consumer, so all of them get through (~1.9 M reach catalog, which serves them from the Redis cache of 04-05); a client loading the home page (18 requests) sees RateLimit-Remaining: 182 and no waiting. Group 2 (orders, 40,000 clients, ~7.5 POSTs each): Kong lets through 60/min per consumer (none of them reaches it), but create_limiter in orders grants 5 in one go and then 1 every 10 s: in a minute, at most 5 + 6 = 11 per client; those who made 7-8 get nearly all of them through (5 immediate + 1 or 2 refilled), and a fraction get a 429 with Retry-After: 10: some 40,000 × ~1.5 ≈ 60,000 rejected and ~240,000 reach orders. That is the desired behaviour: nobody legitimate places 8 orders in a minute apart from a double click, which the 429 with a gentle retry resolves. Group 3 (50 IPs, 4,000 requests each to the catalogue): limit_by: consumer does not apply because the catalogue does not require a token; Kong falls back to the limit per IP (200/min): 50 × 200 = 10,000 get through and 190,000 are rejected; but 10,000 identical requests still arrive and, above all, those 50 hosts could be 5,000 tomorrow. Changes: response caching at the gateway for public GETs (proxy-cache: the 10,000 never even touch catalog), a lower per-IP limit for public routes without a token (60/min is more than enough for a human), reputation-based blocking of repeat-offender ranges and, if it persists, bot protection at the CDN. The per-consumer limit does not stop them because they are not consumers: they do not authenticate, and against anonymous traffic all that is left is IP, fingerprint and cache.
Exercise 2.
(a) SELECT resource, timestamp, context->>'ticket_id' FROM audit WHERE sub = 'mhill' AND action = 'orders.read' AND timestamp BETWEEN ... ORDER BY timestamp, and then join with the support tickets: every read without a ticket_id (or with a ticket that does not mention that order) is suspicious; in addition, count distinct orders per hour and compare it with the average for other operators. (b) The chain: every event with source orders carries prev_hash; verify_chain over the hourly files in km0-audit/orders/ detects any deletion or modification; the files are under object lock in compliance mode, so not even as MinIO administrator could they have altered or deleted them; in Kafka, the ACL on audit.events grants only write to the services and read to the archiver, and even if as an administrator they could delete the topic, the archived files and the signed daily anchor (the day's last hash published externally) would prove the tampering; lastly, their own access to the Kafka console and to MinIO is audited in those systems' logs. (c) "A sub with the operator role reads > 200 different orders in an hour", or its finer variant "reads with no associated ticket_id". (d) record stores the order's identifier (P-2026-000123) and not its content, and the context is only request_id/ticket_id: the query reveals which orders they looked up, not anyone's phone number or address; and the audit index has restricted access that is itself audited, so the investigation leaves its own trail.
Exercise 3.
Spoofing: a modified app publishes positions as van-3-017 without being that courier: the token with courier_id (06-01, exercise 2), and the service ignores the id in the body and uses the one from the token. Tampering: an internal process injects fake positions into delivery.positions so that the dashboard shows deliveries where there are none: producer authentication in Kafka with a certificate and ACL (06-04) plus HMAC of the envelope (06-02). Repudiation: a courier denies having marked order P-2026-000124 as delivered: auditing of delivery.mark_delivered with sub, jti, position at that moment and hash chaining (06-05). Information disclosure: Anna sees the exact position of her courier, and with it the previous customer's house; or delivery.dashboard with the routes of 140 people is readable by any operator and leaks: show Anna only an approximate position and only while her order is "out for delivery" (minimisation, 06-02), retention of hours in delivery.positions, dashboard restricted to the operator role with access auditing; the couriers' positions are employees' personal data with specific obligations (impact assessment). Denial of service: 140 apps stuck in a loop because of a bug (or a malicious app) publish 1,000 positions per second each: rate limiting by sub at the edge (one position every 5 s is enough), maximum size, and a delivery that silently drops whatever exceeds it. Elevation of privilege: a courier calls another's ListDeliveries, or finds that MarkDelivered does not check the assignment: ABAC in the service (06-01) and an automated test for each rule. The most serious is information disclosure: it is the only one with irreversible, direct harm to people (Anna's address, an employee's daily route); the others can be detected and reverted. It is also the one that shows up least in a diagram, because there is no "attacker": just a design that shows more than it needs to.
Conclusion
The edge is where the platform meets the world, which is why it concentrates the cross-cutting responsibilities no service should repeat: an API gateway (Kong, Envoy, NGINX, or a managed one) terminates TLS, validates Keycloak's tokens, routes and versions, applies CORS and security headers, enforces the OpenAPI contract, limits sizes and times, and curbs abuse with rate limiting. Of the algorithms, token bucket is the best fit for an API (it tolerates human bursts, cuts off machines), and in Redis with an atomic Lua script it works for every gateway instance; 429, Retry-After and RateLimit-* tell the client how to behave, and limiting by identity (not just by IP) makes the limit fall on the right party. Auditing is a different category from operational logs: a fixed schema (who, what, when, from where, result), no unnecessary personal data, with sub and jti for correlation, immutable by construction (write-only in Kafka, hash chaining, object lock in MinIO), with restricted access, long retention, and anomaly rules that turn the record into an alert. And all of it as a process: STRIDE before every change, dependencies pinned and scanned, security configuration reviewed as code.
That closes Module 6. Kilometre Zero now has an explicit answer to "who can do what":
| Subject | Authenticates with | Can | Cannot |
|---|---|---|---|
Anna, Mark, Lucy (customer) |
Keycloak (Argon2 password or Google) + 15 min RS256 JWT; optional MFA | View the catalogue; create orders (5 in one go, 1 every 10 s); view their own orders and the approximate position of their own courier | View other people's orders (ABAC); edit products; more than 60 calls/min to orders |
La Vega Farm, Montblanc Dairy, Roble Alto Winery (producer) |
Keycloak + mandatory MFA; group with producer_id |
Edit their own products and adjust their own stock; view orders containing their products; upload photos by presigned URL | Touch another producer's products; see customers; reserve stock |
Couriers of van-3 (courier) |
courier-app (OIDC + PKCE) with courier_id |
Publish their own position; view their own deliveries; mark their own orders as delivered | Impersonate another courier; see the full dashboard |
Jordan, Martha (operator) |
OpenLDAP → Keycloak, mandatory MFA | View any order (audited, with a ticket); cancel; approve producers; view the delivery dashboard | Publish positions; read phone numbers without leaving a record; alter the audit log |
orders (spiffe://km0.internal/orders) |
24 h mTLS certificate issued by Vault PKI; client credentials token for compensations | ReserveStock, ReleaseReservation, GetStock; Charge, Refund; write orders.events and audit.events |
AdjustStock; read delivery.positions; read other services' secrets in Vault |
catalog |
mTLS certificate | GetStock; read inventory.alerts; sign URLs for km0-photos |
Reserve, charge, publish orders |
delivery |
mTLS certificate | Write delivery.positions; read orders.events; read and decrypt phone numbers of orders out for delivery |
Cancel orders; read invoices |
analytics (Airflow, Spark, Flink) |
mTLS certificate; AppRole in Vault; dynamic credentials for km0_analytics with a 2 h TTL |
Read orders.events and delivery.positions; read the pseudonymised lake; write daily_sales and delivery.dashboard |
Read km0_orders; see emails or phone numbers; write to orders.events; keep a password |
| Kong (edge) | Public ACME certificate outwards; mTLS inwards | Terminate TLS, verify JWTs, route, limit, record | Decide fine-grained authorization; have its administration visible from the Internet |
| Anyone on the internal network without an identity | Nothing | Nothing: without a certificate there is no handshake, without a token there is no call | — |
And the controls put in place, lesson by lesson: passwords with Argon2id, RS256 JWTs with strict claims, refresh with rotation and revocation by jti, RBAC and ABAC with least privilege (06-01); AES-GCM, HMAC of events, TLS in gRPC and in every data store, field encryption with envelope encryption, pseudonymisation of the lake, crypto-shredding, no cards at all (06-02); Keycloak as the IdP with OIDC + PKCE, client credentials, JWKS, OpenLDAP for employees, MFA per role, centralised lifecycle (06-03); zero trust with mTLS, internal PKI, service-to-service authorization, Vault with dynamic credentials and auditing of secrets (06-04); gateway, rate limiting, schema validation, immutable auditing and STRIDE (06-05). None of them is infallible; together, in layers, they mean that compromising one piece is not compromising the platform.
The platform is secure. But do we know whether it works? One 429 too many during Grape Harvest Week, an audit chain that breaks, a Vault lease that is not renewed, a certificate that expires on a Sunday: nothing built across six modules raises the alarm by itself when things go wrong. Without instruments, a distributed system fails silently until a customer notices. Module 7 deals with monitoring and maintaining the platform, and begins with monitoring through metrics: what to measure in each service, how to collect it with Prometheus, how to define Kilometre Zero's service level objectives and how to alert before Anna finds out.
Distributed Architectures Course
Module 1: Introduction to Distributed Systems
- Basic Concepts of Distributed Systems
- Distributed System Models
- Advantages and Challenges of Distributed Systems
- The Fallacies of Distributed Computing
- Time, Clocks and Event Ordering
- From Monolith to Distributed Platform: the Kilometre Zero Case
Module 2: Communication in Distributed Systems
- Communication Protocols
- RPC and RMI
- gRPC and Data Serialization
- Messaging and Message Queues
- Asynchronous Communication Patterns
Module 3: Consistency and Replication
- Consistency Models
- The CAP Theorem and PACELC
- Consensus Algorithms
- Data Replication
- Distributed Transactions and Sagas
Module 4: Distributed Storage
- Data Partitioning and Consistent Hashing
- Distributed File Systems
- Object Storage
- Distributed Databases
- Distributed Caches
Module 5: Distributed Computing
- Distributed Computing Models
- MapReduce and Hadoop
- Spark and In-Memory Computing
- Stream Processing
- Job Scheduling and Data Pipelines
Module 6: Security in Distributed Systems
- Authentication and Authorization
- Encryption and Data Protection
- Identity Management
- Service-to-Service Security: mTLS and Secrets Management
- API Gateways, Rate Limiting and Auditing
Module 7: Monitoring and Maintenance
- Monitoring Distributed Systems
- Centralized Logs and Distributed Tracing
- Failure Management and Recovery
- Resilience Patterns: Timeouts, Retries and Circuit Breakers
- Automation and Orchestration
- Testing Distributed Systems and Chaos Engineering
