The previous lesson ended with two questions. The first: do we really need a Deployment on EKS, with its replicas, its probes, its HPA and its on-call rota, to generate three thumbnails every time Montblanc Dairy uploads a photo, or to produce an invoice PDF whenever a payment.confirmed arrives? These are small, sporadic, stateless tasks that respond to an event and finish. The second: why does the aged-cheese photo have to travel from the eu-west-1 region to Anna's phone in Valencia on every visit, paying egress and 40 ms of latency, when it could be stored twenty kilometres away from her? And there is a third, which already surfaced in 08-02: what does van-3 do with its positions during a three-minute tunnel, or the La Vega Farm stall at the Lleida market when the connection drops and customers keep arriving? All three questions have answers that step outside the "service in a cluster in one region" model: serverless, where the provider runs ephemeral functions in response to events and charges per invocation; and edge computing, where compute and data move closer to whoever uses them, be it a CDN, a function at the edge of the network or a device that works offline. This lesson develops both, with their execution model, their limits, their good and bad fits in Kilometre Zero, and the code that implements them: a thumbnails Lambda, an invoices Lambda, the order saga rewritten in Step Functions, a Cloudflare Worker that verifies the JWT at the edge and a local aggregator for the van. The final project (08-05) will bring it all together.
Contents
- Serverless and FaaS: the execution model
- Cold starts, limits, concurrency and pricing
- Good and bad fits: the four Kilometre Zero cases
- Serverless beyond FaaS: queues, databases, API Gateway and Step Functions
- Patterns: fan-out, mandatory idempotency and DLQ
- Observability, local testing, lock-in and cost at scale
- Serverless code: thumbnails, invoices, SAM and the saga in Step Functions
- Edge computing: why bring compute closer
- CDN: caching photos and catalogue near Anna
- Functions at the edge: JWT verification and personalisation
- Edge for IoT: the van and the market stall offline
- The cloud-edge-device continuum and security at the edge
- Common Mistakes and Tips
- Exercises
- Conclusion
- Serverless and FaaS: the execution model
"Serverless" does not mean there are no servers: it means they are not ours and we never see them. The provider provisions them, scales them and retires them; we hand over code and pay for what runs. The purest form is FaaS (Function as a Service): AWS Lambda, Google Cloud Functions, Azure Functions. The shared responsibility table in 08-03 named it; this is its execution model:
- An event happens: an object lands in a bucket, a message enters a queue or a topic, an HTTP request reaches an endpoint, a timer fires.
- The provider looks for a ready instance of the function (a micro-container with the Python runtime and our code). If none is free, it creates one (cold start).
- It invokes the handler (
handler(event, context)) with the serialised event. The function does its work and returns. - The instance stays frozen for a few minutes in case another event arrives; if none does, it is destroyed. There is no state between invocations that can be relied upon (the temporary file system may survive, but it is not guaranteed).
- If a thousand events arrive at once, the provider creates up to a thousand instances in parallel (scaling to thousands); if none arrive, there are none (scaling to zero) and nothing is paid.
The comparison with the Deployment from 07-05 sums up the difference in model:
| Aspect | Service on Kubernetes (07-05) | Serverless function |
|---|---|---|
| Unit | Long-lived container with several replicas | Ephemeral function, one invocation per instance (in Lambda) |
| Scaling | HPA by metric, from n to m replicas, in tens of seconds | Automatic per event, from 0 to thousands, in milliseconds or seconds |
| Cost at rest | The minimum replicas are always paid for | Zero |
| State | In memory while the Pod lives; in Redis or a database | External only (S3, DynamoDB, database) |
| Connections | Persistent connection pool to PostgreSQL, Kafka | Each instance opens its own; thousands of instances exhaust a database's connections (a proxy is needed: RDS Proxy) |
| Duration | Unlimited (a Kafka consumer runs for days) | Limited (15 min in Lambda) |
| Deployment | Image + manifest + rollout | Code package (zip or image) + event definition |
| Operation | Probes, resources, on-call by SLO | No servers to operate; but metrics, errors and DLQs to watch |
- Cold starts, limits, concurrency and pricing
Cold start: creating the instance, loading the runtime and running the initialisation code (imports, connections) before the first invocation. In Python with few dependencies it is 100-300 ms; with Pillow, boto3 and a Kafka client, 1-2 s; inside a VPC, somewhat more. It is acceptable for processing a photo and not for answering Anna. It is mitigated with provisioned concurrency (instances kept warm, which are paid for even when unused: scaling to zero is lost), with small packages, and by moving heavy initialisation outside the handler so it runs once per instance rather than once per invocation.
Limits: maximum time per invocation (15 min in Lambda, 9-60 min in Cloud Functions depending on the generation), memory (from 128 MB to 10 GB; CPU is allocated in proportion to memory, so a slow function sometimes speeds up when given more memory), package size (250 MB uncompressed; or a container image of up to 10 GB), event size (6 MB synchronous, 256 KB asynchronous), temporary space (/tmp of up to 10 GB).
Concurrency: the number of simultaneous instances. It has a limit per account and region (1,000 by default in Lambda, extendable) and can be reserved per function (so that a runaway function does not exhaust the quota of the others) or capped (so as not to exhaust the database behind it: if km0_analytics accepts 100 connections, the function that writes to it cannot have 500 instances). That cap is the bulkhead of 07-04 in serverless form.
Pricing: per number of invocations (in the order of €0.20 per million) plus per GB-second of execution (allocated memory × duration; in the order of €0.000017 per GB-s), with a monthly free tier. A thumbnails function with 512 MB that takes 800 ms costs ≈ €0.000007 per photo: 100,000 photos a month, under €1. That is what makes serverless unbeatable for sporadic workloads and what makes it expensive for constant ones (section 6).
- Good and bad fits: the four Kilometre Zero cases
| Workload characteristic | Good fit for FaaS | Bad fit for FaaS |
|---|---|---|
| Frequency | Sporadic, or with huge and unpredictable spikes | Constant and high (a consumer processing 500 events/s all day) |
| Duration | Seconds, minutes at most | Hours (a Spark job; a long-lived Kafka consumer) |
| State | Stateless; everything in external services | State in memory between requests (the WebSocket subscription table of 08-02) |
| Latency | Tolerates hundreds of ms (background processing) | p99 < 500 ms demanded by people (the orders SLO) |
| Connections | Few, brief, to services that scale (S3, DynamoDB, SQS) | Persistent pool to PostgreSQL, WebSocket sessions, MQTT connections |
| Trigger | A named event (object, message, request, timer) | A loop that polls or a process listening on a port |
With that filter, Kilometre Zero identifies four tasks that today live as code inside the services (or as cron jobs in Kubernetes) and that fit better as functions:
- Thumbnails on uploading a photo to
km0-photos. In 04-03 Montblanc Dairy uploadedphotos/aged-cheese/original.jpgvia a presigned URL and "a thumbnail process" generated the 300 and 800 px versions. That process was a consumer with a loop. As a function:s3:ObjectCreatedevent → Lambda with Pillow → writesthumb-300.jpgandthumb-800.jpg. Sporadic (hundreds of photos a day, with spikes when a producer uploads their whole catalogue), stateless, 1 s per photo. - Invoice PDF on
payment.confirmed. Today a consumer oforders.eventsinsideorders. As a function: Kafka event (MSK as a Lambda event source, in batches) → generate the PDF → write it tokm0-invoices→ notify. At the pace of orders: thousands a day, spikes during a campaign, stateless. - Payment gateway webhooks. The gateway notifies deferred charges and refunds asynchronously with a
POSTto a URL of ours. It is an endpoint that receives few calls, must always be available, verify a signature and publish an event: API Gateway + Lambda, without tying up apaymentsreplica to wait. - Lightweight scheduled tasks. Expiring unconfirmed stock reservations every minute, cleaning up sessions, checking that the Airflow DAG finished: timer (EventBridge) → Lambda. They replace Kubernetes CronJobs that tied up resources to run 200 ms of work.
And what does not move to functions, with the reason: orders and catalog (latency demanded by people, connection pool, Kong and the mesh in front), the WebSocket server of 08-02 (state and long connections; there are managed WebSocket services, but the fan-out between instances and the subscription logic are exactly what does not fit), the Flink consumers (state and checkpoints), the Spark jobs (hours).
- Serverless beyond FaaS: queues, databases, API Gateway and Step Functions
FaaS is the visible part; around it there are services that are "serverless" in the sense of no capacity to provision and pay-as-you-go:
| Service | What it is | Equivalent Kilometre Zero already has | When to prefer it |
|---|---|---|---|
| Queues (SQS, Cloud Tasks) | Managed queue, unlimited scaling, pay per message, with built-in DLQ | RabbitMQ (02-04) | To connect functions to each other and absorb spikes without operating a broker |
| Serverless databases (DynamoDB, Aurora Serverless, Firestore) | Key-value or relational store that scales capacity automatically, pay per request or per consumed capacity | Cassandra (04-04), RDS (08-03) | DynamoDB for function state (idempotency table, checkpoints); Aurora Serverless for databases with highly variable load |
| Managed API Gateway | HTTP endpoint that routes to functions, with authentication (JWT), limits and API keys | Kong (06-05) | To expose functions (webhooks) without going through the cluster; with fewer plugins than Kong |
| Step Functions / Workflows | Stateful workflow orchestrator: defines steps, branches, retries and compensations in JSON, and runs each step by invoking functions or services | order_saga.py with the sagas table (03-05) |
When the saga is composed of functions and you do not want to write or operate the orchestrator |
Step Functions deserves attention because it replaces something the course built with effort. The OrderSagaOrchestrator of 03-05 persisted the state of each saga in the sagas table, retried transient failures, distinguished permanent ones and ran the compensations in reverse order. Step Functions does exactly that as a service: the state of each execution is stored by the provider (with a complete history of every step), retries and Catch blocks are declared per step, and compensations are branches of the graph. What is lost: the orchestrator lives outside the cluster, each step is an invocation (with its latency and its cost), the definition is JSON rather than Python, and it is the most provider-specific thing there is (total lock-in). The full comparison is in section 7, with the saga rewritten.
- Patterns: fan-out, mandatory idempotency and DLQ
Fan-out with queues. An event that must be processed in several ways (an uploaded photo: thumbnails, content analysis, updating the listing) does not trigger three functions directly from S3; it is published to a topic (SNS or EventBridge) that delivers it to one queue per consumer (SQS), and each queue triggers its function. The queues decouple (if the analysis function fails, the thumbnails are generated anyway), absorb spikes (a producer uploading 2,000 photos does not create 2,000 × 3 instances at once: the queue meters them out with the concurrency limit) and provide retries and DLQs.
Mandatory idempotency. The provider delivers events at least once and retries automatically when the function fails or times out: a function that takes 16 minutes because of an oversized input is re-invoked, and if the first instance had written half the result, the second starts from that state. Everything seen in 02-05 about idempotent consumers applies with more force, because here we do not control the retries: every function must be safe under repetition. The techniques: deterministic output keys (the thumbnail is always written to the same key: writing it twice is harmless), an idempotency table in DynamoDB keyed by the event id (invoice F-2026-000124 is generated once), and conditional operations (PutItem with attribute_not_exists).
DLQ. After n retries (configurable per function or per queue), the event goes to a dead letter queue, with an alert (07-01) and a reprocessing procedure: the DLQ of 02-05, managed. Without a DLQ configured, a poison event (a corrupt image that makes Pillow fail) is retried until the policy is exhausted and then lost in silence.
- Observability, local testing, lock-in and cost at scale
Observability. Functions emit logs to CloudWatch (or Cloud Logging) automatically, plus metrics for invocations, errors, duration and throttles. What has to be added is the same as in 07-02: structured logs with the X-Request-Id or the event_id for correlation, and traces with OpenTelemetry (or X-Ray) that link the function to the service that produced the event, because a request from Anna that ends in an invoice crosses orders, Kafka and the Lambda, and without context propagation the trace breaks at Kafka. The services/common/traces.py from 07-02 is packaged with the function as a layer.
Local testing. A function cannot be run "without the provider" except with emulators: AWS SAM (sam local invoke, sam local start-api) runs the function in a container with a sample event; LocalStack emulates S3, SQS, DynamoDB and more in Docker, which allows integration tests with Testcontainers (07-06) without an AWS account. Neither is perfect (IAM permissions and the real limits only show up at the provider), so the pipeline deploys to a real staging environment for the contract tests.
Lock-in. It is the highest of anything seen so far: the event format, the handler, the surrounding services (SQS, DynamoDB, Step Functions) belong to the provider. It is mitigated by separating the handler (a 10-line adapter) from the logic (pure Python functions, testable without a provider), as done in section 7; but it has to be accepted: whoever chooses Step Functions chooses AWS.
Cost at scale. The per-invocation price is unbeatable at low frequency and becomes expensive at high frequency: a 512 MB function that runs constantly (say, 100 invocations/s of 200 ms each, 24 hours a day) consumes ≈ 2.6 million GB-s a month ≈ €45, plus 260 million invocations ≈ €52: about €100/month, against ≈ €30 for a small Pod on a node already paid for. The rough rule: when average utilisation exceeds 30-40% of an equivalent container, the container works out cheaper. That is why thumbnails and invoices are serverless and orders is not.
- Serverless code: thumbnails, invoices, SAM and the saga in Step Functions
The structure added to km0/:
km0/serverless/
├── template.yaml # AWS SAM: functions, events, permissions, DLQ
├── thumbnails/
│ ├── handler.py
│ └── requirements.txt # Pillow, boto3
├── invoices/
│ ├── handler.py
│ └── requirements.txt # reportlab, boto3
└── saga/
└── order_saga.asl.json # Step Functions (Amazon States Language)The S3 → Lambda flow
flowchart LR
Q[Montblanc Dairy<br/>PUT via presigned URL] --> S3[(S3 km0-photos<br/>photos/aged-cheese/original.jpg)]
S3 -- "s3:ObjectCreated:*<br/>prefix photos/, suffix original.jpg" --> L[Lambda thumbnails<br/>Pillow, 1024 MB, 60 s]
L -- "PUT thumb-300.jpg<br/>PUT thumb-800.jpg" --> S3
L -- "photo.processed event" --> EB[EventBridge]
EB --> CAT[catalog: updates listing]
L -. "failure after 2 retries" .-> DLQ[(SQS DLQ thumbnails)]
DLQ -. alert .-> OPS[Alert 07-01]
serverless/thumbnails/handler.py
# km0/serverless/thumbnails/handler.py
"""Generates 300 and 800 px thumbnails when an original is uploaded to km0-photos.
Idempotent: the output keys are deterministic and the original's ETag is stored
as metadata on each thumbnail; if it already matches, nothing is regenerated.
"""
import io, json, os, urllib.parse
import boto3
from PIL import Image
# Initialisation OUTSIDE the handler: runs once per instance (cold start),
# not once per invocation. Clients and anything expensive go here.
s3 = boto3.client("s3")
eventbridge = boto3.client("events")
SIZES = (300, 800)
EVENT_BUS = os.environ.get("KM0_EVENT_BUS", "km0")
def _thumbnail_key(original_key: str, px: int) -> str:
# photos/aged-cheese/original.jpg -> photos/aged-cheese/thumb-300.jpg (convention from 04-03)
prefix = original_key.rsplit("/", 1)[0]
return f"{prefix}/thumb-{px}.jpg"
def _already_generated(bucket: str, key: str, original_etag: str) -> bool:
try:
head = s3.head_object(Bucket=bucket, Key=key)
return head.get("Metadata", {}).get("source-etag") == original_etag
except s3.exceptions.ClientError:
return False
def process_object(bucket: str, key: str, original_etag: str) -> list[str]:
"""Pure business logic: testable without AWS by passing a fake client as s3."""
body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
image = Image.open(io.BytesIO(body))
image = image.convert("RGB") # PNG with alpha -> JPEG
generated = []
for px in SIZES:
target = _thumbnail_key(key, px)
if _already_generated(bucket, target, original_etag): # retry or duplicate: do not redo
generated.append(target)
continue
copy = image.copy()
copy.thumbnail((px, px)) # keeps the aspect ratio; never upscales
output = io.BytesIO()
copy.save(output, format="JPEG", quality=85, optimize=True)
s3.put_object(
Bucket=bucket, Key=target, Body=output.getvalue(), ContentType="image/jpeg",
CacheControl="public, max-age=31536000, immutable", # the CDN (section 9) caches it for a year
Metadata={"source-etag": original_etag}, # idempotency marker
)
generated.append(target)
return generated
def handler(event: dict, context) -> dict:
"""Adapter: translates the S3 event into calls to the logic. 10 lines, no business logic."""
results = []
for record in event["Records"]: # S3 may group several objects in one event
bucket = record["s3"]["bucket"]["name"]
key = urllib.parse.unquote_plus(record["s3"]["object"]["key"]) # keys arrive URL-encoded
etag = record["s3"]["object"].get("eTag", "")
if not key.endswith("/original.jpg"):
continue # defence: the SAM filter already does this, but do not trust it
generated = process_object(bucket, key, etag)
eventbridge.put_events(Entries=[{
"Source": "km0.photos", "DetailType": "photo.processed", "EventBusName": EVENT_BUS,
"Detail": json.dumps({"bucket": bucket, "original": key, "thumbnails": generated}),
}])
print(json.dumps({"level": "info", "msg": "thumbnails generated", "key": key,
"n": len(generated), "request_id": context.aws_request_id})) # structured log (07-02)
results.append(key)
return {"processed": results}Decisions worth highlighting: idempotency is based on the original's ETag stored as metadata on the thumbnail, so that a re-invocation does not redo work while a new photo under the same key (new ETag) does get redone; CacheControl: immutable is the promise from 04-03 never to overwrite keys cached by the CDN, honoured here because the web references thumb-800.jpg?v=<etag>; and the handler contains no business logic, so that process_object can be tested with a fake client and so that lock-in is confined to those lines.
serverless/invoices/handler.py
# km0/serverless/invoices/handler.py
"""Generates the invoice PDF for an order on receiving payment.confirmed from MSK (Kafka).
Lambda receives BATCHES of records per partition. Idempotent by event_id via DynamoDB
(km0-idempotency table): the same invoice is never generated or sent twice.
"""
import base64, io, json, os, time
import boto3
from botocore.exceptions import ClientError
from reportlab.lib.pagesizes import A4
from reportlab.pdfgen import canvas
s3 = boto3.client("s3")
dynamo = boto3.resource("dynamodb").Table(os.environ["IDEMPOTENCY_TABLE"])
INVOICES_BUCKET = os.environ["INVOICES_BUCKET"] # km0-invoices
TTL_S = 30 * 24 * 3600 # the idempotency marker expires after 30 days
def claim(event_id: str) -> bool:
"""Tries to register the event. Returns False if it was already there (duplicate). Atomic in DynamoDB."""
try:
dynamo.put_item(Item={"id": f"invoice#{event_id}", "ttl": int(time.time()) + TTL_S},
ConditionExpression="attribute_not_exists(id)")
return True
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
return False
raise
def generate_pdf(order: dict) -> bytes:
"""Pure logic: a simple PDF with reportlab."""
buf = io.BytesIO()
c = canvas.Canvas(buf, pagesize=A4)
c.setFont("Helvetica-Bold", 16); c.drawString(50, 800, "Kilometre Zero - Invoice")
c.setFont("Helvetica", 11)
c.drawString(50, 775, f"Invoice: F-{order['order_id'][2:]} Order: {order['order_id']}")
c.drawString(50, 760, f"Customer: {order['customer']} Market: {order['market']}")
y = 730
for line in order["lines"]:
c.drawString(60, y, f"{line['units']} x {line['product']} ({line['producer']})")
c.drawRightString(540, y, f"{line['amount_cents'] / 100:.2f} EUR"); y -= 16
c.setFont("Helvetica-Bold", 12); c.drawRightString(540, y - 10, f"Total: {order['total_cents'] / 100:.2f} EUR")
c.showPage(); c.save()
return buf.getvalue()
def handler(event: dict, context) -> dict:
generated, duplicates = 0, 0
# MSK event format: {"records": {"orders.events-3": [ {value: <base64>, ...}, ... ]}}
for partition, records in event["records"].items():
for r in records:
ev = json.loads(base64.b64decode(r["value"]))
if ev["type"] != "payment.confirmed":
continue
if not claim(ev["event_id"]): # Lambda retry or Kafka duplicate (02-05)
duplicates += 1
continue
order = ev["data"]
key = f"invoices/{order['customer']}/{order['order_id']}.pdf" # deterministic key
s3.put_object(Bucket=INVOICES_BUCKET, Key=key, Body=generate_pdf(order),
ContentType="application/pdf", ServerSideEncryption="aws:kms") # encryption (06-02)
generated += 1
print(json.dumps({"level": "info", "msg": "batch processed", "generated": generated,
"duplicates": duplicates, "request_id": context.aws_request_id}))
return {"generated": generated}An important subtlety about the order of operations: claim happens before generating the PDF. If the function dies after claiming and before writing, the invoice will never be generated on a retry (the marker already exists). The alternative (claiming after writing) allows duplicates if it dies in between. Since writing the same S3 key twice is harmless (idempotent in itself), the correct choice here is to claim afterwards, or to use the object's existence as the marker. It is deliberately left as it is for exercise 1, which asks you to fix it: this is the kind of reasoning about the ack point that 02-05 taught, and in serverless there is no excuse for skipping it.
serverless/template.yaml with AWS SAM
# km0/serverless/template.yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: Kilometre Zero event functions (thumbnails, invoices)
Globals:
Function:
Runtime: python3.12
Architectures: [arm64] # Graviton: cheaper per GB-s
Tracing: Active # X-Ray / OpenTelemetry traces (07-02)
Environment:
Variables: { KM0_ENVIRONMENT: prod }
Resources:
ThumbnailsDlq:
Type: AWS::SQS::Queue
Properties: { QueueName: km0-thumbnails-dlq, MessageRetentionPeriod: 1209600 } # 14 days to reprocess
Thumbnails:
Type: AWS::Serverless::Function
Properties:
CodeUri: thumbnails/
Handler: handler.handler
MemorySize: 1024 # Pillow is CPU-bound: more memory = more CPU = shorter duration
Timeout: 60
ReservedConcurrentExecutions: 50 # bulkhead: a producer uploading 2,000 photos does not exhaust the account
DeadLetterQueue: { Type: SQS, TargetArn: !GetAtt ThumbnailsDlq.Arn }
EventInvokeConfig: { MaximumRetryAttempts: 2 } # asynchronous invocation: 2 retries and then the DLQ
Policies:
- S3CrudPolicy: { BucketName: km0-photos-prod } # this bucket only
- EventBridgePutEventsPolicy: { EventBusName: km0 }
Events:
PhotoUploaded:
Type: S3
Properties:
Bucket: !Ref PhotosBucket
Events: s3:ObjectCreated:*
Filter:
S3Key:
Rules:
- { Name: prefix, Value: photos/ }
- { Name: suffix, Value: original.jpg } # does NOT fire for the thumbnails: avoids the infinite loop
PhotosBucket: # referenced from Terraform (08-03) or created here in staging
Type: AWS::S3::Bucket
Properties: { BucketName: km0-photos-prod }
IdempotencyTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: km0-idempotency
BillingMode: PAY_PER_REQUEST # serverless: no capacity to provision
AttributeDefinitions: [{ AttributeName: id, AttributeType: S }]
KeySchema: [{ AttributeName: id, KeyType: HASH }]
TimeToLiveSpecification: { AttributeName: ttl, Enabled: true }
Invoices:
Type: AWS::Serverless::Function
Properties:
CodeUri: invoices/
Handler: handler.handler
MemorySize: 512
Timeout: 120
Environment:
Variables: { INVOICES_BUCKET: km0-invoices-prod, IDEMPOTENCY_TABLE: !Ref IdempotencyTable }
Policies:
- S3WritePolicy: { BucketName: km0-invoices-prod }
- DynamoDBCrudPolicy: { TableName: !Ref IdempotencyTable }
- KMSEncryptPolicy: { KeyId: !ImportValue km0-kms-data }
VpcConfig: # MSK lives in the VPC's data subnets (08-03)
SubnetIds: !Split [",", !ImportValue km0-data-subnets]
SecurityGroupIds: [!ImportValue km0-sg-lambda-msk]
Events:
PaymentConfirmed:
Type: MSK
Properties:
Stream: !ImportValue km0-msk-arn
Topics: [orders.events]
StartingPosition: LATEST
BatchSize: 50 # up to 50 records per invocation
MaximumBatchingWindowInSeconds: 5
ConsumerGroupId: invoices-lambda # one more consumer group on the topic (02-04)
DestinationConfig:
OnFailure: { Destination: !GetAtt InvoicesDlq.Arn }
InvoicesDlq:
Type: AWS::SQS::Queue
Properties: { QueueName: km0-invoices-dlq, MessageRetentionPeriod: 1209600 }Three details of the template.yaml that avoid classic mistakes: the original.jpg suffix filter prevents writing a thumbnail from triggering the function again (an infinite loop that would also be an infinite bill); ReservedConcurrentExecutions bounds the damage of a spike; and the MSK event source turns Lambda into one more consumer group on orders.events, with the same partition and offset model as 02-04, which means that a persistent failure of the function blocks progress on that partition for that group (and only for that group) until the batch goes to the DLQ. It is deployed with sam build && sam deploy --guided the first time, and from the pipeline thereafter.
The order saga in Step Functions
The saga from 03-05 (reserve stock → charge → confirm; compensate in reverse order on a permanent failure), expressed in Amazon States Language, with each step invoking a function (or, in production, the corresponding service via its API):
{
"Comment": "Kilometre Zero order saga (equivalent to order_saga.py, 03-05)",
"StartAt": "ReserveStock",
"States": {
"ReserveStock": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "km0-inventory-reserve",
"Payload": { "reservation_id.$": "$.order_id", "order_id.$": "$.order_id",
"market.$": "$.market", "lines.$": "$.lines" }
},
"ResultPath": "$.reservation",
"Retry": [
{ "ErrorEquals": ["TransientFailure", "Lambda.ServiceException", "Lambda.TooManyRequestsException"],
"IntervalSeconds": 1, "MaxAttempts": 3, "BackoffRate": 2.0 }
],
"Catch": [
{ "ErrorEquals": ["InsufficientStock"], "ResultPath": "$.error", "Next": "RejectOrder" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RejectOrder" }
],
"Next": "Charge"
},
"Charge": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": "km0-payments-charge",
"Payload": { "idempotency_key.$": "States.Format('charge-{}', $.order_id)",
"customer.$": "$.customer", "amount_cents.$": "$.total_cents" }
},
"ResultPath": "$.charge",
"TimeoutSeconds": 10,
"Retry": [
{ "ErrorEquals": ["TransientFailure", "States.Timeout"], "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0 }
],
"Catch": [
{ "ErrorEquals": ["PaymentRejected"], "ResultPath": "$.error", "Next": "ReleaseReservation" },
{ "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseReservation" }
],
"Next": "ConfirmOrder"
},
"ConfirmOrder": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": { "FunctionName": "km0-orders-change-status",
"Payload": { "order_id.$": "$.order_id", "status": "paid" } },
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 5, "BackoffRate": 2.0 } ],
"Next": "Confirmed"
},
"Confirmed": { "Type": "Succeed" },
"ReleaseReservation": {
"Type": "Task",
"Comment": "Compensation C2: idempotent (releasing twice does not duplicate stock)",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": { "FunctionName": "km0-inventory-release",
"Payload": { "reservation_id.$": "$.order_id" } },
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0 } ],
"Next": "RejectOrder"
},
"RejectOrder": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": { "FunctionName": "km0-orders-change-status",
"Payload": { "order_id.$": "$.order_id", "status": "rejected", "reason.$": "$.error.Error" } },
"ResultPath": null,
"Retry": [ { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 2, "MaxAttempts": 10, "BackoffRate": 2.0 } ],
"Next": "Rejected"
},
"Rejected": { "Type": "Fail", "Error": "OrderRejected", "Cause": "Saga compensated" }
}
}Compare it with the OrderSagaOrchestrator of 03-05:
| Aspect | order_saga.py + sagas table (03-05) |
Step Functions |
|---|---|---|
| Saga state | Row in sagas in km0_orders, written by our code on every transition |
Stored by the service; complete history of every execution available for inspection |
| Retries and backoff | Our own code (max_retries, TransientFailure) |
Declarative per step (Retry) |
| Compensations | A _compensate method that walks the completed steps in reverse order |
Catch branches → compensation states; the graph draws the order |
| Timeouts | Those of the gRPC client (07-04) | TimeoutSeconds per step, and for the whole execution |
| Recovery after an orchestrator crash | On start-up, resume() over the sagas in progress |
Automatic: the service never "crashes" as far as we are concerned |
| Latency per step | Milliseconds (direct gRPC call) | Tens of ms per transition + the start-up of each Lambda |
| Cost | That of the orders service |
Per state transition (≈ €0.025 per thousand, standard type): with 5 transitions and 1.2 M orders/month ≈ €150/month |
| Testing | pytest with doubles (SimulatedInventory) and Testcontainers (07-06) |
Local Step Functions emulator or staging; the logic of each step, in Python |
| Portability | Python, anywhere | AWS only |
| Visibility | Whatever gets instrumented (07-01/07-02) | Console with the graph and the state of every execution, out of the box |
Kilometre Zero's decision, which ADR-006 in 08-05 formalises: the order saga stays in orders, because it is on the critical path (latency), has a constant volume (cost) and is already built and tested; Step Functions is reserved for low-frequency, long-running workflows where the visibility and the declarative retries pay off, such as offboarding a producer (cancelling their products, settling their payments, archiving their photos, all with waits of days).
- Edge computing: why bring compute closer
Everything above still lives in the provider's region. Edge computing moves compute and data towards the edge: to a CDN's points of presence (hundreds of cities), to the carrier's antennas or gateways, or to the devices themselves. Four reasons, and for each one a Kilometre Zero case:
| Reason | What it solves | Case |
|---|---|---|
| Latency | Light takes ~10 ms to cover 1,000 km of fibre (one way); a Valencia → Ireland → Valencia request is 40-60 ms before the server does anything | Anna's photos and catalogue from a point of presence in Madrid or Valencia |
| Bandwidth and egress | Every photo served from the region pays egress and occupies the link | The CDN serves 95% of photos without touching S3 (04-03 worked it out) |
| Autonomy | Working when the connection to the cloud fails | The van-3 app in a tunnel; the La Vega Farm terminal at the Lleida market with no coverage |
| Local data | Data that does not need to (or should not) travel whole to the cloud | Aggregating 36 positions into one during the tunnel; filtering on the device whatever adds nothing |
- CDN: caching photos and catalogue near Anna
A CDN (Content Delivery Network: CloudFront, Cloudflare, Fastly, Akamai) is a network of cache servers distributed around the world, in front of the origin. 04-05 placed it as a cache tier and 04-03 justified it by egress; here it gets designed.
How it works. The DNS for km0.example points to the CDN; Anna's browser in Valencia resolves to the nearest point of presence; if the object is in its cache (hit), it is served in 5-10 ms; if not (miss), it is requested from the origin (S3 for photos, Kong for the API), stored according to the cache headers and served. The hit ratio is the metric: with 4,200 products and immutable, it exceeds 95% once warmed up.
Cache keys. The cache key is, by default, the full URL (including query parameters). Two consequences: thumb-800.jpg?v=abc and ?v=def are different objects, which is exactly what is wanted for versioning (04-03); and a URL with irrelevant parameters (?utm_source=...) fragments the cache and lowers the hit ratio, so you configure which parameters form part of the key. Headers such as Accept-Language or cookies can be added to the key (carefully: every value multiplies the variants).
TTL per content type and invalidation:
| Content | Origin | Cache-Control |
Invalidation | Why |
|---|---|---|---|---|
| Photo thumbnails and originals | S3 km0-photos |
public, max-age=31536000, immutable |
Never: the URL changes with the content (?v=<etag>) |
Immutable objects (04-03) |
Product listing (JSON from /api/v1/catalog/products/aged-cheese) |
Kong → catalog |
public, max-age=60, stale-while-revalidate=300 |
By URL, triggered by the stock.updated consumer from 04-05 (in addition to Redis) |
Changes little; a minute of staleness is acceptable; stale-while-revalidate serves the old one while refreshing |
| Market listing | Kong → catalog |
public, max-age=30 |
By TTL | Changes with every publication; cheap to recompute |
Anything under /api/v1/orders |
Kong → orders |
private, no-store |
— | Personal data; never in a shared cache |
Static web assets (JS, CSS from tracking.js) |
S3 | immutable with a hash in the file name |
Never | Build with hashed names |
WebSocket /ws |
delivery |
— | — | The CDN passes it through (proxy) without caching; some terminate TLS and forward it |
Explicit invalidation (purge by URL or by tag) exists but is slow (seconds to minutes to propagate to every point of presence) and is often charged for; the right design avoids it for immutable content and uses it only for semi-static content (the listing) as a complement to the short TTL. It is the same principle as 04-05: "TTL as the safety net, explicit invalidation as the mechanism".
- Functions at the edge: JWT verification and personalisation
Modern CDNs run code at the point of presence (Cloudflare Workers, Lambda@Edge and CloudFront Functions, Fastly Compute): very small functions (milliseconds, a few MB) that see the request before it reaches the origin or the response before it reaches the client. Their uses in Kilometre Zero:
- JWT verification at the edge. A request with an invalid or expired token is rejected in Valencia, without travelling to Ireland or occupying Kong: lower latency for the error and less load on the origin under attack. The edge verifies the signature (with Keycloak's public key, cached) and the expiry; fine-grained authorization stays in the service (06-01), and Kong verifies again (defence in depth, as in 08-02).
- Personalisation without losing the cache. The
aged-cheeselisting is the same for everyone except for the market (price and availability per city). The Worker reads the market from a cookie or from the request's geolocation and adds it to the cache key (/products/aged-cheese?market=valencia), so there is one variant per market rather than one per user. - Redirects and A/B tests. Redirecting
/es/...according toAccept-Language, sending 10% of users to the new product page and measuring, without deploying anything at the origin. - Serving from the edge with a fallback. If the origin fails, serve the last cached version even if it has expired (
stale-if-error): the catalogue stays visible during an incident.
// km0/edge/worker/catalog.js — Cloudflare Worker: verifies the JWT and serves from the cache per market
import { jwtVerify, createRemoteJWKSet } from "jose"; // JOSE library; Keycloak's JWKS is cached at the edge
const JWKS = createRemoteJWKSet(new URL("https://id.km0.example/realms/km0/protocol/openid-connect/certs"));
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
if (!url.pathname.startsWith("/api/v1/catalog/")) return fetch(request); // everything else goes to the origin untouched
// 1. JWT verification at the edge: signature, issuer, audience and expiry (06-01)
const auth = request.headers.get("Authorization") || "";
const token = auth.startsWith("Bearer ") ? auth.slice(7) : null;
if (!token) return new Response(JSON.stringify({ error: "unauthenticated" }), { status: 401 });
let claims;
try {
({ payload: claims } = await jwtVerify(token, JWKS, { issuer: "https://id.km0.example/realms/km0", audience: "web-km0" }));
} catch (e) {
return new Response(JSON.stringify({ error: "invalid_token" }), { status: 401 });
}
// 2. Cache key per market (cookie or the request's country), NOT per user
const market = request.headers.get("Cookie")?.match(/market=([a-z]+)/)?.[1]
|| { ES: "valencia" }[request.cf?.country] || "girona";
const cacheKey = new Request(`${url.origin}${url.pathname}?market=${market}`, { method: "GET" });
// 3. Serve from the point of presence's cache if it is there
const cache = caches.default;
let response = await cache.match(cacheKey);
if (response) return new Response(response.body, { ...response, headers: { ...Object.fromEntries(response.headers), "X-Cache": "HIT" } });
// 4. Miss: go to the origin (Kong) with the market and the token (Kong verifies again: defence in depth)
const origin = new Request(`${url.origin}${url.pathname}?market=${market}`, {
headers: { "Authorization": auth, "X-Request-Id": crypto.randomUUID(), "X-Market": market },
});
response = await fetch(origin);
if (response.ok && (response.headers.get("Cache-Control") || "").includes("public")) {
ctx.waitUntil(cache.put(cacheKey, response.clone())); // store without delaying the response
}
return response;
},
};The Worker contains no business logic and touches no databases: it verifies, decides the cache key and forwards. Anything that needs real state (stock, orders) still goes to the origin. And there is a design constraint the code respects: the response cached per market must not contain user data; if catalog added "your favourites" to the listing, the per-market cache would serve Anna's favourites to Mark. That is why the listing is public and the favourites are a separate, private call.
- Edge for IoT: the van and the market stall offline
The most extreme edge is the device. Two cases in Kilometre Zero require working offline and syncing afterwards:
The van. In 08-02, paho queued positions in memory during the tunnel and sent them all at once on the way out: 36 messages nobody cared about one by one. A local aggregator on the device does three things better: it stores positions in a persistent queue on disk (SQLite) so that an app restart does not lose them, it aggregates locally (while there is no network, it condenses the leg into a single message with distance covered, time and a simplified path), and on regaining the connection it sends the current position first (the urgent part) and the summary afterwards (the historical part). It is the edge as a filter: the cloud receives less data, and more useful data.
The market stall. La Vega Farm sells at the Lleida market with a terminal that deducts local stock. Offline, it keeps selling: it is a replica that accepts writes while disconnected, that is, multi-leader replication (03-04) with guaranteed conflicts (the cloud reserves 3 zucchini for an online order while the stall sells 5 of the 6 that were left). 03-01 provided the tool for making the merge deterministic: CRDTs. A stall sales counter as a G-Counter (increment only; merging means taking the maximum per replica) syncs without conflict; available stock, which is a subtraction, is not a pure CRDT, and is resolved with the business rule from 03-04: the stall has authority over its physical stock (inv-lleida-stall) and the cloud over online reservations, and reconciliation may leave an online order without stock, which the saga rejects with compensation. What the edge cannot do is promise strong consistency offline: it is the partition of 03-02, and the stall chooses availability.
# km0/edge/van/aggregator.py
"""Local aggregator on the van: persistent queue in SQLite, offline aggregation and
sync on reconnection. Complements van_mqtt.py (08-02)."""
import json, math, sqlite3, time
import paho.mqtt.client as mqtt
DB_PATH = "/var/lib/km0/queue.sqlite"
T_POS = "km0/delivery/{id}/position"
T_LEG = "km0/delivery/{id}/leg"
def _distance_m(a, b) -> float:
"""Simplified haversine between (lat, lon) pairs, in metres."""
R = 6_371_000
p1, p2 = math.radians(a[0]), math.radians(b[0])
dp, dl = math.radians(b[0] - a[0]), math.radians(b[1] - a[1])
h = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2
return 2 * R * math.asin(math.sqrt(h))
class PersistentQueue:
"""On-disk queue: survives restarts of the app and of the device."""
def __init__(self, path: str = DB_PATH):
self.db = sqlite3.connect(path)
self.db.execute("PRAGMA journal_mode=WAL") # fast, safe writes
self.db.execute("""CREATE TABLE IF NOT EXISTS pending (
seq INTEGER PRIMARY KEY, timestamp_ms INTEGER, lat REAL, lon REAL, sent INTEGER DEFAULT 0)""")
def enqueue(self, seq: int, timestamp_ms: int, lat: float, lon: float) -> None:
with self.db:
self.db.execute("INSERT OR IGNORE INTO pending VALUES (?, ?, ?, ?, 0)", (seq, timestamp_ms, lat, lon))
def pending(self) -> list[tuple]:
return self.db.execute("SELECT seq, timestamp_ms, lat, lon FROM pending WHERE sent = 0 ORDER BY seq").fetchall()
def mark_sent(self, up_to_seq: int) -> None:
with self.db:
self.db.execute("UPDATE pending SET sent = 1 WHERE seq <= ?", (up_to_seq,))
self.db.execute("DELETE FROM pending WHERE sent = 1 AND timestamp_ms < ?",
(int(time.time() * 1000) - 24 * 3600 * 1000,)) # clean up anything older than a day
class Aggregator:
def __init__(self, courier: str, client: mqtt.Client, queue: PersistentQueue):
self.id, self.client, self.queue = courier, client, queue
self.connected = False
client.on_connect = lambda *a: self._on_connect()
client.on_disconnect = lambda *a: setattr(self, "connected", False)
def _on_connect(self) -> None:
self.connected = True
self.sync()
def new_position(self, seq: int, lat: float, lon: float) -> None:
timestamp_ms = int(time.time() * 1000)
self.queue.enqueue(seq, timestamp_ms, lat, lon) # ALWAYS to disk first
if self.connected:
self._publish_position(seq, timestamp_ms, lat, lon)
self.queue.mark_sent(seq)
# offline: nothing else happens; the position waits on disk
def _publish_position(self, seq, timestamp_ms, lat, lon) -> None:
payload = json.dumps({"courier": self.id, "seq": seq, "lat": lat, "lon": lon, "timestamp_ms": timestamp_ms})
self.client.publish(T_POS.format(id=self.id), payload, qos=1, retain=True)
def sync(self) -> None:
"""On regaining the connection: the urgent part first (current position), then the leg summary."""
pending = self.queue.pending()
if not pending:
return
last = pending[-1]
self._publish_position(*last) # 1. the current position, for Anna's map
if len(pending) > 1: # 2. the offline leg, aggregated into ONE message
points = [(p[2], p[3]) for p in pending]
distance = sum(_distance_m(points[i], points[i + 1]) for i in range(len(points) - 1))
leg = {"courier": self.id, "seq_start": pending[0][0], "seq_end": last[0],
"start_ms": pending[0][1], "end_ms": last[1], "distance_m": round(distance),
"points": len(points), "path": points[::max(1, len(points) // 10)]} # 10 points at most
self.client.publish(T_LEG.format(id=self.id), json.dumps(leg), qos=1)
self.queue.mark_sent(last[0])With the aggregator, exercise 2 of 08-02 gets a different answer: on leaving the tunnel, Anna receives one position (the current one), analytics receives one leg with the distance and ten points (enough for Flink's distance calculation, which can now sum distance_m instead of reconstructing it), and the broker does not receive 36 messages. The seq values remain monotonic and persisted, so the deduplication from 08-02 keeps working. What has to be added on the cloud side is a consumer of the new leg topic in the MQTT → Kafka bridge, feeding a delivery.legs topic.
- The cloud-edge-device continuum and security at the edge
Cloud, edge and device are not alternatives but a continuum: every computation and every piece of data is placed wherever the balance between latency, bandwidth, autonomy, capacity and control is best.
| Tier | What is there | Capacity | Latency to the user | Autonomy | What Kilometre Zero puts there |
|---|---|---|---|---|---|
| Cloud (region) | EKS, RDS, MSK, Cassandra, S3, Spark, Flink, Lambda | Unlimited | 20-80 ms | None without network | Everything transactional and analytical; the truth about orders, payments, online stock |
| Network edge (CDN, PoP) | Cache, Workers, TLS termination, filtering | High, but small functions with no state of their own | 5-15 ms | Serves cache if the cloud fails | Photos, catalogue, JWT verification, per-market personalisation, protection |
| Local edge (market, warehouse) | A terminal or mini-server | Low, with disk | < 1 ms | Total, with later sync | The stall's physical stock (inv-lleida-stall), in-person sales, CRDTs and reconciliation |
| Device (van, phone) | App, SQLite queue, aggregator | Minimal, on battery | 0 | Total, with sync | Positions, leg aggregation, pending commands, Anna's map with the last known position |
flowchart TB
subgraph Cloud[Cloud: eu-west-1]
K[(Kafka)] --> S[Services on EKS] --> D[(RDS, Cassandra, S3)]
L[Lambda thumbnails, invoices]
end
subgraph Edge[Network edge: CDN / PoP Valencia]
C[Cache of photos and catalogue]
W[Worker: JWT, key per market]
end
subgraph Local[Local edge: Lleida market]
T[Stall terminal<br/>local stock, CRDT]
end
subgraph Devices[Devices]
F[van-3<br/>SQLite queue, aggregator]
A["Anna's phone<br/>last known position"]
end
A -- "HTTPS" --> W --> C -. "miss" .-> S
F -- "MQTT, QoS 1" --> K
T -. "periodic sync<br/>CRDT merge" .-> S
S -- "WebSocket" --> A
Security at the edge. The further from the centre, the less physical control: a Worker runs on a third party's infrastructure, the market terminal sits on a table and the van can be stolen. Five rules:
- Nothing secret at the network edge: the Worker verifies signatures with the public key; it never holds the private one or database credentials. The secrets a Worker does need (an API key for the origin) are kept in the CDN's secret store, not in the code.
- The device authenticates with its own revocable identity: a certificate or credential per device (the MQTT password for
van-3from 08-02, issued when it was registered), which is revoked at the broker if the device is lost, and which only authorises its own topic prefix (ACL). - Data at rest encrypted on the device: the SQLite queue with positions and the commands with customer addresses are encrypted (SQLCipher or the operating system's secure store) and deleted once synced.
- Minimum data at the edge: the stall terminal does not need Anna's address or phone number; it receives only what it needs to sell (product, units, price), and what it sends (sales) carries no personal data.
- The edge is not the truth: any data coming from the device is validated in the cloud (the bridge in 08-02 checked that the
courierin the payload matches the one in the topic; alegwith 900 km in 3 minutes is discarded as invalid). A compromised device can lie; the design must limit what a lie can cause.
Common Mistakes and Tips
- Functions for everything. A constant consumer of 500 events/s or a service with a latency SLO is cheaper and more predictable as a container. Serverless for the sporadic, the event-driven and whatever scales to zero.
- Assuming a single execution. The provider retries; a function that is not idempotent generates two invoices or three thumbnails. Deterministic keys, idempotency markers, conditional operations.
- The function that triggers itself. Writing the thumbnail to the same bucket that triggers the function, with no prefix or suffix filter: infinite loop and unlimited bill. Filter on the event and check in the code.
- Database connections from a thousand instances. A thousand Lambdas open a thousand connections to PostgreSQL. Concurrency limit, RDS Proxy, or DynamoDB for function state.
- Initialisation inside the handler. Loading Pillow or creating the S3 client on every invocation multiplies duration and cost. Outside the handler, once per instance.
- No DLQ. A poison event is retried and lost without anyone knowing. DLQ with an alert and a reprocessing procedure, always.
- Caching responses with personal data on the CDN. A
publicon/api/v1/ordersserves Anna's orders to Mark.private, no-storefor everything personal, and a cache key per market, never per user, for what is shared. - Overwriting objects cached by the CDN. The cache never finds out. Immutable keys with a version (
?v=<etag>), as decided in 04-03. - Trusting the device. Validate in the cloud everything that comes from the edge; assume it may be compromised.
- Tip: always separate the handler (provider adapter) from the logic (pure Python). Tests run against the logic; lock-in is confined to the adapter.
- Tip: work out the cost of every function at its real frequency and at ten times that. If at ten times it is still cheap, it is a good case; if it explodes, prepare the exit to a container.
Exercises
Exercise 1: the invoice's ack point
In serverless/invoices/handler.py, claim(event_id) runs before generating and writing the PDF. (a) Describe the concrete failure this produces and how likely it is to happen. (b) Rewrite the relevant part of the handler so that idempotency is correct, reasoning about why writing twice to S3 is acceptable and what happens if the process dies at each possible point. (c) Would the answer be different if, instead of writing a PDF to S3, the function emailed the invoice?
Exercise 2: the CDN and Artisan Cheese Week
During the campaign, the website serves 6 million product views a day; each view downloads a thumb-800 (180 KB) and the JSON listing (4 KB). (a) With the design from section 9 (immutable photos, listing with max-age=60), estimate the expected hit ratio for each type and the daily egress from S3 and from Kong, compared with serving everything from the origin. (b) One day, Montblanc Dairy changes the aged-cheese photo and complains that "some customers still see the old one". What went wrong, if the design is correct? (c) The team proposes adding the user's name to the listing ("Hello, Anna") to personalise it. Explain the impact on the cache and propose an alternative.
Exercise 3: when to use Step Functions
For each workflow, decide whether Kilometre Zero should implement it with the in-house orchestrator from 03-05, with Step Functions, or with a simple event-triggered function, and justify it in terms of cost, latency, duration and visibility: (a) the order confirmation saga (1.2 M a month during a campaign, on the critical path); (b) offboarding a producer (dozens a year: cancel products, wait for in-flight orders to be delivered, settle payments after 30 days, archive photos, notify); (c) retrying deferred charges that the gateway rejects temporarily (hundreds a day, one retry every 6 hours for 3 days).
Solutions
Exercise 1.
(a) If the function dies (120 s timeout on a large batch, network error with S3, instance retirement) after claim and before put_object, the invoice#<event_id> marker is written and the PDF is not. On the Lambda retry (or in the next batch, because the offset did not advance), claim returns False and the invoice is skipped forever: lost invoice, with no error and no alert. The probability per event is low (the window is a few milliseconds between two calls), but with 1.2 M orders a month and instance restarts, it will happen several times a month.
(b) Reorder: generate and write first, mark afterwards; and use the marker only to avoid repeated work, not for correctness.
order = ev["data"]
key = f"invoices/{order['customer']}/{order['order_id']}.pdf"
if already_claimed(ev["event_id"]): # read only, no write: avoids regenerating the PDF if it was already done
duplicates += 1; continue
s3.put_object(Bucket=INVOICES_BUCKET, Key=key, Body=generate_pdf(order), ...) # deterministic key
mark(ev["event_id"]) # PutItem without a condition (or with one, ignoring the failure)Analysis by point of death: before put_object: nothing written, the retry does everything. Between put_object and mark: the PDF exists, the marker does not; the retry regenerates the same PDF and writes it to the same key (S3 replaces the whole object; with versioning one more version is kept, harmlessly) and marks it. After mark: everything done. In no case is the invoice lost, and the worst outcome is a regenerated PDF. Writing twice is acceptable because the operation is naturally idempotent: the same deterministic content (the PDF does not carry the generation time; if it did, it would have to be pinned to the event's timestamp_ms) at the same key.
(c) Yes. Sending an email is not idempotent by nature: sending it twice annoys Anna. Then the marker before sending is necessary to avoid duplicates, but it leaves the loss window open. The solution is the one from 02-05 with the gateway: use an email service that accepts an idempotency key (many offer one per message), so that it can be sent "again" without duplicating; or split it into two steps, generate the PDF (idempotent, in S3) and enqueue the send on a queue with deduplication by id (SQS FIFO with MessageDeduplicationId), where the send is retried safely. Without one of those two things, you have to choose between losing or duplicating, and for an email duplicating is preferable.
Exercise 2.
(a) Photos: 4,200 products, immutable objects, a one-year TTL; once warmed up, every point of presence holds all the thumbnails: hit ratio > 98% (the misses are only the first access to each object at each PoP and new photos). Egress from S3: 6 M × 180 KB ≈ 1.08 TB/day without a CDN; with 98% hits, ≈ 22 GB/day from S3 (plus the CDN-to-user egress, which is usually much cheaper or included). Listings: max-age=60 and 4,200 products × 4 markets = 16,800 variants; 6 M views/day is 70 per second, spread over 16,800 keys: on average each key is requested every 4 minutes, so at most PoPs the listing has expired by the time it is requested again; hit ratio perhaps 30-60% (better for popular products, worse for the long tail). stale-while-revalidate=300 raises that ratio a lot (it serves the expired one and refreshes in the background), at the cost of up to 5 minutes of staleness. Egress from Kong: 6 M × 4 KB = 24 GB/day without a CDN; with 50%, 12 GB. The photo dominates: the CDN eliminates 98% of ~1.1 TB a day; for the listing, the benefit is latency and load on catalog more than egress.
(b) If the design is correct (the new photo has a new ETag and the web references ?v=<new etag>), what went wrong is that the listing containing the photo URL is cached: customers who receive a cached listing (up to 60 s, or up to 5 minutes with stale-while-revalidate) receive the old URL, which still points to an immutable, valid object, the old photo. It is not a failure of the photo but of the listing's TTL, and it is the expected behaviour: "some customers for a few minutes". If the problem lasted hours, the cause would be something else: the web referencing the photo without ?v= (key overwriting, the mistake from 04-03) or the stock.updated consumer invalidating Redis but not the CDN (04-05 warned about invalidating at every tier).
(c) With "Hello, Anna" in the body of the listing, the response is no longer the same for every user in a market: either it is marked private (and the CDN cache is lost: the listing's hit ratio drops to 0 and catalog takes the full 70 req/s) or, worse, it is cached per market and Mark sees "Hello, Anna". The alternative: the listing stays public and per market, and personalisation happens on the client (the browser already has the name in the JWT or in the session and renders it) or via a small second call marked private (/api/v1/me) that the CDN never caches. It is the Worker's principle: separate the shared, cacheable part from the personal one.
Exercise 3.
(a) In-house orchestrator in orders. High and constant volume (≈ €150/month in Step Functions transitions alone, plus the invocations), on the critical path (every transition adds tens of ms to the p99, compromising the 500 ms SLO), duration of seconds, and already built and tested with Testcontainers. Visibility comes from the traces of 07-02 and the sagas table.
(b) Step Functions. Dozens a year (negligible cost), duration of weeks (30-day waits that an in-house orchestrator would have to manage with persisted state and timers: Step Functions has the Wait state with dates and executions of up to a year), outside any critical path, and with enormous visibility value (seeing which step each producer's offboarding is at, who approved it, why the settlement failed). Each step invokes the APIs of the existing services. The lock-in is accepted because the workflow is peripheral.
(c) A timer-triggered function (or a delayed queue), neither Step Functions nor the orchestrator. Hundreds a day with one retry every 6 hours for 3 days is a delayed-retry queue pattern: SQS with DelaySeconds (maximum 15 minutes, so it gets chained) or, simpler still, a pending_charges table with a next_attempt column and a Lambda scheduled every 15 minutes that processes the due ones with the Idempotency-Key from 02-05 and publishes payment.confirmed or payment.rejected when finished. Step Functions with a 6-hour Wait would work, but it is a state machine for what is a loop with a date; the cost and complexity are not justified, and the table provides the visibility.
Conclusion
Serverless and edge computing widen Kilometre Zero's decision space in two directions opposite to the cluster in one region. With FaaS, the provider runs ephemeral functions in response to events, scales from zero to thousands and charges per invocation and GB-second; in exchange it imposes cold starts, time and memory limits, the absence of state and, above all, retries we do not control, which turn the idempotency of 02-05 from good practice into an obligation. With that filter, the thumbnails for km0-photos, the invoices on payment.confirmed, the gateway webhooks and the scheduled tasks leave the services and become functions defined with SAM, with a DLQ, a concurrency limit and filters that prevent loops, while orders, the WebSocket of 08-02, Flink and Spark stay where they were. The surrounding serverless services (queues, DynamoDB, API Gateway, Step Functions) complete the model, and the saga of 03-05 rewritten in Amazon States Language shows precisely what is gained (managed state, declarative retries, visibility) and what is paid (latency, cost per transition, lock-in), which leaves the order saga in orders and reserves Step Functions for long, sporadic workflows. In the other direction, the edge brings data and compute closer to whoever uses them: the CDN serves immutable photos and per-market listings with TTL and invalidation designed per content type, the Worker verifies the JWT and decides the cache key without touching the origin, the van's aggregator turns 36 tunnel positions into one position and one leg, and the market stall keeps selling offline with CRDTs and authority-based reconciliation, accepting the partition of 03-02 rather than denying it. The cloud-edge-device continuum places every piece of data where the balance is best, and security at the edge starts from the premise that the edge can lie.
With this, every piece of Kilometre Zero is on the table: from the bounded context cut of 08-01 to the cache in Valencia and the SQLite queue in the van. The last lesson puts them together: the complete architecture in a single diagram, the journey of one of Anna's orders end to end with everything it leaves in its wake, the decisions and their trade-offs revisited as ADRs, the five symptoms from 01-06 with their resolution, the assessment against the course's criteria, what is left out, the project the student can build on their laptop and the guide for going deeper. It is the Final Project: Kilometre Zero End to End.
Distributed Architectures Course
Module 1: Introduction to Distributed Systems
- Basic Concepts of Distributed Systems
- Distributed System Models
- Advantages and Challenges of Distributed Systems
- The Fallacies of Distributed Computing
- Time, Clocks and Event Ordering
- From Monolith to Distributed Platform: the Kilometre Zero Case
Module 2: Communication in Distributed Systems
- Communication Protocols
- RPC and RMI
- gRPC and Data Serialization
- Messaging and Message Queues
- Asynchronous Communication Patterns
Module 3: Consistency and Replication
- Consistency Models
- The CAP Theorem and PACELC
- Consensus Algorithms
- Data Replication
- Distributed Transactions and Sagas
Module 4: Distributed Storage
- Data Partitioning and Consistent Hashing
- Distributed File Systems
- Object Storage
- Distributed Databases
- Distributed Caches
Module 5: Distributed Computing
- Distributed Computing Models
- MapReduce and Hadoop
- Spark and In-Memory Computing
- Stream Processing
- Job Scheduling and Data Pipelines
Module 6: Security in Distributed Systems
- Authentication and Authorization
- Encryption and Data Protection
- Identity Management
- Service-to-Service Security: mTLS and Secrets Management
- API Gateways, Rate Limiting and Auditing
Module 7: Monitoring and Maintenance
- Monitoring Distributed Systems
- Centralized Logs and Distributed Tracing
- Failure Management and Recovery
- Resilience Patterns: Timeouts, Retries and Circuit Breakers
- Automation and Orchestration
- Testing Distributed Systems and Chaos Engineering
