In 06-01 we took the decision: the basket and the sessions leave PostgreSQL. The numbers behind it
were overwhelming —180,000 writes a day, 92 % of them updates to the same record, access always by
key, zero exploratory queries and a 78 GB table of which 55 GB are baskets abandoned more than two
months ago. That table is the one that triggers the VACUUM that consumes the most I/O in
mercadofresco-pedidos.
Amazon DynamoDB is AWS's managed key-value and document database. There are no servers to run,
no versions to upgrade and no connections to exhaust: there is an HTTP API and a data model that, if
it is used well, answers in less than 10 milliseconds regardless of whether the table holds a thousand
items or a thousand million. That "if it is used well" is the entire lesson: DynamoDB punishes
relational design without mercy and rewards design driven by the access pattern. Here Luis models
mercadofresco-carritos from the queries the application really runs, works out the capacity with the
figures from the Friday peak and switches on the TTL that turns the 55 GB of rubbish into a problem
that can no longer repeat itself.
Cost warning. On-demand tables cost nothing if they are not used, but they do cost storage (0.25 USD per GB per month) and PITR backups; provisioned ones charge for reserved capacity even when no traffic arrives. When you are done, delete the test tables. Fictitious data.
Contents
- What DynamoDB is and what problem it solves
- Data model: table, item, attributes and types
- Primary key: partition and sort
- Partitions, hashing and hot keys
- Designing from the queries, not from the entities
- Single-table design applied to
mercadofresco-carritos - Write operations and condition expressions
- Reading:
GetItem,Queryand whyScanis almost always a mistake - Batch operations and transactions
- Secondary indexes: GSI and LSI
- Capacity, RCU and WCU with the Friday figures
- Bursts, throttling and retries with exponential backoff
- Eventual consistency and strong consistency
- TTL: the end of the zombie baskets
- Streams, backups, encryption and global tables
- Access from boto3 and from Lambda
- Cost compared with keeping it in RDS
- The migration, step by step
- Common mistakes and tips
- Exercises
- Conclusion
What DynamoDB is and what problem it solves
DynamoDB is a fully managed, serverless service. You do not choose an instance size, you do not apply a patch, you do not size the storage. You create a table and you write to it. The structural difference with RDS sums up why it fits the basket:
| RDS PostgreSQL | DynamoDB | |
|---|---|---|
| Unit of scale | Instance (vertical) | Partitions (horizontal, automatic) |
| Connections | Limited (200 on db.t3.small) |
None: signed HTTP requests |
| Latency as volume grows | Degrades | Constant by design |
| Queries | Arbitrary SQL | Only by key and defined indexes |
| Schema | Fixed | Only the primary key is mandatory |
| Data expiry | A process you have to write | Native TTL, free of charge |
| Maintenance | VACUUM, versions, windows |
None |
The connections row matters more than it looks: the mercadofresco-rds-conexiones-altas alarm from
05-01 exists because the instance has a hard ceiling and the Lambda functions brush against it at the
Friday peak. DynamoDB has no such concept —every operation is an independent HTTP call authenticated
with IAM— so a thousand concurrent Lambdas exhaust nothing.
Data model: table, item and attributes
Three concepts and no more. A table is a collection of items with no schema beyond the primary key. An item is the equivalent of a row and takes up at most 400 KB, attribute names included. An attribute is a name-value pair, and two items in the same table can have completely different attributes.
The absence of a schema is freedom and it is danger. Nothing stops you storing precio as a number
in one item and as a string in another; validation is the application's responsibility. At
MercadoFresco that validation lives in a single data access layer that all the code uses, precisely so
that it does not depend on the discipline of whoever writes each endpoint.
Primary key: partition and sort
The primary key is the only mandatory structure and the hardest decision to reverse: changing it means creating another table and migrating.
A simple key is just the partition key (PK, or hash key): it identifies an item uniquely and is accessed by exact equality, nothing more. A composite key adds the sort key (SK, or range key): every item with the same PK lives together, physically ordered by SK. That enables the most powerful operation in DynamoDB, retrieving a whole range of related items in a single operation.
PK = CLIENTE#4471 SK = CARRITO#2026-08-02T18:04:11Z SK = CARRITO#2026-07-28T09:12:40Z SK = PERFIL SK = SESION#a91f...
A single operation can ask for "all the items of CLIENTE#4471 whose SK begins with CARRITO#,
ordered from most recent to oldest, the first 5". That query was, in PostgreSQL, a SELECT … WHERE id_cliente = ? ORDER BY fecha DESC LIMIT 5 with its index; here it is a direct access to a contiguous
block of disk.
Data types
| Category | Types | Notation | Notes |
|---|---|---|---|
| Scalars | String, Number, Binary, Boolean, Null | S, N, B, BOOL, NULL |
Numbers with 38 digits of precision |
| Documents | List, Map | L, M |
Nestable up to 32 levels |
| Sets | Strings, Numbers, Binaries | SS, NS, BS |
No duplicates, no ordering |
Two practical warnings. There is no date type: you use ISO 8601 strings (2026-08-02T18:04:11Z),
which sort alphabetically the same way they sort chronologically, or Unix numbers when they have to be
compared. And an empty string is accepted in non-key attributes, but not in key attributes: an
empty id_sesion raises a validation error.
Partitions, hashing and hot keys
DynamoDB spreads the data by applying a hash function to the value of the partition key. The result determines which physical partition the item falls into. Each partition supports at most 3,000 RCU and 1,000 WCU, and that limit explains almost every real performance problem.
graph LR
A["PK = CLIENTE#4471"] -->|hash| P1[Partition 1]
B["PK = CLIENTE#8802"] -->|hash| P2[Partition 2]
C["PK = CLIENTE#1043"] -->|hash| P1
D["PK = CLIENTE#9917"] -->|hash| P3[Partition 3]
P1 --> L1["Each partition: max. 3,000 RCU / 1,000 WCU"]
P2 --> L1
P3 --> L1
A hot key is a PK value that concentrates a disproportionate share of the traffic. The whole table can have capacity to spare and still throttle because everything goes to one partition.
| Hot key | Why it fails | Fix |
|---|---|---|
PK = FECHA#2026-08-02 for the day's orders |
All the day's traffic in one partition | PK = PEDIDO#<id> and a GSI to query by date |
PK = ESTADO#pendiente |
Millions of items under one value | PK = PEDIDO#<id>, sparse GSI with the pending ones only |
PK = TIENDA#mercadofresco |
A single PK for everything | Spread it by customer or by product |
PK = PRODUCTO#<id> on a viral product |
One product absorbs everything | Spread the write: PRODUCTO#<id>#<0-9> and read all 10 |
In mercadofresco-carritos the PK is CLIENTE#<id>, with tens of thousands of distinct values and
traffic spread naturally. It is the easy case, and it is worth saying that it is easy because the
access pattern was easy: if the access had been "give me every basket abandoned today", the design
would be a different one.
Designing from the queries, not from the entities
The correct procedure has three steps and none of them is drawing entities.
Step 1: list every query, extracted from the application's real logs:
| # | Query | Daily frequency |
|---|---|---|
| C1 | Get a customer's active basket | 210,000 |
| C2 | Add, change the quantity of or remove a line | 180,000 |
| C3 | Get the web session by its identifier | 340,000 |
| C4 | List a customer's last 5 baskets (customer service) | 400 |
| C5 | Mark a basket as converted into an order | 4,000 |
| C6 | Delete baskets abandoned more than 30 days ago | Continuous |
Step 2: for each query, decide how it is resolved. The rule is that every frequent query must
be resolved with a GetItem or a Query, on the table or on an index. If any of them requires a
Scan, the model is wrong. Step 3: design the key so that this holds. And only then look at which
entities are left.
Single-table design applied to mercadofresco-carritos
Single-table design consists of keeping several types of entity in the same table, telling them
apart by prefixes in the key. It sounds odd coming from SQL and it answers a very specific reason:
there is no JOIN in DynamoDB, so the way to retrieve related entities in a single operation is
for them to share a partition key.
| Entity | PK | SK | Attributes |
|---|---|---|---|
| Basket | CLIENTE#<id> |
CARRITO#<ts_iso> |
lineas (L of M), importe, estado, expira_en |
| Session | CLIENTE#<id> |
SESION#<id_sesion> |
ip, agente, ultima_actividad, expira_en |
| Light profile | CLIENTE#<id> |
PERFIL |
ciudad, franja_reparto, alergenos |
graph TD
subgraph "Partition CLIENTE#4471"
P["SK = PERFIL · Valencia · 18-21h slot"]
C1["SK = CARRITO#2026-08-02T18:04:11Z · activo · 34.20 EUR"]
C2["SK = CARRITO#2026-07-28T09:12:40Z · convertido · 51.90 EUR"]
S1["SK = SESION#a91f3c · ultima_actividad 18:06:02"]
end
Q1["C1 and C4: Query PK + SK begins_with CARRITO#<br/>ScanIndexForward false, Limit 1 or 5"] --> C1
Q2["C3: GetItem with exact PK and SK"] --> S1
With this design, one single Query on CLIENTE#4471 returns the profile, the active basket and the
sessions —everything the shop needs to render the header— in one operation and a few milliseconds. In
PostgreSQL that was three queries against three tables.
When single-table design does not pay off: when the entities are never queried together, when they
have very different life cycles and capacity patterns, or when the team does not yet master the model.
Putting orders and catalogue in one table "because it is best practice" is taking on the complexity
without collecting the benefit. Here it is justified because C1, C3 and C4 share id_cliente.
Write operations: PutItem, UpdateItem, DeleteItem
import boto3
from datetime import datetime, timezone, timedelta
from decimal import Decimal
session = boto3.Session(profile_name="mercadofresco-dev", region_name="eu-west-1")
table = session.resource("dynamodb").Table("mercadofresco-carritos")
now = datetime.now(timezone.utc)
ts = now.strftime("%Y-%m-%dT%H:%M:%SZ")
# PutItem: creates the item or REPLACES COMPLETELY the one holding that key.
# That "replaces completely" is the number one source of data loss:
# the attributes you do not send disappear.
table.put_item(Item={
"PK": "CLIENTE#4471",
"SK": f"CARRITO#{ts}",
"estado": "activo",
"importe": Decimal("0.00"),
"lineas": [],
# expira_en is a Unix number in seconds: that is what the TTL knows how to read.
"expira_en": int((now + timedelta(days=30)).timestamp()),
})UpdateItem is the operation to use 90 % of the time: it modifies specific attributes without
reading the item first and it is atomic, so two concurrent requests incrementing a counter give
the right result with no locks.
# Add a line to the basket and add up the amount, in a single atomic operation.
response = table.update_item(
Key={"PK": "CLIENTE#4471", "SK": f"CARRITO#{ts}"},
UpdateExpression=(
"SET #lineas = list_append(if_not_exists(#lineas, :vacia), :linea), "
" #actualizado = :ahora ADD #importe :precio"
),
# Names go through aliases because many words are reserved in DynamoDB
# ("estado", "name", "size"...). Always using aliases avoids surprises.
ExpressionAttributeNames={
"#lineas": "lineas", "#importe": "importe", "#actualizado": "actualizado_en"},
ExpressionAttributeValues={
":vacia": [],
":linea": [{"sku": "PESC-SALM-001", "uds": 2, "precio": Decimal("12.40")}],
":precio": Decimal("24.80"),
":ahora": ts},
ReturnValues="ALL_NEW", # returns the resulting item, with no extra read
)The four verbs: SET assigns or creates (SET estado = :s), ADD adds to a number or adds to a set
(ADD importe :p), REMOVE deletes an attribute or a list element (REMOVE lineas[2]) and DELETE
removes elements from a set (DELETE etiquetas :e).
DeleteItem deletes by key. In mercadofresco-carritos it is barely used: baskets are deleted by
the TTL and converted ones are marked, not removed.
Condition expressions and conditional writes
A condition expression makes the write apply only if it is met; if it is not, the operation fails
with ConditionalCheckFailedException. It is DynamoDB's optimistic locking mechanism and it replaces
the SELECT … FOR UPDATE of the relational world.
from botocore.exceptions import ClientError
try:
# Mark the basket as converted ONLY if it is still active. This stops two
# presses of the "Pay" button generating two orders from the same basket.
table.update_item(
Key={"PK": "CLIENTE#4471", "SK": f"CARRITO#{ts}"},
UpdateExpression="SET #e = :nuevo, id_pedido = :pedido",
ConditionExpression="attribute_exists(PK) AND #e = :esperado",
ExpressionAttributeNames={"#e": "estado"},
ExpressionAttributeValues={
":nuevo": "convertido", ":esperado": "activo", ":pedido": "PED-84213"},
)
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
# Not a system error: it is the correct outcome of a race.
print("The basket had already been converted; the order is not duplicated.")
else:
raiseCommon conditions: attribute_exists(PK) for "update only if it exists",
attribute_not_exists(PK) for "create only if it does not exist" —the idempotency pattern picked up
again in 07-05— comparators (<, >, BETWEEN) and size().
Reading: GetItem, Query and why Scan is almost always a mistake
from boto3.dynamodb.conditions import Key, Attr
# GetItem: one item by its complete key. The cheapest and fastest operation.
web_session = table.get_item(
Key={"PK": "CLIENTE#4471", "SK": "SESION#a91f3c"},
ConsistentRead=False, # eventual: half the cost, enough here
).get("Item")
# Query: items with the SAME partition key, filtering by the sort key.
# This is C1: the customer's most recent active basket.
result = table.query(
KeyConditionExpression=Key("PK").eq("CLIENTE#4471") & Key("SK").begins_with("CARRITO#"),
FilterExpression=Attr("estado").eq("activo"),
ScanIndexForward=False, # descending: the SK timestamp gives the most recent first
Limit=1,
)
basket = result["Items"][0] if result["Items"] else NoneThe difference between the two expressions is the one that costs the most money when it is not
understood: KeyConditionExpression acts before reading —it chooses what is read, only on PK and
SK, and charges only for what it returns— whereas FilterExpression acts afterwards —it discards
results, accepts any attribute, and charges for everything read, returned or not. Filtering by
estado when the partition holds 3 baskets is irrelevant; doing it over a partition with 50,000
items means paying for 50,000 reads to return 1.
Scan reads the whole table and then applies the filter. With 78 GB migrated, a Scan is some
10 million RCU: more than 1.25 USD per run and several minutes. And the worst part is not the cost: it
is that it consumes the capacity the real operations need, throttling the shop.
Scan is acceptable in three cases: small, stable tables (a few thousand items), out-of-hours batch
processes with Segment/TotalSegments to parallelise, and one-off exports. Any other use means an
index is missing or the key is badly designed.
Batch operations and transactions
| Operation | Limit | Atomic | Cost | Use at MercadoFresco |
|---|---|---|---|---|
BatchGetItem |
100 items / 16 MB | No | Normal | Load several profiles at once |
BatchWriteItem |
25 items / 16 MB | No | Normal | Initial load of the migration |
TransactWriteItems |
100 items / 4 MB | Yes | ×2 | Convert basket and mark session |
TransactGetItems |
100 items / 4 MB | Yes | ×2 | Coherent read of several items |
Batches are not atomic: if 3 of 25 writes fail, the other 22 are applied and the failed ones come
back in UnprocessedItems, which you have to retry because the SDK does not do it for you.
client = session.client("dynamodb")
# TransactWriteItems: all or nothing. Converting the basket and closing the session
# have to happen together. It costs twice the capacity; use it only when needed.
client.transact_write_items(TransactItems=[
{"Update": {
"TableName": "mercadofresco-carritos",
"Key": {"PK": {"S": "CLIENTE#4471"}, "SK": {"S": f"CARRITO#{ts}"}},
"UpdateExpression": "SET #e = :c",
"ConditionExpression": "#e = :a", # the condition aborts the WHOLE transaction
"ExpressionAttributeNames": {"#e": "estado"},
"ExpressionAttributeValues": {":c": {"S": "convertido"}, ":a": {"S": "activo"}},
}},
{"Update": {
"TableName": "mercadofresco-carritos",
"Key": {"PK": {"S": "CLIENTE#4471"}, "SK": {"S": "SESION#a91f3c"}},
"UpdateExpression": "REMOVE carrito_activo",
}},
])An important conceptual limit: a DynamoDB transaction does not allow intermediate logic. You cannot read, decide in the code and write inside the same transaction. That is why 06-01 ruled DynamoDB out for the order confirmation flow.
Secondary indexes: GSI and LSI
A secondary index lets you query by attributes that are not the primary key:
| GSI (global) | LSI (local) | |
|---|---|---|
| Index PK | Any attribute | The same as the table |
| Index SK | Any attribute | Another attribute |
| When it is created | At any time | Only when creating the table |
| Maximum | 20 (can be raised) | 5 |
| Capacity | Its own and independent | Shared with the table |
| Consistency | Eventual only | Eventual or strong |
| Partition limit | None | 10 GB per PK |
The practical rule: use a GSI unless you need a strongly consistent read by an alternative
attribute. LSIs are decided on the first day and for ever, and their limit of 10 GB per partition is
a silent trap. mercadofresco-carritos needs a GSI for C6 and for customer service:
# GSI "gsi-estado-fecha": lists baskets by status and date range without
# going through the table. With on-demand capacity there is nothing to size.
aws dynamodb update-table --table-name mercadofresco-carritos \
--attribute-definitions AttributeName=estado,AttributeType=S \
AttributeName=actualizado_en,AttributeType=S \
--global-secondary-index-updates '[{"Create": {
"IndexName": "gsi-estado-fecha",
"KeySchema": [{"AttributeName": "estado", "KeyType": "HASH"},
{"AttributeName": "actualizado_en", "KeyType": "RANGE"}],
"Projection": {"ProjectionType": "INCLUDE",
"NonKeyAttributes": ["PK", "importe"]}}}]' \
--region eu-west-1 --profile mercadofresco-devProjections: KEYS_ONLY copies only the keys (minimum storage, useful when locating and then
reading is enough), INCLUDE copies the keys plus the chosen attributes (the usual case) and ALL
copies the whole item, duplicating the table. A GSI with ALL over 78 GB adds 78 GB and consumes
WCU on every write to the base table; choosing INCLUDE with the three attributes actually drawn on
screen usually cuts that cost by 80 %.
Watch out for the GSI's hot key: estado has few distinct values, so gsi-estado-fecha concentrates
traffic. It is fine for low-frequency batch processes; it must not be used on the shop's critical
path.
Capacity: on-demand versus provisioned
| On-demand | Provisioned | |
|---|---|---|
| You pay | Per request | Per reserved capacity per hour |
| Scaling | Instant | With auto scaling, in minutes |
| Unit price | ~6.9× more expensive per operation | Cheaper if you use it up |
| Risk | A surprise bill | Throttling |
| When | Irregular or unknown traffic | Predictable, sustained traffic |
You can switch mode, but only once every 24 hours, so it is not a lever for the Friday peak. For MercadoFresco the recommendation is to start on-demand during the migration and the first two months: we do not know the real traffic of the new table and the cost of getting provisioned capacity wrong is throttling the basket on a Friday. With two months of metrics, you recalculate.
RCU and WCU with the Friday figures
Capacity units are DynamoDB's currency. 1 WCU is one write per second of up to 1 KB (a 3.5 KB write costs 4 WCU). 1 RCU is one strongly consistent read per second of up to 4 KB, or two eventual reads (an eventual read of 7 KB costs 1 RCU). Transactional operations, both reads and writes, cost double.
Calculation for the Friday peak, 17:00-21:00, with 900 orders/hour:
| Operation | Requests/s at peak | Average size | Consistency | Units |
|---|---|---|---|---|
| C2 Update basket | 12 | 3 KB | — | 12 × 3 = 36 WCU |
| C5 Convert (transactional) | 0.25 | 3 KB | — | 0.25 × 3 × 2 = 1.5 WCU |
| Write/refresh session | 20 | 1 KB | — | 20 WCU |
| C1 Read basket | 24 | 3 KB | eventual | 24 × (4÷4) ÷ 2 = 12 RCU |
| C3 Read session | 40 | 1 KB | eventual | 40 × 1 ÷ 2 = 20 RCU |
| C4 Customer service | 0.1 | 15 KB | eventual | 1 RCU |
Peak totals: ≈58 WCU and ≈33 RCU. With provisioned capacity and auto scaling at 70 % utilisation,
you would size it to around 85 WCU and 50 RCU in the Friday window, dropping to 15 WCU and 10 RCU in
the small hours. The conclusion worth internalising: the workload that was suffocating
db.t3.small fits in fewer than 100 capacity units. The problem was never the volume; it was that
those writes shared an engine, memory and VACUUM with the order transactions.
Bursts, throttling and retries with exponential backoff
DynamoDB accumulates burst capacity: the capacity unused over the last 300 seconds is saved up and can be spent in one go. It is a useful cushion and it must not be designed for: it is not guaranteed.
When the available capacity is exceeded, the operation is rejected with
ProvisionedThroughputExceededException (or ThrottlingException). Causes, in order of real
frequency: a hot key, insufficient provisioned capacity, a GSI with less capacity than the base table,
and growth faster than auto scaling reacts.
from botocore.config import Config
# The SDK retries by itself, but conservatively by default. The "adaptive"
# mode adjusts the send rate to the throttling observed, which is exactly
# what you want on a table shared by many Lambdas.
config = Config(
retries={"max_attempts": 10, "mode": "adaptive"},
connect_timeout=1, read_timeout=3, # fail fast: a Lambda must not hang
)
table = session.resource("dynamodb", config=config).Table("mercadofresco-carritos")Exponential backoff with jitter is essential: without the random part, every client retries at the same time and reproduces the burst that caused the problem.
Minimum CloudWatch watch towards alertas-mercadofresco: ThrottledRequests > 0 in 5 minutes,
sudden rises in UserErrors (almost always a deployment with a bug),
SuccessfulRequestLatency p99 > 25 ms and ConsumedWriteCapacityUnits above 80 % of what is
provisioned both on the table and on each index.
Eventual consistency and strong consistency
Every write is replicated across three AZs. An eventual read may reach a replica that does not
yet have the latest change; the typical lag is under a second. Eventual is the default and costs
0.5 RCU per 4 KB; strong (ConsistentRead=True) costs 1 RCU, has slightly more latency and is not
available on GSIs.
Applied to MercadoFresco, it is the 06-01 rule —eventual to show, strong to decide— turned into code: drawing the header with the number of items goes eventual, but reading the basket just before converting it into an order goes strong, because what is charged depends on that read.
TTL: the end of the zombie baskets
This is the feature that on its own justifies the migration. The TTL (Time To Live) automatically deletes items whose expiry attribute —a Unix number in seconds— has passed, and the deletion is free: it consumes no WCU.
aws dynamodb update-time-to-live --table-name mercadofresco-carritos \
--time-to-live-specification "Enabled=true, AttributeName=expira_en" \
--region eu-west-1 --profile mercadofresco-devFour things to know so you do not get a nasty surprise:
- Deletion is not immediate. DynamoDB removes items within a typical window of 48 hours. If the application must not see what has expired, you have to filter it on read as well.
- The attribute has to be a number in seconds. Milliseconds, or an ISO string, make the TTL ignore the item silently. It is the most common mistake.
- Deletions appear in Streams with a
userIdentityof typeService, which makes it possible to archive the abandoned basket before it disappears. - It does not replace a documented retention policy. Under the GDPR you have to be able to explain how long data is kept and why; the TTL implements the policy, it does not define it.
Compared with what exists today —a scheduled nightly DELETE that costs I/O, locks and VACUUM,
that nobody notices when it fails and that has let the table grow to 78 GB— the TTL is free, cannot
fail silently and keeps the size stable by design.
Streams: reacting to changes
DynamoDB Streams publishes an ordered record of every modification, with 24 hours of retention
and four possible views: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE and NEW_AND_OLD_IMAGES. A Lambda
function subscribes to the stream and reacts to each change.
Planned uses: archiving into mercadofresco-informes-analitica the baskets the TTL is about to
delete —Sara wants to analyse abandonment in 06-04— and feeding business metrics. Event orchestration
on a larger scale, with routing and several decoupled consumers, is EventBridge and it is covered in
07-03; Streams is the low-level mechanism tied to this table.
Backups, encryption and global tables
Backups. PITR restores to any second within the last 35 days, is enabled with
aws dynamodb update-continuous-backups --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true and always restores to a new table: it does not overwrite.
Encryption. Always on at rest, with no option to turn it off. With the customer managed key
alias/mercadofresco-datos from 04-02 you get what MercadoFresco's criteria demand: control over
rotation, a record of every use in trail-mercadofresco and the ability to revoke access to the data
by revoking permissions on the key. Global tables, finally, replicate active-active across regions
with last-writer-wins conflict resolution; MercadoFresco operates only in eu-west-1 and does not
need them, but they are the equivalent of Aurora Global Database (06-03) should there ever be
operations in France.
Access from boto3 and from Lambda
resource converts types automatically (Decimal ↔ N) and is the interface for business logic;
client exposes the raw API with the explicit descriptor ({"S": "..."}) and is the only one that
covers transactions and administration. A warning: resource demands Decimal and rejects
float with Float types are not supported. It is a deliberate nuisance: floating-point amounts
are a bug in any system that handles money.
Least-privilege policy for the mercadofresco-estado-pedido Lambda:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OperacionesDelCarrito",
"Effect": "Allow",
"Action": ["dynamodb:GetItem", "dynamodb:Query",
"dynamodb:PutItem", "dynamodb:UpdateItem"],
"Resource": [
"arn:aws:dynamodb:eu-west-1:111122223333:table/mercadofresco-carritos",
"arn:aws:dynamodb:eu-west-1:111122223333:table/mercadofresco-carritos/index/*"
],
"Condition": {
"ForAllValues:StringEquals": {
"dynamodb:LeadingKeys": ["${aws:PrincipalTag/cliente}"]
}
}
},
{
"Sid": "SinScanNiBorradoNiAdministracion",
"Effect": "Deny",
"Action": ["dynamodb:Scan", "dynamodb:DeleteTable", "dynamodb:DeleteItem"],
"Resource": "*"
}
]
}Three deliberate decisions. dynamodb:LeadingKeys limits operations to the partition keys that match
a tag on the principal: it is per-customer isolation at the IAM level, not in the code. The Deny
on Scan is not paranoia, it is economics: it stops a rushed deployment putting a Scan on the
critical path. And the Deny on DeleteItem forces deletion to always be the TTL.
Cleanup. If you have created tables to practise, delete them with
aws dynamodb delete-table. An empty on-demand table costs pennies, but a test one with PITR enabled and a few GB loaded does show up on the bill.
Cost compared with keeping it in RDS
Estimated monthly figures for eu-west-1, with the workload described:
| Item | Today in mercadofresco-pedidos |
DynamoDB on-demand | DynamoDB provisioned |
|---|---|---|---|
| Writes (5.4 M/month) | Included | 4.05 USD | — |
| Reads (17.4 M/month) | Included | 0.44 USD | — |
| Reserved capacity | — | — | ~9.20 USD (85 WCU / 50 RCU) |
| Storage | 78 GB × 0.127 = 9.91 USD | 8 GB × 0.25 = 2.00 USD | 2.00 USD |
| PITR | Included in RDS | 1.60 USD | 1.60 USD |
| Attributable share of the instance | ~35 % of 118 USD = 41.30 USD | 0 | 0 |
| Total | ≈51.21 USD | ≈8.09 USD | ≈12.80 USD |
Two observations. First, 8 GB versus 78 GB: the TTL removes 90 % of the storage, because the
abandoned baskets stop piling up, and that fact on its own almost pays for the migration. Second, and
more important, the 41.30 USD figure is not the real saving: once that workload is released,
mercadofresco-pedidos stops needing 35 % of its capacity, which opens the door to reducing its size
or to absorbing growth without having to scale. The value of the migration is not in the DynamoDB line
of the bill, but in the line that disappears from the relational bill. And there is a benefit that
appears in no column at all: the VACUUM that competed for I/O with the orders on Friday afternoons
disappears.
The migration, step by step
# Create the on-demand table, encrypted with the project key and tagged.
aws dynamodb create-table \
--table-name mercadofresco-carritos \
--attribute-definitions AttributeName=PK,AttributeType=S AttributeName=SK,AttributeType=S \
--key-schema AttributeName=PK,KeyType=HASH AttributeName=SK,KeyType=RANGE \
--billing-mode PAY_PER_REQUEST \
--sse-specification Enabled=true,SSEType=KMS,KMSMasterKeyId=alias/mercadofresco-datos \
--stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES \
--tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
Key=Componente,Value=carrito Key=Propietario,Value=luis \
Key=CentroCoste,Value=plataforma \
--region eu-west-1 --profile mercadofresco-dev| Phase | Action | Exit criterion |
|---|---|---|
| 1 | Enable TTL, PITR and throttling alarms | Alarms verified in alertas-mercadofresco |
| 2 | Dual write; read from PostgreSQL | 7 days with a write error rate <0.01 % |
| 3 | Migrate only the active baskets of the last 30 days | A sample of 1,000 keys with no discrepancies |
| 4 | Read from DynamoDB with a fallback to PostgreSQL | 7 days with fallbacks <0.1 % |
| 5 | Remove the fallback and the write to PostgreSQL | TiempoConfirmacionPedido p99 stable or better |
| 6 | Export sesiones to S3 and delete it |
Backup verified; RDS WriteIOPS trending down |
And the 55 GB of abandoned baskets are not migrated: they are exported in Parquet to
mercadofresco-informes-analitica —Sara will analyse them in 06-04— and discarded: a migration is the
only cheap chance to leave the rubbish behind. And the switch that makes all of this reversible
without deploying is the /mercadofresco/produccion/carrito/origen parameter in Parameter Store
(04-03), with values postgres, ambos or dynamodb, read at process start-up and cached 60 seconds.
Common Mistakes and Tips
Using PutItem to update. PutItem replaces the whole item: the attributes you do not send
disappear. If two processes do a PutItem on the same basket, the second wipes out what the first
wrote. To modify, always UpdateItem.
Filtering with FilterExpression thinking it saves money. The filter is applied after reading
and paying. If you filter a lot, the model is wrong: that is a missing index.
Putting a Scan on the critical path. It works perfectly in development with 40 items and brings
the shop down with 4 million. The explicit Deny in the IAM policy is the best prevention.
Choosing the partition key badly. It is the only decision you cannot change without migrating.
Before creating the table, write the queries and check all of them resolve with GetItem or Query.
Setting the TTL in milliseconds or in ISO 8601. It has to be a Unix number in seconds; if it is not, the TTL deletes nothing and says nothing. And it is not an exact timer: it has a window of up to 48 hours, so if the logic demands that expired items are not seen, filter them on read too.
Ignoring UnprocessedItems in batches. BatchWriteItem returns the items it could not write and
does not retry them itself. In a migration, that oversight means lost baskets.
Creating GSIs with an ALL projection out of convenience. It doubles the storage and consumes WCU
on every write to the base table. INCLUDE with what is really drawn is usually enough.
Tip: measure from day one. Enable ReturnConsumedCapacity="TOTAL" in development and log the
units consumed per operation. An operation that consumes 40 RCU to return one basket points to a model
problem that costs money in production.
Tip: use DAX only when you can prove it. DynamoDB Accelerator is an in-memory cache in front of DynamoDB, with microsecond latencies and the same API. It costs from around 90 USD/month per node and only pays off with very intense repetitive reads. The basket is not that: each customer reads their own. The catalogue would be, and for that MercadoFresco will use ElastiCache (06-05), which also serves data that is not in DynamoDB.
Exercises
Exercise 1: modelling the order history
Customer service needs a mercadofresco-pedidos-consulta table in DynamoDB, fed from Aurora, that
resolves these four queries without a Scan: (1) given an id_pedido, return the complete order
(18,000/day); (2) given a customer and a date range, list their orders from most recent to oldest
(2,400/day); (3) list the orders in status en_reparto for a specific city (350/day); (4) given an
id_pedido, return the history of status changes (1,200/day).
Define: the table's PK and SK, the indexes needed with their projection, how each query is resolved, and justify why no partition key is hot.
Exercise 2: calculating capacity and deciding the mode
MercadoFresco is launching a Black Friday campaign with these forecasts for
mercadofresco-carritos: 4 hours of peak with 3,500 orders/hour (against the usual 900), an operation
mix identical to a normal Friday, and normal traffic for the other 20 hours. Prices in eu-west-1:
on-demand 0.7452 USD per million writes and 0.1491 USD per million eventual reads; provisioned
0.00065 USD per WCU-hour and 0.00013 USD per RCU-hour.
Calculate: (a) the WCU and RCU needed at the peak; (b) the cost of the whole day in on-demand mode; (c) the cost in provisioned mode with auto scaling, stating which values you would configure; (d) which mode you choose and why; (e) which two DynamoDB limits could appear and how you prevent them.
Exercise 3: diagnosing throttling
Three weeks after the migration, on a Friday at 18:20, the alarms go off. ThrottledRequests on
mercadofresco-carritos: 4,200 in 5 minutes. ConsumedWriteCapacityUnits on the table: 61 WCU out of
the 120 provisioned. ConsumedWriteCapacityUnits on the gsi-estado-fecha index: 98 WCU out of the
100 provisioned. A deployment that morning added a historial attribute (a list of status changes) to
each basket, and the average item size went from 3 KB to 11 KB. SuccessfulRequestLatency shows no
change.
Answer: (a) why the table throttles if it is consuming half its capacity; (b) what part the new attribute plays; (c) three measures ordered from fastest to most structural; (d) what would have prevented the incident before the deployment.
Solutions
Solution 1
The mercadofresco-pedidos-consulta table, single-table design:
| Entity | PK | SK |
|---|---|---|
| Order | PEDIDO#<id> |
META |
| Status change | PEDIDO#<id> |
ESTADO#<ts_iso> |
Indexes: gsi-cliente-fecha (PK id_cliente, SK fecha_pedido, INCLUDE of amount, status and
number of lines) resolves query 2; gsi-reparto-ciudad (PK ciudad_estado, SK fecha_pedido,
INCLUDE of id_pedido and courier) resolves query 3.
Resolution: query 1 is a GetItem with PK=PEDIDO#<id>, SK=META. Query 2, a Query on
gsi-cliente-fecha with Key("id_cliente").eq(...) & Key("fecha_pedido").between(...) and
ScanIndexForward=False. Query 3, a Query on gsi-reparto-ciudad with
Key("ciudad_estado").eq("VALENCIA#en_reparto"). Query 4, a Query on the table with
Key("PK").eq("PEDIDO#<id>") & Key("SK").begins_with("ESTADO#").
Why no key is hot: PEDIDO#<id> has as many values as there are orders, spread evenly.
id_cliente spreads across tens of thousands. ciudad_estado is the only one at risk —few cities—
but it is sparse: the attribute is only written while the order is out for delivery and is removed
on delivery, so the index holds a few hundred items at any moment and receives 350 queries a day.
Sparse indexes are the right technique for querying by transient statuses without creating a
permanently hot partition.
Solution 2
(a) Capacity at the peak. The multiplication factor is 3,500 ÷ 900 = 3.89.
| Operation | Requests/s | Units |
|---|---|---|
| Update basket | 46.7 | 140 WCU |
| Convert (transactional) | 0.97 | 5.8 WCU |
| Sessions | 77.8 | 78 WCU |
| Total writes | ≈224 WCU | |
| Read basket | 93.4 | 47 RCU |
| Read session | 155.6 | 78 RCU |
| Total reads | ≈125 RCU |
(b) On-demand. Peak: 3.23 M writes and 1.80 M reads in 4 h. The rest of the day, with the normal load of 58 WCU and 33 RCU: 4.18 M writes and 2.38 M reads. Totals 7.41 M writes (5.52 USD) and 4.18 M reads (0.62 USD) = ≈6.14 USD for the day.
(c) Provisioned. With auto scaling at 70 % you would need 320 WCU and 180 RCU at the peak, and 85/50 for the rest: 4 h × (320 × 0.00065 + 180 × 0.00013) = 0.93 USD, plus 20 h × (85 × 0.00065 + 50 × 0.00013) = 1.24 USD = ≈2.17 USD for the day. But you have to schedule the scaling in advance: auto scaling reacts in minutes and the Black Friday peak rises in seconds.
(d) Choice: on-demand. It costs 3.97 USD more for one single day. In exchange it removes the risk of throttling the basket on the biggest revenue day of the year and does not require getting right a forecast nobody has ever made. Saving 4 USD while risking Black Friday sales is the perfect example of misdirected optimisation; the provisioned capacity conversation happens in January, with data.
(e) Two limits. First, the partition: 1,000 WCU and 3,000 RCU. With CLIENTE#<id> well
spread it is not a risk, unless a test process concentrates traffic on a few customers; you prevent it
by checking that synthetic traffic uses varied identifiers. Second, the capacity of on-demand
tables, which scales fast but can throttle when faced with instantaneous multiples far above the
previous peak; you prevent it by configuring warm throughput in advance —or by warming the table with
a rising load— and with adaptive retries in the client.
Solution 3
(a) Why the table throttles. The table is not throttling: the index is. A GSI has its own
independent capacity, and gsi-estado-fecha is at 98 % of its 100 WCU. When a GSI cannot absorb its
writes, DynamoDB rejects the writes on the base table, because it cannot leave the index
permanently out of sync. The table's spare capacity is irrelevant: the bottleneck is downstream. It is
the most frequent GSI failure mode and the least intuitive.
(b) The new attribute. The item went from 3 KB to 11 KB, so every write went from 3 WCU to 11
WCU: a multiplication by 3.7. And if the GSI's projection is ALL, the index replicates the
historial attribute even though nobody queries it there, inheriting the whole increase. You are
paying index capacity to copy data that no index query uses.
(c) Three measures. Immediate (minutes): raise the GSI's provisioned capacity to 300 WCU; it
restores service on the spot at negligible extra cost. Short term (same day): since the projection
cannot be modified, create a new GSI with an INCLUDE of the attributes needed —without historial—
migrate the queries and delete the old one. Structural: take the status history out of the item and
model it as items of its own with SK = ESTADO#<ts>, as in solution 1; the item goes back to 3 KB and
the history grows without fattening anything. That is the right fix: an attribute that grows
unbounded inside an item is always a modelling error, and it was also getting close to the hard
limit of 400 KB.
(d) What would have prevented it. A load test in pre-production with the real item size; an alarm
on ConsumedWriteCapacityUnits of the index at 80 %, and not only of the table —the
mercadofresco-produccion dashboard watched the table, not the GSI—; and a design review that would
have spotted the unbounded attribute. All three are cheap; the incident happened on a Friday at
18:20.
Conclusion
The basket and the sessions are no longer in PostgreSQL. You know what DynamoDB is and why it fits here: no servers, no connections to exhaust, with latency that stays constant as volume grows and writes that scale horizontally instead of demanding a bigger instance. You have mastered the data model —table, item of up to 400 KB, attributes without a schema— and the hardest decision to reverse: the primary key, with its partition key spread by hash across partitions of 3,000 RCU and 1,000 WCU, and its sort key that physically groups what is related. You can recognise and fix a hot key, including spreading writes when a product goes viral.
Above all, you know how to design from the queries: list the basket's six real queries before
drawing anything, check that all of them are resolved with GetItem or Query, and only then write
the model. Out of that comes mercadofresco-carritos with PK = CLIENTE#<id> and SK = CARRITO#<ts>
/ SESION#<id> / PERFIL, a single-table design that returns in one operation what used to be
three queries against three tables —and with the criteria for knowing when that design does not pay.
You can handle the operations: UpdateItem with its four verbs instead of the PutItem that wipes
what you do not send, condition expressions as optimistic locking so that two presses of the pay
button do not generate two orders, Query versus Scan and the difference that costs the most money
—KeyConditionExpression chooses what is read, FilterExpression discards what you have already paid
for—, batches with their UnprocessedItems and TransactWriteItems at double the cost and with no
intermediate logic. And you know when a GSI and when an LSI, with the real effect of projections.
And you have the arithmetic: 58 WCU and 33 RCU cover the Friday peak —the workload that was
suffocating db.t3.small fits in fewer than 100 capacity units— on-demand during the migration
because saving four dollars while risking sales is bad optimisation, bursts and throttling with
adaptive retries, eventual consistency to show and strong to decide, and the TTL that turns the
78 GB into 8 GB and makes zombie baskets impossible ever again. With Streams ready for 07-03, PITR
enabled, encryption with alias/mercadofresco-datos and an IAM policy that explicitly denies Scan
and DeleteItem. Cost: from 51 USD a month to 8 USD, and the disappearance of the VACUUM that
competed with the orders on Friday afternoons.
The main workload is still there. mercadofresco-pedidos no longer carries the basket, but it is
still a PostgreSQL instance with Multi-AZ and one replica, with storage growing 12 GB a month, a
failover of a couple of minutes and a cost that is paid in full in the small hours too. In 06-03,
"Amazon Aurora", we will see the natural evolution of that database: the separation of compute and
storage, the volume distributed in six copies across three availability zones, read replicas with
millisecond lag, Serverless v2 so that the night-time trough stops costing the same as the Friday
peak, fast cloning so that Luis can test with realistic data without doubling the cost, and the
migration from the current RDS with minimal downtime.
AWS Course
Module 1: Introduction to AWS
- What Is AWS?
- Setting Up Your AWS Account
- AWS Global Infrastructure
- The AWS Management Console
- AWS CLI and SDKs
Module 2: Core AWS Services
Module 3: Networking and Content Delivery
Module 4: Security and Identity
- AWS Identity and Access Management (IAM)
- AWS Key Management Service (KMS)
- Secrets Manager and Parameter Store
- AWS Shield
- AWS WAF
Module 5: Monitoring and Management
Module 6: Databases
Module 7: Application Integration
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- Integration Patterns: Idempotency, Retries and Dead-Letter Queues
