Module 5 ended with an uncomfortable diagnosis: MercadoFresco's next bottleneck is the database. Not
because mercadofresco-pedidos is badly configured —it has Multi-AZ, a read replica, automated
backups and PITR since 02-04— but because it is doing four different jobs and only one of them is
its own.
The temptation at this point is to open the console and start creating services. DynamoDB sounds fast, Redshift sounds analytical, ElastiCache sounds like a cache. That is exactly how you end up with six databases and the same problems, plus the cost of maintaining them.
This lesson creates no resources. It is the lesson in which Marta sits down with the data module 5 gave her —CloudWatch metrics, X-Ray traces, ALB logs— and decides, with criteria she can explain to the board, which engine gets each workload and why. The next four lessons carry out that decision. This one justifies it.
Note on cost. This is a design lesson: nothing is created and therefore nothing is billed. The cost figures that appear are estimates for the
eu-west-1region with fictitious data, and they serve to compare orders of magnitude, not to build a budget.
Contents
- Why "one database for everything" ends up failing
- The three symptoms of coupled workloads
- Polyglot persistence: one tool per job
- The nine database families
- Master table: family, AWS service and MercadoFresco case
- The first criterion: the access pattern, not the data model
- Volume, growth and target latency
- Known queries versus exploratory queries
- Consistency: the CAP theorem with a box of strawberries
- Transactions, schema and cost
- Decision tree
- OLTP versus OLAP
- Normalisation, denormalisation and the NoSQL change of mindset
- MercadoFresco's four workloads, measured
- The reasoned decision, workload by workload
- How to migrate without stopping the shop
- AWS DMS and SCT: the migration tools
- Common mistakes and tips
- Exercises
- Conclusion
Why "one database for everything" ends up failing
When we created mercadofresco-pedidos back in 02-04, PostgreSQL was the right answer. There was one
application, a handful of tables and a team of three. A well normalised relational database handled
the catalogue, the orders, the basket and the reports with the same SELECT.
The problem is not that the decision was a bad one. It is that workloads diverge over time and the database does not. Eighteen months later, inside that same instance, these live together:
- Tiny, constant writes that expire on their own (the basket).
- Transactions that have to be exact for ever (the orders).
- Identical reads repeated thousands of times a minute (the catalogue).
- Sweeps over millions of rows to compute sums (Sara's reports).
Each of those four things has an engine that does it well. None of the four is PostgreSQL doing everything at once.
The three symptoms of coupled workloads
Putting incompatible workloads in the same engine always produces the same three symptoms. Recognising them is half the diagnosis.
1. Performance interference. One workload degrades another by competing for the same finite
resource. At MercadoFresco, Sara's monthly report keeps an execution plan with a Seq Scan over
pedidos and lineas_pedido running for 40 seconds; meanwhile PostgreSQL's buffer cache fills up
with historical pages that evict the catalogue's hot pages, and the shop's latency rises even
though the report does not touch a single shop table. It is the most counter-intuitive effect of
coupling: the damage does not travel through locks, it travels through shared memory.
2. Availability coupling. If the instance goes down, all four workloads go down. A badly judged
DROP INDEX, a query that exhausts the connections or a version upgrade affects everything alike.
The mercadofresco-pedidos-lectura replica mitigates this only for reads, and not for all of them.
3. Lowest-common-multiple scaling. When one single workload needs more resources, you have to
scale the entire instance. Sara's reports need memory; the basket needs write IOPS; the catalogue
needs CPU to parse the same query thousands of times over. As they cannot be scaled separately, you
end up paying for a db.r6g.2xlarge so that one of the four workloads runs well for three hours a
month.
graph TD
A[MercadoFresco shop] --> DB[(mercadofresco-pedidos<br/>PostgreSQL 16 db.t3.small)]
B[Basket and sessions] --> DB
C[Product catalogue] --> DB
D[Reports from Sara] --> DB
DB --> E[Interference: the report evicts<br/>the catalogue pages from the cache]
DB --> F[Coupling: one outage, four services down]
DB --> G[Scaling: you pay for the peak of the most demanding workload]
Polyglot persistence: one tool per job
Polyglot persistence is the name of the principle that says an application can —and often
should— use several data stores, each one chosen because it fits a particular workload. It is not a
microservices fad: it is the storage version of an idea we already accept without argument elsewhere.
Nobody keeps the catalogue photos in PostgreSQL: they have been in mercadofresco-catalogo-fotos, on
S3, since 02-03. That was already polyglot persistence.
Its trade-offs are real and they need saying before you sell it:
| For | Against |
|---|---|
| Each workload performs at the best its engine allows | More services to operate, watch and patch |
| Independent scaling and cost per workload | More security surface (IAM, encryption, network) |
| One failure does not drag the other workloads down | Coherence between stores: there is no JOIN |
| You can change one piece without touching the rest | The team has to know several data models |
| Performance interference disappears | Coordinated backups and restores |
The practical rule: add an engine when you can name the specific workload that justifies it and measure the improvement. If you cannot do both, do not add it. At MercadoFresco we will go from one database to four stores, and each one will have a module 5 measurement behind it.
The nine database families
It is worth understanding what each family models before looking at product names. The data model determines which queries are cheap and which are expensive, and that is the only thing that matters.
Relational
Models entities and relationships in tables with rows and columns, a fixed schema and integrity
constraints. The engine guarantees ACID properties and lets you combine tables with a JOIN at query
time. It shines when the data has complex relationships, when the queries are not known in advance
and when transactional correctness is non-negotiable. It does not shine when writes have to scale
horizontally, when the schema changes every week or when the volume per query runs to millions of
aggregated rows.
Key-value
Models a giant dictionary: a key returns an opaque value. There are no JOINs and no queries by
arbitrary fields. In exchange, lookup by key costs a constant amount and the engine can spread the
keys over hundreds of partitions with no coordination. It shines in sessions, baskets, profiles and
any "give me record X" access. It does not shine when you need to ask "give me every record that
meets Y" without having planned for it.
Document
Models nested JSON documents with a flexible schema and indexes over internal fields. It is a key-value store that also knows how to look inside the value. It shines in catalogues with heterogeneous attributes —a hake has "weight" and "catch area", a cheese has "maturing" and "milk"— and in data that is always read whole. It does not shine in massive aggregations, nor when the relationships between documents are the heart of the problem.
In-memory
Models data structures (strings, hashes, lists, sorted sets) that live in RAM, with microsecond latencies and optional persistence. It shines as a cache, as a session store, as a counter and as a real-time leaderboard. It does not shine as the source of truth for data you cannot afford to lose, nor when the data set does not fit in memory at a reasonable cost.
Columnar or data warehouse
Models facts and dimensions and stores data by column instead of by row, which makes it possible to read only the columns you need and to compress them enormously. It shines in aggregations over hundreds of millions of rows. It does not shine when reading or writing one particular row, which is precisely what a shop does all day long.
Time series
Models time-stamped measurements: metrics, telemetry, sensors. It optimises sequential writing, retention by age and time-window functions. It shines at "the temperature of the cold room every 10 seconds for two years". It does not shine on data with no dominant time dimension to organise it around.
Graph
Models nodes and edges and makes variable-depth traversals cheap: "customers who bought the same
as this customer and live in their city, three hops away". In a relational engine those are recursive
JOINs that blow up. It shines in recommendations, fraud detection and social networks. It does not
shine when the problem is tabular and the relationships are a single hop deep.
Search
Models inverted text: for each term, the list of documents that contain it, with relevance, tolerance for typing mistakes, synonyms and facets. It shines in the shop's search box —"sarmon" has to return "salmon"— and in log analysis. It does not shine as a transactional source of truth.
Ledger
Models an immutable, cryptographically verifiable record of changes. It shines when you have to prove to a third party that a history has not been altered. It does not shine as a general-purpose database, and it is worth knowing that Amazon QLDB is end of support: for new cases, AWS recommends Aurora PostgreSQL with insert-only tables and signatures.
Master table: family, AWS service and MercadoFresco case
| Family | AWS service | Typical latency | Ad hoc queries | MercadoFresco case |
|---|---|---|---|---|
| Relational | Amazon RDS / Aurora | 1-10 ms | Yes, full SQL |
Orders, invoices, stock: the source of truth (06-03) |
| Key-value | Amazon DynamoDB | <10 ms, stable | No, by key only | Basket and sessions (06-02) |
| Document | Amazon DocumentDB | 1-10 ms | Partial | Product records with heterogeneous attributes (future) |
| In-memory | ElastiCache / MemoryDB | <1 ms | No | Catalogue cache and web sessions (06-05) |
| Columnar | Amazon Redshift | Seconds | Yes, analytical | Sara's reports over 18 months (06-04) |
| Time series | Amazon Timestream | ms | Time-based | Temperature telemetry from refrigerated vans |
| Graph | Amazon Neptune | ms | Traversals | "People who bought this also bought…" |
| Search | Amazon OpenSearch Service | ms | Text and facets | The shop's search box with typo tolerance |
| Ledger | (QLDB, end of support) | — | — | Traceability of a fresh-produce batch |
The four in bold are this module's. The rest are named so that you know they exist and not so that you use them: every service you add is one more thing to operate, and MercadoFresco does not have the team for nine.
The first criterion: the access pattern, not the data model
The most widespread mistake when choosing a database is to start by drawing the entities. In the relational world it works, because the relational model is designed precisely so that you can normalise first and query however you like afterwards. In the other families it does not work, and the right order is the opposite one.
Before choosing an engine, answer these questions with numbers for each workload:
- Which operations happen, how often and in what read-to-write proportion?
- Which field is used for access? Always the same one, or does it change?
- How many items does a typical operation return: one, a hundred, a million?
- What is the maximum acceptable latency at the 99th percentile?
- What happens if a piece of data is lost? And if a value from two seconds ago is read?
- How much does it grow per month and how big will it be in three years?
Those six answers, and not the entity-relationship diagram, decide the engine. Module 5 gave us all six for MercadoFresco's four workloads: that is what makes this decision defensible.
Volume, growth and target latency
Volume and growth decide whether a single-node engine will do. A properly sized relational server handles tens of terabytes without difficulty; the problem appears when what has to scale is the writing, because a relational database scales writes vertically —a bigger machine— until you run out of machines. Distributed key-value engines scale writes horizontally by adding partitions, and that is what makes them different.
Target latency must always be expressed in percentiles, never as an average. In 05-01 we saw that
the average of TiempoConfirmacionPedido was 240 ms and the p99 was 1,900 ms: the average hid the
problem. The orders of magnitude worth memorising:
| Source | Typical latency | What it means for the shop |
|---|---|---|
| In-memory cache (same AZ) | 0.2-1 ms | Imperceptible |
| Managed key-value | 3-10 ms | Imperceptible within one request |
| Relational with an index | 1-10 ms | Fine, if there are few queries |
Relational with a Seq Scan |
100 ms-60 s | The p99 from 05-02 |
| Columnar store | 1-30 s | Acceptable in a report, lethal in a product page |
A product page that runs 30 relational queries of 8 ms takes 240 ms in the database alone. The same page served from cache takes less than 10 ms. That is the whole argument of 06-05.
Known queries versus exploratory queries
This criterion splits the world in two and almost nobody states it explicitly.
- Queries known in advance. Today, at design time, you know the ten queries the application will run over the next two years: "give me customer X's basket", "give me order Y". With that you can model so that every query is a direct access. This is DynamoDB territory.
- Exploratory queries. Sara does not know today what she will ask in March. She needs to cross anything with anything. That calls for an engine that accepts arbitrary SQL: relational for small volumes, columnar for large ones.
Choosing DynamoDB for an exploratory workload produces a Scan over the whole table, which is slow
and expensive. Choosing relational for a very high-volume known workload produces what MercadoFresco
has. The same data can live in both places: orders are transactional in Aurora and analytical in
Redshift, and that is not a contradiction but the design.
Consistency: the CAP theorem with a box of strawberries
The CAP theorem says that a distributed system cannot simultaneously guarantee consistency (Consistency), availability (Availability) and network partition tolerance (Partition tolerance). Since network partitions are not optional —cables get cut— the real choice is between consistency and availability when a partition happens.
With fresh produce you see it straight away. There are 3 boxes of strawberries left and two customers ask for them at the same time from different nodes that have lost contact with each other:
| Choice | Behaviour | Consequence for MercadoFresco |
|---|---|---|
| Consistency (CP) | The isolated node refuses the operation until contact is restored | Nobody oversells; some customers see an error when adding to the basket |
| Availability (AP) | Both nodes accept and reconcile afterwards | Nobody sees errors; 4 boxes can be sold out of 3 |
The right answer depends on the operation, not on the company:
- Confirming the order and deducting stock: strong consistency, no argument. Selling strawberries that do not exist means a call, a refund and a lost customer. It goes to Aurora, with transactions.
- Showing "only a few units left" on the product page: eventual consistency, perfectly acceptable. A counter running two seconds behind harms nobody. It goes to the cache.
- Saving the basket: eventual consistency is acceptable, with a strong read only at the final checkout step. It goes to DynamoDB, which lets you ask for each read either way.
This distinction —consistency per operation, not per system— is the concept that unlocks the most performance in practice and the hardest one to accept coming from the relational world.
Transactions, schema and cost
Transactions. Ask whether several writes must succeed or fail together. Confirming an order means
deducting stock, creating the order, recording the payment and generating the invoice: either all
four or none. That calls for an engine with real ACID and a natural BEGIN … COMMIT. DynamoDB has
TransactWriteItems with limits (up to 100 items, no intermediate logic) and it works for bounded
cases, not for a complete business flow.
Schema. Do all the rows have the same fields? MercadoFresco's catalogue does not: fish has a
landing port and a minimum size, wine has an appellation and a vintage. In a relational engine those
are null columns or attribute tables; in a document or key-value store it is natural. But a flexible
schema does not mean no schema: it means the application validates the schema instead of the engine,
and if nobody validates it, precio ends up a number in some records and a string in others.
Cost. The models are structurally different and are not compared by looking at the hourly price:
| Model | You pay for | Good when | Bad when |
|---|---|---|---|
| Instance (RDS, provisioned Aurora) | Instance hours, in use or not | Stable, predictable load | Load with deep troughs |
| Serverless by capacity (Aurora Serverless v2) | ACU-hours consumed | Variable load with a night-time trough | Flat 24×7 load |
| Per request (DynamoDB on-demand) | Every read and write | Irregular or unpredictable traffic | Huge, constant volume |
| Per query (Athena) | TB scanned | Occasional queries | Many queries a day |
| Per memory (ElastiCache) | Node hours | A hot, bounded working set | Cold, bulky data |
Decision tree
graph TD
A[New workload] --> B{Is it analytical:<br/>aggregating millions of rows?}
B -->|Yes| C{Frequent queries<br/>or occasional ones?}
C -->|Frequent| D[Redshift]
C -->|Occasional, over S3| E[Athena]
B -->|No| F{Is it always accessed<br/>by a known key?}
F -->|No| G{Text search<br/>or deep relationships?}
G -->|Text| H[OpenSearch]
G -->|Relationships| I[Neptune]
G -->|Neither: ad hoc SQL| J[Aurora / RDS]
F -->|Yes| K{Can it be lost<br/>without consequences?}
K -->|Yes, it is a copy| L[ElastiCache]
K -->|No, it is the truth| M{Does it need transactions<br/>across several entities?}
M -->|Yes| J
M -->|No| N[DynamoDB]
The tree is a guide, not a law. Its value lies in the order of the questions: first analytical or transactional, then key access or free-form query, then truth or copy, and only at the end transactions. That order mirrors the real impact of each decision.
OLTP versus OLAP
The first fork in the tree deserves some expansion, because it is the one that explains 80 % of MercadoFresco's problems.
| OLTP (transactional) | OLAP (analytical) | |
|---|---|---|
| Typical question | "Give me order 84,213" | "Average basket by city over 18 months" |
| Rows touched | 1-100 | 10⁶-10⁹ |
| Columns touched | All of those in the row | 3-6 out of 40 |
| Expected latency | Milliseconds | Seconds or minutes |
| Concurrency | Thousands of sessions | Dozens |
| Writes | Constant and small | Periodic bulk loads |
| Storage | By rows | By columns |
| Normalisation | High (3NF) | Low (star) |
| Service | Aurora, RDS, DynamoDB | Redshift, Athena |
When Sara runs her monthly report on mercadofresco-pedidos she is asking an OLTP engine to do OLAP
work. PostgreSQL does it —it is a good engine— but it reads whole 40-column rows to use 4, compresses
nothing and does not parallelise. That it takes 40 seconds and wipes the cache is not a fault: it is
the predictable consequence of using the wrong tool.
Normalisation, denormalisation and the NoSQL change of mindset
In relational, normalising —each fact in one place only, with no duplication— is the right way to
design. It avoids update anomalies: if a product's name changes, it changes in one row and that is
it. The price of that purity is that rebuilding complete information requires a JOIN in every query.
In key-value and document stores you denormalise deliberately: the data is duplicated so that a
single read returns everything you need, because there is no JOIN and because reading is the
expensive operation to optimise.
| Normalised | Denormalised | |
|---|---|---|
| Duplicated data | No | Yes, on purpose |
| Read cost | Several JOINs |
One read |
| Write cost | One write | Several coordinated writes |
| Risk | Slow queries | Copies that diverge |
| Adding new queries | Easy | May require remodelling |
The change of mindset NoSQL demands comes down to four reversals of the relational habit:
- You design from the queries, not from the entities.
- Duplicating is not a mistake, it is the technique.
- Integrity is guaranteed by the application, not by the engine.
- New queries are not free: an unforeseen query may require a new secondary index or a remodelling.
The classic failure —and we will see it in 06-02— is bringing a relational mindset to DynamoDB: one
table per entity, Scan to search and manual joins in the code. The result is slower and more
expensive than PostgreSQL, and the wrong conclusion is often "DynamoDB is no good".
MercadoFresco's four workloads, measured
These are the real figures module 5 left on the table. They are fictitious, but they are the kind of number you need to have before deciding.
Workload A: orders and stock
- 900 orders/hour at the Friday peak; around 4,000 a day.
- Each confirmation is 6 writes across 4 tables, inside one transaction.
- Frequent exploratory queries from customer service: "this customer's orders in March".
- Volume: 340 GB, +12 GB/month.
- Losing a record: unacceptable. Consistency: strong.
Workload B: basket and sessions
- 180,000 writes/day in
sesiones; 92 % of them are updates to the same record. - Access always by
id_clienteorid_sesion. Zero exploratory queries. - The table has grown to 78 GB, of which 71 % are baskets abandoned more than 60 days ago.
- The
VACUUMon that table is the single most I/O-hungry operation in the whole instance. - Losing a record: annoying, not serious. Consistency: eventual except at the payment step.
Workload C: catalogue
- 4,100 reads/minute at peak hour; 94 % of the queries return exactly the same result as the previous one.
- Prices and descriptions change once a day, at 06:00, in the market load.
- Volume: 1.2 GB. It fits in memory several times over.
- Target latency: <5 ms. Consistency: eventual, with minutes of slack.
Workload D: reports
- 3-8 reports a day from Sara, with peaks at month end.
- Each one aggregates between 8 and 18 months: from 6 to 40 million rows.
- It uses 4-6 columns out of the 40 available.
- Current duration: from 40 s to 6 min, with measured impact on the shop's latency.
- Consistency: yesterday's data is perfectly good enough.
The reasoned decision, workload by workload
| Workload | Dominant pattern | Engine chosen | Deciding reason | Lesson |
|---|---|---|---|---|
| A. Orders and stock | Transactional OLTP, ad hoc SQL | Aurora PostgreSQL | ACID transactions and free-form queries; compatible with no rewriting | 06-03 |
| B. Basket and sessions | Key-value, heavy writing, ephemeral data | DynamoDB | Key access, write scale and a TTL that deletes by itself | 06-02 |
| C. Catalogue | Identical repeated reads | ElastiCache (Redis) | <1 ms latency and 94 % of the reads offloaded | 06-05 |
| D. Reports | OLAP, massive aggregation | Redshift Serverless | Columnar and MPP; and above all, out of production | 06-04 |
It is worth making explicit what is not chosen and why, because a decision without discarded alternatives is not a decision:
- The basket does not go to ElastiCache even though it is fast: it is customer data we do not want to lose in a failover, and we need it to expire in an auditable way. Redis could do it; DynamoDB with TTL does it as well, with disk durability and without sizing memory.
- Orders do not go to DynamoDB: customer service runs queries nobody foresaw, and order confirmation is a four-entity transaction with intermediate logic.
- The reports are not solved by the read replica: the replica removes the blocking, but it still reads by rows, uncompressed and without parallelism. Going from 40 s to 35 s is not the solution.
- The catalogue is not fixed with more CPU on the database: the problem is not that the query is slow, it is that it runs 4,100 times a minute to return the same thing.
And an important note on ordering: Aurora first, then the rest. Migrating the main engine before splitting workloads avoids doing two migrations over the same data. In practice, MercadoFresco will run DynamoDB (06-02) and Aurora (06-03) in parallel because they touch disjoint data, and Redshift (06-04) and ElastiCache (06-05) afterwards, because both read from what came before.
How to migrate without stopping the shop
None of these migrations can be done with a long outage. There are three strategies, and they combine.
Dual write
The application writes to the old store and the new one at the same time, reads from the old one, and when the new one has matched for weeks, the read is switched over.
sequenceDiagram
participant App as Shop
participant PG as PostgreSQL (old)
participant DDB as DynamoDB (new)
Note over App: Phase 1 · dual write, old read
App->>PG: write basket
App->>DDB: write basket
App->>PG: read basket
Note over App: Phase 2 · new read with fallback
App->>DDB: read basket
alt not found
App->>PG: read basket (fallback)
end
Note over App: Phase 3 · the new store only
App->>DDB: read and write
Its three traps: you have to decide what happens if the second write fails (almost always the right thing is to log the error and carry on, not to break the order); you have to migrate the history separately, because dual writing only covers what is new; and you have to compare both stores with an automated process before trusting them.
Strangler pattern
Instead of migrating everything at once, you intercept one feature at a time and redirect it to the
new system, until the old one runs out of work and is switched off. It is the pattern this module
follows: the basket goes first, then the reports, then the catalogue, and mercadofresco-pedidos
ends up doing only what it was good at.
Controlled cutover
For workloads that tolerate a window —the reports, for instance— it is enough to export, load and switch the destination. It is the simplest one and it should be used whenever possible: the complexity of dual writing is only justified when an outage is unacceptable.
| Strategy | Outage | Complexity | Rollback | When to use it |
|---|---|---|---|---|
| Controlled cutover | Minutes or hours | Low | Easy | Reports, internal workloads |
| Dual write | Zero | High | Medium | Basket, sessions |
| Strangler | Zero | Medium | Easy per feature | Phased migration |
| Promoted replica | 1-2 minutes | Medium | Hard after promoting | Compatible engine change |
AWS DMS and SCT: the migration tools
AWS Database Migration Service (DMS) copies data between stores and, more importantly, can replicate changes continuously (CDC) after the initial load: the source carries on in production while the target is kept in sync. It supports heterogeneous sources and targets —PostgreSQL to Aurora, PostgreSQL to DynamoDB, PostgreSQL to Redshift or to S3— which is exactly the range of options this module needs.
# Structure of a DMS task: a replication instance, two endpoints and a task.
# 1) The replication instance lives in the VPC, in the data subnets.
aws dms create-replication-instance \
--replication-instance-identifier dms-mercadofresco \
--replication-instance-class dms.t3.medium \
--replication-subnet-group-identifier sng-mercadofresco-datos \
--no-publicly-accessible \
--tags Key=Proyecto,Value=mercadofresco Key=Entorno,Value=produccion \
Key=Componente,Value=migracion Key=Propietario,Value=marta \
Key=CentroCoste,Value=plataforma \
--region eu-west-1 --profile mercadofresco-dev
# 2) The source endpoint points at the current RDS. The password is NOT written here:
# it references the Secrets Manager secret created in 04-03.
aws dms create-endpoint \
--endpoint-identifier origen-mercadofresco-pedidos \
--endpoint-type source --engine-name postgres \
--secrets-manager-secret-id mercadofresco/produccion/rds/mfadmin \
--secrets-manager-access-role-arn arn:aws:iam::111122223333:role/rol-dms-secretos \
--database-name pedidos \
--region eu-west-1 --profile mercadofresco-devThree details that decide whether the task works or not:
dms.t3.mediumis the replication instance, not the target. You pay by the hour while the task lives and it must be deleted at the end: it is the costliest oversight in any migration.- The
full-load-and-cdctask type does a full load and then continuous changes. It requires the source to haverds.logical_replicationenabled, which forces a reboot of the instance: plan for it beforehand, do not discover it on migration day. - DMS does not migrate indexes, procedures or constraints by default: it copies data. The schema is prepared separately.
AWS Schema Conversion Tool (SCT) translates the schema and the code —views, functions, procedures— between different engines, and produces a report of what it cannot convert automatically. For MercadoFresco, going from PostgreSQL to Aurora PostgreSQL, it is not needed: the engine is the same. SCT is the tool for heterogeneous migrations, typically Oracle or SQL Server to PostgreSQL.
Cost and cleanup warning. A migration leaves expensive residue: DMS replication instances, manual snapshots, target clusters used for testing and intermediate buckets. Before closing any migration, review the
Componente=migraciontag in Cost Explorer (11-03) and delete everything that carries it. And do not delete the source until you have at least one tested restore.
Common Mistakes and Tips
Choosing by fashion. "Let's use DynamoDB because it scales." Scales what, exactly, and from what figure? If a workload does 40 writes per second, PostgreSQL handles it with one hand tied. The question is not whether an engine scales, but whether your workload needs that scale. Fashion costs dearly because the cost is not the bill: it is the team learning a new model while incidents need attending.
Migrating without measuring first. Without module 5's baseline —p99 latency, IOPS, cache hits, report duration— you cannot prove the migration improved anything. And if you cannot prove it, the next architecture proposal does not get approved. Capture the metrics the week before, not the day after.
Using NoSQL with a relational mindset. One DynamoDB table per entity, Scan to search and joins
in the application code. It is slow, expensive and fragile. If you cannot write down the list of
queries the table has to serve, you are not ready to design it.
Confusing "can" with "should". PostgreSQL has a JSONB type, full-text search, geospatial types
and even vector extensions. It can do almost anything. That does not mean it should: the question is
whether it does it well enough for your volume. With a 1.2 GB catalogue, PostgreSQL's text search
is more than enough; with 300 GB and facets, it is not.
Adding an engine with no operating plan. Every new store needs: tested backups, CloudWatch
alarms, a least-privilege IAM policy, encryption with alias/mercadofresco-datos, complete tags and
somebody who knows how to restore it at three in the morning. If you cannot commit to those six
things, do not add it.
Forgetting that there are now two truths. As soon as the basket lives in DynamoDB and the order
in Aurora, there is no JOIN or transaction that spans both. You have to decide explicitly what
happens if one of the two writes fails. Module 7 is about exactly that.
Tip: write the decision down. A one-page document per workload —measured pattern, options considered, engine chosen, deciding criterion, how success will be measured— is what turns a migration into engineering. A year from now, when somebody asks why the basket is in DynamoDB, that page is worth more than anybody's memory.
Exercises
Exercise 1: classifying five new workloads
MercadoFresco is planning five features. For each one, state the database family, the AWS service and the deciding criterion (just one, the one that weighs most):
- Fresh-produce traceability. Every batch of fish has to be able to prove its journey from the market to the health authority. 400 batches/day, extremely rare queries but legally mandatory, and the history cannot be altered.
- Van temperature. 30 vehicles send the temperature of the compartment every 10 seconds. The curve for the last 24 hours is queried and 2 years are kept for regulatory reasons.
- The shop's search box. Customers type "pepp", "sarmon" or "cured sheep cheese" and expect relevant results, with filters by category, allergens and price range.
- "Customers like you also bought". Recommendations based on joint purchases, up to three hops of distance between customers and products.
- Notifications sent. A record of every email and SMS sent: 50,000/day, queried only by
id_clientefor customer service, and deleted after 90 days.
Exercise 2: applying CAP to a specific decision
MercadoFresco wants to launch "last units": when fewer than 5 units of a product are left, the product page shows a live counter. The team proposes two designs:
- Design A: the product page queries the real stock in the transactional database on every load.
- Design B: stock is published into a cache with a 30-second TTL, and the page reads from there.
Answer: (a) what does each design choose in CAP terms and what is sacrificed?; (b) with 4,100 reads per minute at peak hour, what load does each one add to the database?; (c) what happens to a customer who adds the last box of strawberries to the basket in design B?; (d) propose a design C that combines both and say exactly at which point of the purchase flow the strong read is needed.
Exercise 3: the basket migration plan
Write the migration plan for the sesiones table (78 GB, 180,000 writes/day) from PostgreSQL to
DynamoDB, with no outage. It has to include: the strategy chosen and why; the phases with their
measurable exit criteria; what is done with the 55 GB of baskets abandoned more than 60 days ago; how
you verify that both stores match; the rollback plan at each phase; and the three module 5 metrics
you will use to prove to Marta that the migration was worth it.
Solutions
Solution 1
| # | Family | Service | Deciding criterion |
|---|---|---|---|
| 1 | Relational with an immutable record | Aurora PostgreSQL with insert-only tables | Verifiability to a third party; QLDB is end of support, so the current route is insert-only plus signature, with trail-mercadofresco as audit backup |
| 2 | Time series | Amazon Timestream | Retention by age: 259,200 points/day that are almost always queried within a recent window and afterwards merely kept; the memory→magnetic lifecycle is done by the engine |
| 3 | Search | Amazon OpenSearch Service | Typo tolerance and facets: "sarmon" has to return "salmon", and a LIKE will not give you that |
| 4 | Graph | Amazon Neptune | Variable-depth traversals: three hops in SQL means recursive JOINs that grow exponentially |
| 5 | Key-value | DynamoDB | Key-only access and expiry: the same pattern as the basket; a 90-day TTL solves the deletion with no process at all |
An important nuance for case 1: even though it is technically food traceability, the volume is
ridiculous (400 batches/day). It does not justify a new engine. The operationally correct answer
is to solve it in Aurora with an insert-only table and triggers that prevent UPDATE and DELETE.
That is exactly the lesson's advice: do not add an engine if you can name why not.
Solution 2
(a) CAP. Design A chooses consistency: the value shown is always the real one, at the price of every visit to a product page depending on the availability and the latency of the database. Design B chooses availability and latency: the page answers in less than a millisecond even when the database is saturated, at the price of showing a value that can be up to 30 seconds old. What is sacrificed in B is momentary accuracy; what is sacrificed in A is performance and isolation from failures.
(b) Load added. Design A: 4,100 additional queries per minute, some 68 per second, on the instance that is already the bottleneck. Design B: with a 30-second TTL, and even if every product is queried constantly, the database receives at most 2 queries per minute per product; for a catalogue of 600 active products, around 1,200 per minute in the worst case, and in practice far fewer because only the products somebody actually looks at get refreshed. The reduction is one to two orders of magnitude.
(c) The case of the last box. The customer sees "2 left", adds to the basket and reaches checkout. Between their read and their purchase, another customer took both. In design B the error must not appear when adding to the basket, but at order confirmation, where a strongly consistent transaction deducts stock and fails if there is none left. The correct experience is a clear message —"they sold out while you were completing your order"— and the suggestion of an alternative product. An order must never be confirmed by reading from the cache.
(d) Design C. Eventual reads from cache to display (product page, listings, search box) and strong reads and writes inside a transaction to commit (the `UPDATE stock SET unidades = unidades
- :n WHERE id = :p AND unidades >= :n` at order confirmation, checking that it affected one row). The exact point where the strong read is needed is order confirmation, not the basket and not the product page. This is the general pattern: eventual to read, strong to decide.
Solution 3
Strategy: dual write combined with the strangler pattern. An outage is not acceptable —a lost basket is a lost sale— and the workload is one of continuous writing, so the controlled cutover is ruled out. The strangler pattern orders the phases by feature; dual writing guarantees that nothing is lost during the transition.
Phases and exit criteria:
| Phase | What is done | Measurable exit criterion |
|---|---|---|
| 0 | Model the mercadofresco-carritos table from the real queries extracted from the logs |
A closed list of queries; none of them requires a Scan |
| 1 | Dual write; read from PostgreSQL only; DynamoDB errors are logged but break nothing | 7 days with a DynamoDB write error rate <0.01 % |
| 2 | Migrate the useful history with DMS in full-load mode | 100 % of the active baskets of the last 30 days present in both |
| 3 | Read from DynamoDB with a fallback to PostgreSQL if not found | 7 days with fallbacks <0.1 % of reads |
| 4 | Remove the fallback and the write to PostgreSQL | 7 days with no incidents; TiempoConfirmacionPedido p99 stable or better |
| 5 | Delete the sesiones table after a final backup |
Backup verified in mercadofresco-copias-basedatos |
The 55 GB of abandoned baskets: they are not migrated. Migrating them would cost writes and storage for data nobody is going to query. They are exported to S3 in Parquet as a historical archive (in case Sara wants to analyse basket abandonment in Redshift, 06-04) and discarded from the target. The new table is born with a 30-day TTL, so that the problem cannot happen again. This is the most important point of the exercise: a migration is the only cheap chance to leave the rubbish behind.
Verification: a daily process that takes a random sample of 1,000 keys from PostgreSQL, looks
them up in DynamoDB and compares field by field, publishing a custom metric
MercadoFresco/Migracion/DiscrepanciasCarrito in the MercadoFresco/Tienda namespace, with an alarm
to alertas-mercadofresco if it goes above 5 in 24 hours. Compare the sample, not the total:
comparing 78 GB daily costs more than the migration.
Rollback: in phases 1 and 2 it is immediate because PostgreSQL is still the read source —just
turn off the dual write. In phase 3 it is a configuration flag in Parameter Store
(/mercadofresco/produccion/carrito/origen) that is changed without deploying. From phase 4 onwards
rollback is no longer trivial: that is why phase 4 does not start until seven clean days in phase 3.
The three metrics: (1) the p99 of TiempoConfirmacionPedido in the MercadoFresco/Tienda
namespace, which should fall once the session writes leave the instance; (2) WriteIOPS and
CPUUtilization of mercadofresco-pedidos, which should drop once 180,000 daily writes and the
associated VACUUM disappear; and (3) the monthly cost with the Componente=basedatos tag in Cost
Explorer, comparing the previous month with the following one. All three were already instrumented
back in module 5: that is why this migration can be defended with data.
Conclusion
This lesson has not created a single resource, and it is probably the most important one in the
module. You know why one database for everything ends up failing —performance interference
through the shared cache, availability coupling and lowest-common-multiple scaling— and you know
polyglot persistence with its honest trade-offs: more services to operate, more security surface
and the disappearance of the JOIN between stores.
You have the map of the nine families —relational, key-value, document, in-memory, columnar, time series, graph, search and ledger— with their AWS service and the MercadoFresco case each one would apply to, and you know that only four make it into this module because the other five do not yet have a workload that justifies them.
And above all you have the right order of the questions: first the access pattern —not the data model—, then volume and latency expressed in percentiles, then whether the queries are known in advance or exploratory, then consistency decided per operation and not per system —eventual to show that few strawberries are left, strong to deduct them— and only at the end transactions, schema and cost. With the OLTP versus OLAP distinction as the first fork, and with the change of mindset that NoSQL demands: you design from the queries, duplicating is the technique and integrity is guaranteed by the application.
The decision is made and it is the module's road map: DynamoDB for the basket and the sessions, because access is exclusively by key, writes scale and a TTL resolves the 55 GB of rubbish; Aurora for orders and stock, because of ACID transactions and exploratory queries with nothing to rewrite; Redshift Serverless for Sara's reports, because it is columnar, massively parallel and, above all, out of production; and ElastiCache for the catalogue, because the problem was never that the query was slow but that it repeated 4,100 times a minute to return the same thing. Together with the strategies for getting there without stopping the shop: dual write, the strangler pattern and a controlled cutover whenever possible, with DMS and its continuous replication as the tool and SCT reserved for the heterogeneous migrations MercadoFresco does not need.
We start executing with the most independent workload, and the one that relieves the current instance
most. In 06-02, "Amazon DynamoDB", we will take the basket and the sessions out of PostgreSQL: we
will model the mercadofresco-carritos table from its queries and not from its entities, we will see
partition and sort keys, single-table design, Query versus Scan, secondary indexes, on-demand
versus provisioned capacity with the arithmetic of the Friday peak, and the TTL that will make the
55 GB of abandoned baskets a problem that can no longer exist.
AWS Course
Module 1: Introduction to AWS
- What Is AWS?
- Setting Up Your AWS Account
- AWS Global Infrastructure
- The AWS Management Console
- AWS CLI and SDKs
Module 2: Core AWS Services
Module 3: Networking and Content Delivery
Module 4: Security and Identity
- AWS Identity and Access Management (IAM)
- AWS Key Management Service (KMS)
- Secrets Manager and Parameter Store
- AWS Shield
- AWS WAF
Module 5: Monitoring and Management
Module 6: Databases
Module 7: Application Integration
- Amazon SQS
- Amazon SNS
- Amazon EventBridge
- AWS Step Functions
- Integration Patterns: Idempotency, Retries and Dead-Letter Queues
