Escena Viva already persists to MongoDB and the controllers do not know it exists. Now comes the part of the job that separates an application that works from an application that holds up: how data relates to itself and how you query it once the questions stop being trivial.

We will make the most important modeling decision in the document world — embed or reference — see how populate really works and why it is not a JOIN, measure the N+1 problem, defend a deliberate denormalization, build Escena Viva's reports with the aggregation framework — retiring the ones we produced in Module 3 by reading CSV files — and learn to look at indexes with explain() instead of guessing.

1. Embedding versus referencing
2. Escena Viva's decisions, justified one by one
3. References with ref and populate
4. The N+1 problem: measuring it and solving it
5. Deliberate denormalization
6. The aggregation framework as a pipeline
7. Real Escena Viva reports
8. $lookup versus populate
9. Indexes taken seriously, and explain()
10. Partial indexes, write cost, text and geospatial

In a relational database this decision does not exist: everything is split into tables and joined with a JOIN. In MongoDB you have both options, and choosing badly is the number-one cause of projects that "get slow for no apparent reason".

Embedding means putting the child data inside the parent document, like our sessions inside the Event. Referencing means storing them in another collection and keeping only their identifier, like Order.userId.

The 16 MB per document limit is not a recommendation: it is a hard BSON limit. A document that grows indefinitely will hit it, and the failure will arrive in production, with real data, on some ordinary day. But it hurts long before that: MongoDB rewrites the whole document on every update, so a 2 MB document to which you add a small element moves 2 MB of disk and network. There is a third way, the subset pattern: embed the few children that are always displayed (the last 3 reviews) and reference the rest.

Sessions are embedded in the event. Bounded cardinality: between 1 and 7 in our catalog, maybe 60 for a play in a long run. Ridiculous size: five fields, around 150 bytes. Always accessed together: GET /events/evt-001 returns the event with its sessions and the web page shows all of them. They are not shared: a session makes no sense outside its event. The only serious objection is volatility — sold changes on every purchase and that rewrites the document — but since it is tiny the cost is negligible. Embedding wins clearly.

Orders are referenced. Growth with no ceiling: a festival can generate 20,000. Inside the event, the document would grow until it burst and every purchase would rewrite megabytes. On top of that they are queried on their own ("my orders", "today's orders") without starting from the event, and they have their own life cycle with four statuses.

Tickets are referenced. It is the highest-volume entity and they are accessed by their code at the door, individually, without knowing the order. They have their own cycle (valid → used | cancelled).

The user is referenced from the order. A user has N orders and we do not want to duplicate their name and email in each one: if the email changes, every one of them would have to be rewritten.

The solid arrows are data living in the same document; the dashed ones are references that require an extra query. That visual distinction is exactly the difference in cost.

We already declared the references: userId: { type: Schema.Types.ObjectId, ref: 'User' }. The ref says which model is on the other side; populate replaces the identifier with the document.

And now the important part, the one the documentation mentions in passing and that ruins applications: populate is not a JOIN. MongoDB joins nothing. Mongoose runs your query, collects the identifiers from the populated field, fires a second query (User.find({ _id: { $in: [...] } })) and stitches the results together in memory, inside your Node process. Consequences:

• A populate over 50 orders is 2 queries, not 50: Mongoose groups the identifiers. A two-level nested population is 3 queries.
• The stitching cost is paid by your event loop, not by the database server.
• You cannot filter the parent by a field of the child. "Orders whose user is an organizer" cannot be expressed with populate; the match option does not discard parents, it leaves the field as null and the order still shows up. For that, $lookup with $match.

It appears when, in order to resolve a list of N items, you fire one query per item. It is extremely easy to write without noticing:

With a modest latency of 2 ms per query, that is 202 ms of pure sequential waiting for a response that should take 5 ms. And it scales with traffic: 100 simultaneous requests are 10,100 queries. Measuring it is trivial and you should do it before optimizing anything: mongoose.set('debug', true) prints every query sent to the engine.

session.sold is redundant information: it could be computed by counting valid tickets. We store it anyway, and not out of laziness.

In favor: checking capacity before selling is the platform's most frequent and most latency-sensitive operation, and reading an integer from an already-loaded document costs nothing, whereas counting tickets requires an aggregation over millions of documents. The catalog shows "88 tickets left" on every card: with the counter, the listing comes out of a single read. And it enables the conditional atomic update that will solve overselling, because you can only compare against a field that exists in the document.

The price, stated plainly: there are two sources of truth for the same fact and they can diverge. If tickets are created without incrementing the counter, or the other way round, or a process dies between the two operations, the catalog lies. And no engine mechanism prevents it: MongoDB knows nothing about that relationship. That price is paid at three levels:

1. Atomicity of the set: reserving capacity, creating the order and issuing tickets must happen all or nothing. That is lesson 07-06.
2. Periodic reconciliation: a scheduled job that recounts and corrects, leaving a trace. The safety net.
3. A single write door: nobody touches sold or creates tickets outside the repository. If ten places touch the counter, incoherence is a matter of time.

If Module 3 left you comfortable with pipeline() and streams, aggregation will feel familiar: it is a pipeline of stages where each one receives documents, transforms them and passes the result to the next. The difference is that it runs inside the database server, next to the data, and not in your process.

Two golden rules about ordering, because performance depends on it. $match as early as possible: it is the only stage that can use indexes, and only if it comes first; filtering at the end means processing the entire collection. And $project to drop heavy fields early: fewer bytes down the pipeline and less memory, because every stage has a 100 MB limit (which can be lifted with allowDiskUse: true, but if you need it, rethink the query).

In Module 3 we computed occupancy by reading data/sales.csv with streams. That was an excellent pipeline exercise, but the data now lives in the database and the engine aggregates it better than we do.

With our seed catalog (7 sessions, total capacity 3000, 1811 sold), this pipeline returns the three venues sorted by revenue and an overall occupancy of 60.4%.

And a complete dashboard in a single query with $facet: each branch receives the same input documents and produces its own array, replacing three round trips to the database with one. It is the perfect stage for control panels.

Practical criterion: populate to serve documents to the API; $lookup for reports and whenever you need to filter or aggregate over the relationship. One detail that trips people up: in $lookup, from is the real collection name (users, plural and lowercase, exactly as Mongoose pluralizes it from the User model), not the model name.

An index is a B-tree sorted by the values of one or more fields, with pointers to the documents. Without one, "give me the events at Teatro Almendra" forces reading every document: a collection scan, or COLLSCAN. With one, the engine walks down the tree: an IXSCAN.

In a compound index the order of the fields rules, just like a phone book sorted by last name and then first name: it works for searching by last name, and by last name + first name, but not by first name alone. That is the prefix principle, applied to eventSchema.index({ status: 1, venue: 1, title: 1 }):

The mnemonic is ESR: Equality-compared fields first, then the Sort field and finally the Range ones. A range field before the sort field forces the engine to sort in memory, which is exactly what you want to avoid.

You read it with two indicators. stage: 'COLLSCAN' means there is no usable index: on a small collection it does not matter, on a large one it is an alarm. And the totalDocsExamined / nReturned ratio: the ideal is 1, examining exactly what you return; if you examine 50,000 to return 20, the index is not doing its job even though it exists. One excellent special case: if the index contains every field the query needs (filter and projection), MongoDB answers without touching the documents — a covered query — and totalDocsExamined is 0.

A normal unique index rejects duplicates including nulls: if ten tickets have externalCode: null, the second one already violates uniqueness. The solution is the partial index, which only indexes the documents matching a filter.

And the uncomfortable reminder: every index is paid for on every write. Inserting a document with six indexes means seven structures to update. It takes memory — ideally indexes fit in RAM — and disk. An index no query uses is pure cost, and MongoDB lets you audit it with Event.collection.aggregate([{ $indexStats: {} }]): if accesses.ops is still 0 after weeks in production, it is surplus.

There can only be one text index per collection, and for serious search (synonyms, spell correction, tuned relevance) the usual approach is to delegate to a specialized engine such as Elasticsearch or Meilisearch, rather than forcing MongoDB.

• Embedding something that grows without bound. The day an event accumulates 30,000 embedded orders, it will pass 16 MB and become unwritable. There is no patch, there is a migration.
• Believing populate is a JOIN. They are extra queries, and inside a loop they turn into N+1.
• Filtering by a populated field. populate with match does not discard parents: it leaves the field as null. Use $lookup + $match.
• Putting $match at the end of the aggregation. You lose the index and process the whole collection.
• Forgetting $unwind before grouping by a field inside an array. You would be summing whole arrays instead of their elements.
• Creating indexes "just in case". Each one slows writes down. Create them from measured queries and audit them with $indexStats.
• Tip: run explain('executionStats') on any query in a hot route, before calling it done.
• Tip: an aggregation with three nested $lookups is usually the sign that this subsystem would fit better in a relational model. A good moment to read the next lesson without prejudice.

Write an aggregation summaryByOrganizer() returning, for each organizerId, the number of published events, the total number of sessions, tickets sold, revenue in cents and average occupancy, sorted by revenue descending.

This code lists the tickets of a session along with the buyer's name. State how many queries it fires for 200 tickets and rewrite it with a single one.

Escena Viva adds an "upcoming sessions with tickets at a venue" screen: it filters by status: 'published' and venue, requires sessions with dateTime >= now and sorts by sessions.dateTime ascending. Propose the index, justify the order with the ESR rule and say what you would expect to see in explain().

Exercise 1.

Exercise 2. It fires 401 queries: 1 for the tickets, 200 for the orders and 200 for the users. With $lookup it is resolved in one:

Exercise 3. Index: { status: 1, venue: 1, 'sessions.dateTime': 1 }. By ESR, status and venue are compared by equality and go first; sessions.dateTime plays the role of both sort and range, and goes last. Because it is a field inside an array of subdocuments, MongoDB treats it as a multikey index. In explain('executionStats') we would expect stage: 'IXSCAN', the absence of an in-memory SORT stage — the ordering is served by the index itself — and a totalDocsExamined / nReturned ratio close to 1.

You have learned to model relationships in MongoDB with judgment: when to embed and when to reference, with the criteria table and Escena Viva's decisions justified one by one. You know what populate really does — extra queries stitched together in your process, not a JOIN — how to detect and kill an N+1, and why we keep sold denormalized while consciously accepting the coherence debt it creates. With the aggregation framework you have built the reports that used to come out of a CSV file in Module 3: revenue by venue, occupancy by category, session ranking and sales by month, plus a complete dashboard with $facet. And you no longer assume anything about indexes: you design them with the ESR rule and verify them with explain().

You have also seen the limits. Nested $lookups, referential integrity nobody guarantees, a sold <= capacity invariant that depends on your discipline rather than on the engine. None of this disqualifies MongoDB — it is Escena Viva's official persistence layer and it works — but it leaves a legitimate question hanging: what would this look like in a relational system?

In the next lesson we answer that by modeling the same domain in PostgreSQL with Sequelize: tables, foreign keys, normalization, a CHECK (sold <= capacity) constraint the engine enforces no matter what, include as a real single-query JOIN, parameterized SQL versus injection, and a second implementation of src/repositories/events.js that will prove the architecture of lesson 07-01 was not theory.