CineClick now has tracked experiments and a champion model, versioned and loadable by alias. Before serving it in module 4, one silent risk remains: the model doesn't consume raw data, it consumes features, and those features will have to be recomputed in production. If the production computation differs in anything from training's — a rounding, a default value, a time window — the model will receive numbers different from the ones it learned on, and its predictions will degrade with no visible error whatsoever. This problem is called training/serving skew, and the industry has built an entire category of tools to attack it: feature stores. In this lesson you'll understand what they are, which problems they genuinely solve (skew, point-in-time correctness, reuse), and — just as important — when you do not need one. For CineClick we'll make a deliberately modest decision that module 4 will build on.

1. Training/serving skew: the silent corruption
2. The ratio_tickets case: one feature, two implementations
3. What a feature store is: registry, offline store, and online store
4. Point-in-time correctness: temporal leakage
5. Components and flow of a feature store
6. The landscape: Feast and the managed offerings
7. A critical look: when you don't need one (and what CineClick does)
8. Closing module 3

Let's recap CineClick's chain: prepare_data cleans the CSV, build_features (the code in features.py) transforms raw columns into the features the model expects, train fits the RandomForest. All of that happens at training time, with pandas, over a historical file.

Now picture module 4's service in production: a request arrives with the raw data of one customer and it must be answered in milliseconds. Someone has to compute that customer's features at that moment, in different code and a different context. That's where the crack opens:

• Two implementations: the training one (pandas, batch) and the serving one (maybe plain Python, maybe another team, maybe another language). Two codebases that "do the same thing" diverge sooner or later.
• Silent failure: if the production feature is 0.8 where training would have said 1.2, there's no exception, no error log. The model predicts; it simply predicts worse. It's the worst kind of bug: the one that doesn't warn you.
• Late detection: it gets discovered weeks later, when someone notices that retention campaigns perform below what the offline metrics promised — if anyone notices at all.

This is problem #7 from module 1's diagnosis ("features are computed in the notebook and nobody knows how to recreate them elsewhere"), and it's one of the biggest model killers in the industry. The classic paper on technical debt in ML lists it among the costliest patterns precisely because of its invisibility.

Let's make it concrete. The retention team suggests a promising new feature: the customer's friction intensity, ratio_tickets = support_tickets / tenure_months — 6 tickets over 36 months is not the same as 6 tickets in 3.

Laura implements it in features.py for training:

Months later (a hypothetical but painfully realistic scenario), another developer implements the computation in the prediction service, reading the feature's description in a document:

Both implementations are reasonable. Both pass their tests. And they don't compute the same thing: for a brand-new customer with 3 tickets, training said 3/1 = 3.0 (extreme friction, a strong churn signal) and production says 0.0 (no friction at all). Precisely in the segment where the feature is most informative — new customers with problems — the production model receives the opposite value from the one it learned. Nobody will ever see an error; only a model that "doesn't work as well in production".

The possible remedies, from least to most machinery:

1. A single source of truth in code: training and serving import the same function from the same package. No feature is ever implemented twice.
2. A feature store: feature definitions live in a central system that computes them and serves them to both worlds.

Let's look at option 2 seriously before deciding.

A feature store is a layer of infrastructure dedicated to managing features as first-class assets. Its three pieces:

• Definition registry: a central catalog where each feature is defined once — name, the entity it belongs to (the customer), type, computation logic, owner. It's the single source of truth: training and serving consume the same definition, so implementation skew disappears by construction. It also enables reuse: if tomorrow CineClick builds a second model (say, upgrade propensity), it reuses ratio_tickets instead of reimplementing it.
• Offline store: historical feature values, timestamped, optimized for bulk reads. It's where training datasets are built from — and its superpower is point-in-time correctness, which deserves its own section.
• Online store: the current feature values per entity, in a low-latency store (typically Redis or similar). When customer 4812's request arrives, the service doesn't compute anything heavy: it does a millisecond lookup and gets the same features, computed by the same logic, that fed training.

The offline store solves a second problem, subtler than skew: temporal data leakage. An example with CineClick:

Suppose we train with March labels: churned=1 means "this customer cancelled in March". To build the dataset, someone queries support_tickets from the operational database... today, in July. Customer 4812 cancelled in March after opening 4 furious tickets that very March, already mid-cancellation, and then 2 more in April fighting over the refund. The dataset says support_tickets=6 for a customer labeled with March data — but at the moment of the real prediction (February, when they could still be retained) they had 0.

The model learns a rule that's terrific in training and useless in production: "many tickets ⇒ churn"... because many of those tickets are a consequence of the cancellation, not a prior signal. Inflated offline metrics, disappointing real performance. It's the temporal equivalent of training with the answer leaked in.

The solution is called a point-in-time join (or time-travel join): for each training row, the offline store retrieves the value each feature had on the label's date, and not a day later:

Doing this by hand with pandas is possible (merge_asof, surgical care with the dates) but it's a minefield; feature stores give it to you solved because they store the history with timestamps. It is, arguably, their most valuable technical contribution. (An honest note about our project: churn_customers.csv is a static snapshot with no timestamps, so this risk is latent rather than active; it will become very real when we discuss retraining on continuously arriving data in module 6.)

The full flow: the definitions (in code, versioned in Git) get registered; materialization pipelines compute the values from the sources and write them to both stores; training reads from the offline store with point-in-time joins; serving reads from the online store with lookups. Training/serving symmetry is guaranteed by construction — that is the feature store's contract.

Without going into tutorials (that's not this lesson's goal), the lay of the land:

So the idea of a "declarative definition" is visible in code, this is how a feature view is declared in Feast — read it as a conceptual sketch, not a tutorial:

From a definition like this, the same system answers get_historical_features(...) (offline, point-in-time) and get_online_features(...) (online, in milliseconds). The definition is one; the consumers, two.

And now, the part many courses leave out. A feature store is serious infrastructure: an online store to operate and pay for, materialization pipelines to orchestrate and watch, a framework to learn, and a new source of incidents ("yesterday's features didn't materialize"). Adopting one without needing it is pure operational debt — the symmetric error to the one this course has been fighting since module 1: the notebook without engineering is as bad as engineering without a problem to justify it.

Let's run CineClick through this table: one model, one team (Laura, Marc, and you), features computed in microseconds from the customer's own columns, no history aggregations, no streaming. Right-hand column across the board. Decision: CineClick does not adopt a feature store today. But the skew problem is real and must be closed regardless, so we adopt the proportionate remedy — option 1 from section 2, elevated to a project rule:

features.py is the single source of truth for features. Every transformation from raw data to features lives in src/cineclick_churn/features.py, inside the installable package. The training pipeline imports it (it already does, via the build_features stage), and module 4's prediction service will import exactly the same function — same package, same version, same .clip(lower=1). Reimplementing a feature outside that module is forbidden.

This is a perfectly dignified "poor man's feature store": it solves implementation skew through the same mechanism (a single definition) at zero infrastructure cost. It doesn't provide point-in-time correctness or an online store — the day CineClick has three models, aggregation features over streaming viewing events, and a latency requirement that forces precomputation, the table above will say otherwise and the migration to Feast will be natural, because the definitions are already centralized and tested. The right architecture is not the most complete one, it's the one proportionate to the problem — revisited when the problem changes.

• Adopting a feature store "because that's what people do": if your case sits in the table's right-hand column, you end up operating Redis and materialization pipelines to serve features an import would have solved. Evaluate signals, not fashions.
• The opposite: denying the skew problem because you don't use a feature store: skew isn't cured by ignoring it. If there's no store, there must be another guarantee of a single definition (a shared package, as at CineClick) and, in module 5, tests that watch over it.
• Copying "just this one function" from features.py into the service "to avoid depending on the whole package": that is exactly how the second implementation of ratio_tickets is born. The package dependency is the guarantee, not a nuisance.
• Building historical datasets by querying current values: the temporal leakage from section 4. If your labels have dates, your features must be those of that date. With or without a feature store, always ask yourself: "was this value known at prediction time?".
• Confusing a feature store with "the features table in the warehouse": a shared table helps with reuse, but it gives you no point-in-time joins, no online serving, no definition registry. It may be enough — but be aware of what you're missing.
• Forgetting that feature definitions are versioned too: if ratio_tickets changes formula, models trained with the old formula still expect it. Definitions go in Git (with or without a store) and the model's lineage (previous lesson) must make it possible to know which version it was trained with.

1. Hunting the skew. Beyond the tenure_months=0 case, find two other subtle ways the serving implementation of ratio_tickets from section 2 could diverge from training's, even after fixing the zero case. Hint: think about data types and about what support_tickets means in each context.

2. Temporal leakage diagnosis. Marc proposes adding the feature weekly_hours_last_month (average viewing hours over the last month) using the current viewing table for labels from three months ago. Explain (a) what value this feature would have in training for a customer who cancelled ten weeks ago, (b) why that inflates the offline metrics, and (c) what the dataset would need in order to build this feature correctly.

3. Feature store, yes or no? For each scenario, decide using the signals table and justify in one or two sentences: (a) a fintech with 14 models sharing transactional-behavior features aggregated over 1/7/30-day windows, served online with a 50 ms budget; (b) a two-person team with a monthly batch purchase-propensity model over a static CRM extract; (c) CineClick two years from now, with five models (churn, upgrade, recommendation...) sharing near-real-time aggregates of viewing events.

1. Examples of subtle divergence (there are more): (a) types and rounding — in pandas, division produces float64 from integer columns; if the service receives the JSON with support_tickets as a string ("3"), or someone rounds to 2 decimals "to clean up the response", the values stop being bit-for-bit identical, and near a threshold in the trees' splits the prediction can flip. (b) source semantics — in training, support_tickets comes from the historical CSV (accumulated tickets as per the extract); in production, the service might read from the live support system, which perhaps also counts opened-then-cancelled tickets or excludes billing ones. Same field name, different population. The moral: skew isn't just different code, it's also a different source — which is why the single-source rule covers the function and the contract of its inputs (the model signature from the registry helps with the latter).

2. (a) Close to 0: the customer has been gone for ten weeks, so their viewing "over the last month" (counted from today) is nil — when the label was generated, they were an active customer with real hours. (b) The model learns "weekly_hours_last_month ≈ 0 ⇒ churn", a nearly perfect rule in the dataset... that is useless in production, because there the feature is computed for still-active customers: the offline metric measures the ability to recognize already consummated cancellations, not to predict them. (c) A viewing history with timestamps and a point-in-time computation: for each customer, the average hours of the month before their label date. Exactly the join an offline store gives you for free — and which, without one, demands a careful merge_asof over the history.

3. (a) Yes, clearly: many models sharing expensive features (time windows), online serving with strict latency — every signal in the left-hand column at once. (b) No: one model, batch, static data, minimal team; a shared features.py and single-source discipline are enough, and the store's operational cost never pays off. (c) Probably yes, and it's the moment to re-evaluate: aggregation features over events shared by five models trip the signals of reuse, computation cost, and point-in-time. The advantage of today's decision is that the migration starts from definitions already centralized and tested in features.py, not from five divergent copies.

Module 3 closes the chapter on "managing what gets learned". MLflow's churn-cineclick experiment stores every run with its parameters, metrics, commit, and data hash — the question "what was v2?" can never exist again. The registry turns the winner into model churn-cineclick version 2, promoted to @champion (recall 0.68, precision 0.55, F1 0.61) with documented human review and rollback one alias away. And features have a single source: features.py, in the installable package, shared by training and by the serving that's coming — with clear criteria for which signals would justify, tomorrow, a real feature store. Review the inventory: reproducible code, versioned data, a declared pipeline, experiments with memory, a champion model with a name, a version, and a signature, features with no double implementation. Everything a production service needs to load is identified and traceable — models:/churn-cineclick@champion is waiting for someone to call it. Module 4 does exactly that: choose the right deployment pattern (batch, online, or streaming), stand up a prediction service with FastAPI that loads the champion and imports the features from the single source, and package it with Docker so it runs the same anywhere. Laura has a model; what's missing is for CineClick to be able to use it.