Last case study of the module. In module 5 you trained classifiers, regressors, networks and clustering on Rutalia's 2,000-delivery dataset — in a notebook, with clean data and a metric at the end. This lesson is about what happens afterwards: when that model has to make decisions every minute, with data that changes, users who depend on it and consequences when it gets things wrong. We will see what changes from the lab to production, tour the major application domains connecting them with what you already built (developing one new piece for real: the collaborative filtering behind recommender systems), study the real life cycle of a model — drift, retraining, monitoring — learn from famous failures, and finish with what no metric captures: the ethics and regulation of automated decisions about people.

1. From notebook to production: what changes
2. Domain map: where each module 5 algorithm works
3. Developed case: a recommender system with collaborative filtering
4. The real life cycle: drift, retraining and monitoring
5. Famous failures and their lessons
6. Ethics and regulation: deciding about people
7. The Rutalia case: two uses of the same model, two verdicts
8. Closing the module: the four case studies

The model is the same; the environment changes everything:

The practical consequence: in production, the model is the small piece. Most of the system is data plumbing (the 06-03 you just saw: ingestion, deduplication, aggregation), validation, monitoring and governance. The teams that fail usually fail at the plumbing, not the algorithm.

Three modeling notes before the developed case:

• ETA in logistics is the production version of what you already did: the delivery_minutes model from 05-03 serving predictions in real time. Delivery and navigation apps do exactly that, with more variables (live traffic) and continuous retraining.
• Fraud is the kingdom of the accuracy trap from 05-02: with 0.3% positives, the model "it is never fraud" has 99.7% accuracy and zero value. In production you operate on precision/recall, choosing the threshold based on the asymmetric cost (letting a fraud through costs X, bothering a legitimate customer costs Y) — and DBSCAN (05-05) provides the unsupervised route: new fraud is not labeled, but it usually shows up as noise that fits no behavioral pattern.
• Vision and language: overview only. The networks you wrote in numpy in 05-04 are, conceptually, what is inside — backprop and gradient descent do not change; what changes is the architecture (convolutions, attention), the scale (billions of parameters) and the hardware. We do not develop it here: the point of the case study is that you already know the basic mechanism.

The new piece of the lesson, lightweight but complete. Rutalia delivers for a marketplace and wants to recommend products: "customers who bought this also bought...". The classic technique is collaborative filtering: it needs to know nothing about the products (no category, no price) — only the user-item matrix of interactions. The hypothesis: users with similar histories will keep resembling each other in the future.

Does the recipe sound familiar? It is the k-NN from 05-01 with two changes: the "space" is the rows of the user-item matrix, and the distance is replaced by cosine similarity (the angle between vectors, which ignores how much each user buys and looks only at the pattern). And it is also, deep down, the friend recommendation from 06-02 — overlapping neighborhoods — with a matrix instead of a graph.

Read the result for U-305 (backpack, case): their neighbors are U-301 (shares both) and U-302 (shares the backpack), so it strongly recommends the bottle — the "backpack+case" pattern pulls along U-301's. Nobody recommends the charger to them: its buyers resemble U-305 in nothing. That is collaborative filtering: the co-purchase structure does all the work, without a single label or product description.

What separates this toy from a real system (and where pieces of the course fit):

• Scale: millions of users × millions of items, a matrix that is 99.9% empty. Sparse representations, hashing (01-04) and approximate neighbors are used; the batch similarity computation is a divide-group-combine job (06-03).
• Cold start: a new user has no row. It is covered with popularity or with item attributes (content) until they build up a history.
• Feedback loops: the system recommends → the user sees only what was recommended → future data reflects the recommendations, not the preferences. The biases of this lesson start right here.

In the lab, the flow was linear: data → train → evaluate → done. In production it is a cycle, because the world signs no stability contract with your model:

Drift is the enemy's name, in two flavors:

• Data drift: the input distribution changes. Rutalia opens delivery in a new city: the distances and zones no longer look like those in the 2,000-delivery dataset. The model does not "break": it answers confidently about a world that no longer exists — it is extrapolating, the sin you already saw in the regression of 05-03.
• Concept drift: the relationship between input and output changes. Same traffic, same distances, but the downtown streets are rebuilt: the same features now imply different times.

Standard defenses, all with tools you already have:

• Monitor the input, not just the output: compare the recent distribution of each variable with the training one (means, percentiles, histograms — the single-pass summaries from 06-03). Data drift alarms need no labels and warn you before the error goes up.
• Monitor two layers of metrics: the model's (prediction error once the real label arrives: the observed delivery_minutes) and the business's (complaints about unnotified delays, click-through rate of the recommendations). They can diverge — a model can improve its MSE and worsen the experience if it improves where it does not matter. The business metric rules.
• Retrain deliberately: on a schedule (weekly/monthly) or triggered by drift alarms. Always against a temporal validation set — train on the past, validate on the present; never the other way around (it is the temporal version of the train/test split from 05-02).
• Reproducible pipeline: versioned data, versioned code, fixed seeds, tagged model. If the week-12 model fails, you must be able to rebuild it exactly to diagnose. Without reproducibility there is no debugging, only archaeology.

Generic cases, all documented in the industry over and over:

• Bias in the data → biased model. A company trains a résumé filter on its historical hires; since it historically hired few profiles from a certain group, the model learns to penalize features correlated with that group. The model did not "get it wrong": it learned exactly what was in the data — the historical bias was the dominant pattern. Lesson: data is a mirror of the past; if the past is unfair, the model automates the unfairness with a veneer of objectivity. Auditing the data (whom does it represent? whom does it not?) comes before auditing the model.
• Data leakage: the model that copied. A hospital model "predicted" a disease perfectly… using a variable that was only filled in after the diagnosis. At Rutalia: predicting the delay using the actual delivery time, or a delay average computed over the ENTIRE history (including the record's own future). Spectacular test results, useless in production. Lesson: for every variable, ask "would it be available at prediction time?". A result that is too good is not celebrated: it is investigated.
• Correlation ≠ causation. The model detects that deliveries with gift wrapping arrive later and someone proposes "removing gift wrapping to reduce delays". But the wrapping does not cause the delay: gift orders concentrate in peak-demand campaigns. Removing the wrapping will move nothing. Lesson: the models of module 5 are excellent correlation machines for predicting; using them to intervene ("if I change X, Y will happen") requires causal analysis or a controlled experiment (an A/B test). Prediction and intervention are different questions.

When a model's output affects people — a price, a loan, a penalty, an opportunity — the conversation stops being purely technical:

• Transparency: whoever receives the decision must be able to know that an automated system was involved and, in essence, which factors weighed in. Interpretable models (the trees and regressions of 05-02/05-03, whose coefficients and rules can be read) have a real advantage here over blacker boxes: at roughly equal performance, the interpretable one is usually the better choice for sensitive decisions.
• Human oversight: decisions with significant effects on people should not be fully autonomous. The healthy pattern is the one you already saw in 06-01 with optimization: the system recommends, the person decides — and the person must have real information and authority to contradict it, not be a rubber stamp.
• Regulation: in Europe, the GDPR limits decisions based solely on automated processing with significant effects, and the EU AI Act classifies systems by risk level and imposes on high-risk ones (employment, credit, essential services...) obligations of risk management, data quality, documentation and human oversight. You do not need to memorize the articles; you need the reflex: if your model scores people, assume you are on regulated ground and that the design decisions (which data, which metric, which threshold) will have to be explained to someone. For the details of a specific case, consult your organization's legal specialists — that is their terrain, not the algorithm's.

The delay classifier from 05-02 (will this delivery be late?) can be deployed in two ways. Same model, same F1 — and opposite verdicts:

This contrast is the synthesis of the lesson: the ethical question is not "is the model good?" but "what decision does it feed and on whom does the error fall?". The same F1 can be a good product or a legal and moral problem, depending on the use. And note the technical irony of use B: the delay depends on the zone, the zone was assigned by an algorithm, and the model would charge it to the courier — one system penalizing a person for another system's decisions. When you chain algorithms (which is exactly what this module teaches), you also chain their responsibilities.

The pattern common to all four: no real problem was solved with one algorithm. It was solved by modeling (translating the business into canonical problems), combining (chaining pieces from different modules) and measuring (honest baseline, business metric, temporal validation). That triad is what this module added on top of the previous five.

• Optimizing the model metric and ignoring the business one. A better F1 that does not move customer complaints is a vanity number. Define the business metric before training.
• Evaluating without respecting time. Mixing past and future in train/test (or letting a "from the future" variable sneak in: leakage) inflates the results. Temporal validation whenever the data has dates.
• Trusting a model that extrapolates. If the current input does not resemble the training input (drift), the model's confidence is worthless. Monitor the input distribution, not just the error.
• Using predictions to intervene. "The model says X is associated with delay" does not imply that changing X reduces delays. For interventions, experiment (A/B) or analyze causally.
• Automating the decision when automating the recommendation was enough. The jump from "the system suggests" to "the system decides" is the one that triggers the ethical and regulatory risk. Take it only deliberately, with oversight and an appeal path.
• Tip: before any deployment, write on one page: what the system decides, on whom each type of error falls, how its degradation will be detected and who can stop it. If you cannot write that page, you are not ready for production.

1. Item-item recommender. The filtering in the example compares users. Implement it the other way around: cosine similarity between columns (items), and recommend to each user the items most similar to the ones they already bought. Compare the recommendations for U-305 with those of the user-user approach. Why is the item-item approach usually preferred in production when there are far more users than items?
2. Drift detector. Using the generator of the 2,000-delivery dataset from 05-01, create a "new month" with distances increased by 40% (the city grew). Without using the label, detect the drift by comparing the mean and the 10th/50th/90th percentiles of each variable between the history and the new month, and define an alarm threshold. Then check how much the error of the 05-03 regression model worsens on the new month.
3. Hunt the leakage. A colleague proposes these variables to predict delay at the moment the order is assigned: (a) the order's distance, (b) planned departure time, (c) the courier's average delay over the last 30 days, (d) actual delivery minutes, (e) the zone's average delay computed over the entire dataset, (f) day of the week. Classify each as valid, direct leakage or subtle leakage, and justify it.

1. For U-305 (backpack, case), the co-purchase matrix gives the bottle as the item closest to theirs (co-purchased with the backpack and with the case via U-301) — matching the user-user approach, as is usual with small matrices. The industrial preference for item-item: with far more users than items, the item-item similarity matrix is small and stable (aggregate tastes about an item change slowly; a user changes with every purchase), so it can be precomputed in batch (06-03) and served instantly; it is also explainable: "because you bought a backpack" is a justification the user understands — transparency, which connects with the ethics section.
2. Sketch: generate hist and new_month (with distance * 1.4), and for each variable compute (new_mean - hist_mean) / hist_std and the percentile differences. The distance will jump several standard deviations (a clear alarm with a threshold like |z| > 0.5); the other variables will not. When the old model is applied to the new month, the delivery_minutes error rises noticeably, because the model extrapolates outside the range of distances it saw (05-03). The order matters: the input alarm fires before any label exists to measure the error with — that head start is the value of drift monitoring.
3. (a) valid — known at assignment time. (b) valid — planned, not actual. (c) valid with care — it is legitimate if the average uses only the 30 days prior to that order; if it is computed over a window that includes the order itself or later dates, it becomes subtle leakage. Also, since it scores a person, it carries the ethical implications of this lesson. (d) direct leakage — it is (almost) the answer: it does not exist until the delivery is finished. (e) subtle leakage — the aggregate "over the entire dataset" includes the future of the record being predicted; it will look wonderful in test and will not be available as such in production. It must be computed only with data prior to each order. (f) valid. General rule applied: reconstruct the moment of prediction and ask each variable "did you already exist?".

With this lesson, the case-study module closes. You have taken the machine learning of module 5 into the real world: you saw what changes from notebook to production (the data moves, the metric that rules is the business one, the model is the small piece of the system), you developed a recommender system by collaborative filtering reusing the k-NN of 05-01 with cosine similarity over the user-item matrix, you learned to watch for drift and to retrain with temporal validation, you extracted the lessons of the famous failures — bias in the data, leakage, correlation that is not causation — and you contrasted two uses of Rutalia's same delay model: notifying a customer (low risk, good product) versus penalizing a courier (an automated decision about a person, biased by another algorithm's decisions and on regulated ground). The whole module leaves a single idea repeated four times: real problems are not solved with one algorithm, but by modeling, combining and measuring — optimization in industry (06-01), graphs in social networks (06-02), data that does not fit in one machine (06-03) and models with consequences (06-04). There is nothing left of the course to introduce to you: in module 7, the case study is yours to choose. You will define an integrative final project (07-01), develop it by combining the pieces of these six modules (07-02) and present it by measuring results as we have done here (07-03). The tools are yours; now, go build.