The previous lesson left the signals switched on: operational proxies that warn when "something moves" in the model's inputs or outputs. This lesson puts rigor on that movement. We'll see why a model degrades without anyone deploying anything, the four types of drift with concrete CineClick examples, how to quantify it with PSI computed by hand (to truly understand what it measures), and how to finally honor the commitment we made in module 1: Evidently as the detection tool, run every week by Prefect, with an alert to #ml-alerts. We close with the playbook for what to do when the alarm goes off — because detecting drift is easy; interpreting it well is what separates a useful system from a false-alarm factory.

1. The model degrades without anyone touching the code
2. The four types of drift, with CineClick examples
3. Univariate detection: PSI step by step
4. Multivariate detection in broad strokes
5. Evidently: drift as a reproducible report
6. Scheduling the vigilance: Evidently inside Prefect
7. The playbook: what to do when the alarm goes off

Recall the idea we planted in module 1: traditional software fails when someone changes the code; an ML model can fail when the world changes, with the code intact. CineClick's champion v2 is a snapshot of the churn patterns that existed in the data it was trained on. That snapshot doesn't expire with time, it expires with distance: the further today's reality drifts from the photographed reality, the worse it generalizes.

Formally, the model learned an approximation of P(y|X) over a sample of P(X). There are two ways for that to stop holding:

• P(X) changes: the customers arriving no longer resemble those in training. The model is opining on terrain it doesn't know.
• P(y|X) changes: the same customers now behave differently. The learned pattern is outright false.

Neither of the two breaks the service, fails a test, or fires a 5xx. That's why this problem is invisible to everything built up to module 5, and why 06-01's prediction table is this lesson's raw material.

Marketing launches a three-month half-price promo and a new cohort walks in: young customers, almost all on the basic plan, with very low tenure_months and few weekly_hours so far. The distribution of the input features is no longer the training one. The model isn't necessarily wrong about them — but it is extrapolating, and its reliability on that cohort is an unknown.

Before anyone looks at the features, the histogram of emitted probabilities changes shape: the positive rate jumps from 14% to 28%. Prediction drift is usually the first visible symptom — it's the shadow that data drift casts onto the model's output — and it's dirt cheap to watch (we've been doing it since 06-01 with the 10–25% band). That said: it's a symptom, not a diagnosis; you have to look at the inputs to know the cause.

CineClick raises the price of all plans by 15%. Loyal customers, with long tenure and many weekly hours — the profile the model learned as "safe" — start cancelling. Those customers' features haven't changed at all: weekly_hours is still high, support_tickets still low. What has changed is the relationship between those features and churn. The learned pattern stopped being true, and no inspection of P(X) will detect it.

The overall churn proportion moves (from the training-time ~15% to 22%, say, because of an economic downturn), even though the mechanism relating features and churn stays roughly the same. In practice it usually comes mixed with the previous types; its most treacherous effect is miscalibrating the threshold: a 0.5 calibrated for a world with 15% churn no longer cuts where it should.

The last column is what organizes all the work: data drift and prediction drift are detected today, with the prediction table. Concept drift is only confirmed when the labels arrive — that is, with 06-01's ground truth job. The two mechanisms are complementary, not alternatives: data drift is the early warning system; the real metrics are the judge.

The Population Stability Index (PSI) is the classic metric for quantifying how much a variable's distribution has moved between a reference population (the champion's training data) and a current one (say, the last week of the prediction table). We're going to compute it by hand on weekly_hours, because understanding the calculation is worth more than memorizing the threshold.

The recipe:

1. Split the variable's range into bins, defined on the reference (typically 10 quantiles).
2. Compute what proportion of the reference falls into each bin (expected) and what proportion of the current data falls into those same bins (observed).
3. For each bin: (observed - expected) * ln(observed / expected). Sum over all bins.

And we apply it with fictional data mimicking the promo scenario:

How to read the number, with the industry's conventional thresholds:

Intuition for the formula: each bin contributes according to how much its proportion changed (observed - expected) weighted by in which relative direction (ln(observed/expected)). Both factors always have the same sign, so every bin adds positively: PSI = 0 only if nothing moved. Our 0.196 says "moderate change, not yet critical" — consistent with a new cohort that's a minority of the total population.

A common alternative: the Kolmogorov-Smirnov (KS) test, which compares the cumulative distributions and yields a p-value (scipy.stats.ks_2samp). It works well, with one caveat: at a streaming platform's volumes, KS declares tiny, irrelevant differences "significant" (with enormous n, everything is significant). That's why industrial practice prefers PSI, which measures the magnitude of the change, not statistical significance. For the categorical features (plan, payment_method), PSI is computed the same way using each category as a bin.

PSI looks at each feature separately, and that has a blind spot: the marginal distributions can hold steady while the correlations change (for example, premium-plan customers used to be the heaviest watchers; now they're not — each marginal intact, the joint structure broken).

The conceptually most elegant technique for that: train a classifier whose task is to distinguish whether a row comes from the reference or from the current data (label 0 = reference, 1 = current). If the best possible classifier can't beat an AUC of ~0.5, the two populations are indistinguishable — no drift. If it reaches 0.75, something has changed in the joint structure, and its feature importances tell you where to look. It's the "domain classifier" approach: current vs. reference.

We stay at the concept level: for the size and risk of the CineClick case, per-feature univariate PSI plus prediction drift covers the vast majority of real cases, and the domain classifier remains an investigation tool for when the univariates don't explain what you're seeing.

Computing PSI by hand educates, but maintaining the calculation by hand for 7 features, with visualizations, per-type tests, and a structured JSON for automation, doesn't scale. Here we honor module 1's commitment: Evidently is CineClick's drift tool.

Two ingredients:

• Reference: the training data of champion v2. Not a loose copy: the exact version, retrieved with DVC (dvc pull of the commit v2 was trained with). If the reference isn't versioned, the drift detector measures against an unknown measuring stick.
• Current: the last week of 06-01's prediction table, which stores the features exactly as they entered the model.

What each piece does:

• DataDefinition tells Evidently which column is what. It matters because the appropriate statistical test depends on the type (numerical vs. categorical); Evidently picks automatically based on type and sample size (Wasserstein, KS, Jensen-Shannon, PSI…).
• DataDriftPreset is a bundle of tests: per-column drift + a global summary.
• The .dict() is the key output for automation. From it we extract two numbers: how many columns drifted and drift_share, the fraction of columns with detected drift.

Reading the HTML: one row per feature with its test, its drift score, and a detected/not detected verdict, plus the overlaid reference vs. current distributions. In our promo scenario you'd see tenure_months and weekly_hours in red, plan shifted toward basic, and support_tickets calm — the exact "new cohort" pattern, which you'll recognize in seconds with the plots in front of you.

A report someone has to remember to generate is the problem this module came to eliminate. We turn it into a weekly Prefect flow, just like 05-03's scoring, on the same cineclick-pool work pool:

It gets deployed with the same pattern as weekly-churn-scoring: cron "0 7 * * 1" (Mondays at 07:00 Europe/Madrid, one hour after the scoring, so the last week is complete and fresh). Retries, run history, and alerts: all the 05-03 scaffolding, reused.

Important detail: the alert carries the what (drifted features and magnitude) and the where to go next (report + playbook). An alert that only says "there's drift" forces you to start the investigation from scratch at 07:00 on a Monday.

The reflex reaction — "there's drift, let's retrain" — is wrong often enough to deserve a procedure. When the alert arrives, the order of questions is this:

The three outcomes, in detail:

1. Upstream data error. The most common case in real life: a schema change, weekly_hours starting to arrive in minutes, a migration that fills nulls with 0. The pandera validation (customer_schema, 05-01) catches type and range changes, but a distribution change with valid values slips past it — that's what the drift detector is for. The solution is to fix the data, never to retrain on it.
2. Legitimate new cohort with no impact (yet). The promo brings real customers; the model extrapolates over them. If the proxies stay within band and there's no sign of degradation, you document and watch. Retraining immediately would be premature: the cohort doesn't have labels to learn from yet (they've just arrived — none has had time to cancel!).
3. Drift with performance impact. Here retraining does come into play. But when exactly, with what trigger policy and with what pipeline — that's the systematic decision the next lesson builds.

And the section's final reminder: everything above detects changes in P(X) and P(ŷ). Concept drift — the pattern that stops being true with normal-looking inputs, like the price increase — is invisible to Evidently without labels. Only 06-01's ground truth job confirms it, when real recall falls week after week with PSI calm. That's why the single dashboard has both rows: proxies and real metrics need each other.

• Using "last month's data" as the reference instead of the training data. Against a sliding reference, a slow, sustained degradation never triggers the alarm: every week looks like the previous one. The canonical reference is the champion's training data (versioned with DVC); a sliding reference can be added as a second view to catch abrupt changes.
• Confusing prediction drift with concept drift. Outputs changing doesn't mean the learned pattern is false: it's almost always data drift reflected. Concept drift is only confirmed with labels.
• Retraining as a reflex to every alert. If the cause is a data bug, retraining learns it and consolidates it. Playbook first, always.
• Textbook thresholds without context. PSI's 0.1/0.25 is a reasonable convention to start with, not a law. Calibrate with your history: compute week-vs-week PSI during healthy periods and see how much "background noise" your business has (seasonality, campaigns) before setting the alert threshold.
• Running the detector on the raw data instead of the model's features. What degrades the model is drift in its input space (including ratio_tickets, which is derived). That's why the prediction table stores the processed features.
• Ignoring sample size. With few current rows (a holiday, a traffic dip), the tests get noisy. Require a minimum row count before trusting the result, or accumulate more days.

1. Classify the drift. For each CineClick scenario, identify the type (data drift, prediction drift, concept drift, label shift) and whether Evidently would detect it without labels: (a) a hugely successful exclusive series makes low-hours customers stop cancelling for two months; (b) the app team changes the sign-up form and payment_method now arrives as "CARD" instead of "card" for new customers; (c) after a TV campaign, the median tenure_months of incoming requests drops from 26 to 9.

2. PSI by hand. With the lesson's psi() function, generate a reference rng.normal(10, 2, 10_000) and three current scenarios: (a) rng.normal(10, 2, 2_000), (b) rng.normal(11, 2, 2_000), (c) rng.normal(14, 3, 2_000). Compute the PSI for each, classify it against the threshold table, and explain the why of each result.

3. Design the reference. When a new champion v3 (trained on more recent data) gets promoted in the future, what must happen to the drift detector's reference, and how would you implement it so it happens with no manual intervention? Hint: think about what the registry "knows" about each version.

1. (a) Concept drift: P(y|X) changes — the "few hours" profile no longer implies risk the way it used to. Evidently on the inputs would see nothing (the customers look the same); it would show up as a drop in real precision in the 06-01 job (we predicted cancellations that don't happen). (b) Data drift from an upstream error: a new category appears in P(X). Evidently does detect it (drift in the categorical payment_method), and the pandera customer_schema would probably catch it first if the category isn't allowed. Playbook outcome: fix upstream, don't retrain. (c) Legitimate data drift (new cohort): detectable without labels via PSI on tenure_months; it will almost certainly come with prediction drift, because low tenure pushes churn probabilities upward.

2. Ballpark values (they vary with the seed): (a) PSI ≈ 0.01–0.03 → stable; it's the same distribution, and the small residual value is sampling noise — a good illustration that PSI never comes out exactly 0. (b) PSI ≈ 0.25–0.30 → on the border of significant change; the mean shifted half a standard deviation: the central bins' proportions redistribute appreciably. (c) PSI > 1 → drastic change; the mean shifted two deviations and the variance grew: a good share of the current mass falls into the reference's extreme bins, and those enormous observed/expected ratios blow up the sum.

3. The reference must become v3's training dataset, at the exact moment of the promotion — not before (we'd measure the current champion with the future's yardstick) and not never (we'd measure v3 against v2's data, generating permanent phantom drift). Implementation: when each version is registered in MLflow, the dvc_data_md5 of its training dataset is stored as a tag (the lineage the retraining flow will formalize in 06-03). The flow's load_reference task carries no fixed path: it queries the registry for models:/churn-cineclick@champion, reads that tag, and does dvc pull of that exact version. That way, promoting the champion updates the reference automatically, without touching the flow.

You can now put a name and a number on what in 06-01 was just "something moves". The four types of drift are delimited — data drift and prediction drift, detectable today without labels; concept drift and label shift, which only the ground truth confirms — and CineClick has a detector in production: PSI understood from the formula up, Evidently comparing every Monday the champion's versioned reference against the last week of the prediction table, and an alert in #ml-alerts with a playbook attached when drift_share crosses the threshold. The lesson deliberately leaves a door open: the playbook ends at "retraining candidate", but it doesn't say when the trigger fires, how the new dataset gets built without committing the temporal leakage we announced in 03-03, or how the result reaches production without skipping module 5's safeguards. That is exactly the content of the next lesson: automated retraining, the piece that truly closes the lifecycle we drew in 01-02.