The previous lesson's playbook ended at a box with its own name: "retraining candidate → systematic decision". This lesson builds that decision and everything that comes after. We'll look at the three possible trigger policies and which one CineClick picks, we'll finally flesh out the flows/retraining.py sketch we left in 05-03 — solving along the way the temporal leakage we announced in 03-03 —, and we'll make explicit the principle that governs the whole design: the system retrains and proposes, but it never promotes on its own. By the end, the lifecycle we drew in 01-02 will be truly closed, and CineClick will have crossed into maturity level 2.

1. The three trigger policies
2. CineClick's policy
3. The new dataset: point-in-time correctness (03-03's leakage, solved)
4. The complete retraining pipeline
5. Evaluation: frozen test and fresh test
6. Register as challenger, never as champion
7. Retraining doesn't always fix things; and it isn't free
8. The closed loop: CineClick reaches level 2

When should a model be retrained? There are three families of answers:

They aren't mutually exclusive, and in fact the scheduled + degradation combination is the most common pattern in industry: the scheduled one as a safety net ("the model will never be older than X"), the degradation one as a fast reflex ("if something goes wrong, we don't wait for the cycle").

CineClick adopts exactly that combination, with numbers you already know — they're aligned with the thresholds from 06-01, 06-02, and the business criterion CI has turned into a blocking check since 05-01:

• Scheduled trigger: monthly retraining (first Tuesday of the month, after Monday's scoring and drift detection).
• Extraordinary trigger, if any of these happens:
  • PSI > 0.25 on some relevant feature (06-02's "significant change" threshold), once the playbook has ruled out an upstream data error;
  • real recall < 0.60 two weeks in a row in 06-01's ground truth job — the same 0.60 that is a blocking threshold in CI, because the business criterion is one and the same no matter where it's measured. Two weeks are required so as not to react to the noise of one weak week.

The extraordinary trigger is not 100% automatic: the alert lands in #ml-alerts, a human runs the 06-02 playbook (upstream bug? legitimate cohort?) and, if the outcome is "retraining candidate", launches the flow with one click from the Prefect UI. The expensive, mechanical part (building the dataset, training, evaluating, registering) is indeed 100% automatic; the judgment of "does retraining make sense?" remains human. It's the same philosophy as module 5: automate the execution, not the decision.

Here we pay a debt we've been carrying since module 3. We announced it in 03-03: the static churn_customers.csv we've been working with has no timestamps, and that hid a problem that only surfaces now, when moving to continuous data. If we build the retraining dataset with each customer's "current state", we commit temporal leakage: the features of a customer who cancelled in March would include support tickets from April — information posterior to the event we want to predict. The model would learn to "predict the past with data from the future", shine in validation, and flop in production, where the future isn't available.

The rule is called point-in-time correctness: for each training example, the features are computed exclusively from data prior to a reference instant, and the label is observed after that instant. We define, for each customer:

• cutoff_date: the instant when we "take the photo" (for cancellations, a margin before the cancellation; for active customers, a sampling date).
• Features: aggregates over events with timestamp < cutoff_date.
• Label: was there a cancellation in the window (cutoff_date, cutoff_date + 30 days]? (the same 30-day window as 06-01's evaluation job — the prediction horizon is one and the same across the whole system).

The essence is in the two query("timestamp < cutoff_date") calls and in computing tenure_months relative to the cutoff: without timestamps in the events, this operation is impossible — that's why the static CSV shielded us from the problem for five modules and why it blows up exactly here. In SQL it would be the same pattern: JOIN ... ON e.timestamp < c.cutoff_date before the GROUP BY. Note that ratio_tickets needs no changes: features.py remains the single source of derived features (03-03), and since it's computed from columns already cut in time, it inherits the correctness.

There is a second contaminant left, already familiar from 05-04: the retention campaign itself. Customers flagged at risk get offered a discount; if it works, they don't cancel — and their label says "no churn" thanks to our intervention, not because the model was wrong. Training on those customers teaches the model that their very risk profile "doesn't churn". The solution has been deployed since module 1 and formalized in 05-04: the permanent control group, which never receives the discount. The control group's labels are the only unintervened ones, and they feed both the real evaluation (06-01) and, with appropriate weighting, retraining. It's the course's best example of why MLOps decisions get made before you need them: a control group can't be created retroactively.

Now, at last, we flesh out the flows/retraining.py sketch we noted in 05-03 (dvc pull → dvc repro → register as challenger). The full version:

Nothing here is new as a piece — and that's the beauty of it. dvc repro is module 2's pipeline; the tracking and the registry are module 3's; the thresholds are 05-01's CI thresholds; Prefect, its retries and its alerts are from 05-03. The flow merely chains what's already built. When a retraining pipeline requires new pieces, it's usually a symptom that the earlier modules left holes.

The flow evaluates against two test sets, and the reason deserves its own section:

• Frozen test (data/processed/test.csv, versioned since module 2): the same exam v1 and v2 passed. It's the only way to compare versions against each other on equal footing — if each version takes a different exam, "v3 has better recall than v2" means nothing. The blocking thresholds are applied against this test.
• Fresh test (fresh_test.csv, drawn from the control group's most recent labels, disjoint from training): it measures what really matters going forward — does the candidate work on the current world, promo cohort included? A candidate can shine on the frozen test (which ages along with the world) and stumble on the fresh one.

Over time, the frozen test gets renewed too (when enough drift accumulates, a new one is frozen and the live versions are re-evaluated on it), but that's a documented decision — an ADR, as we'll see in 06-04 —, never a silent change.

The rule was already written in the 05-03 sketch and now it makes full sense: the flow registers the candidate as @challenger and never touches @champion. Why so categorical, if the candidate cleared the thresholds?

Because the offline thresholds are a necessary condition, not a sufficient one. Between "it passed the exam" and "it deserves the traffic" lies everything module 5 built: the documented human review of the promotion PR (docs/model-promotion.md + scripts/promote_champion.py, 05-02), the mandatory two weeks of shadow for every challenger, and the 5% → 25% → 100% canary with guardrail metrics (05-04). A system that self-promotes skips exactly the safeguards that exist because offline tests don't capture online behavior. Automated retraining doesn't create a new path to production: it feeds the path that already exists.

The lineage tags (dvc_data_md5, time_window, flow_run) are the other half of the rule: when someone asks in six months "which exact data was v3 trained on and why did it exist?", the answer is in the registry, not in Laura's memory. In 06-04 this will become a centerpiece of auditability.

Two warnings that separate a mature system from one that shoots itself in the foot:

Retraining doesn't always fix things. If the detected drift was an upstream data bug (payment_method arriving as "CARD", hours in minutes), retraining doesn't eliminate the problem: it learns it. The new model fits the corrupted data, the tests pass (the fresh test is corrupted too), and the bug gets consolidated with a bow on top. That's why the 06-02 playbook comes always before the trigger: the human reviewing the alert rules out the bug before launching the flow. It's the underlying reason the extraordinary trigger isn't 100% automatic.

Retraining isn't free. Three costs that grow with frequency:

• Compute: full dataset + hyperparameter search + double evaluation, every time.
• Risk: every new model is an opportunity for regression. More versions per year = more shadows, more canaries, more occasions for rollback.
• Review fatigue: if there's a challenger awaiting human review every week, the review becomes a rubber stamp — and a safeguard that gets approved without looking is no longer a safeguard.

CineClick's balance — monthly + extraordinary with a human filter — follows from these costs: frequent enough that the model doesn't age (the observed drift is on the scale of weeks/months, not hours), and low enough that every challenger gets a genuine review. A homepage recommendation system would need a different answer; the trigger policy is a business decision in technical clothing.

With this piece, the lifecycle we drew in 01-02 — which back then was a promise — is truly closed:

Notice the two diamonds: they are the only points where the loop stops to wait for a person, and they sit exactly where there is judgment to exercise — interpreting the drift and approving the promotion. Everything else flows on its own.

And now, the promised recap against 01-03's maturity ladder. Level 2 demanded: automated, signal-triggered training (✔ this flow, with its two policies), model monitoring in production (✔ 06-01), drift detection that feeds decisions (✔ 06-02), and continuous model deployment with safeguards (✔ module 5). At the end of module 5 we said CineClick was missing "the system watching on its own": the last three lessons have given it the eyes and the reflexes. CineClick is at level 2. What remains of the course adds no automation: it adds governance (06-04) and perspective (06-05).

• Auto-promoting "because the metrics are better". Offline metrics don't see latency, or behavior under real traffic, or per-segment disagreements. The shadow and the canary exist because the exam and the job are different things. Challenger always; champion, only a human.
• Building the dataset from the customers' current state. It's temporal leakage in its most common and most silent form: excellent validation metrics, disappointing production. If your churn dataset has no cutoff_date column (or equivalent), be suspicious.
• Evaluating only against the frozen test. You'll approve models that are excellent for 2025. The fresh test is what measures 2026.
• Retraining as an automatic response to any drift alert. If it was a data bug, you've just fed it to the model. Playbook first.
• Registering without lineage. A challenger without dvc_data_md5 or a time window is an orphan model: when something goes wrong, you won't be able to reproduce it or explain where it came from. The tags cost four lines; write them always.
• Choosing the retraining frequency by intuition. Look at it as a business decision: speed of the observed drift, compute cost, the team's real review capacity. And revisit it when those variables change.

1. Pick the policy. For each system, propose a trigger policy (or combination) and justify it with the pros/cons table: (a) a payment fraud detection model, where adversarial patterns change in days; (b) a real-estate appraisal model whose labels take months to arrive (the actual sale price); (c) CineClick's churn if the business were expanding to a new country next quarter.

2. Hunt the leakage. Marc proposes simplifying build_pit_dataset like this: "for customers who cancelled, we use their aggregates as of the cancellation date as features; come on, a day here or there doesn't matter". Explain the two problems this introduces relative to the lesson's version and how each would manifest (one in validation, the other in production).

3. The suspiciously good challenger. The monthly flow registers v4 with recall 0.81 and precision 0.63 on the frozen test — way above v2's 0.68/0.55. Before celebrating, which three checks would you make using the system's lineage and artifacts, and what would you expect to find in each one if something fishy is going on?

1. (a) Drift/degradation-based with aggressive thresholds and short windows, plus a weekly scheduled one as a net: adversarial drift is fast and waiting for the monthly cycle gifts days to the fraudsters; the cost of retraining is lower than the cost of undetected fraud. (b) Scheduled with a long period (quarterly) + by volume of new labels: with ground truth that takes months, a trigger based on real-metric degradation would react far too late anyway, and triggering on input drift without being able to confirm impact produces blind retrainings; the label volume guarantees each cycle learns something new. (c) Keep monthly + extraordinary, but plan a proactive retraining once enough history from the new country has accumulated — and until then, watch that cohort as a separate segment on the dashboard: it's legitimate data drift announced in advance, the ideal case for getting ahead instead of reacting.

2. Problem 1 — leakage toward the label: with cutoff_date = cancellation_date, the features capture behavior during the cancellation process (the ticket spike of someone calling to cancel, the hours collapse of someone who has already decided to leave). The model learns to recognize "customer in the act of cancelling", not "customer who will cancel": spectacular recall in validation. Problem 2 — uselessness in production: the service scores customers before they decide to leave; the goodbye signals the model learned to look for don't exist yet at prediction time, so real performance sinks. On top of that, the room to act disappears: predicting the cancellation on cancellation day leaves no time for the retention campaign, which was the business purpose. The cutoff 30 days before (the horizon) solves both.

3. First: with the dvc_data_md5 tag, retrieve the exact dataset and check the overlap between training and test.csv (any customer_id in both? did dvc repro accidentally regenerate the test?); if there are shared rows, there's your culprit. Second: review the point-in-time construction for that window — a big recall jump is the classic signature of temporal leakage (exercise 2); compare the cutoff_date distribution and audit a sample of cancelled customers. Third: in the MLflow run, compare hyperparameters and features against v2 (did a suspiciously predictive new column sneak in, like "retention discount applied", which is a consequence of the prediction and not a cause?). If all three come out clean, then yes: congratulations — and into the shadow for two weeks like everyone else, where a leaky model would, by the way, give itself away with massive disagreements.

The loop is closed. CineClick will retrain every month — or sooner, if PSI exceeds 0.25 or real recall spends two weeks below 0.60 — with a flow that builds the dataset with point-in-time correctness (03-03's debt, settled: features only from data prior to the cutoff, clean labels thanks to the control group), re-runs the same reproducible pipeline as always, evaluates against the frozen test and a fresh one, and registers the result as @challenger with its full lineage. Never as champion: from challenger to traffic, the path is still module 5's — human review, two weeks of shadow, canary with guardrail metrics. With that, the lifecycle from 01-02 stops being a drawing and CineClick finally crosses into level 2 of the 01-03 maturity ladder. But a system that decides about people and improves itself raises questions no dashboard answers: why was retention offered to this customer and not that one? Does it treat all segments equally well? What do we document, what do we retain, what does regulation demand? That — governance, documentation, and compliance — is the next lesson.