The CD pipeline from lesson 05-02 left an announced limitation: after staging and the approval, the new version reaches production all at once, on every replica. For API code that's a bounded risk; for a model it's another story, because the versioned test set — however rigorous — is not real traffic. A challenger with 0.71 recall offline can behave strangely with the customers arriving today through the cancellation flow: distributions the historical data didn't contain, rare combinations, different volumes. This lesson presents the three strategies for letting a new model earn trust gradually: shadow (it predicts in the shadows, with no consequences), canary (it receives a growing percentage of real traffic), and A/B (a controlled experiment to answer a business question). All three are implemented on top of what we already have, and they close the module — and with it, an entire level of MLOps maturity.

1. The problem: offline doesn't guarantee online
2. Shadow deployment: predicting without consequences
3. What gets compared after two weeks in the shadows
4. Canary release: real traffic, little by little
5. Guardrail metrics and automatic rollback
6. A/B testing: when the question is about the business
7. Comparison and CineClick's decision
8. Closing the module: where we stand on the maturity ladder

The challenger's offline metrics are computed on the versioned test set: a split of historical data, with random_state=42, frozen with DVC. That makes it comparable (all candidates are measured on the same ground), but it doesn't make it representative of the future. Concrete reasons why a good offline model can disappoint online:

• Real traffic is not a random sample of the historical data. The /predict endpoint mostly receives customers in the cancellation flow — a heavily biased subset of the customer base. The test set contains a bit of everything.
• The world moved on. Between the dataset's cutoff and today there were campaigns, price increases, a new plan. The test set is a snapshot; traffic is the movie.
• Effects that only exist in production. The new model's latency under load, interaction with 04-05's TTL cache, extreme values that validation permits but training barely saw.
• The ML metric is not the business metric. The model exists to retain customers; recall and precision are approximations. F1 can go up without moving — or while worsening — retention (we'll see it in the A/B).

Operational conclusion: staging + smoke test verify that the system works; they don't verify that the new model decides well on today's traffic. For that you have to expose it to real traffic — the question is how much risk you accept while you check. The three strategies are three answers to that question:

In a shadow deployment, the challenger receives a copy of all the traffic: for every real request its prediction is also computed, logged, and discarded — the response to the customer always comes from the champion. Functional risk: zero. Information gained: how the challenger would behave on exactly production's traffic.

There are two ways to set it up: a second deployment that receives duplicated traffic (at the infrastructure level, with a mirror in the ingress/service mesh), or — the simple option CineClick will use — having the service itself load both models and log both probabilities. With the per-prediction JSON logging that has existed since 04-02, the implementation is small:

Details that matter:

• SHADOW_MODE is configuration, not code: it's switched on by adding the variable to the churn-api-config ConfigMap and restarting — no image release. Switching it off is just as cheap.
• The features are computed once and reused for both models: same feature cost, double cost only for inference.
• The response and the probability field in the log do not change meaning: any existing dashboard or consumer keeps working; the shadow only adds fields.
• The real cost is latency and CPU: the second inference adds a few milliseconds within the budget (p95 = 92 ms against a 300 ms budget: plenty of headroom, but you measure it, you don't assume it — that's what shadow_latency_ms is logged for).
• Strictly speaking, an extra predict_proba in the same process can also fail; it gets wrapped in a try/except that logs the error and never affects the response (left as exercise 1).

The shadow doesn't issue a verdict by itself: it accumulates data for an analysis. At CineClick the fixed period is two weeks (covering two full weekly cycles, weekend pattern included), and the analysis looks at three things in the JSON logs:

1. Probability distribution. Does the challenger assign probabilities with a reasonable shape, similar to the champion's or shifted in an explicable way? A challenger that concentrates everything in 0.45-0.55, or that pushes the positive rate from the expected ~15% to 40%, is suspect even if its offline metrics were good.
2. Champion/challenger disagreements. In what percentage of requests do they cross the threshold in opposite directions? Those discordant cases are the magnifying glass: they get extracted (via input_hash) and Laura reviews them — does the challenger get right what the champion got wrong, or the other way around? An honest note: in churn the true label takes weeks to arrive, so this review is qualitative and about plausibility; confirmation with labels comes later (and systematizing it is module 6's business).
3. Added latency. Is shadow_latency_ms behaving? Does the combined p95 stay within the 300 ms budget? A challenger 4 times slower is an operational problem even if it predicts wonderfully.

If all three come out healthy, the challenger has demonstrated that it works on real traffic. What the shadow cannot demonstrate is that its decisions produce better outcomes — its predictions were never used, so they caused nothing. For that you have to give it real traffic: canary or A/B.

The canary (after the mine canary) routes a small, growing percentage of real traffic to the new model: at CineClick, 5% → 25% → 100%, advancing only if the guardrail metrics stay healthy at each step (typically 24-48 h per step). If something degrades, you go back to 0% in seconds and only 5% of requests were ever affected.

Two levels where you can implement it:

• Kubernetes level: two deployments (churn-api-stable and churn-api-canary) behind the same Service; splitting by replica count gives coarse percentages (1 canary pod out of 4 ≈ 25%). For fine percentages and real weight control you use an ingress with weight support or a service mesh (Istio, Linkerd) — they exist, they work, and we're not going to tutorialize them here: at CineClick's size that would be more infrastructure than problem.
• Application level (CineClick's option): the service loads both models and decides per request which one to use. The key requirement: the assignment must be stable per customer — the same customer must get the same model for as long as the canary lasts, so their experience is coherent and the groups are comparable. A random() per request doesn't satisfy that; a hash of the customer_id does:

And in the endpoint:

Properties of this design, worth savoring:

• Stability: customer C-0042 always falls into the same bucket, on any replica, today and tomorrow.
• Monotonicity when scaling up: when CANARY_PERCENTAGE goes from 5 to 25, buckets 0-4 stay in the canary and 5-24 get added. Nobody bounces between models when you climb a step.
• Operation without a release: the percentage lives in the churn-api-config ConfigMap; climbing a step means editing one value and restarting. The rollback (to 0) too.
• The group field in the JSON log is what allows comparing metrics per group — without it, the canary is blind traffic.

A canary without automatic surveillance is roulette with extra steps. Guardrail metrics are the small set of signals that, if they degrade in the canary group relative to the stable group, abort the progression without waiting for a human:

The first two are service metrics; the third is the interesting one in ML: it needs no labels (which in churn take weeks) and still catches the big disasters instantly. The mechanics of automatic rollback: a recurring check compares the metrics by group and, if a threshold is violated in a sustained way, sets CANARY_PERCENTAGE=0 and alerts — the human arrives afterwards, with the incident already contained. It's the natural evolution of the "semi-automatic" rollback we noted in 05-02: here the trigger does get automated, because the criterion is objective and the cost of a false positive (pulling a healthy canary) is minimal. How these metrics get collected and queried continuously — dashboards, real alerting — is exactly the topic of 06-01; here the contract is enough: no guardrail metrics defined before you start, no canary.

Shadow and canary answer questions about system behavior: does it predict sensibly? Does it hold up? But the question that really matters to CineClick is a different one: does the new model retain more customers? That's a causal question about a business outcome, and only an experiment answers it: the A/B test.

The design for CineClick:

• Unit: the customer (assigned by the same stable hash as the canary — the code already exists).
• Group A: its at-risk customers are identified with the champion; group B: with the challenger. In both, the identified ones receive marketing's retention offer.
• Control group: a fraction of at-risk customers who receive no offer. The problem we diagnosed in module 1 resurfaces here: the intervention contaminates the data — if an at-risk customer gets a discount and stays, did the model "fail" at predicting their churn, or did the offer work? Without a control it's impossible to separate model quality from the offer's effect, and along the way the control provides the clean data future retrainings will need.
• Primary metric: retention rate 30 days after the offer. Secondary: cost of the offers issued, revenue retained.
• Minimum duration: fixed before starting, by calculating how many customers are needed to detect the effect that matters (with ~2,000 at-risk customers/month per group, detecting a 5-point retention improvement takes on the order of a month; smaller effects, considerably more). Stopping the experiment "while B is winning" is the classic way to fool yourself.

The intuition: if B retains 23% and A retains 20%, is that a real improvement or sampling noise? The operational question is "if there were no real difference, how odd would it be to see this difference by chance?" — that's the p-value. With two proportions, the standard test is a chi-squared test on the contingency table:

The practical reading: p = 0.023 < 0.05 → if the models were equally good, a difference like this would show up by chance only ~2 out of every 100 times; it's reasonable to attribute it to the challenger. Three nuances that prevent 90% of the misuses: the 0.05 threshold is convention, not physics; with enormous samples almost everything comes out "significant" — also ask whether the 3 points matter to the business (here: 60 extra customers retained per month, clearly yes); and checking the p-value every day and stopping when it crosses 0.05 invalidates it — duration fixed in advance.

The most instructive possible outcome: the challenger has better offline recall and worse retention in the A/B. How? For example: the challenger identifies more actual churners (recall ↑) but the ones it adds are customers already determined to leave, whom the offer doesn't move — while it stops flagging waverers the champion did capture and the offer did retain. The ML metric measures prediction accuracy; the business metric measures the effect of the action taken with the prediction. When they collide, business wins — it's the only reason the model exists. This scenario, impossible to see in the test set and invisible to the shadow, is the definitive justification for the A/B.

They are not mutually exclusive alternatives but layers: each one filters what the previous one can't see. The policy CineClick puts in writing:

1. Every challenger aspiring to @champion spends 2 weeks in the shadows (on top of the 05-01 offline gates and the 05-02 approval). Cheap, zero risk, catches the disasters.
2. The promotion is executed as a 5% → 25% → 100% canary with this lesson's guardrail metrics and automatic rollback to 0%.
3. A/B testing reserved for business decisions: threshold changes (CHURN_THRESHOLD), changes to the retention offer, or challengers whose value case is disputed. With a permanent control group, which additionally protects the future training data.

It's worth looking back with module 1's yardstick. CineClick started at level 0: notebook, hands, and memory. Modules 2-4 built the pieces (package, DVC, MLflow, service, image, Kubernetes) and this module has made them move on their own:

• CI (05-01): every change runs code tests, data tests (pandera), and model tests (smoke, behavior, business thresholds).
• CD (05-02): two decoupled pipelines — image by tag with approval, model by alias with a human in the loop — each with its own rollback.
• Orchestration (05-03): the Monday batch runs by itself, retries, alerts, and leaves a trail.
• Release (05-04): new models earn trust in layers — shadow, canary with guardrails, A/B for the business.

That's a long level 1, brushing against 2. What exactly is missing for level 2? That something triggers all of this without a person deciding it: today, retraining is launched by hand, the shadow is switched on by hand, and nobody knows — with data — when the champion starts to age. The missing piece isn't more automation: it's observation. Knowing how the service and the model behave in production, detecting when the data drifts, and using those signals to fire the pipelines this module left ready. That's module 6.

• Skipping the shadow "because the offline metrics are so good". The shadow exists precisely because that sentence is not an argument: it's the hypothesis to test. Two weeks of shadow cost almost nothing; an unhinged champion in the cancellation flow costs customers.
• A canary with random per-request assignment. The same customer gets model A at 10:00 and model B at 10:05: an incoherent experience and incomparable groups. Stable hash of customer_id, always.
• Using Python's hash() for routing. Since Python 3.3, string hash() changes between processes (PYTHONHASHSEED): each replica would assign different buckets and the "stable assignment" would be an illusion. hashlib is deterministic.
• A canary without guardrail metrics defined up front. If you decide the thresholds by looking at the live canary's data, you'll decide whatever confirms your urge to promote. Guardrails and thresholds in writing before the 5%.
• Stopping the A/B as soon as the p-value crosses 0.05. Peeking inflates false positives: checking every day, the probability of crossing 0.05 by pure chance at some point is much higher than 5%. Duration fixed in advance; analysis at the end.
• A/B without a control group. You'll measure which model flags customers "better suited to the offer", but not whether the offer itself adds anything, and you'll contaminate the history for future retrainings (module 1's lesson). The control isn't optional: it's the price of being able to learn.
• Tip: write the release policy (the three points from the previous section) into the repo, next to 05-02's promotion procedure. When someone proposes "promote it straight through, we're in a hurry" six months from now, the cost of the exception will be visible and the decision, conscious.

Harden the shadow mode: rewrite the shadow-inference block so that (a) a challenger failure can never affect the response, (b) the failure is recorded in the JSON (shadow_error), and (c) if the challenger fails more often than it should, you have a way to detect it afterwards from the logs.

With CANARY_PERCENTAGE=25, state with reasoning whether these customers get the canary, using customer_bucket: (a) a customer whose bucket is 4; (b) one whose bucket is 24; (c) one whose bucket is 25; (d) what happens to each when the step rises to 100? And if in the middle of the 25% step there's a rollback to 0?

In the retention offer A/B: group A (champion) 1,800 customers with the offer and 342 retained; group B (challenger) 1,750 and 368 retained. Compute the rates, run the chi-squared test with scipy, and interpret the result the way you'd tell it to the head of retention (no jargon). What decision do you recommend?

Solution 1

(a) The try/except wraps only the shadow inference: whatever happens, the champion's response is already computed and gets returned intact. (b) The exception type is logged — enough for diagnosis without dumping giant tracebacks on every log line. (c) Since it's a field in the structured JSON, the shadow's failure rate is obtained by counting lines with shadow_error over the total — and a challenger that fails on 3% of real requests has just handed you extremely valuable information before touching a single user: that's exactly what the shadow was for.

Solution 2

The rule is bucket < percentage. (a) bucket 4 < 25 → canary (and it already was at 5%: buckets 0-4 entered at the first step). (b) bucket 24 < 25 → canary; it entered at the 25% step. (c) bucket 25 is not < 25 → stable; it will enter when the percentage exceeds 25 (in our plan, straight at 100%). (d) With 100, every bucket (0-99) satisfies the condition → everyone on the canary, which at that point becomes the new stable and gets consolidated by moving @champion (the 05-02 flow). With a rollback to 0, no bucket satisfies bucket < 0 → everyone returns to the champion instantly, with one ConfigMap edit and no release. The beauty of the design is that all four answers come out of a single line of code and are identical on every replica.

Solution 3

Rates: A 19.0%, B 21.0%. The p-value ≈ 0.14 means: if both models were equally good at retaining, we'd see a difference like this (or bigger) by pure chance roughly one in seven times. That's too frequent to rule out chance. Translation for the head of retention: "B looks somewhat better, but with these numbers we can't tell it apart from luck; it's not a 'no', it's a 'we don't know yet'". Recommendation: do not promote yet on this evidence — if the experiment's planned duration was longer, continue until it completes (without watching the p-value along the way); if it's already over, the real improvement, if any, is probably smaller than what the experiment was sized to detect, and it's time to decide whether such a small improvement justifies a longer experiment or sticking with the champion.

Module 5 began with a list of manual chores and ends with a system that moves on its own and — harder — that changes with a safety net. CI runs the code, data, and model tests on every change, with the business criterion turned into a blocking threshold; CD delivers along two decoupled roads — the image by tag with environment approval, the model by alias with documented human review — each with a rehearsed rollback; Prefect launches the Monday 06:00 scoring with retries, alerts, and a trail; and this final lesson has supplied the piece that was missing between "approved in staging" and "serving 100%": two weeks of shadow for every challenger, a 5% → 25% → 100% canary with guardrail metrics and automatic rollback to zero, and A/B with a control group when the question belongs to the business — because retention, not recall, is the reason this model exists. CineClick now lives at a very long maturity level 1, brushing against 2. What it lacks to cross over has a name: none of this triggers itself. The shadow gets switched on by hand, retraining waits for someone to ask for it, and the only way to know whether the champion is aging is for a human to remember to look. Module 6 builds the system's eyes — service and model monitoring, data and concept drift detection — and with them, its reflexes: retraining when it's due, governing what gets deployed, and documenting what gets decided. The pieces already move on their own; now they have to learn to watch.