In lesson 04-02 we discovered that logistic regression does not predict classes but probabilities (predict_proba), and that the 0.5 threshold used to turn them into "churn / no churn" is our decision — a business lever. In the previous lesson we compared models with that threshold nailed to 0.5; in this one we set it free. The ROC curve evaluates a classifier at every threshold at once, and the AUC condenses that curve into a single number that lets us compare models independently of the threshold. Then we will walk the road back: choosing the concrete operating threshold that suits MercaFresh's retention campaign given its costs and budget. We will close with the precision-recall curve, the preferable alternative under extreme imbalance, and with the limitations of AUC.

1. The decision threshold as a free parameter
2. TPR and FPR: the two coordinates of the ROC
3. Building the ROC curve point by point (by hand)
4. AUC: the probabilistic interpretation
5. ROC and AUC in scikit-learn: comparing models
6. Choosing the operating threshold from business costs
7. The precision-recall curve as an alternative
8. Limitations of AUC

Let's recall the mechanism from 04-02:

Lowering the threshold to 0.3 makes the model flag more customers as churn: it catches more real departures (recall goes up) at the cost of more false alarms (precision goes down). Raising it to 0.7 does the opposite. The same trained model generates infinitely many different classifiers, one per threshold. Evaluating only the 0.5 one is watching a single frame of a movie; the ROC curve shows the whole movie.

For each threshold, two rates are computed from the confusion matrix of 06-02:

• TPR (true positive rate) = TP / (TP + FN). This is exactly the recall: of the customers who leave, what fraction do we detect? We want it high.
• FPR (false positive rate) = FP / (FP + TN). Of the loyal customers, what fraction do we bother with a false alarm? We want it low.

Note the elegance: each rate is computed within its own class (TPR among the actual positives, FPR among the actual negatives), so neither depends on the dataset's churn proportion — a detail that will matter at the end of the lesson.

Nothing cements the idea like computing it by hand. Take 10 MercaFresh test customers with their churn probabilities according to the model, sorted from highest to lowest probability (4 actual churners, 6 loyal):

Now we slide the threshold down from the top: at each value, every customer with probability ≥ threshold is predicted as churn, and we count TP, FP → TPR = TP/4, FPR = FP/6:

The ROC curve is the plot of these points (FPR on the X axis, TPR on the Y axis), from (0,0) — an impossibly high threshold: nobody gets flagged — to (1,1) — threshold zero: everybody flagged. Every time the sliding threshold "swallows" an actual churner, the curve moves up; every time it swallows a loyal customer, it moves right. A good model ranks the churners ahead of the loyal customers, so its curve rises early and steeply before moving right: it hugs the top-left corner.

Two reference points on the chart:

• The diagonal from (0,0) to (1,1) is chance: a "model" assigning random probabilities rises and moves right at the same pace. Any useful curve must sit above it; a curve below the diagonal indicates a model that ranks backwards (flipping its predictions would make it useful!).
• The ideal point is (0, 1): 100% of departures detected, 0% of loyal customers bothered. No real model reaches it, but the distance to that corner summarizes the quality of each threshold.

The AUC (Area Under the Curve) is the area under the ROC curve: a number between 0 and 1 that condenses performance across all thresholds.

Its most useful interpretation is probabilistic: the AUC is the probability that, picking at random one customer who left and one who stayed, the model assigns a higher churn probability to the one who left. In other words, it measures the quality of the ranking, not of the classification. Let's verify it with our table: there are 4 × 6 = 24 possible (churn, loyal) pairs; counting how many the model orders correctly (the churner with a higher probability than the loyal customer): C1 and C2 beat all 6 loyal customers (12 pairs), C4 beats 5 (all except C3), C7 beats 3 (C8, C9, C10). Total: 12 + 5 + 3 = 20 correctly ordered pairs out of 24 → AUC = 20/24 ≈ 0.83, which matches exactly the area under the stepped curve above.

This reading explains why AUC is ideal for comparing models before deciding on a threshold: a model with a better AUC ranks customers by risk better, and that ranking is the raw material of any campaign ("call the highest-risk customers first").

Important details:

• roc_curve returns three arrays: fpr, tpr and the thresholds corresponding to each point — we will use them next to choose the operating threshold.
• roc_auc_score must receive the probabilities, never predict() (with hard 0/1 classes the "curve" would have a single useful point and the AUC would come out distorted).
• For models without a well-calibrated predict_proba (like SVC), use decision_function, which also ranks.
• If two curves cross, one model may be better in the low-FPR zone and the other in the high-FPR zone: the overall AUC can tie while, for your operating range, one of the two is clearly preferable. Look at the curve, not just the number.

AUC compares models; to use the chosen model you must fix a threshold. The MercaFresh scenario: the retention team can call 600 customers a month (limited budget) out of a base of 10,000, with expected churn of 20%. Two complementary approaches:

Approach 1 — capacity: if you can only call 600, the threshold is simply the one that flags the 600 highest-probability customers:

Approach 2 — costs: with explicit costs (call + voucher ≈ €15; a lost customer ≈ €300 of annual value; retention works, say, on 30% of those contacted in time), you can compute the expected profit at each threshold and keep the maximum:

The usual conclusion in churn: the optimal threshold ends up well below 0.5 (often 0.2–0.35), because an FN costs 20 times more than an FP. The default 0.5 is not neutral: it presupposes symmetric costs that almost never match the business. And a methodological note: the threshold is chosen on validation data, not test data (06-01's rule: the test set takes part in no decisions).

Under heavy imbalance (positives at 1–5%: fraud, rare diseases... or a very low monthly churn), the ROC can paint an overly optimistic picture: since FPR is computed over the huge negative class, a "low" FPR of 5% can mean thousands of false alarms burying the true positives. The precision-recall curve (precision on Y, recall on X, one point per threshold) uses precision instead of FPR, and precision does feel every false positive directly.

Rule of thumb: churn at 20% → the ROC works fine; positives below ~5% or interest focused exclusively on the minority class → add (or prioritize) the precision-recall curve. Here the baseline is not the diagonal, but the horizontal line at the prevalence (0.20 in our case): that is the precision of flagging at random.

• It is blind to costs: it summarizes all thresholds equally, including ones you would never use. Two models with the same AUC can perform very differently in your operating zone (low FPR, if the budget is tight).
• It does not measure calibration: it evaluates the ordering, not whether "0.7" really means a 70% probability. A model can have AUC 0.85 with systematically inflated probabilities; if you are going to use the probabilities as such (expected-profit calculations), calibration matters.
• Optimistic under extreme imbalance, as we have just seen: complement it with precision-recall.
• A number is no substitute for the curve: with crossing curves, the scalar summary hides exactly the difference that affects you.
• None of this invalidates it: as a threshold-independent model-comparison metric it remains the standard (which is why scoring="roc_auc" fits naturally into the cross-validation of 06-03).

• Passing predict() instead of predict_proba() to roc_auc_score. With hard classes the AUC is computed over a two-value ranking and comes out artificially low. Probabilities, always.
• Keeping the 0.5 threshold "because it's the default". In problems with asymmetric costs (nearly all real ones), 0.5 is an inherited arbitrary choice, not a decision.
• Choosing the threshold by looking at the test set. The threshold is one more modeling decision: it is made on validation (or CV) and verified exactly once on test.
• Comparing AUCs to three decimal places. Lesson 06-03 applies here too: an AUC of 0.842 is not better than one of 0.838 without looking at the variability across folds.
• Using ROC as the only lens under extreme imbalance. Pair it with the precision-recall curve.
• Tip: hand the business the curve with two or three operating points annotated ("with threshold 0.3: we detect 78% of departures by calling 1,900 customers") — it is infinitely more actionable than "AUC = 0.84".

With this mini-table (3 actual churners, 3 loyal), compute the (FPR, TPR) points for thresholds 0.9, 0.6 and 0.3, and the AUC via the correctly-ordered-pairs counting method.

MercaFresh's churn model has AUC = 0.84. Explain to an executive, without jargon, what that 0.84 means, and why that number alone does not say how many customers to call each month.

Write the code that, given probs (probabilities on validation) and y_val, finds the lowest threshold whose recall is ≥ 0.80 and reports how many customers would need to be contacted with it (over a base of 10,000 with the same distribution). Hint: roc_curve gives you tpr and thresholds aligned.

Solution 1. Positives = 3 (A, C, D), negatives = 3 (B, E, F).

• Threshold 0.9 → flagged {A}: TP = 1, FP = 0 → (FPR, TPR) = (0, 0.33).
• Threshold 0.6 → flagged {A, B, C}: TP = 2, FP = 1 → (0.33, 0.67).
• Threshold 0.3 → flagged {A, B, C, D}: TP = 3, FP = 1 → (0.33, 1.00).

(Churn, loyal) pairs = 9. Correctly ordered: A beats B, E, F (3); C beats E, F (2); D beats E, F (2). Total 7/9 → AUC ≈ 0.78.

Solution 2. "If we pick at random one customer who ended up cancelling and another who stayed, the model assigns a higher risk to the one who cancelled 84% of the time: it ranks customers well by churn risk." It doesn't say how many to call because AUC evaluates the entire ordering without fixing any cut-off point: how many we call depends on where we cut that ranked list, and that is a business decision (campaign budget, cost per contact, customer value), not a property of the model.

Solution 3.

Since roc_curve orders the thresholds from highest to lowest and tpr grows along the array, the first index with tpr >= 0.80 corresponds to the highest threshold (and therefore fewest contacts) that reaches that recall; it is the cheapest point that meets the detection goal.

We have set the decision threshold free: TPR and FPR describe each possible cut-off, the ROC curve draws them all (built by hand, point by point, over ten MercaFresh customers), and the AUC summarizes it with a memorable interpretation — the probability of ranking a churn/loyal pair correctly. With roc_curve and roc_auc_score we compared models beyond the default 0.5, chose the operating threshold on business criteria (campaign budget, cost of each type of error), and learned when the precision-recall curve is the better lens and what AUC does not tell you. We now have the complete evaluation arsenal; what remains is the diagnostic question that runs through it all: when a model performs poorly — or suspiciously well — is it because it memorizes, or because it doesn't learn enough? Overfitting and underfitting, the central diagnosis of all Machine Learning, close the module in the next lesson.