We closed the previous lesson with a table in which logistic regression got 87.1 % of the orders right and the "never returned" baseline 83.6 %. Diego summed it up like this: "three and a half points; I don't think it's worth setting all this up." He was right to distrust the figure, but for the opposite reason to the one he had in mind: accuracy is a crude measure that, with imbalanced classes, hides what really matters. In this lesson we will learn to evaluate models like professionals: the confusion matrix and the four types of outcome, each with its cost in euros; precision, recall, specificity and F1; the ROC and precision-recall curves and the AUC; choosing the threshold according to cost (bringing back the human review band of 02-04); the regression metrics (MAE, RMSE, R²) for demand forecasting; and the validation strategies (hold-out, cross-validation, stratified and temporal) that stop a single lucky split from fooling us. It matters because evaluation is what turns an experiment into a business decision: without appropriate metrics and honest validation there is no way to know whether a model deserves to be deployed.

1. Why accuracy misleads
2. The confusion matrix and the cost of each error
3. Precision, recall, specificity and F1
4. ROC curve, AUC and precision-recall curve
5. Choosing the threshold according to cost
6. Regression metrics: MAE, RMSE and R²
7. Validation: hold-out, cross-validation, stratified and temporal
8. Mandatory baseline and the three sets: training, validation and test
9. Proper comparison of the models from 04-04
10. Common Mistakes and Tips
11. Exercises
12. Conclusion

Accuracy is the proportion of correct answers. It is what score returns in classification and what we have used so far. Its problem appears as soon as the classes are imbalanced: at NovaMarket 16.4 % of orders are returned, so a "model" that always says "not returned" gets 83.6 % right. If the rate were 8 %, it would get 92 %; in card fraud detection, where fraud is 0.1 %, a useless model would have 99.9 % accuracy. The figure is high and the model is good for nothing, because it does not detect a single case of what we are looking for.

The underlying lesson: accuracy treats all errors the same, but for NovaMarket failing to anticipate a return is not the same as reviewing an order that was fine. To reason properly, the types of hit and miss have to be separated.

The confusion matrix cross-tabulates what the model predicted with what actually happened. For the returns problem (positive class = "returned"):

Each cell has a different business cost, and putting numbers on them is the most important conversation Marta and Diego have in the whole project:

• An FN (a return that was not anticipated) costs NovaMarket about €30: return transport, refurbishment, handling, and the discount at which the opened product is resold.
• An FP (false alarm) costs about €5: the time of Diego's team reviewing the order and the small friction with a customer who had done nothing unusual (remember 02-04: if those frictions concentrate on one group, the cost is also ethical and legal).
• A TP allows action (confirming the order with the customer, reinforcing the packaging, holding shipment until verified) and avoids a good part of those €30; a TN costs nothing.

With scikit-learn, on the pipeline from 04-03 and the logistic regression from 04-04:

Output:

Reading: of the 750 test orders, 123 were returned. The model detected 43 (TP), missed 80 (FN) and raised 17 false alarms (FP). In euros: 17 × 5 + 80 × 30 = €2,485. The same function with the "always no" baseline (TP = 0, FN = 123) gives €3,690, and with Diego's rule (amount > 300: 19 TP, 104 FN, 15 FP) gives €3,195. Now the comparison has a language Diego understands: the model saves €1,205 per 750 orders compared with doing nothing, and €710 compared with his rule; with 3,000 orders a day, about €4,800 a day against the baseline. And we have not tuned the threshold yet (section 5).

The standard metrics derive from the confusion matrix. With our numbers (TN 610, FP 17, FN 80, TP 43):

Now the model's real story is visible: when it flags an order it is right 72 % of the time (good precision), but it only detects 35 % of the returns (low recall). With accuracy there was no way of knowing. Precision and recall are in tension: to detect more returns you have to flag more orders, and among them there will be more false alarms; F1 summarises the trade-off in a single number and is the most common metric for comparing classifiers with imbalanced classes when costs are not available.

classification_report prints all of this per class:

Notice that the majority class has an F1 of 0.93 and the minority one 0.47: the "weighted" averages (weighted by size) hide how badly the class that matters is doing; the "macro" one (simple mean) reflects it better.

All the metrics above depend on the threshold with which we turn the probability into a class (0.5 by default). The curves evaluate the model for all thresholds at once:

• The ROC curve plots, for each threshold, the true positive rate (recall) against the false positive rate (1 − specificity). A random model gives the diagonal; a perfect one rises along the left up to the top corner. The AUC (area under the curve) summarises the curve in a number between 0.5 (chance) and 1 (perfect), and has an intuitive interpretation: it is the probability that the model gives a higher score to a randomly chosen actual return than to a randomly chosen normal order. It depends neither on the threshold nor on the class imbalance, which makes it ideal for comparing models.
• The precision-recall curve plots precision against recall for each threshold. It is more informative than ROC when the positive class is rare, because it focuses on what happens with that class; its summary is the average precision (AP). The baseline of a random model is not 0.5, but the positive rate (0.164 here).

Output:

An AUC of 0.84 is a clearly useful model (in 84 % of return/non-return pairs it ranks them correctly). The three points show the trade-off: lowering the threshold from 0.5 to 0.2, recall rises from 35 % to 72 % in exchange for bothering 17.5 % of normal orders. Which is preferable the curve does not say: the costs do.

With the business_cost function we can sweep thresholds and choose the one that minimises the euros:

Output:

With costs of €5/€30, the optimal threshold is 0.2: cost €1,610 against the €2,485 of the default threshold, 35 % less. The price is flagging 200 orders out of 750 (27 %), which Diego finds operationally excessive. This is where the human review band of 02-04 fits in: instead of a single boundary, three zones. With probability ≥ 0.5 action is automatic (60 orders, 43 actual returns); between 0.2 and 0.5 the order goes to human review (140 orders, of which 45 were returned: the reviewer finds almost one return in every three); below 0.2 it is let through (550 orders, 35 returns that slip by). If a review costs €3 and the reviewer gets it right, the total cost is around €1,555, and moreover the system generates hand-labelled data to improve the model. Note the logical order: first the model is measured with AUC and curves, then the threshold is decided with costs and operational capacity; the threshold is never tuned "to make the accuracy look nice".

For demand forecasting there are no classes: the error is a distance between forecast and actual. Three metrics:

On the linear model with seasonality from 04-04 (trained on weeks 1-78, evaluated on 79-104):

Output:

The model is off by about 17 units per week on average (on sales of 550-670), the somewhat larger RMSE (20) indicates that there is some week with a large error (the maximum deviation is 43 units, in week 95), and it explains 83 % of the variability. The naive baseline ("next week we will sell the same as this one") has an MAE of 39 and a negative R²: worse than predicting the mean of the period. The simple line of 04-02 had MAE 47 and R² −0.44. Without a baseline, an MAE of 17 means nothing; with it, we know the model cuts the error to less than half.

So far we have used a single training/test split (hold-out). It is quick, but the result depends on which orders landed on each side: repeating the split with ten different seeds, the AUC of the logistic regression swings between 0.80 and 0.86. With a single test set we could have been lucky (or unlucky) and drawn wrong conclusions when comparing two models that differ by 0.02.

k-fold cross-validation solves the problem: the set is split into k parts (folds), training happens k times using each part once as test and the remaining k−1 as training, and the k measures are averaged. All the data are used both for evaluating and for training, and the deviation between folds indicates the uncertainty. Variants:

• Stratified (StratifiedKFold): each fold keeps the class proportion; essential with imbalanced classes.
• Temporal (TimeSeriesSplit): for series, the folds respect the order: training on the past and evaluating on the next block, never the other way round.
• Leave-one-out, groups (GroupKFold, so that orders from the same customer are not spread between training and test): other variants for specific cases.

Output:

Explanation:

• cross_val_score receives the complete pipeline, and in each fold it refits the imputation, the scaling and the model only on the training part of that fold: cross-validation inherits the leakage protection of 04-03. scoring="roc_auc" selects the metric; there are dozens ("f1", "recall", "neg_mean_absolute_error"...; scikit-learn uses the "higher is better" convention, hence the negative sign on errors).
• The mean AUC of the logistic regression is 0.836 ± 0.016. When we compare two models, a difference smaller than that deviation is not conclusive.
• On the series, each fold trains on more history and evaluates the following quarter, exactly as the model will be used in production. The first fold, trained on a single year, is the worst (25 units): with one year the yearly seasonality cannot be estimated well. If instead of TimeSeriesSplit we used a shuffled KFold, the mean MAE would drop to about 16.7 units, an optimistic estimate that would not hold in production.

Two rules of discipline close the lesson:

Always a baseline. Before celebrating any metric, compute the same metric for the dumbest reasonable model: DummyClassifier(strategy="most_frequent") in classification (always the majority class), the mean or the last value in regression, and the manual rule currently in force (Diego's). A model is justified by its distance from the baseline, in the metric and in euros.

Three sets, not two. If we use the test set to choose between models, thresholds and hyperparameters, we no longer have an honest measure: every decision we take while looking at it "leaks" information from the test set into the model, and the final result will be optimistic. That is why the professional workflow separates three sets:

• Training: for fitting parameters (fit).
• Validation: for taking decisions (which algorithm, which threshold, which hyperparameters). In practice, cross-validation on the training set plays this role without the need for a separate part.
• Test: touched once only, at the end, to estimate the performance of the model already chosen. In this lesson we have used the test set to explore thresholds for teaching purposes; in a real project that exploration would be done with cross-validation. The next lesson (04-06) develops hyperparameter tuning with this scheme.

We repeat the comparison from 04-04, this time with the appropriate metrics and the business cost:

Output:

The table tells a much richer story than accuracy alone:

• k nearest neighbours looked "only" 3 points worse than logistic regression in accuracy; in reality it detects 6.5 % of the returns (recall 0.065) and its AUC of 0.74 leaves it far behind: it is almost the baseline with make-up on.
• The depth-4 tree has lower accuracy than the forest, but better recall and F1 and lower cost (€2,495 versus €2,830) with the default threshold. Going by accuracy we would have chosen the forest.
• Logistic regression wins on everything (AUC 0.844, cost €2,485) and, after tuning the threshold (section 5), drops to €1,610: 56 % less than the baseline. In cross-validation, the advantage of the logistic regression over the forest (0.836 versus 0.822 mean AUC) is of the order of the deviation between folds, so both are reasonable candidates and in 04-06 we will see whether the forest improves when its hyperparameters are tuned.
• The baseline has AUC 0.5 and cost €3,690: it is the reference against which everything else is justified.

Marta can now answer Diego in his own language: "the model, with the threshold set properly, saves about €2,000 per 750 orders compared with doing nothing, and with the review band no more than 140 orders per 750 reach you for review".

• Reporting only accuracy. With imbalanced classes it is almost always misleading. Show the confusion matrix, precision, recall, F1 and AUC, and if you can, euros.
• Confusing precision and recall. Precision: of what I flag, how much is right. Recall: of what exists, how much I find. Which matters more depends on the cost of FP versus FN.
• Leaving the threshold at the default 0.5. The threshold is a business decision; sweep it with the cost function and consider a human review band.
• Using the test set to decide. Every look at the test set to choose something contaminates it. Decide with cross-validation; the test set, once.
• Shuffling time series. Use TimeSeriesSplit; a shuffled KFold gives an optimism that is paid for in production.
• Comparing models on a single split. Small differences may be noise; use cross-validation and look at the deviation between folds.
• Forgetting the baseline. An MAE of 17 or an AUC of 0.84 mean nothing without knowing what the dumb model and the current rule do.
• Ignoring groups. If the same customer has orders in training and in test, the model can "recognise" them; GroupKFold by customer_id avoids it.

Exercise 1. Diego revises the costs: after talking to logistics, an unanticipated return costs €20 (not 30) and a false alarm €8 (the customer has to be called). Recompute the threshold table with business_cost(m, cost_fp=8, cost_fn=20). Does the optimal threshold change? In which direction and why?

Exercise 2. Compute with 5-fold stratified cross-validation the F1 and recall of the logistic regression and of the depth-4 tree (scoring="f1" and scoring="recall"). Which of the two has the better mean recall? Does the difference exceed the deviation between folds? Comment on whether your decision would change with respect to the table in section 9.

Exercise 3. In demand forecasting, replace the "last week" baseline with the mean of the last 4 training weeks and with "the same week of the previous year" (units of week −52). Compute the MAE of each on weeks 79-104. Which is better? Why does the previous-year one come out so badly on this series, and what would have to be corrected for it to be a reasonable baseline?

Solution 1. With FP at €8 and FN at €20, the cost of letting a return slip drops and that of a false alarm rises, so it pays to flag less. The costs become: 0.1 → €2,104; 0.2 → €1,596; 0.3 → €1,600; 0.4 → €1,644; 0.5 → €1,736. The numerical minimum is still at 0.2, but now 0.3 is only €4 away and 0.4 €48 away (with the original costs, 0.3 cost €370 more than 0.2): the curve has flattened and shifted towards higher thresholds. In general, the more expensive the FP relative to the FN, the higher the threshold should be set (more precision, less recall), and vice versa. When the cost curve is flat around the optimum, the operational criterion decides: 0.3 flags 127 orders against the 200 of 0.2 for practically the same cost, so Diego would choose 0.3.

Solution 2. With StratifiedKFold(5, shuffle=True, random_state=42), the logistic regression obtains a mean F1 of 0.50 (± 0.04) and a mean recall of 0.39 (± 0.04); the depth-4 tree, F1 0.44 (± 0.07) and recall 0.33 (± 0.07) (the values may vary by a few hundredths depending on the version). The logistic regression is better on average in both metrics, but the recall difference (0.05) is of the order of the deviation between folds, and the tree is moreover much more variable from one fold to the next (the instability of trees we discussed in 04-04). The decision does not change with respect to section 9: the logistic regression is still preferable on AUC, F1 and stability, but cross-validation forces us to say it precisely: "it detects somewhat more returns and more stably; with a single split that advantage might not show".

Solution 3. The mean of the last 4 training weeks (about 649 units) gives an MAE of about 35 units, slightly better than "last week" (39) because it averages out the weekly noise. "The same week of the previous year" gives an MAE of about 118 units: it comes out very badly because the series has a trend of 2.5 units per week, that is, about 130 units more than a year earlier, and the baseline ignores that growth. To make it reasonable you would have to add the trend (for instance, the mean growth observed between the two years) or express it as "previous year × growth factor"; that turns it into a seasonal baseline that is very hard to beat, and it is the one Marta should use to justify the model to Diego. The lesson: the baseline must also be sensible; an absurd baseline makes any model look good.

In this lesson we have learned to evaluate honestly. Accuracy misleads with imbalanced classes (83.6 % without detecting a single return); the confusion matrix separates TP, FP, FN and TN, and the function business_cost translates them into euros (€30 per unanticipated return, €5 per false alarm), Diego's language. From there come precision, recall, specificity and F1, and the ROC curve (with the AUC, 0.84 for the logistic regression) and precision-recall curve, which evaluate all thresholds at once. Then we chose the threshold by cost (0.2 instead of 0.5, 35 % less cost) and combined it with the human review band of 02-04. For demand we used MAE, RMSE and R² against a baseline. And we replaced the single split with stratified and temporal cross-validation (TimeSeriesSplit), we demanded a baseline and we separated training, validation and test, with the test set reserved for a single final look. Comparing the models from 04-04 with these tools has changed the ranking: the logistic regression is still first, the tree overtakes the forest on cost and k nearest neighbours is unmasked.

One question has been open since 04-04: the random forest and the unlimited tree got far more right on training than on test, and k nearest neighbours with k = 1 got 100 % on training and 81 % on test. In the last lesson of the module, Overfitting, Regularisation and Hyperparameter Tuning, we will give that phenomenon a name, see how to detect it with learning and validation curves, which techniques control it (regularisation, pruning, ensembles) and how to choose the hyperparameters with GridSearchCV without touching the test set, closing the complete workflow that Marta now commands.