The Project | About Us | Contribute | Donations | License

HOME

In the previous lesson we learned which data to evaluate a model on; now it's time to decide which number to use. So far we have used accuracy (classification) and MSE (regression) instrumentally, without questioning them. That ends here: we will see that accuracy can be a blatant trap with imbalanced classes — MercaFresh's churn hovers around 20% — that the confusion matrix distinguishes types of error with very different business costs (which is worse: calling a loyal customer, or losing one who was about to leave?), and that regression comes with several coexisting metrics (MAE, MSE, RMSE, R²) with different interpretations and units. We will close with dummy models, scikit-learn's embodiment of the baseline concept we introduced in lesson 01-04: no metric means anything in a vacuum; it only means something compared against a reference.

The confusion matrix: the four fates of a prediction
Accuracy and its trap with imbalance
Precision, recall and F1: formulas and business interpretation
classification_report and choosing a metric for the problem
Regression metrics: MAE, MSE, RMSE and R²
The baseline: DummyClassifier and DummyRegressor

The confusion matrix: the four fates of a prediction

In a binary problem like churn (1 = the customer cancels, 0 = they stay), every prediction can end up in one of four fates, depending on what the model predicted and what actually happened:

	Predicted: churn (1)	Predicted: stays (0)
Actual: churn (1)	TP (true positive)	FN (false negative)
Actual: stays (0)	FP (false positive)	TN (true negative)

TP (true positive): we predicted churn and the customer did indeed leave. Correct.
TN (true negative): we predicted they would stay and they stayed. Correct.
FP (false positive): we predicted churn, but the customer was loyal. The "false alarm": MercaFresh spends a phone call and maybe a discount on someone who wasn't going anywhere.
FN (false negative): we predicted they would stay and they left. The most expensive error here: we lose the customer without ever trying to retain them.

This is not new to you: in lesson 02-05, with Bayes' theorem, we analyzed a detector with false positives and saw that the cost of each type of error is different, and that with rare events false alarms can dominate. The confusion matrix is exactly that analysis, applied to a classifier.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y  # as in 06-01
)

model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

cm = confusion_matrix(y_test, y_pred)
print(cm)
# [[TN  FP]
#  [FN  TP]]   ← careful: sklearn orders the classes 0, 1 (rows = actual, columns = predicted)

ConfusionMatrixDisplay(cm, display_labels=["Stays", "Churn"]).plot()

Suppose that over 2,000 test customers we get:

	Pred: churn	Pred: stays
Actual: churn (400)	TP = 240	FN = 160
Actual: stays (1,600)	FP = 120	TN = 1,480

Every binary classification metric is computed from these four numbers.

Accuracy and its trap with imbalance

Accuracy is the fraction of total correct predictions:

$$\text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{240 + 1480}{2000} = 0.86$$

86% sounds good... until we remember the imbalance. With 20% churn, an absurd "model" that always predicts "stays" is right 80% of the time:

import numpy as np
y_pred_lazy = np.zeros_like(y_test)          # always predicts "stays"
print((y_pred_lazy == y_test).mean())        # ≈ 0.80

That model has 80% accuracy and is completely useless: it doesn't detect a single at-risk customer (TP = 0). The trap gets worse as the imbalance grows: in fraud detection with 1% positives, "there is never fraud" reaches 99% accuracy. The moral: with imbalanced classes, accuracy on its own is misleading; we need metrics that look at each class separately.

Precision, recall and F1: formulas and business interpretation

Precision: when the model raises the alarm, is it right?

$$\text{precision} = \frac{TP}{TP + FP} = \frac{240}{240 + 120} = 0.67$$

Of all the customers the model flagged as churn, what fraction was really leaving? At MercaFresh: if the retention team calls the flagged customers, precision measures what percentage of calls is not wasted. Low precision means many false alarms: campaign budget burned on loyal customers (plus the risk of annoying them).

Recall: of those who leave, how many do we catch?

$$\text{recall} = \frac{TP}{TP + FN} = \frac{240}{240 + 160} = 0.60$$

Of all the customers who actually cancelled, what fraction did the model detect? A recall of 60% means that 40% of real departures go unnoticed: customers lost without anyone trying to retain them. It is also called sensitivity or true positive rate (a name under which it will reappear in the ROC curve of 06-04).

The tug of war, and the business question

Precision and recall are in tension: to raise recall you must flag more customers (a more sensitive alarm), which produces more false positives and lowers precision, and vice versa. Which one to prioritize? It depends on the costs:

Cost of an FP: one more call from the retention team plus, perhaps, a €10 voucher. Cheap.
Cost of an FN: losing a customer whose average annual value at MercaFresh can run to hundreds of euros. Expensive.

With this cost structure, for churn it usually pays to prioritize recall: better a few extra calls than valuable customers slipping away in silence. In other problems it's the other way around (a spam filter with low precision buries legitimate email). In 06-04 we will see that the decision threshold of logistic regression (04-02) is precisely the lever for shifting this balance.

F1: the one-number summary

When a single figure combining both is needed, the harmonic mean of precision and recall is used:

$$F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} = 2 \cdot \frac{0.67 \cdot 0.60}{0.67 + 0.60} \approx 0.63$$

The harmonic mean punishes imbalance between the two: if either one is close to 0, F1 collapses even if the other is perfect (unlike the arithmetic mean). A model only gets a good F1 if it is reasonable at both.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

print("Accuracy :", accuracy_score(y_test, y_pred))
print("Precision:", precision_score(y_test, y_pred))
print("Recall   :", recall_score(y_test, y_pred))
print("F1       :", f1_score(y_test, y_pred))

`classification_report` and choosing a metric for the problem

scikit-learn summarizes all of the above, per class, in a single call:

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred, target_names=["Stays", "Churn"]))

              precision    recall  f1-score   support

       Stays       0.90      0.93      0.91      1600
       Churn       0.67      0.60      0.63       400

    accuracy                           0.86      2000
   macro avg       0.78      0.76      0.77      2000
weighted avg       0.85      0.86      0.86      2000

How to read it:

Each row gives precision/recall/F1 treating that class as the "positive" one. support is how many actual examples of that class are in the test set.
macro avg: plain average across classes (all weigh the same — useful with imbalance so the minority class isn't hidden).
weighted avg: average weighted by support (with imbalance it resembles accuracy and can gloss over a poor minority class).
Note that the majority class sports an F1 of 0.91 while churn sits at 0.63: the per-class report uncovers what the overall accuracy was hiding.

Which metric to choose? A practical guide:

Situation	Main metric	Why
Balanced classes, errors of similar cost	Accuracy	Simple and sufficient
The expensive error is the false alarm (spam, blocking legitimate purchases)	Precision	Minimize FP
The expensive error is the missed positive (churn, diagnosis, fraud)	Recall	Minimize FN
Imbalance, and you want a single balanced number	F1	Combines P and R
Comparing models independently of the threshold	ROC-AUC	Lesson 06-04

Regression metrics: MAE, MSE, RMSE and R²

Let's switch problems: predicting the monthly spend of each MercaFresh customer, the regression we set up in 04-01. Here there are no hits and misses, but distances between the predicted value ($\hat{y}_i$) and the actual one ($y_i$).

MAE (mean absolute error): $\frac{1}{n}\sum |y_i - \hat{y}_i|$. It reads directly in the target's units: "on average we're off by €18 a month". It treats all errors equally.
MSE (mean squared error): $\frac{1}{n}\sum (y_i - \hat{y}_i)^2$. By squaring, it punishes large errors much harder (a €100 error weighs as much as 100 errors of €10) and its units are €², hard to interpret. It is the metric linear regression minimizes internally (04-01).
RMSE: $\sqrt{MSE}$. It recovers the original units (€) while keeping the sensitivity to large errors. Always RMSE ≥ MAE; a wide gap between the two betrays the existence of huge one-off errors (outliers).
R² (coefficient of determination): the fraction of the target's variance that the model explains. 1.0 = perfect prediction; 0.0 = as good as always predicting the mean; it can be negative on test if the model is worse than the mean. It is dimensionless, which allows comparing problems on different scales.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

reg = LinearRegression().fit(X_train_s, y_train_s)   # monthly spend (EUR)
y_pred_s = reg.predict(X_test_s)

mae  = mean_absolute_error(y_test_s, y_pred_s)
mse  = mean_squared_error(y_test_s, y_pred_s)
rmse = np.sqrt(mse)
r2   = r2_score(y_test_s, y_pred_s)

print(f"MAE : {mae:.2f} EUR      ← 'typical' average error")
print(f"MSE : {mse:.2f} EUR²     ← hard to interpret directly")
print(f"RMSE: {rmse:.2f} EUR      ← like MAE but penalizes large errors")
print(f"R²  : {r2:.3f}           ← fraction of variance explained")

Metric	Units	Sensitive to outliers	When to prefer it
MAE	The target's (€)	Not especially	Direct interpretation; large errors aren't "worse per unit"
MSE	Target² (€²)	Very	Internal optimization; technical comparison
RMSE	The target's (€)	Very	Reporting when large errors are especially costly
R²	Unitless	Moderately	Communicating "how much the model explains"; comparing datasets

At MercaFresh, an MAE of €18 on an average monthly spend of €120 (~15% relative error) is a message the business understands instantly; an MSE of 850 €² is not.

The baseline: DummyClassifier and DummyRegressor

In lesson 01-04 we introduced the concept of a baseline: before celebrating any metric, you must ask "better than what?". scikit-learn embodies it with deliberately dumb models:

from sklearn.dummy import DummyClassifier, DummyRegressor
from sklearn.metrics import f1_score, mean_absolute_error

# Classification: always predict the majority class ("stays")
dummy_clf = DummyClassifier(strategy="most_frequent")
dummy_clf.fit(X_train, y_train)
print("Dummy accuracy:", dummy_clf.score(X_test, y_test))          # ≈ 0.80
print("Dummy F1      :", f1_score(y_test, dummy_clf.predict(X_test)))  # 0.0 !

# Regression: always predict the mean spend
dummy_reg = DummyRegressor(strategy="mean")
dummy_reg.fit(X_train_s, y_train_s)
print("Dummy MAE:", mean_absolute_error(y_test_s, dummy_reg.predict(X_test_s)))

The takeaway is revealing: the dummy reaches 80% accuracy (the imbalance trap, quantified!) but an F1 of 0 on the churn class. Our real model, at 86% accuracy, only beats the dummy by 6 accuracy points... but it goes from 0.0 to 0.63 F1, and that is where the real value lies. Practical rules:

Always compute the dummy baseline before evaluating your first model.
Report your model's metrics alongside the baseline's: "MAE of €18 versus €41 for the mean baseline" is a result; a bare "MAE of €18" is not.
If your model doesn't clearly beat the dummy, something is wrong: features with no signal, a leak in the opposite direction, a bug... or a problem that simply cannot be predicted with that data.

Common Mistakes and Tips

Reporting only accuracy with imbalanced classes. The star mistake. Always accompany it with the confusion matrix and the minority class's F1 (or precision/recall).
Mixing up precision and recall. Mnemonic: precision = of my positive predictions, how many were correct?; recall = of the real positives, how many did I retrieve?
Choosing the metric after seeing the results. Deciding "my metric is whichever comes out best" is self-deception. Fix the metric (with the business) before comparing models.
Interpreting MSE in the target's units. MSE is in squared units; for communication use MAE or RMSE.
Assuming high R² = useful model. An R² of 0.95 with data leakage (06-01) is worth nothing; and an R² of 0.3 can be extremely valuable in a very noisy problem, if it clearly beats the baseline.
Tip: the confusion matrix is your default X-ray: almost any question about a binary classifier ("where does it fail? who does it confuse?") is answered by looking at it before any aggregate metric.

Exercises

Exercise 1

A churn model evaluated on 1,000 customers (200 actual churn) produces: TP = 150, FN = 50, FP = 100, TN = 700. Compute accuracy, precision, recall and F1 by hand. What accuracy would the majority-class dummy get? Is the model worth it?

Exercise 2

MercaFresh evaluates two monthly-spend models. Model A: MAE = €15, RMSE = €45. Model B: MAE = €18, RMSE = €22. Explain what the gap between MAE and RMSE reveals about each model, and reason about which you would choose if large prediction errors (e.g., underestimating a big customer's spend by €200) are especially costly.

Exercise 3

Write the code that trains a LogisticRegression and a DummyClassifier(strategy="most_frequent") on X_train, y_train (churn), and prints the classification_report on test for both. State which two numbers in the report you would compare to decide whether the model adds value.

Solutions

Solution 1.

Accuracy = (150 + 700) / 1000 = 0.85
Precision = 150 / (150 + 100) = 0.60
Recall = 150 / (150 + 50) = 0.75
F1 = 2 · (0.60 · 0.75) / (0.60 + 0.75) = 0.9 / 1.35 ≈ 0.67
Majority-class dummy: gets the 800 "stays" right → accuracy = 0.80.

The model only beats the dummy by 5 accuracy points, but that framing flatters the dummy: the model detects 75% of real departures (the dummy, 0%). Given churn's asymmetric costs (losing a customer ≫ one extra call), it is worth it, and a recall of 0.75 with precision 0.60 is a reasonable trade-off for a retention campaign.

Solution 2. The RMSE − MAE gap measures how much large errors weigh. Model A has the better typical error (MAE 15 < 18) but an RMSE three times its MAE (45 vs. 15): it makes some very large one-off errors. Model B is slightly worse "day to day" but its errors are homogeneous (22 vs. 18: no catastrophes). If large errors are especially costly, pick model B: it sacrifices €3 of average error in exchange for eliminating the extreme failures.

Solution 3.

from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import classification_report

model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)

for name, m in [("Logistic regression", model), ("Dummy", dummy)]:
    print(f"===== {name} =====")
    print(classification_report(y_test, m.predict(X_test),
                                target_names=["Stays", "Churn"],
                                zero_division=0))

I would compare above all the recall and the F1 of the "Churn" class: in the dummy both are 0.0 (it flags no one), so any clearly positive value from the model proves real detection ability — something the accuracy (0.80 vs. ~0.86) barely lets you see.

Conclusion

We now know how to measure: the confusion matrix separates the four fates of a prediction and connects with the error-cost analysis we did with Bayes in 02-05; accuracy misleads under churn's imbalance; precision and recall translate errors into business language (wasted calls versus lost customers) and F1 summarizes them; in regression, MAE/RMSE speak in euros and R² in explained variance; and DummyClassifier/DummyRegressor turns the baseline from 01-04 into code, giving context to any figure. But all these metrics were computed on a single train/test split, and that split depends on the luck of a seed: with a different random_state, different numbers. How reliable, then, is a metric computed only once? That worry has a name and a solution — cross-validation — and it is exactly the topic of the next lesson.

Evaluation metrics

Contents

The confusion matrix: the four fates of a prediction

Accuracy and its trap with imbalance

Precision, recall and F1: formulas and business interpretation

Precision: when the model raises the alarm, is it right?

Recall: of those who leave, how many do we catch?

The tug of war, and the business question

F1: the one-number summary

`classification_report` and choosing a metric for the problem

Regression metrics: MAE, MSE, RMSE and R²

The baseline: DummyClassifier and DummyRegressor

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

Evaluation metrics

Contents

The confusion matrix: the four fates of a prediction

Accuracy and its trap with imbalance

Precision, recall and F1: formulas and business interpretation

Precision: when the model raises the alarm, is it right?

Recall: of those who leave, how many do we catch?

The tug of war, and the business question

F1: the one-number summary

classification_report and choosing a metric for the problem

Regression metrics: MAE, MSE, RMSE and R²

The baseline: DummyClassifier and DummyRegressor

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

`classification_report` and choosing a metric for the problem