The Project | About Us | Contribute | Donations | License

HOME

Throughout the module the same unnamed symptom has kept appearing: k nearest neighbours with k = 1 got 100 % right on training and 81 % on test (04-04); the tree with no depth limit, 100 % and 82.5 %; the random forest, 0.908 on training against 0.859 on test (04-05). In every case the model learned the data it saw "too well" and the data it did not see worse. That phenomenon is called overfitting, and its opposite, the model too simple to learn even what is there, underfitting. This last lesson of the module explains both with the intuition of the bias-variance trade-off, teaches how to diagnose them with learning and validation curves, presents the techniques for fighting them (more data, fewer features, L1/L2 regularisation, tree pruning, ensembles, early stopping) and, finally, tackles methodically the question we have been postponing since 04-01: how to choose the hyperparameters (GridSearchCV, RandomizedSearchCV, nested validation) without contaminating the test set. It matters because overfitting is the most common way to fail in production: a model that is excellent on Marta's laptop and mediocre on tomorrow's orders. We will close with the complete workflow that Marta now commands and with the bridge to module 5.

Underfitting and overfitting: the tree that memorises the orders
The bias-variance trade-off
Diagnosis: learning curves and validation curves
Techniques against overfitting
L1 and L2 regularisation in code
Hyperparameters versus parameters: how to choose them
Grid and random search with GridSearchCV and RandomizedSearchCV
Nested validation and why the test set is touched once only
The complete workflow of the module, at a glance
Common Mistakes and Tips
Exercises
Conclusion

Underfitting and overfitting: the tree that memorises the orders

We train decision trees of increasing depth on the pipeline from 04-03 and measure accuracy and AUC on training and on test:

from novamarket_ml import generate_orders_ml, dirty_orders, prepare_orders, build_preprocessing
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score

X, y = prepare_orders(dirty_orders(generate_orders_ml(3000, 42), 42))
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

print("depth  acc_train  acc_test  auc_train  auc_test  leaves")
for depth in [1, 2, 3, 4, 6, 8, 12, None]:
    p = Pipeline([("prep", build_preprocessing()),
                  ("model", DecisionTreeClassifier(max_depth=depth, random_state=42))]).fit(Xtr, ytr)
    tree = p.named_steps["model"]
    print(f"{str(depth):5s}  {p.score(Xtr, ytr):9.3f}  {p.score(Xte, yte):8.3f}  "
          f"{roc_auc_score(ytr, p.predict_proba(Xtr)[:, 1]):9.3f}  "
          f"{roc_auc_score(yte, p.predict_proba(Xte)[:, 1]):8.3f}  {tree.get_n_leaves():6d}")

Output:

Depth	Train accuracy	Test accuracy	Train AUC	Test AUC	Leaves
1	0.846	0.827	0.611	0.570	2
2	0.861	0.840	0.743	0.746	4
3	0.878	0.849	0.799	0.793	8
4	0.886	0.855	0.830	0.792	15
6	0.900	0.844	0.871	0.769	38
8	0.924	0.840	0.924	0.693	89
12	0.965	0.811	0.984	0.634	200
No limit	1.000	0.788	1.000	0.648	299

Three zones:

Depth 1-2: underfitting. The model is too simple to capture the relationship (a single question is not enough, as we saw in 04-01): it performs badly on training and on test.
Depth 3-4: the sweet spot. Training and test go hand in hand and the test AUC is at its maximum (0.79).
Depth 6 onwards: overfitting. Training performance keeps rising to 100 % (the unlimited tree has 299 leaves for 2,249 orders: many leaves contain a single order and "remember" it), while test performance drops: AUC 0.65, barely better than depth 2. The tree has learned the noise of the generator's random draw (04-01), which is different in the new orders.

The practical definition: there is overfitting when the gap between training and validation performance grows as model complexity increases; there is underfitting when both are low. And a consequence worth engraving: training performance is no use for choosing the model; it always rises with complexity.

The bias-variance trade-off

The classical explanation decomposes a model's error into two sources:

Bias: error from assumptions that are too rigid. A model with high bias (a straight line for a curved relationship, a depth-1 tree) is wrong systematically, whatever it does with the data. This is underfitting.
Variance: error from excessive sensitivity to the specific training data. A model with high variance (an unlimited tree, k = 1) changes completely if trained on another sample, because it follows the noise. This is overfitting.

The usual image is the target:

	Low variance	High variance
Low bias	All shots in the centre: the ideal model	Shots spread around the centre: right on average but each specific model is unpredictable (overfitting)
High bias	Shots clustered but far from the centre: always wrong in the same way (underfitting)	Shots scattered and off-centre: the worst of both

As complexity increases (depth, smaller k, more features, less regularisation) bias goes down and variance goes up; the total error, the sum of both plus the irreducible noise, is U-shaped, and the minimum is the sweet spot we are looking for. The noise is never eliminated: in our data the label is drawn with a probability, so not even the perfect model would get 100 % right; chasing it is the direct route to overfitting.

Diagnosis: learning curves and validation curves

Two plots answer the two diagnostic questions. Both are computed with cross-validation (04-05), never with the test set.

3.1 Learning curve: would more data help?

learning_curve trains the model with increasing fractions of the training set and measures performance on training and on validation for each size:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve, StratifiedKFold
from sklearn.linear_model import LogisticRegression

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
pipe_log = Pipeline([("prep", build_preprocessing()), ("model", LogisticRegression(max_iter=1000))])
sizes, auc_tr, auc_va = learning_curve(pipe_log, X, y, cv=cv, scoring="roc_auc",
                                       train_sizes=np.linspace(0.1, 1.0, 6), random_state=42)
print(pd.DataFrame({"n_train": sizes, "auc_train": auc_tr.mean(1).round(3), "auc_val": auc_va.mean(1).round(3)}))

plt.plot(sizes, auc_tr.mean(1), "o-", label="training")
plt.plot(sizes, auc_va.mean(1), "o-", label="validation")
plt.xlabel("training orders"); plt.ylabel("AUC"); plt.legend(); plt.show()

Output (logistic regression):

   n_train  auc_train  auc_val
0      239      0.921    0.611
1      671      0.842    0.821
2     1103      0.839    0.831
3     1535      0.839    0.835
4     1967      0.831    0.836
5     2399      0.841    0.836

How to read the plot: with 239 orders, the training curve is very high (0.92) and the validation curve very low (0.61): with little data, even logistic regression overfits. From about 1,100 orders both curves converge around 0.83-0.84 and flatten out: more data would no longer help this model, and to improve you would have to change family or add features. If you repeat the calculation with the unlimited tree, the training curve stays at 1.000 for any size and the validation curve at 0.65: a large, persistent gap, the signature of overfitting, in which more data would help a little (the gap would close slowly) but the effective thing is to limit the model.

Reading summary: large gap between curves = variance (overfitting); curves together but low = bias (underfitting); curves together and high = good.

3.2 Validation curve: which value of the hyperparameter?

validation_curve fixes the data size and varies a hyperparameter:

from sklearn.model_selection import validation_curve

depths = [1, 2, 3, 4, 5, 6, 8, 10, 15, 20]
pipe_tree = Pipeline([("prep", build_preprocessing()), ("model", DecisionTreeClassifier(random_state=42))])
auc_tr, auc_va = validation_curve(pipe_tree, X, y, param_name="model__max_depth",
                                  param_range=depths, cv=cv, scoring="roc_auc")
print(pd.DataFrame({"depth": depths, "auc_train": auc_tr.mean(1).round(3), "auc_val": auc_va.mean(1).round(3)}))
# plt.plot(depths, auc_tr.mean(1), "o-", label="training"); plt.plot(depths, auc_va.mean(1), "o-", label="validation")

Output:

   depth  auc_train  auc_val
0      1      0.672    0.657
1      2      0.756    0.745
2      3      0.804    0.789
3      4      0.833    0.793
4      5      0.854    0.789
5      6      0.881    0.775
6      8      0.938    0.722
7     10      0.976    0.680
8     15      0.999    0.667
9     20      1.000    0.656

In the plot, the training curve rises monotonically towards 1 and the validation curve draws an inverted U with its maximum at depth 4 (0.793); underfitting to the left, overfitting to the right. With cross-validation, the choice (depth 4) is reliable, not the product of a lucky split. Note the parameter name: "model__max_depth" with a double underscore, the syntax for reaching a hyperparameter inside a Pipeline (step model, parameter max_depth); we will use it in the searches.

Techniques against overfitting

Technique	Idea	Where	Comment
More data	With more examples, the noise weighs less	Any model	Only if the learning curve shows a gap; usually the most expensive
Fewer features	Remove noisy or redundant columns (04-03)	Any model, especially k-NN and linear ones	Selection by importance, correlation or L1
L2 regularisation (Ridge)	Penalise the sum of squared coefficients: all shrink towards 0	Linear regression (`Ridge`, `alpha`), logistic (`C` = 1/strength), SVM (`C`), networks	Default in `LogisticRegression`
L1 regularisation (Lasso)	Penalise the sum of absolute values: many coefficients become exactly 0	`Lasso` (`alpha`), logistic with `penalty="l1"`	Performs automatic feature selection
Limiting trees	`max_depth`, `min_samples_leaf`, `min_samples_split`, cost-complexity pruning (`ccp_alpha`)	Trees and their ensembles	What we did in section 1
Ensembles	Average many high-variance models (bagging, forests)	Trees above all	Reduce variance without raising bias
Early stopping	Stop iterative training when the validation error stops falling	Boosting, neural networks	Central in module 5
Cross-validation	Does not prevent overfitting, but detects it and avoids overfitting the choice of hyperparameters to a single split	Always	Section 7
Data augmentation, dropout	Techniques specific to deep learning	Module 5

Regularisation deserves a separate explanation. When training a logistic regression the log loss is minimised (04-01, 04-04). Regularising means adding to that loss a penalty for large coefficients: loss + λ · Σ coef² (L2) or loss + λ · Σ |coef| (L1). It is exactly the penalty technique we used in 03-04 for order assignment: turning a preference ("moderate coefficients") into a term of the objective. A huge coefficient means the model leans very heavily on one column, often to explain a few noisy examples; penalising it forces more "spread-out", smoother explanations, which generalise better. In scikit-learn the parameter is called alpha in Ridge/Lasso (higher, more regularisation) and C in LogisticRegression/SVC (it is the inverse: lower, more regularisation).

L1 and L2 regularisation in code

We vary C in the logistic regression of the pipeline from 04-03 and watch the coefficients:

from sklearn.model_selection import cross_val_score

print("C       auc_train  auc_val  sum|coef|  coef≈0")
for C in [0.001, 0.01, 0.1, 1, 10, 100]:
    p = Pipeline([("prep", build_preprocessing()), ("model", LogisticRegression(C=C, max_iter=1000))])
    auc_val = cross_val_score(p, Xtr, ytr, cv=cv, scoring="roc_auc").mean()
    p.fit(Xtr, ytr)
    coef = p.named_steps["model"].coef_[0]
    print(f"{C:<7}  {roc_auc_score(ytr, p.predict_proba(Xtr)[:, 1]):9.3f}  {auc_val:7.3f}  "
          f"{np.abs(coef).sum():9.2f}  {(np.abs(coef) < 0.05).sum():6d}")

Output:

C       auc_train  auc_val  sum|coef|  coef≈0
0.001        0.792    0.781       0.71      17
0.01         0.826    0.812       2.63      11
0.1          0.838    0.828       5.03       7
1            0.839    0.829       5.87       6
10           0.839    0.829       9.61       3
100          0.839    0.829       9.65       3

With C = 0.001 (very strong regularisation) the coefficients almost vanish (sum 0.71; 17 of 21 practically null) and the model underfits (AUC 0.78). Between 0.1 and 100 the validation performance is the same (0.829): on these data the logistic regression does not overfit appreciably, so regularisation hardly matters, except that the coefficients grow (5 → 9.6) without improving anything. L2 here is a cheap safety net rather than a necessity. With L1 (LogisticRegression(penalty="l1", solver="liblinear", C=...)) the effect is different: with C = 0.05 only 7 non-zero coefficients remain (amount, num_items, delivery_days, amount_per_item, new_customer, category_electronics, category_home) with a validation AUC of 0.821, almost that of the full model: L1 has selected features automatically and has discarded, among others, the three postcode_zone columns, in line with what we saw in 04-03. For regression, Ridge(alpha=...) and Lasso(alpha=...) do the same with demand; exercise 2 explores it.

Hyperparameters versus parameters: how to choose them

Recall the distinction from 04-01: parameters (coefficients, node thresholds, centroids) are learned by fit from the data; hyperparameters (max_depth, min_samples_leaf, n_estimators, C, alpha, n_neighbors, n_clusters) we fix before training and they control the complexity or behaviour of the algorithm. There is no formula for choosing them: you have to try values and measure with validation, which is what we have just done by hand with the validation curve. The principles:

Choose with validation (cross-validation on the training set, or a separate validation set), never with the test set.
Try a reasonable grid of values, on a logarithmic scale when the parameter is continuous (C at 0.001, 0.01, 0.1, 1, 10).
Prefer, at equal performance, the simplest model (less depth, more regularisation).
Check that the optimum is not on the edge of the grid; if it is, extend it.

Grid and random search with `GridSearchCV` and `RandomizedSearchCV`

GridSearchCV automates the principles above: it receives a pipeline, a dictionary of hyperparameters with the values to try, a cross-validation strategy and a metric; it trains every combination, keeps the best and retrains the pipeline with it on the whole training set. We apply it to the random forest, which in 04-05 was behind the logistic regression and showed signs of overfitting:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

grid = {
    "model__n_estimators": [100, 300],
    "model__max_depth": [3, 5, 8, None],
    "model__min_samples_leaf": [1, 5, 20],
}                                                    # 2 x 4 x 3 = 24 combinations
pipe_rf = Pipeline([("prep", build_preprocessing()), ("model", RandomForestClassifier(random_state=42))])
search = GridSearchCV(pipe_rf, grid, cv=cv, scoring="roc_auc", n_jobs=-1)
search.fit(Xtr, ytr)                                 # 24 combinations x 5 folds = 120 trainings

print("Best hyperparameters:", search.best_params_)
print("Cross-validation AUC:", round(search.best_score_, 3))

results = pd.DataFrame(search.cv_results_)
columns = ["param_model__max_depth", "param_model__min_samples_leaf", "param_model__n_estimators",
           "mean_test_score", "std_test_score", "rank_test_score"]
print(results[columns].sort_values("rank_test_score").head(4).round(3).to_string(index=False))
print(results[columns].sort_values("rank_test_score").tail(2).round(3).to_string(index=False))

# ONCE only, at the end: the test set
print("Test AUC of the best forest:", round(roc_auc_score(yte, search.predict_proba(Xte)[:, 1]), 3))

Output:

Best hyperparameters: {'model__max_depth': 5, 'model__min_samples_leaf': 1, 'model__n_estimators': 300}
Cross-validation AUC: 0.819
 param_model__max_depth  param_model__min_samples_leaf  param_model__n_estimators  mean_test_score  std_test_score  rank_test_score
                      5                              1                        300            0.819           0.017                1
                      5                              5                        300            0.819           0.017                2
                      5                              1                        100            0.819           0.016                3
                      5                              5                        100            0.817           0.017                4
                   None                              1                        300            0.799           0.027               23
                   None                              1                        100            0.797           0.027               24
Test AUC of the best forest: 0.838

Reading:

best_params_ says the best combination is depth 5, minimum leaf 1 and 300 trees; best_score_ is its mean AUC in cross-validation (0.819). cv_results_ is a table with all the combinations: the mean and deviation of the metric over the 5 folds and the ranking. It always deserves a look: the top four rows are within 0.002 of each other (less than the deviation, 0.017), so "depth 5" is what matters and the rest is indifferent; and the worst are the forests with no depth limit, which overfit (0.797 with a larger deviation, 0.027).
The forest with default hyperparameters (no depth limit) had a validation AUC of 0.797 and a test AUC of 0.811; the tuned one rises to 0.838 on test. Tuning the hyperparameters has turned the forest into a competitor of the logistic regression (0.844), and with the cost function of 04-05 its cost at threshold 0.2 drops from €2,830 (default, threshold 0.5) to €1,860.
search behaves like an already-trained model (predict, predict_proba): internally it has retrained the best pipeline on the whole training set. The test set has been used once only, on the last line.

When the grid is large (five hyperparameters with six values each make 7,776 combinations), exhaustive search is unfeasible: the combinatorial explosion of 03-01. RandomizedSearchCV tries n_iter combinations at random from distributions we give it, and in practice finds solutions just as good at a fraction of the cost, because usually only one or two hyperparameters matter:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

distributions = {
    "model__n_estimators": randint(100, 500),            # random integer between 100 and 499
    "model__max_depth": [3, 4, 5, 6, 8, 10, None],
    "model__min_samples_leaf": randint(1, 40),
    "model__max_features": ["sqrt", 0.3, 0.5],
}
random_search = RandomizedSearchCV(pipe_rf, distributions, n_iter=20, cv=cv, scoring="roc_auc",
                                   random_state=42, n_jobs=-1).fit(Xtr, ytr)
print(random_search.best_params_, round(random_search.best_score_, 3))
# {'model__max_depth': 5, 'model__max_features': 0.5, 'model__min_samples_leaf': 11,
#  'model__n_estimators': 187}  0.822   -> test AUC 0.840

With 20 combinations (100 trainings) it finds depth 5 again and a slightly better validation AUC (0.822). It is the same idea as the metaheuristics of 03-04: when you cannot walk the whole space, sample intelligently. There are more sophisticated methods (Bayesian optimisation, HalvingGridSearchCV, libraries such as Optuna) that are mentioned in 07-03.

Nested validation and why the test set is touched once only

There is a subtlety worth understanding even if it is not always applied. When GridSearchCV tries 24 combinations and keeps the best, the best_score_ (0.819) is slightly optimistic: among 24 candidates, the winner wins partly on merit and partly by luck on those specific folds. It is a small overfitting of the selection to the validation data. To estimate performance without that bias, nested cross-validation is used: an outer cross-validation loop and, inside each outer fold, a complete GridSearchCV (inner loop) that chooses the hyperparameters only with the training data of that fold. In scikit-learn it is one line: cross_val_score(GridSearchCV(...), X, y, cv=outer_cv). It is expensive (5 × 120 trainings in our example) and in applied projects it is often replaced by the simpler scheme of the previous section, which is correct as long as the final rule is respected:

The test set is touched once only. Everything we have done in this lesson (curves, regularisation, grids) has been decided with cross-validation on the training set; the test set has only appeared at the end to give the figure Marta will take to Diego (AUC 0.838 for the tuned forest, 0.844 for the logistic regression). Had we chosen the hyperparameters by looking at the test set, that figure would no longer be an honest estimate of what will happen tomorrow, but the best of many looks, and deployment would bring a disappointment. It is the same discipline that in 04-05 separated training, validation and test.

The complete workflow of the module, at a glance

Marta has walked, with NovaMarket's orders, through the whole workflow we drew in 04-01:

flowchart LR
    A[Define T, E, P<br/>04-01, 04-02] --> B[Data and quality<br/>02-03]
    B --> C[Preprocessing in a Pipeline<br/>04-03]
    C --> D[Choose algorithms<br/>04-04]
    D --> E[Cross-validation,<br/>metrics and cost<br/>04-05]
    E --> F[Diagnosis and<br/>hyperparameter tuning<br/>04-06]
    F -->|not good enough| C
    F -->|ready| G[Test once,<br/>threshold, deployment<br/>and monitoring<br/>08-01]

Step	What Marta did	Result
Define the problem	Binary classification "returned"; baseline: Diego's rule and "always no"	Clear T, E, P; costs €5/€30
Prepare the data	`dirty_orders` → cleaning, engineering, `ColumnTransformer` + `Pipeline`	21 features, no leaks
Choose algorithms	Logistic, k-NN, tree, forest, SVM	Logistic and forest as candidates
Evaluate	Confusion matrix, AUC, F1, cost; stratified cross-validation	Logistic AUC 0.84; threshold 0.2 (€1,610 versus €3,690 for the baseline)
Tune	Validation curves; `GridSearchCV` on the forest	Tuned forest AUC 0.84; both valid
Decide	Test once only; human review band	Model ready for 08-01

The same workflow, with generate_weekly_demand, TimeSeriesSplit and MAE instead of AUC, serves for demand forecasting; and with generate_customers_ml, the elbow and the silhouette, for segmentation. In 09-02 you will practise it end to end with more exercises.

Common Mistakes and Tips

Choosing the model by its training performance. It always rises with complexity; only validation tells the truth.
Misreading the learning curve. Curves together and flat: more data will not help, change model or features. Large gap: limit the model or get more data.
Confusing the direction of C and alpha. High alpha = more regularisation; high C = less regularisation.
Linear-scale grids for multiplicative parameters. C in [1, 2, 3, 4] explores almost nothing; use [0.001, 0.01, 0.1, 1, 10].
Accepting an optimum on the edge of the grid. If the best is the highest value tried, extend the grid.
Looking at the test set to decide. Every look contaminates it. Grids and curves with cross-validation; test at the end, once.
Tuning dozens of hyperparameters with little data. The tuning itself overfits the validation; use small grids and, if the result matters a lot, nested validation.
Forgetting the simple model. After all the tuning, the forest (0.838) ties with the untuned logistic regression (0.844): at equal performance, choose the simpler, more interpretable one.

Exercises

Exercise 1. Repeat the validation curve of section 3.2 with KNeighborsClassifier, varying n_neighbors over [1, 3, 5, 9, 15, 25, 51, 101] (parameter "model__n_neighbors"). Describe the shape of the two curves: where is the overfitting, where the underfitting and what is the optimal k? Compare with section 3 of 04-04, where k = 15 looked best with a single split.

Exercise 2. On NovaClean demand with the features from 04-04 (week, sin, cos, black_friday) plus five columns of pure noise (rng.normal(size=(len(d), 5)) with np.random.default_rng(0)), train LinearRegression, Ridge(alpha=10) and Lasso(alpha=5) on weeks 1-78 and evaluate the MAE on 79-104. Which one handles the noise columns best? Look at the Lasso coefficients: how many are exactly zero? Scale the features first (StandardScaler) so that alpha affects them all equally.

Exercise 3. Extend the forest's grid search with "model__max_features": ["sqrt", 0.5] (48 combinations) and change the metric to scoring="f1". Do the best hyperparameters change? Why can the optimum differ depending on the metric, and which would you choose for NovaMarket knowing that the final decision is taken with a threshold and cost (04-05)?

Solutions

Solution 1. The k-NN training curve starts at 1.000 for k = 1 (each order is its own neighbour) and drops as k grows (0.93 with k = 3, 0.85 with k = 15, 0.81 with k = 101); the validation curve starts very low at k = 1 (AUC 0.57: pure overfitting), rises with k (0.70 at k = 5, 0.75 at k = 15, 0.77 at k = 25) and flattens at 0.79 for k = 51 and k = 101. In this range the drop from underfitting has not appeared yet; a k of several hundred (averaging almost the whole set) would be needed to see it. With the 21 columns of the pipeline, the optimum is a k quite a bit larger than the 15 that looked good in 04-04 with 4 columns: the more noisy columns, the more neighbours are needed to average out the noise; and even so k-NN stays below the logistic regression and the forest, which confirms the reading of 04-05.

Solution 2. With five noise columns and only 26 test weeks, the MAEs are close and the order may surprise: linear gives 16.1, Ridge(alpha=10) 18.2 and Lasso(alpha=5) 17.3 (against 16.8 for the model without noise). What is revealing are the coefficients: linear regression assigns the noise coefficients of up to ±4 units (which on this specific test set, by chance, do not get in the way), Ridge shrinks them but also shrinks the signal (cos goes from −45 to −39) and Lasso sets exactly four of the five to zero (the fifth is left at −0.1) while keeping trend, seasonality and Black Friday. The difference becomes obvious as the noise increases: with 20 noise columns linear rises to an MAE of 22 and Ridge to 23, while Lasso stays at 17.6 zeroing 18 of 20; with 50 columns, linear reaches 26, Ridge 23 and Lasso is still at 17.4 with 48 of 50 at zero. This is L1's automatic feature selection. If you raise alpha a lot, Lasso starts zeroing useful coefficients too and the MAE shoots up: underfitting from excess regularisation, the other arm of the U. And an extra lesson: with a 26-week test set, differences of one or two MAE units are evaluation noise; to claim anything you have to use TimeSeriesSplit (04-05).

Solution 3. With scoring="f1" the optimum changes: in our tests a forest with no depth limit wins, with min_samples_leaf=5, max_features=0.5 and 100 trees (mean F1 0.51), precisely the kind of configuration that with AUC ended up at the tail. The reason is that F1 is computed with the default threshold of 0.5: shallow forests with large leaves produce moderate probabilities that rarely exceed 0.5, so they flag few orders and their recall (and F1) is low even though they rank the orders by risk better (higher AUC). AUC measures the quality of the ranking, independent of the threshold; F1 measures one specific point of that ranking. Since at NovaMarket the threshold will be chosen afterwards with the cost function and the review band, the coherent thing is to tune the hyperparameters with AUC (or with average precision, if you prefer to focus on the positive class) and leave the threshold for the end.

Conclusion

In this lesson we have given a name to the symptom we had been carrying since 04-04: overfitting (the unlimited tree that memorises the 2,249 orders with 299 leaves: AUC 1.0 on training and 0.65 on test) and its opposite, underfitting, explained with the bias-variance trade-off. We have learned to diagnose them with learning curves (does more data help?) and validation curves (which value of the hyperparameter?), to fight them with more data, fewer features, L2 and L1 regularisation (C, alpha; L1 selected 7 of 21 columns), limits on trees, ensembles and early stopping, and to choose the hyperparameters with GridSearchCV and RandomizedSearchCV, reading best_params_ and cv_results_, with nested validation as a refinement and the test set reserved for a single final look. The random forest, tuned, has caught up with the logistic regression, and Marta now has the complete workflow, from problem definition to validated model, on the module's three use cases: returns, demand and segmentation.

With this we close module 4. You have gone from Mitchell's definition of learning and the first fit (04-01) to the supervised, unsupervised and reinforcement paradigms (04-02), data preparation in a leak-free Pipeline (04-03), the classical algorithms and their intuitions (04-04), honest evaluation with confusion matrix, AUC, cost in euros and cross-validation (04-05) and, in this lesson, the control of overfitting and hyperparameter tuning. Throughout the module the idea with which we closed module 3 has reappeared: learning is optimising. fit minimises a loss on the training data; regularising means adding a penalty to the objective, as in 03-04; and searching for hyperparameters is a search, exhaustive or random, in a space that explodes combinatorially. In module 5, Neural Networks and Deep Learning, we will take those ideas to the extreme: models with thousands or millions of parameters, in which overfitting is the permanent enemy (hence the early stopping, dropout and data augmentation we have left noted) and in which optimisation is done, step by step, following the gradient of the loss: the gradient descent and backpropagation we anticipated in 03-04 and in the logistic regression of 04-04. Marta and Diego, with the returns predictor now validated, will ask themselves in module 5 whether a neural network can read the reviews in reviews.csv and the photos of the incidents, things the algorithms of this module cannot do.

Overfitting, Regularisation and Hyperparameter Tuning

Contents

Underfitting and overfitting: the tree that memorises the orders

The bias-variance trade-off

Diagnosis: learning curves and validation curves

3.1 Learning curve: would more data help?

3.2 Validation curve: which value of the hyperparameter?

Techniques against overfitting

L1 and L2 regularisation in code

Hyperparameters versus parameters: how to choose them

Grid and random search with `GridSearchCV` and `RandomizedSearchCV`

Nested validation and why the test set is touched once only

The complete workflow of the module, at a glance

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Fundamentals of Artificial Intelligence (AI)

Module 1: Introduction to Artificial Intelligence

Module 2: Basic Principles of AI

Module 3: Algorithms in AI

Module 4: Machine Learning

Module 5: Neural Networks and Deep Learning

Module 6: Logic and Expert Systems

Module 7: Tools and Programming Languages in AI

Module 8: Projects and Case Studies

Module 9: Exercises and Practice

Module 10: Additional Resources

Overfitting, Regularisation and Hyperparameter Tuning

Contents

Underfitting and overfitting: the tree that memorises the orders

The bias-variance trade-off

Diagnosis: learning curves and validation curves

3.1 Learning curve: would more data help?

3.2 Validation curve: which value of the hyperparameter?

Techniques against overfitting

L1 and L2 regularisation in code

Hyperparameters versus parameters: how to choose them

Grid and random search with GridSearchCV and RandomizedSearchCV

Nested validation and why the test set is touched once only

The complete workflow of the module, at a glance

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Fundamentals of Artificial Intelligence (AI)

Module 1: Introduction to Artificial Intelligence

Module 2: Basic Principles of AI

Module 3: Algorithms in AI

Module 4: Machine Learning

Module 5: Neural Networks and Deep Learning

Module 6: Logic and Expert Systems

Module 7: Tools and Programming Languages in AI

Module 8: Projects and Case Studies

Module 9: Exercises and Practice

Module 10: Additional Resources

Grid and random search with `GridSearchCV` and `RandomizedSearchCV`