This whole module — and a good part of the course — has been running up a tab: alpha in regularization, C in logistic regression and SVMs, K in K-NN, max_depth in trees, learning rate and n_estimators in boosting, layers and dropout in networks… So far we've chosen them by eye or with manual loops like those of 06-05. This lesson closes the module by automating that search: you'll learn to distinguish parameters from hyperparameters, to use GridSearchCV and RandomizedSearchCV on the full MercaFresh churn pipeline, to understand Bayesian search, to avoid the trap of scoring with the same validation that picked the winner, and to manage your compute budget wisely.

1. Parameters vs. hyperparameters
2. Manual search and its limits
3. GridSearchCV in detail
4. Grid search on the full churn pipeline
5. RandomizedSearchCV: when sampling is better
6. Bayesian search: searching with memory
7. Nested validation: don't score with the judge who chose
8. Good search practices
9. Closing the module: the full arsenal

The distinction underpins everything:

• Parameters: the algorithm learns them from the data during fit. You don't choose them.
• Hyperparameters: you set them before training; they configure how the algorithm learns.

The mnemonic rule: if it appears in the constructor (Ridge(alpha=...)), it's a hyperparameter; if it appears as an attribute with a trailing underscore after training (ridge.coef_), it's a learned parameter. And the methodological principle that governs the lesson: parameters are fitted with the train set; hyperparameters are chosen with validation — never with the test set, which stays under lock and key for the final verdict (06-01).

We've already done hyperparameter search three times without naming it: the validation curves over max_depth in 06-05, the alpha loop in 07-01 and the η/n_estimators combinations in 07-03. The manual method — loop, CV, jot down results — works, but scales terribly:

• Combinatorial explosion: with 4 hyperparameters and 5 values each you get 625 combinations; by hand, impossible.
• Interactions: the best learning_rate depends on n_estimators (we saw it in 07-03); exploring each knob separately misses the cross combinations.
• Discipline slips: in hand-rolled loops it's easy to accidentally tune with test data, or to scale outside the CV (the leak of 06-01).
• Irreproducibility: "I tried stuff and kept this one" is not an auditable procedure.

sklearn packages the solution into two tools that do exactly what we were doing by hand, but exhaustively, in parallel and leak-free.

GridSearchCV takes an estimator, a grid of values per hyperparameter, and a CV strategy; it trains and evaluates every combination with cross-validation and keeps the best one.

Its pieces:

• param_grid: a dictionary {hyperparameter: list of values}. The Cartesian product defines the combinations.
• cv: the cross-validation strategy — for churn, the StratifiedKFold of 06-03.
• scoring: the metric that decides (06-02): "f1", "roc_auc", "neg_root_mean_squared_error"… Choosing this metric well is choosing what "better" means; for MercaFresh's imbalanced churn, F1 or AUC, not accuracy.
• refit=True (the default): after the search, it automatically retrains the best combination on the whole train set — the resulting object is already a model ready to predict.
• n_jobs=-1: parallelizes across all cores; the combinations are independent of each other.

And its outputs:

• best_params_: the winning combination.
• best_score_: its mean CV score.
• cv_results_: the complete table of the search (all combinations, means, deviations, timings) — turn it into a DataFrame and study it: it's worth more than the winner alone.
• best_estimator_: the retrained model (if refit=True).

The golden rule of 06-03 still applies: what you search over is the full pipeline, not the bare model, so the preprocessing is refitted inside each fold without leaks. The hyperparameters of a pipeline step are named with the step__parameter notation (double underscore) we met back then — which even lets you treat preprocessing decisions as just more hyperparameters.

Points worth rereading:

• The real cost is combinations × folds training runs (60 here): the grid grows multiplicatively and the CV multiplies it again. Budget before launching.
• search is used afterwards like a normal model (predict, predict_proba): thanks to refit, it carries inside the best pipeline retrained on the whole train set.
• Always look at std_test_score: a "winner" by 0.002 of mean with deviations of 0.03 is a technical tie; in that case pick the simplest/most regularized combination, not the first row of the table.
• The test F1 usually lands somewhat below best_score_ — and that gap has a name and its own section further down.

The exhaustive grid dies of combinatorics: with the gradient boosting of 07-03 we'd want to explore learning_rate, max_iter, max_depth, min_samples_leaf, subsample… 5 values of each = 3,125 combinations × 5 folds = 15,625 training runs.

RandomizedSearchCV changes strategy: instead of trying everything, it samples n_iter combinations at random from distributions you define. Decisive advantages:

• Fixed budget: you decide how many training runs you pay for (n_iter=50), no matter how big the space is.
• Continuous distributions instead of lists: loguniform(0.001, 0.3) for the learning rate explores the whole range, not 5 scattered points.
• Better coverage of what matters: in practice only a few hyperparameters are influential; random sampling tries many different values of each hyperparameter, while the grid wastes budget repeating the same values of the influential one for every value of the irrelevant one.

Rule of thumb: grid for small, discrete spaces (2-3 hyperparameters with few reasonable values, like the logistic example); randomized for everything else, and especially for boosting and networks.

Grid and randomized share a naivety: each combination is chosen without looking at previous results. Bayesian search (or Bayesian optimization) is the natural evolution: it builds a probabilistic model of the "hyperparameters → score" function — yes, an ML model to optimize ML models — and uses it to decide which combination to try next, balancing exploiting the zones that look good and exploring the uncertain ones. It's the same belief-updating-with-evidence logic of Bayes' theorem (02-05), applied to the search.

On tight budgets (dozens of evaluations of an expensive model), it usually finds better combinations than pure chance. The reference library is Optuna (also scikit-optimize); its API integrates well with sklearn and adds extras like discarding mid-training the combinations that are going badly (pruning). We won't develop it here: conceptually you already have the essentials, and its syntax can be learned in an afternoon when you need it.

An important subtlety that already peeked out in the grid example: best_score_ is an optimistic estimate of real performance. Why? Because you tried many combinations on the same CV and kept the one that scored best on that particular CV: part of its edge is merit and part is luck with those folds. It's the same selection bias that in 06-01 led us to separate validation and test — resurfacing one level up.

The clean solution is nested validation (nested CV): an outer CV loop to estimate performance, and inside each outer fold, a complete inner search loop that chooses the hyperparameters. That way, the data doing the scoring never took part in the choosing:

The cost multiplies (5 × 60 = 300 fits in our example), so in practice it's reserved for when the performance figure must be rigorous — comparing algorithms for a report, or estimating how the model will perform before committing to the business. For day-to-day work, this lesson's scheme is enough: search with CV on the train set and deliver the final verdict a single time on the untouched test set.

1. Coarse to fine. First pass with wide ranges and few points (or a moderate n_iter); look at cv_results_, locate the good zone and launch a second, refined pass around it. Two cheap searches beat one very expensive one.
2. Log scale for the multiplicative ones. alpha, C, learning_rate, gamma act by orders of magnitude: use np.logspace / loguniform. For the structural ones (max_depth, n_neighbors), linear scale.
3. Budget before launching. Compute combinations × folds × time_per_fit and decide whether you can afford it; if not, shrink the grid, lower the folds (5 → 3) or switch to randomized. Time a single fit first with %timeit or time.perf_counter.
4. Search less where it matters less. Not every hyperparameter deserves a grid: in Random Forest, n_estimators is set high and that's that (07-02); in boosting, early stopping picks n_estimators for you (07-03) — remove them from the search space.
5. Fix random_state everywhere (CV, models, search) so the search is reproducible and comparisons are fair.
6. Save cv_results_. The full landscape says things the winner keeps quiet: which hyperparameters truly matter, where the model is robust and where fragile.

This module answered the question we opened it with — "can I do even better?" — with five complementary techniques:

And the workflow that ties them together for a tabular problem like MercaFresh's churn: Dummy baseline (06-02) → simple interpretable model → Random Forest → gradient boosting with early stopping → hyperparameter search on the finalist → a single verdict on the test set.

• Searching hyperparameters using the test set. The capital sin. If the test set influences any decision — hyperparameters, features, threshold — it stops measuring generalization. Test set untouched until the end, a single evaluation.
• Passing the bare model instead of the pipeline. If you scale beforehand and outside the search, every CV fold sees statistics from the rest: the leak of 06-01 in its subtle version. Always search over the Pipeline with the step__parameter notation.
• Linear grids for multiplicative hyperparameters. C in [1, 2, 3, 4, 5] explores a tiny corner of the useful range; np.logspace(-3, 2, 6) explores five orders of magnitude at the same cost.
• Taking best_score_ as the expected performance. It's optimistic due to selection bias; the honest number comes from the final test or from nested validation.
• Crowning winners by differences within the noise. Compare mean_test_score alongside std_test_score; in a technical tie, the simpler model wins.
• Tip: treat the search like a scientific experiment: hypotheses (reasoned ranges, not arbitrary ones), a reproducible procedure (fixed seeds), archived results (cv_results_ to CSV). Your three-months-from-now self — or the model's auditor — will thank you.

1. Grid on the churn K-NN. Build a StandardScaler + KNeighborsClassifier pipeline and search with GridSearchCV (scoring="f1", the lesson's stratified CV) over clf__n_neighbors in [3, 5, 9, 15, 25, 41] and clf__weights in ["uniform", "distance"]. Report best_params_, best_score_ and the test F1. How many training runs did the search execute?
2. Grid vs. randomized on the same budget. On the example's HistGradientBoostingClassifier, compare: (a) a GridSearchCV with 3 values of learning_rate × 3 of max_depth (9 combinations) and (b) a RandomizedSearchCV with n_iter=9 over the example's continuous distributions. Same cv, same scoring. Repeat (b) with three different random_state values. Who wins and what stability do you observe?
3. The size of the optimism. For the example's logistic-regression search, compute: (a) best_score_, (b) the nested-validation F1 with cross_val_score(search, ...), and (c) the test F1. Order the three figures and explain each difference using the lesson's concepts.

1. Structure: Pipeline([("scaler", StandardScaler()), ("clf", KNeighborsClassifier())]) and the grid with the clf__ notation. With these data, the winner tends to sit around n_neighbors between 15 and 41 with weights="distance" (the churn data is noisy and wide neighborhoods average better), with a CV F1 close to the logistic model's. Training runs: 6 × 2 = 12 combinations × 5 folds = 60 fits (plus 1 final one from the refit). The test F1 should land in the vicinity of best_score_, slightly below.
2. With only 2 well-bounded hyperparameters, the grid is competitive and sometimes wins; the randomized search comes very close and, depending on the seed, beats it — its 9 points cover learning-rate values the grid never even tries. Across seeds, the randomized best_score_ fluctuates (typically in the second-third decimal place): with such a low n_iter, the luck of which combinations get drawn matters. A double moral: in small spaces the grid is defensible; as soon as the space grows, the randomized search's continuous coverage wins — and with a larger n_iter, its variance across seeds shrinks.
3. The expected order is best_score_ ≥ nested F1 ≈ test F1. (a) > (b): best_score_ carries the selection bias — the winning combination is a winner partly through luck with those specific folds, and the outer CV of the nested scheme, which played no part in the choice, discounts it. (b) ≈ (c): both are honest estimates on data used in no decision; they differ only by sampling noise (the test is a single split; the nested scheme averages five). If in your run (c) comes out above (a), that's a lesson too: with datasets this size, the noise between splits can exceed the biases we measure.

You've closed the circle module 6 opened: parameters are learned by fit, hyperparameters are chosen by a disciplined search — GridSearchCV for small spaces, RandomizedSearchCV with distributions and a budget for large ones, Bayesian search when each training run is expensive —, always over the full pipeline, always with CV on the train set, with nested validation as the referee when the figure must be beyond reproach and the test set untouched for the final verdict. With that, module 7 is complete: regularization to restrain, ensembles and boosting to combine, deep networks for unstructured data and systematic optimization to squeeze it all. The MercaFresh team finally has its best churn model, tuned and evaluated honestly; but a model that lives in a notebook retains no customer. The best model in the world is worth nothing if it doesn't reach production, and that is what module 8 is about: frameworks, deployment, monitoring and the ethical considerations of putting machine learning in front of real people.