We closed the previous module with a question left hanging: "can I make my model even better?". This module's first answer is regularization, the technique that tames overfitting with surgical precision instead of a sledgehammer. In lesson 06-05 we saw that an overfitted model memorizes the noise in the training data; in linear models that overfitting has a very recognizable signature: enormous coefficients with opposite signs that cancel each other out. Regularization attacks that signature head-on by adding a complexity penalty to the cost function. In this lesson you'll understand Ridge (L2), Lasso (L1) and Elastic Net, the role of the alpha hyperparameter, and you'll apply all three techniques to predicting the monthly spend of MercaFresh customers.
Contents
- Overfitting in linear models: runaway coefficients
- The core idea: penalizing complexity
- Ridge (L2): shrink without zeroing out
- Lasso (L1): automatic feature selection
- Elastic Net: the best of both worlds
- The alpha hyperparameter and coefficient paths
- Scale before you regularize
- Regularization at MercaFresh: predicting monthly spend
- The same idea in classification: the C parameter
- Comparison table and how to choose
Overfitting in linear models: runaway coefficients
In 04-01 we trained linear regressions by minimizing the mean squared error, and we already warned that regularization would extend that idea. Why is it needed? Because ordinary linear regression (OLS) has a weak spot: correlated features.
Remember lesson 02-03: at MercaFresh, orders_per_month and products_per_month are strongly correlated (whoever places more orders buys more products). When two columns contain almost the same information, the linear model has infinitely many nearly equivalent ways to split the weight between them:
spend = 20·num_orders + 5·num_productspredicts almost the same asspend = 500·num_orders - 155·num_products
Both fit the training set equally well, but the second one is a ticking time bomb: its giant coefficients amplify any small variation in the input, and on new data the error explodes. It is exactly the high-variance symptom we diagnosed in 06-05: the model is hypersensitive to the particular sample it was trained on.
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(42)
n = 60
# Two nearly redundant features (correlation ~0.98)
num_orders = rng.poisson(4, n).astype(float)
num_products = num_orders * 5 + rng.normal(0, 1, n)
# The true spend depends mostly on the number of orders
spend = 22 * num_orders + rng.normal(0, 8, n)
X = np.column_stack([num_orders, num_products])
ols = LinearRegression().fit(X, spend)
print(ols.coef_) # e.g. [17.3, 0.95] — or far more extreme values depending on the sampleWith only 60 customers and nearly duplicated features, if you repeat the experiment with a different seed you'll see the coefficients swing wildly between runs. A model whose parameters depend that much on the luck of the sample cannot be trusted.
The core idea: penalizing complexity
The solution is surprisingly simple. Instead of minimizing only the error, we minimize:
$$\text{Cost} = \underbrace{\text{Fit error (MSE)}}{\text{do I predict well?}} + \alpha \cdot \underbrace{\text{Penalty(coefficients)}}{\text{am I simple?}}$$
- The first term pushes the model to fit the data (business as usual).
- The second term punishes large coefficients: the bigger the weights, the higher the cost.
- alpha (α) is the hyperparameter that sets the balance: the "price" the model pays for each unit of complexity.
The model can no longer afford coefficients of 500 and -155 "just because": it will only keep a large weight if the improvement in error makes it worthwhile. This reduces variance in exchange for a small increase in bias — the trade-off we formalized in 06-05, now with a continuous control knob.
The three techniques in this lesson differ only in how they measure the size of the coefficients:
| Technique | Penalty | Term formula |
|---|---|---|
| Ridge | L2 (squares) | $\alpha \sum_j w_j^2$ |
| Lasso | L1 (absolute values) | $\alpha \sum_j |w_j|$ |
| Elastic Net | L1 + L2 mix | $\alpha \left( r \sum_j |w_j| + \frac{1-r}{2} \sum_j w_j^2 \right)$ |
Note: the penalty never includes the intercept; punishing the baseline level of the prediction would make no sense.
Ridge (L2): shrink without zeroing out
Ridge penalizes the sum of squares of the coefficients. Since squaring punishes large values disproportionately, Ridge hates extreme coefficients and prefers to spread the weight evenly across correlated features.
Key properties:
- It shrinks all coefficients toward zero, but never makes them exactly zero: shrinking an already small weight yields less and less quadratic saving, so zeroing it out entirely is never worth it.
- It is the most stable option under multicollinearity: in the earlier example, Ridge would split the weight between
num_ordersandnum_productsrobustly. - It has a closed-form analytical solution, so it is fast and numerically stable.
from sklearn.linear_model import Ridge
ridge = Ridge(alpha=1.0).fit(X, spend)
print(ridge.coef_) # moderate coefficients, stable across runsLasso (L1): automatic feature selection
Lasso penalizes the sum of absolute values. The geometric difference seems minor, but it has an enormous consequence: with L1, shrinking a coefficient from 0.1 to 0 saves exactly as much as shrinking it from 5.1 to 5. That is why Lasso does find it worthwhile to set coefficients exactly to zero when a feature contributes little.
The result is that Lasso performs automatic feature selection: the final model uses only a subset of columns, and the rest are literally eliminated (coefficient = 0). This connects directly with the feature engineering of 03-06: there we built features by hand (RFM, ratios, aggregates) without knowing which would be useful; Lasso gives us a data-driven mechanism to discard the ones that don't earn their keep.
from sklearn.linear_model import Lasso
lasso = Lasso(alpha=1.0).fit(X, spend)
print(lasso.coef_) # e.g. [21.8, 0.0] — num_products eliminated!An important nuance: when two features are highly correlated, Lasso tends to keep one and zero out the other, and which one it picks can depend on the luck of the sample. Ridge, by contrast, splits the weight between both. Neither strategy is "the right one" in the abstract: it depends on whether you want a compact model (Lasso) or a stable one (Ridge).
Elastic Net: the best of both worlds
Elastic Net combines both penalties with a second hyperparameter, l1_ratio (r in the formula above):
l1_ratio=1→ pure Lasso.l1_ratio=0→ pure Ridge.- Intermediate values → it still zeroes out irrelevant features (the L1 inheritance) but treats correlated features as a group instead of picking one arbitrarily (the L2 inheritance).
from sklearn.linear_model import ElasticNet
enet = ElasticNet(alpha=1.0, l1_ratio=0.5).fit(X, spend)
print(enet.coef_)It is the recommended option when you have many features, suspect some are surplus, and there are correlated groups on top — a very common situation after a generous feature-engineering session.
The alpha hyperparameter and coefficient paths
alpha controls the strength of the punishment:
- alpha = 0: no penalty → ordinary linear regression (with all its problems).
- small alpha: gentle penalty, coefficients nearly unconstrained.
- large alpha: harsh penalty, ever smaller coefficients; at the extreme, they all tend to zero and the model predicts roughly the mean (pure underfitting, like the
DummyRegressorof 06-02).
The best way to visualize it is the path plot: how each coefficient evolves as alpha increases.
import matplotlib.pyplot as plt
alphas = np.logspace(-2, 3, 100) # from 0.01 to 1000, on a log scale
coefs_ridge, coefs_lasso = [], []
for a in alphas:
coefs_ridge.append(Ridge(alpha=a).fit(X, spend).coef_)
coefs_lasso.append(Lasso(alpha=a, max_iter=10000).fit(X, spend).coef_)
fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharey=True)
for ax, coefs, title in [(axes[0], coefs_ridge, "Ridge"),
(axes[1], coefs_lasso, "Lasso")]:
ax.plot(alphas, coefs)
ax.set_xscale("log")
ax.set_xlabel("alpha (log scale)")
ax.set_title(f"Coefficient paths — {title}")
ax.axhline(0, color="gray", lw=0.5)
axes[0].set_ylabel("Coefficient value")
plt.show()What you'll see in the plot:
- In Ridge, the curves glide smoothly toward zero but never touch it: they approach asymptotically.
- In Lasso, the curves hit zero and stay there: past a certain alpha, each feature "dies" and the model simplifies itself in stages.
And how do you pick the best alpha? You try a grid of values with cross-validation (06-03). In this lesson we'll do it with a manual loop; the systematic, automated search (GridSearchCV and friends) is the subject of lesson 07-05.
Scale before you regularize
A critical point that picks up lesson 03-05: regularization punishes the numerical size of the coefficients, and that size depends on each feature's scale.
If total_spend is measured in euros (values in the thousands) and frequency in orders/month (single-digit values), the former will need a tiny coefficient and the latter a large one for the same real effect. The penalty would unfairly crush frequency purely because of its scale. Bottom line:
Always standardize the features (StandardScaler) before Ridge, Lasso or Elastic Net. And, as we learned in module 3, inside a
Pipelineso the scaling is fitted on the train set only and there is no leakage (06-01).
Regularization at MercaFresh: predicting monthly spend
Let's apply everything to MercaFresh's regression problem: predicting a customer's monthly spend from behavioral features, several of them correlated with each other (as we discovered in the correlation matrix of 02-03).
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
# --- Synthetic MercaFresh dataset (behavioral features) ---
rng = np.random.default_rng(7)
n = 300
frequency = rng.gamma(3, 1.5, n) # orders/month
products = frequency * 6 + rng.normal(0, 2, n) # correlated with frequency
tenure = rng.uniform(1, 60, n) # months as a customer
avg_order_spend = rng.normal(35, 8, n) # EUR per order
web_visits = frequency * 4 + rng.normal(0, 3, n) # correlated with frequency
noise_1 = rng.normal(0, 1, n) # irrelevant features
noise_2 = rng.normal(0, 1, n)
# True spend: depends on frequency and order value; the rest is redundant or noise
monthly_spend = frequency * avg_order_spend + rng.normal(0, 15, n)
X = pd.DataFrame({
"frequency": frequency, "products": products,
"tenure": tenure, "avg_order_spend": avg_order_spend,
"web_visits": web_visits, "noise_1": noise_1, "noise_2": noise_2,
})
y = monthly_spend
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
# --- Tournament: OLS vs Ridge vs Lasso vs Elastic Net, always with scaling ---
models = {
"OLS": LinearRegression(),
"Ridge": Ridge(alpha=10),
"Lasso": Lasso(alpha=1.0, max_iter=10000),
"Elastic Net": ElasticNet(alpha=1.0, l1_ratio=0.5, max_iter=10000),
}
for name, model in models.items():
pipe = Pipeline([("scaler", StandardScaler()), ("reg", model)])
# 5-fold cross-validated RMSE, as in 06-03
rmse = -cross_val_score(pipe, X_train, y_train, cv=5,
scoring="neg_root_mean_squared_error").mean()
pipe.fit(X_train, y_train)
coefs = pipe.named_steps["reg"].coef_.round(1)
print(f"{name:12s} CV-RMSE: {rmse:6.2f} EUR coefs: {coefs}")A typical outcome of this experiment:
- OLS spreads weight across
frequency,productsandweb_visits(redundant) and even assigns some to the noise features. - Ridge shrinks all the weights and stabilizes the split among the correlated ones; it usually improves the validation RMSE slightly.
- Lasso zeroes out
noise_1,noise_2and often one of the redundant features: the resulting model is shorter and more readable for MercaFresh's business team ("spend is explained by frequency and average order value"). - Elastic Net lands between the two: it eliminates the noise but keeps part of the correlated group.
Notice that the comparison is done with cross-validation on the train set, keeping the test set for the final verdict — the discipline of module 6 is not abandoned just because we're using more advanced techniques.
The same idea in classification: the C parameter
Regularization is not exclusive to regression. In fact, you've already been using it without knowing:
- In 04-02 we saw that sklearn's
LogisticRegressionhas aCparameter. Well:LogisticRegressioncomes L2-regularized by default, andCis exactly the inverse of alpha:C = 1/α. - In 04-04, the
Cparameter of SVMs played the same role: controlling the balance between fitting the train set well and keeping the model simple (a wide margin).
| Parameter | Small value | Large value |
|---|---|---|
alpha (Ridge/Lasso/EN) |
Little regularization → overfitting risk | Heavy regularization → underfitting risk |
C (LogisticRegression/SVM) |
Heavy regularization → underfitting | Little regularization → overfitting |
Beware of the inversion: raising alpha and raising C have opposite effects. Also, LogisticRegression(penalty="l1", solver="liblinear") gives you a classifier with Lasso-style feature selection — useful in MercaFresh's churn problem if you want a model that uses few RFM variables.
from sklearn.linear_model import LogisticRegression
# Churn with L1 regularization: zeroed coefficients = discarded features
clf = Pipeline([
("scaler", StandardScaler()),
("logreg", LogisticRegression(penalty="l1", C=0.1, solver="liblinear")),
])Comparison table and how to choose
| Criterion | Ridge (L2) | Lasso (L1) | Elastic Net |
|---|---|---|---|
| Coefficients exactly zero | No | Yes | Yes |
| Feature selection | No | Automatic | Automatic (by groups) |
| Highly correlated features | Splits the weight (stable) | Picks one almost at random | Tends to keep the group |
| Number of hyperparameters | 1 (alpha) |
1 (alpha) |
2 (alpha, l1_ratio) |
| When to use it | Many useful, correlated features; priority: stability | You suspect few features matter; priority: interpretability | Many features, some irrelevant and others in correlated groups |
Rule of thumb: start with Ridge as the safe option; move to Lasso if you need a compact, explainable model; use Elastic Net when Lasso behaves erratically because of correlation between features.
Common Mistakes and Tips
- Regularizing without scaling. The number one mistake. Without
StandardScaler, alpha punishes features for their unit of measurement, not for their relevance. Always use aPipeline. - Confusing alpha and C. In regression (Ridge/Lasso), more alpha = more regularization. In classification (
C), it's the other way around. Always check which direction you're turning the knob. - Reading a Lasso zero as "this variable has no real-world effect". Lasso says the feature adds nothing given that the others are already in; with correlated features, the eliminated one may be just as causal as the survivor. Causal inference is a different discipline.
- Choosing alpha by looking at the training error. The train set will always prefer alpha = 0. Alpha is chosen with (cross-)validation, never with the training error.
- Forgetting
max_iterin Lasso/ElasticNet. They use iterative optimization (coordinate descent) and with small alphas they may fail to converge; if you see warnings, raisemax_iter. - Tip: always explore alpha on a logarithmic scale (
np.logspace), because its effect is multiplicative: the relevant difference is between 0.01, 0.1, 1 and 10, not between 1 and 2.
Exercises
- Paths on MercaFresh. With this lesson's monthly-spend dataset (the 7 features), plot the Lasso coefficient paths for
alphas = np.logspace(-2, 2, 100)(standardizing first). In what order do the features "die"? Does it match what you know about how the spend was generated? - Ridge against multicollinearity. Generate 20 different samples of the small dataset from section 1 (changing the seed) and on each one fit OLS and Ridge with
alpha=10(with scaling). Compute the standard deviation of the first coefficient across the 20 runs for each model. Which one is more stable? - L1 on churn. On the churn dataset with the RFM features from module 3, train
LogisticRegression(penalty="l1", solver="liblinear")withCin[0.01, 0.1, 1, 10]inside a pipeline with scaling. For each C, count how many coefficients end up at zero and compute the F1 with stratified cross-validation (06-03). Which C would you choose and why?
Solutions
- Standardize with
StandardScalerand fit aLassofor each alpha, storingcoef_. When you plot the 7 curves you'll see thatnoise_1andnoise_2die almost immediately (small alphas), then the redundant ones fall (web_visits,products— Lasso usually keepsfrequencyfrom the correlated group) and the last survivors arefrequencyandavg_order_spend, exactly the two variables the spend was generated from. The order in which the paths die is itself a relevance ranking. - Structure: a
for seed in range(20)loop, regenerate the data withnp.random.default_rng(seed), fit both models on standardized features and storecoef_[0]. With OLS the standard deviation of the coefficient comes out several times larger than with Ridge (with data this correlated it can be an order of magnitude). Conclusion: Ridge doesn't just improve prediction; it makes the model reproducible, which is what reducing variance means. - With
C=0.01(very strong regularization) almost all coefficients end up at zero and the F1 falls toward theDummyClassifierbaseline (06-02): underfitting. WithC=10hardly anything is zeroed out and the result resembles unregularized logistic regression. The sweet spot is usually atC=0.1–1: F1 comparable to the maximum with 2-4 RFM features eliminated. You'd choose the C with the best CV F1 and, in a practical tie, the more regularized one (simpler model). The systematic search for this kind of decision is what we'll automate in 07-05.
Conclusion
Regularization turns the bias-variance trade-off of 06-05 into a continuous dial: the cost function now rewards fit and simplicity at the same time, and alpha sets the balance. Ridge (L2) shrinks coefficients and stabilizes models with correlated features; Lasso (L1) additionally sets coefficients to exact zero, giving us automatic feature selection for free; Elastic Net blends the two. You've seen that the idea isn't new to you: the C parameter of logistic regression and SVMs was regularization in disguise. And you've confirmed two non-negotiable disciplines: scale inside a Pipeline and choose alpha with cross-validation, never with the train set. Regularizing improves a model by restraining it; the next lesson explores the opposite and complementary route: improving by combining many models so their errors cancel each other out. Welcome to Ensemble Learning.
Machine Learning Course
Module 1: Introduction to Machine Learning
- What is Machine Learning?
- History and evolution of Machine Learning
- Types of Machine Learning
- Applications of Machine Learning
- The Machine Learning project workflow
Module 2: Foundations of Statistics and Probability
- Basic statistics concepts
- Probability distributions
- Correlation and covariance
- Statistical inference
- Bayes' theorem
Module 3: Data Preprocessing
- Data cleaning
- Handling missing data
- Data transformation
- Encoding categorical variables
- Normalization and standardization
- Feature engineering
Module 4: Supervised Machine Learning Algorithms
- Linear regression
- Logistic regression
- Decision trees
- Support Vector Machines (SVM)
- K-Nearest Neighbors (K-NN)
- Naive Bayes
- Neural networks
Module 5: Unsupervised Machine Learning Algorithms
- Clustering: K-means
- Hierarchical clustering
- Principal Component Analysis (PCA)
- DBSCAN clustering
- Data visualization with t-SNE and UMAP
Module 6: Model Evaluation and Validation
- Data splitting: training, validation and test
- Evaluation metrics
- Cross-validation
- ROC curve and AUC
- Overfitting and underfitting
Module 7: Advanced Techniques and Optimization
- Regularization: Ridge, Lasso and Elastic Net
- Ensemble Learning
- Gradient Boosting
- Deep neural networks (Deep Learning)
- Hyperparameter optimization
Module 8: Model Implementation and Deployment
- Popular frameworks and libraries
- Deploying models to production
- Model maintenance and monitoring
- Ethical and privacy considerations
Module 9: Hands-On Projects
- Project 1: Housing price prediction
- Project 2: Image classification
- Project 3: Sentiment analysis on social media
- Project 4: Fraud detection
- Project 5: Customer segmentation