In 09-01 you went back to the algorithms of module 3 and confirmed that discipline (represent well, define the objective, validate) matters more than the specific algorithm. Now it is time for the same discipline with the machine learning of module 4: five practices with the NovaMarket generators you already know (generate_orders_ml, dirty_orders, prepare_orders, build_preprocessing, generate_weekly_demand, generate_customers_ml), each with a baseline, an honest comparison and a business decision at the end. The reference figures from module 4 (AUC 0.844, cost €2,485 → €1,610 with threshold 0.2, demand MAE 16.8, three customer segments) are your yardstick; in several practices you will discover that "improving" is harder than it looks.
How to work through the lesson: each practice has a statement with specific questions; write your solution before reading ours, run it, and compare figures (they vary by a few hundredths depending on the scikit-learn version). You need the novamarket_ml.py module from 04-03/04-05 (the functions listed above) in the same folder, plus pandas, numpy and scikit-learn. Guideline time: 30-45 minutes per practice.
Contents
- Practice 1: exploring and cleaning a dirty batch with a decision report
- Practice 2: a new feature in the returns pipeline, compared honestly
- Practice 3: demand forecasting with calendar and lags, linear versus gradient boosting
- Practice 4: customer segmentation with k-means and one action per segment
- Practice 5: cost-based threshold tuning and a fairness check across zones
- Common Mistakes and Tips
- Conclusion
- Practice 1: exploring and cleaning a dirty batch with a decision report
Reminder (04-03, sections 2-3 and 8). dirty_orders produces a "realistic" copy of the orders with nulls, duplicates, an outlier, dates, text and a leak (return_reason, which only exists if the order was returned). Before modelling you have to measure each problem and decide what to do with it, without looking at the label more than strictly necessary.
Statement. Marta receives the September-November orders.csv batch (dirty_orders(generate_orders_ml(3000, 42), 42)). Write a script that produces a quality report with: (a) shape and types; (b) nulls per column, as a count and a percentage, and a check of whether the nulls in delivery_days are spread equally between returned and non-returned orders; (c) duplicated rows (exact and by order_id); (d) amount outliers with the 3 × IQR rule and by eye; (e) category counts and date range; (f) the columns that would be leaks. Then write clean_batch(batch), which returns the clean batch and a dictionary of what it did, and justify each decision in a table.
Hints: isna().sum(), duplicated(keep=False), quantile([0.25, 0.75]), pd.crosstab(batch["return_reason"], batch["returned"]).
Solution
import numpy as np, pandas as pd
from novamarket_ml import generate_orders_ml, dirty_orders, prepare_orders
batch = dirty_orders(generate_orders_ml(3000, 42), 42)
print(batch.shape); print(batch.dtypes.to_dict())
nulls = batch.isna().sum(); nulls = nulls[nulls > 0]
print(pd.DataFrame({"nulls": nulls, "%": (nulls / len(batch) * 100).round(1)}))
print("returned if delivery_days null:", batch.loc[batch["delivery_days"].isna(), "returned"].mean().round(3),
"| not null:", batch.loc[batch["delivery_days"].notna(), "returned"].mean().round(3))
print("Exact duplicates:", batch.duplicated().sum(), "| repeated ids:", batch["order_id"].duplicated().sum())
print(batch[batch.duplicated(keep=False)].sort_values("order_id")[["order_id", "customer_id", "order_date", "amount", "returned"]])
q1, q3 = batch["amount"].quantile([0.25, 0.75]); ceiling = q3 + 3 * (q3 - q1)
print(f"IQR: q1={q1:.1f} q3={q3:.1f} 3xIQR ceiling={ceiling:.1f} | above: {(batch['amount'] > ceiling).sum()}")
print(batch.loc[batch["amount"] > 1000, ["order_id", "amount", "num_items", "category", "returned"]])
print("Maximum without the outlier:", batch.loc[batch["amount"] < 5000, "amount"].max())
for c in ["category", "postcode_zone", "payment_method", "shipping_type"]:
print(c, batch[c].value_counts(dropna=False).to_dict())
print("Dates:", batch["order_date"].min().date(), "->", batch["order_date"].max().date())
print(pd.crosstab(batch["return_reason"], batch["returned"]))
print("Return rate:", batch["returned"].mean().round(3), "| by zone:", batch.groupby("postcode_zone")["returned"].mean().round(3).to_dict())
def clean_batch(batch, amount_ceiling=5000):
report = {}
clean = batch.drop_duplicates(); report["duplicates_removed"] = len(batch) - len(clean)
outlier = clean["amount"] > amount_ceiling
report["amount_outliers_removed"] = int(outlier.sum()); clean = clean[~outlier].copy()
report["nulls_pipeline_will_impute"] = clean.isna().sum()[lambda s: s > 0].to_dict()
clean = clean.drop(columns=["return_reason"]); report["leaks_removed"] = ["return_reason"]
return clean, report
clean, report = clean_batch(batch); print(clean.shape, report)
X, y = prepare_orders(batch); print("prepare_orders ->", X.shape, "rate", y.mean().round(4))(3003, 13)
nulls %
amount 56 1.9
delivery_days 153 5.1
payment_method 100 3.3
returned if delivery_days null: 0.209 | not null: 0.162
Exact duplicates: 3 | repeated ids: 3
IQR: q1=63.8 q3=163.6 3xIQR ceiling=463.1 | above: 13
order_id amount num_items category returned
659 P100017 99999.0 2 electronics 0
Maximum without the outlier: 632.61
payment_method {'card': 1624, 'paypal': 730, 'bizum': 310, 'cash_on_delivery': 239, nan: 100}
Dates: 2025-09-01 -> 2025-11-29
returned 0 1
return_reason
2509 0
arrived late 0 124
defective 0 126
didn't like it 0 131
size/model 0 113
Return rate: 0.165 | by zone: {'A': 0.162, 'B': 0.168, 'C': 0.164}
(2999, 12) {'duplicates_removed': 3, 'amount_outliers_removed': 1, 'nulls_pipeline_will_impute': {'amount': 56, 'delivery_days': 153, 'payment_method': 100}, 'leaks_removed': ['return_reason']}
prepare_orders -> (2999, 15) rate 0.1644| Finding | Decision | Justification |
|---|---|---|
3 exact duplicated rows (same order_id, customer, date, amount) |
Remove | An order cannot appear twice; counting it double biases training and evaluation |
amount = €99,999 in P100017 (2 electronics items) |
Remove the row | It is a data-entry error, not an order: the next maximum is €632. The remaining 12 above 3 × IQR (up to €632) are legitimate expensive orders and are kept |
Nulls: delivery_days 5.1 %, payment_method 3.3 %, amount 1.9 % |
Impute inside the Pipeline (median / most frequent) |
They are few and you cannot afford to lose 10 % of the rows; imputing in the pipeline avoids leaks. The return rate when delivery_days is null (0.209 versus 0.162, over 153 rows) is within the margin of chance, but it is the check you have to make: if it were large and persistent, the null would be informative and a delivery_days_missing indicator would be worthwhile (variant) |
return_reason only filled in if returned = 1 |
Drop the column | Data leakage: it is known after the return |
order_id, customer_id, order_date |
Not direct features | They are used to derive (weekday, per-customer history) and then discarded |
| Categories with no typos; rate per zone ≈ 16 % in A, B and C | Nothing | Consistent with how the data is built (the zone has no influence) |
Feedback
- Typical mistake: removing the 13 "outliers" from the IQR rule. Statistical rules point at candidates; the decision comes from context: a €600 order is normal in electronics. Just as bad is imputing the 99,999 with the median instead of removing it (the data-entry error would still be in the statistics of that row's other columns).
- Another one: imputing with
fillnabefore splitting; doing it in thePipelineis what protects you from leaks (04-03). - Variants: add
delivery_days_missingas a binary feature and measure in 09-02.2 whether it contributes; simulate a batch with 30 % ofamountnull and decide again (with that many nulls, median imputation impoverishes the variable); write the report as a reusablequality_report(df)function innovamarket_ai/.
- Practice 2: a new feature in the returns pipeline, compared honestly
Reminder (04-03 section 9, 04-05 sections 5 and 8). The reference pipeline (build_preprocessing() + logistic regression, 21 columns) obtains AUC 0.844 on the 750-order test set and 0.836 ± 0.016 in 5-fold stratified cross-validation; with threshold 0.2 the cost is €1,610 on the test set. An honest comparison uses the same split and, better, cross-validation; and it decides with the business metric, not only with the AUC.
Statement. Logistic regression cannot on its own model the interaction "new customer and expensive order" that the hidden truth of the data contains (remember 04-01). Add three candidates: new_x_amount (= new_customer × amount), risk_category (1 if electronics or computing) and days_x_new (= new_customer × delivery_days). For each one: (a) extend the preprocessing (ColumnTransformer) without touching the rest; (b) measure AUC with StratifiedKFold(5, shuffle=True, random_state=42) and the business cost with cross_val_predict at thresholds 0.2, 0.3 and 0.5; (c) compare with the reference in the same validation and also on the split from 04-05 (train_test_split(..., test_size=0.25, random_state=42, stratify=y)); (d) decide whether Marta should change the production model.
Solution
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, RepeatedStratifiedKFold, cross_val_score, cross_val_predict, train_test_split
from sklearn.metrics import roc_auc_score, confusion_matrix
from novamarket_ml import build_preprocessing, NUMERIC, BINARY, NOMINAL, ORDINAL
def business_cost(m, cost_fp=5, cost_fn=30):
tn, fp, fn, tp = m.ravel(); return fp * cost_fp + fn * cost_fn
def add_features(X):
X = X.copy()
X["new_x_amount"] = X["new_customer"] * X["amount"].fillna(X["amount"].median())
X["risk_category"] = X["category"].isin(["electronics", "computing"]).astype(int)
X["days_x_new"] = X["new_customer"] * X["delivery_days"].fillna(X["delivery_days"].median())
return X
def build_preprocessing_v2(extra_numeric=(), extra_binary=()):
return ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), NUMERIC + list(extra_numeric)),
("bin", "passthrough", BINARY + list(extra_binary)),
("nom", Pipeline([("impute", SimpleImputer(strategy="most_frequent")), ("onehot", OneHotEncoder(handle_unknown="ignore"))]), NOMINAL),
("ord", OrdinalEncoder(categories=[["standard", "fast", "urgent"]]), ORDINAL)])
X2 = add_features(X)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
def evaluate(name, prep, Xd):
pipe = Pipeline([("prep", prep), ("model", LogisticRegression(max_iter=2000))])
aucs = cross_val_score(pipe, Xd, y, cv=cv, scoring="roc_auc")
prob = cross_val_predict(pipe, Xd, y, cv=cv, method="predict_proba")[:, 1]
costs = {t: int(business_cost(confusion_matrix(y, (prob >= t).astype(int)))) for t in (0.2, 0.3, 0.5)}
print(f"{name:22s} AUC CV {aucs.mean():.4f} ± {aucs.std():.3f} cost CV {costs}")
evaluate("reference", build_preprocessing(), X)
evaluate("+ new_x_amount", build_preprocessing_v2(["new_x_amount"]), X2)
evaluate("+ risk_category", build_preprocessing_v2([], ["risk_category"]), X2)
evaluate("+ days_x_new", build_preprocessing_v2(["days_x_new"]), X2)
evaluate("+ all three", build_preprocessing_v2(["new_x_amount", "days_x_new"], ["risk_category"]), X2)
# repeated cross-validation (5 x 4) with paired differences per fold
rcv = RepeatedStratifiedKFold(n_splits=5, n_repeats=4, random_state=1)
ref = Pipeline([("prep", build_preprocessing()), ("model", LogisticRegression(max_iter=2000))])
new = Pipeline([("prep", build_preprocessing_v2(["new_x_amount"])), ("model", LogisticRegression(max_iter=2000))])
a, b = cross_val_score(ref, X, y, cv=rcv, scoring="roc_auc"), cross_val_score(new, X2, y, cv=rcv, scoring="roc_auc")
print(f"20 folds: ref {a.mean():.4f} new {b.mean():.4f} difference {np.mean(b - a):+.4f} ± {np.std(b - a):.4f} in favour in {(b > a).sum()}/20")
# the split from 04-05
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
for name, prep, tr, te in [("reference", build_preprocessing(), Xtr, Xte),
("+ new_x_amount", build_preprocessing_v2(["new_x_amount"]), add_features(Xtr), add_features(Xte))]:
p = Pipeline([("prep", prep), ("model", LogisticRegression(max_iter=2000))]).fit(tr, ytr); prob = p.predict_proba(te)[:, 1]
print(f"{name:22s} AUC test {roc_auc_score(yte, prob):.4f} cost test", {t: int(business_cost(confusion_matrix(yte, (prob >= t).astype(int)))) for t in (0.2, 0.5)})
p = new.fit(X2, y); names, coef = p["prep"].get_feature_names_out(), p["model"].coef_[0]
print("Largest coefficients:", [(n, round(float(c), 2)) for n, c in sorted(zip(names, coef), key=lambda t: -abs(t[1]))[:5]])reference AUC CV 0.8365 ± 0.016 cost CV {0.2: 6635, 0.3: 7220, 0.5: 9455}
+ new_x_amount AUC CV 0.8409 ± 0.014 cost CV {0.2: 6725, 0.3: 7410, 0.5: 9325}
+ risk_category AUC CV 0.8365 ± 0.016 cost CV {0.2: 6635, 0.3: 7220, 0.5: 9450}
+ days_x_new AUC CV 0.8366 ± 0.016 cost CV {0.2: 6630, 0.3: 7200, 0.5: 9490}
+ all three AUC CV 0.8406 ± 0.014 cost CV {0.2: 6680, 0.3: 7420, 0.5: 9305}
20 folds: ref 0.8333 new 0.8379 difference +0.0046 ± 0.0060 in favour in 16/20
reference AUC test 0.8444 cost test {0.2: 1610, 0.5: 2485}
+ new_x_amount AUC test 0.8334 cost test {0.2: 1710, 0.5: 2360}
Largest coefficients: [('num__new_x_amount', 0.83), ('num__amount', 0.76), ('num__delivery_days', 0.66), ('nom__category_electronics', 0.54), ('bin__new_customer', 0.45)]Reading, in the order in which it pays to reason:
risk_categorychanges nothing (0.8365 and the same costs): it is a linear combination of the one-hot columns ofcategorythat already exist; logistic regression gains no information from a redundant column.days_x_newdoes not contribute either (the hidden truth does not contain that interaction).new_x_amountdoes improve the AUC in a small but consistent way: +0.0046 on average, in favour in 16 of 20 folds, and it becomes the largest coefficient in the model (0.83), as you would expect from the interaction the hidden truth contains. But the improvement does not translate into euros at the working threshold: at 0.2 the cross-validated cost is slightly worse (€6,725 versus €6,635), and it only improves at 0.5 (where nobody operates).- The single split from 04-05 says the opposite of the cross-validation (0.833 versus 0.844): with 750 orders and 123 returns, ±0.01 of AUC is sampling noise. That is why in 04-05 we insisted on cross-validation.
- Decision: do not change the production model. The gain is marginal in the technical metric and nil or negative in the business one; all else being equal, the simpler one wins (04-06). Marta records the feature as a candidate for the next retraining, when there is more data and it can be evaluated with the real cost.
Feedback
- Typical mistake: comparing the new pipeline with the AUC "from memory" (0.844) obtained on a different split; the comparison is only valid in the same validation. Another one: creating the interaction with
amountwithout imputing and letting the numeric branch'sSimpleImputerimpose the median of a product (here we make it explicit withfillnain the function). - The AUC going up while the cost does not go down is not a contradiction: the AUC measures the global ordering of all pairs; the cost at 0.2 depends on how a few orders get reordered around that threshold.
- Variants: try
HistGradientBoostingClassifier, which captures interactions without being given them; adddelivery_days_missingfrom practice 1; repeat withn_repeats=10and watch the deviation of the difference narrow.
- Practice 3: demand forecasting with calendar and lags, linear versus gradient boosting
Reminder (04-04 section 2, 04-05 sections 6 and 8). The weekly demand for the NovaClean robot (104 weeks) has a trend (2.5/week), yearly seasonality and a Black Friday spike. Linear regression with week, sin, cos and black_friday gives MAE 16.8 on weeks 79-104 and 18.6 on average with TimeSeriesSplit(4, test_size=13); the "last week" baseline gives 39. Temporal folds train on the past and evaluate the following quarter.
Statement. (a) Write build_features(d, lags=(1, 2, 4), with_black_friday=True), which adds calendar (sin, cos, black_friday), the lags lag_k (units k weeks ago) and mean_4 (mean of the previous 4 weeks), and drops the initial rows without lags. (b) With TimeSeriesSplit(n_splits=4, test_size=13), compute the mean MAE of: the naive baseline (lag_1), the mean_4 baseline, the linear model with calendar only, the linear model with calendar + lags, and HistGradientBoostingRegressor (max_iter=200, learning_rate=0.05, max_depth=3, min_samples_leaf=5) with calendar + lags. (c) Repeat linear and HGB without black_friday and look at the specific error in week 100 (the second Black Friday). (d) Explain the results and choose a model for Marta.
Solution
from novamarket_ml import generate_weekly_demand
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_absolute_error
d = generate_weekly_demand(104, 42)
def build_features(d, lags=(1, 2, 4), with_black_friday=True):
f = d.copy()
f["sin"], f["cos"] = np.sin(2 * np.pi * f["week"] / 52), np.cos(2 * np.pi * f["week"] / 52)
f["black_friday"] = ((f["week"] - 1) % 52 == 47).astype(int)
for k in lags: f[f"lag_{k}"] = f["units"].shift(k) # units k weeks ago
f["mean_4"] = f["units"].shift(1).rolling(4).mean() # mean of the previous 4
calendar = ["week", "sin", "cos"] + (["black_friday"] if with_black_friday else [])
lag_cols = [f"lag_{k}" for k in lags] + ["mean_4"]
return f.dropna().reset_index(drop=True), calendar, lag_cols
tscv = TimeSeriesSplit(n_splits=4, test_size=13)
def mae_cv(f, cols, make_model=None, name=""):
maes = []
for tr, te in tscv.split(f):
pred = f.loc[te, cols[0]] if make_model is None else make_model().fit(f.loc[tr, cols], f.loc[tr, "units"]).predict(f.loc[te, cols])
maes.append(mean_absolute_error(f.loc[te, "units"], pred))
print(f"{name:42s} MAE per block {np.round(maes, 1)} mean {np.mean(maes):.1f}")
lin = LinearRegression
hgb = lambda: HistGradientBoostingRegressor(max_iter=200, learning_rate=0.05, max_depth=3, min_samples_leaf=5, random_state=0)
f, cal, lg = build_features(d)
print("Test blocks:", [(int(f.loc[te, 'week'].min()), int(f.loc[te, 'week'].max())) for _, te in tscv.split(f)])
mae_cv(f, ["lag_1"], None, "Naive baseline (last week)")
mae_cv(f, ["mean_4"], None, "4-week mean baseline")
mae_cv(f, cal, lin, "Linear calendar only (04-05)")
mae_cv(f, cal + lg, lin, "Linear calendar + lags")
mae_cv(f, cal + lg, hgb, "HGB calendar + lags")
f0, cal0, lg0 = build_features(d, with_black_friday=False)
mae_cv(f0, cal0 + lg0, lin, "Linear WITHOUT black_friday")
mae_cv(f0, cal0 + lg0, hgb, "HGB WITHOUT black_friday")
for name, (ff, cols) in {"linear with BF": (f, cal + lg), "linear without BF": (f0, cal0 + lg0), "HGB with BF": (f, cal + lg)}.items():
tr, te = ff["week"] <= 91, ff["week"] > 91
m = (lin if "linear" in name else hgb)().fit(ff.loc[tr, cols], ff.loc[tr, "units"])
err = pd.Series(np.abs(ff.loc[te, "units"].values - m.predict(ff.loc[te, cols])), index=ff.loc[te, "week"])
print(f"{name:17s} weeks 92-104: MAE {err.mean():.1f} | error week 100 (Black Friday): {err[100]:.1f} | without 100: {err.drop(100).mean():.1f}")Test blocks: [(53, 65), (66, 78), (79, 91), (92, 104)] Naive baseline (last week) MAE per block [31.3 18.7 15.8 52.8] mean 29.7 4-week mean baseline MAE per block [27.4 19.7 12.1 50.8] mean 27.5 Linear calendar only (04-05) MAE per block [38.1 19. 13.2 20.1] mean 22.6 Linear calendar + lags MAE per block [66.4 19. 13.7 19.8] mean 29.7 HGB calendar + lags MAE per block [49.8 50.5 12.4 48.9] mean 40.4 Linear WITHOUT black_friday MAE per block [168.4 16.5 14.5 40. ] mean 59.9 HGB WITHOUT black_friday MAE per block [49.8 50.5 12.4 48.9] mean 40.4 linear with BF weeks 92-104: MAE 19.8 | error week 100 (Black Friday): 26.4 | without 100: 19.2 linear without BF weeks 92-104: MAE 40.0 | error week 100 (Black Friday): 214.5 | without 100: 25.5 HGB with BF weeks 92-104: MAE 48.9 | error week 100 (Black Friday): 129.9 | without 100: 42.2
Four conclusions Marta must be able to explain:
- The linear model with calendar still wins (22.6). The lags do not improve it (blocks 2-4 unchanged) and they make it worse in the first block (66.4): with only 48 weeks of training and the Black Friday spike sitting inside
lag_1,lag_4andmean_4of weeks 49-52, the lag coefficients are poorly estimated. And they cannot help much: the noise in this series is independent from one week to the next (the autocorrelation of the calendar model's residuals is 0.09), so the lags contain no information the calendar does not already have. In real series with runs (promotions that drag on, stock-outs) they would. - Gradient boosting loses clearly (40.4): a tree model does not extrapolate the trend (it predicts at most the maximum seen in training, and the series grows by 2.5 units/week) and with 48-87 rows it is starved of data. With
min_samples_leaf=1it drops to 34.8; with the classic fix (linear for trend and seasonality, HGB only for the residuals) to 24.7: it still does not beat the linear model on its own. - Black Friday: removing the variable sends the linear model's error in week 100 through the roof (214 units: it predicts ~600 and 825 are sold) and contaminates the fit of the first block (168), because the week-48 spike is fitted as if it were seasonality. For HGB nothing changes: with
min_samples_leaf=5, a tree cannot isolate a leaf with a single Black Friday week, so it ignores the variable even when given it. It is an example of why business rules ("week 47 sells ×1.4") are sometimes worth more than the model. - Choice: the linear model with calendar, MAE ≈ 20 units on sales of 550-800, versus 27-30 for the baselines. And a maintenance rule: retrain every quarter, because the most recent 13-week block is always outside the training set.
Feedback
- Typical mistake: building the lags and not dropping the first rows (they are left as
NaN), or computing them withshift(-1)(using the future: leakage). Another one: evaluating with a shuffledKFold: the MAE would drop to ~17 and would not hold in production (04-05). - Notice that the naive baseline is worse in blocks 1 and 4 (31 and 53): those are the ones containing Black Friday, and "repeat last week" fails twice (it does not anticipate it and then prolongs it).
- Variants: add
lag_52(same week of the previous year) plus the trend (exercise 3 of 04-05); usegenerate_weekly_demand(208)(4 years) and check whether HGB gets closer with more data; tryRidgewith the lags to stabilise the first block.
- Practice 4: customer segmentation with k-means and one action per segment
Reminder (04-02 section 5, 04-04 section 8). k-means groups by distance to k centroids; it requires scaling the variables; inertia always drops as k rises, so you look for the "elbow", and you complement it with the silhouette coefficient (higher, better separation) and with business interpretability. generate_customers_ml(600, 42) has three hidden types (occasional, regular, big spenders).
Statement. (a) Scale annual_spend, num_orders and tenure_months and compute inertia and silhouette for k from 2 to 8. (b) Choose k and justify it. (c) Describe each segment (size, means, mean ticket = spend/orders, percentage of total spend) and propose a business action per segment. (d) Check what happens with k = 4 (which segment splits and why?) and what happens if you do not scale.
Solution
from novamarket_ml import generate_customers_ml
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
cl = generate_customers_ml(600, 42)
scaler = StandardScaler(); Xc = scaler.fit_transform(cl)
print("| k | inertia | silhouette |")
for k in range(2, 9):
km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(Xc)
print(f"| {k} | {km.inertia_:.0f} | {silhouette_score(Xc, km.labels_):.3f} |")
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(Xc)
cl["segment"] = km.labels_; cl["mean_ticket"] = cl["annual_spend"] / cl["num_orders"]
profile = cl.groupby("segment").agg(n=("annual_spend", "size"), spend=("annual_spend", "mean"), orders=("num_orders", "mean"),
ticket=("mean_ticket", "mean"), tenure=("tenure_months", "mean")).round(1)
profile["% total spend"] = (cl.groupby("segment")["annual_spend"].sum() / cl["annual_spend"].sum() * 100).round(1)
print(profile)
km4 = KMeans(n_clusters=4, n_init=10, random_state=42).fit(Xc)
print(pd.crosstab(cl["segment"], km4.labels_, rownames=["k=3"], colnames=["k=4"]))
print(cl.groupby(km4.labels_)[["annual_spend", "num_orders", "tenure_months"]].mean().round(1))
km_unscaled = KMeans(n_clusters=3, n_init=10, random_state=42).fit(cl[["annual_spend", "num_orders", "tenure_months"]])
print(pd.crosstab(cl["segment"], km_unscaled.labels_, rownames=["scaled"], colnames=["unscaled"]))| k | inertia | silhouette |
|---|---|---|
| 2 | 796 | 0.554 |
| 3 | 400 | 0.572 |
| 4 | 299 | 0.534 |
| 5 | 254 | 0.524 |
| 6 | 219 | 0.416 |
| 8 | 174 | 0.398 |
n spend orders ticket tenure % total spend segment 0 351 185.8 2.5 95.0 9.5 20.5 1 52 2206.7 20.0 117.4 38.9 36.1 2 197 699.9 9.2 88.0 28.5 43.4 k=4 0 1 2 3 k=3 0 23 328 0 0 1 0 0 51 1 2 85 0 0 112 annual_spend num_orders tenure_months 0 694.2 9.1 16.7 1 152.9 2.1 9.5 2 2229.8 20.1 38.6 3 699.1 9.0 36.2 unscaled 0 1 2 scaled 0 27 0 324 1 3 49 0 2 187 1 9
Inertia (1,800 with k = 1) falls to 796 with 2 groups and to 400 with 3, and then drops slowly (299, 254, 219): the elbow is at 3, and the silhouette also peaks at 3 (0.572). With another generator seed (generate_customers_ml(600, 7)) the silhouette peaks at 3 again (0.583): the conclusion is stable. The segments and their actions:
| Segment | Profile | Proposed action |
|---|---|---|
| 0 · Occasional (351 customers, 58 %) | 2-3 orders/year, €186 of spend, 9 months of tenure; 20 % of spend | Activation: second-purchase coupon and recommendations (case 1); measure conversion to "regular" at 6 months |
| 2 · Regular (197, 33 %) | 9 orders/year, €700, ticket €88, 28 months; 43 % of spend | Loyalty: free shipping via subscription, replenishment reminders (NovaBrew capsules, NovaClean filters); it is the segment where preventing churn is worth most |
| 1 · Big spenders (52, 9 %) | 20 orders/year, €2,200, ticket €117, 39 months; 36 % of spend | Personal management: early access to launches (NovaBook, NovaView), priority handling of incidents; watch for churn signals one by one |
With k = 4 no new type appears: the regular segment is split by tenure (17 versus 36 months), a division that may be useful for marketing but that the silhouette does not support. Without scaling, the variable with the largest variance (annual_spend, deviation 599 versus 6 and 13) dominates the distance and 36 customers change group: the groups become "spend brackets" and num_orders and tenure barely count.
Feedback
- Typical mistake: choosing k solely by minimum inertia (it would always be the maximum k) or presenting the centroids on the standardised scale (
scaler.inverse_transform(km.cluster_centers_)returns them to euros and orders). - Another one: interpreting the segment numbers (0, 1, 2) as an order; they are arbitrary labels that change with the seed. Name them by their profile.
- Variants: add
mean_ticketas a fourth variable and see whether the partition changes; tryAgglomerativeClusteringand compare; compute the annual value that would be lost if 10 % of the regulars became occasional (that is the argument for Diego).
- Practice 5: cost-based threshold tuning and a fairness check across zones
Reminder (04-05 section 5, 02-04 sections 9-10). The threshold is not 0.5 by decree: it is chosen by sweeping thresholds with business_cost (€5 per false alarm, €30 per unanticipated return) and adding a human review band. And every decision about people is checked by group: flag-rate parity with the four-fifths rule (ratio of the least-flagged group to the most-flagged ≥ 0.8). postcode_zone has no influence on the label by construction, so a well-built model should come out at parity.
Statement. With the reference pipeline trained on the split from 04-05: (a) compute the table of FP, FN, flagged and cost for thresholds from 0.05 to 0.70 with three cost structures: (€5, €30), (€8, €20) and (€5, €50); at which threshold is the minimum of each? (b) With threshold 0.2, compute the flag rate per zone on the test set, the impact ratio and a bootstrap interval for the ratio; repeat with the cross-validated predictions over the 2,999 orders. (c) Look at the zone coefficients in the model, and at the false negative and false positive rates per zone. (d) Simulate a scenario in which zone C had many more new customers and see what happens to the ratio: how would you detect it and what would you do if the result were not at parity?
Solution
X, y = prepare_orders(dirty_orders(generate_orders_ml(3000, 42), 42))
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
pipe = Pipeline([("prep", build_preprocessing()), ("model", LogisticRegression(max_iter=1000))]).fit(Xtr, ytr)
prob = pipe.predict_proba(Xte)[:, 1]
rows = []
for t in np.round(np.arange(0.05, 0.71, 0.05), 2):
m = confusion_matrix(yte, (prob >= t).astype(int))
rows.append({"threshold": t, "FP": m[0, 1], "FN": m[1, 0], "flagged": int((prob >= t).sum()),
"cost 5/30": business_cost(m), "cost 8/20": business_cost(m, 8, 20), "cost 5/50": business_cost(m, 5, 50)})
table = pd.DataFrame(rows); print(table.to_string(index=False))
for c in ["cost 5/30", "cost 8/20", "cost 5/50"]:
print(c, "-> minimum", table[c].min(), "€ at threshold", table.loc[table[c].idxmin(), "threshold"])
def flag_rates(zones, flagged):
t = pd.Series(flagged).groupby(np.asarray(zones)).mean()
return t.round(3).to_dict(), round(float(t.min() / t.max()), 3)
te = Xte.assign(prob=prob, y=yte.values, flagged=(prob >= 0.2).astype(int))
print("Test 750, threshold 0.2:", flag_rates(te["postcode_zone"], te["flagged"]), "| n per zone:", te.groupby("postcode_zone").size().to_dict())
print("Actual return rate per zone (test):", te.groupby("postcode_zone")["y"].mean().round(3).to_dict())
rng = np.random.default_rng(0)
ratios = [flag_rates(*te.iloc[rng.integers(0, len(te), len(te))][["postcode_zone", "flagged"]].T.values)[1] for _ in range(300)]
print("Bootstrap ratio 2.5-97.5 %:", np.percentile(ratios, [2.5, 97.5]).round(3))
prob_cv = cross_val_predict(Pipeline([("prep", build_preprocessing()), ("model", LogisticRegression(max_iter=1000))]), X, y, cv=cv, method="predict_proba")[:, 1]
print("CV 2999, threshold 0.2:", flag_rates(X["postcode_zone"], (prob_cv >= 0.2).astype(int)))
print("FNR per zone:", te[te.y == 1].groupby("postcode_zone")["flagged"].apply(lambda s: round(1 - s.mean(), 3)).to_dict(),
"| FPR per zone:", te[te.y == 0].groupby("postcode_zone")["flagged"].mean().round(3).to_dict())
names, coef = pipe["prep"].get_feature_names_out(), pipe["model"].coef_[0]
print("Zone coefficients:", {n.split("_")[-1]: round(float(c), 3) for n, c in zip(names, coef) if "zone" in n})
X_sim = Xte.copy(); in_c = X_sim["postcode_zone"] == "C" # scenario: 60 % of C becomes new customers
X_sim.loc[in_c, "new_customer"] = np.where(rng.random(in_c.sum()) < 0.6, 1, X_sim.loc[in_c, "new_customer"])
print("Proxy scenario (more new customers in C):", flag_rates(X_sim["postcode_zone"], (pipe.predict_proba(X_sim)[:, 1] >= 0.2).astype(int))) threshold FP FN flagged cost 5/30 cost 8/20 cost 5/50
0.05 347 8 462 1975 2936 2135
0.10 218 18 323 1630 2104 1990
0.15 149 25 247 1495 1692 1995
0.20 112 35 200 1610 1596 2310
0.30 60 56 127 1980 1600 3100
0.50 17 80 60 2485 1736 4085
0.70 3 107 19 3225 2164 5365
cost 5/30 -> minimum 1495 € at threshold 0.15
cost 8/20 -> minimum 1596 € at threshold 0.2
cost 5/50 -> minimum 1990 € at threshold 0.1
Test 750, threshold 0.2: ({'A': 0.235, 'B': 0.288, 'C': 0.29}, 0.813) | n per zone: {'A': 310, 'B': 257, 'C': 183}
Actual return rate per zone (test): {'A': 0.129, 'B': 0.175, 'C': 0.208}
Bootstrap ratio 2.5-97.5 %: [0.579 0.93 ]
CV 2999, threshold 0.2: ({'A': 0.269, 'B': 0.268, 'C': 0.269}, 0.994)
FNR per zone: {'A': 0.225, 'B': 0.311, 'C': 0.316} | FPR per zone: {'A': 0.156, 'B': 0.203, 'C': 0.186}
Zone coefficients: {'A': 0.017, 'B': 0.073, 'C': -0.094}
Proxy scenario (more new customers in C): ({'A': 0.235, 'B': 0.288, 'C': 0.475}, 0.495)Reading:
- Threshold. With a step of 0.05 the minimum for (5, 30) is at 0.15 (€1,495), not at 0.2 (the table in 04-05 only looked at tenths): 247 flagged out of 750, a third. With (8, 20) the minimum moves to 0.2 (false alarms cost more and returns less, so fewer are flagged), and with (5, 50) it drops to 0.1 (flagging is cheap and missing is very expensive). The threshold is a business decision parameterised by the costs; the model does not change. Given Diego's review capacity (≤ 300/day out of 3,000 orders, 10 %), 0.15 would flag too many and the 0.2-0.5 band from 08-01 remains the operational one.
- Parity on the test set: 0.235 / 0.288 / 0.290 → ratio 0.813, just above 0.8, and with a bootstrap interval from 0.58 to 0.93: 750 orders are not enough to claim either parity or disparity. With the 2,999 cross-validated orders the rates are 0.269 / 0.268 / 0.269 (ratio 0.994): the model is at parity, as it had to be by construction, and the deviation on the test set is sampling noise (in that test set, zone C happened by chance to have more actual returns, 0.208 versus 0.129 in A). The zone coefficients are ≈ 0 (0.017 / 0.073 / −0.094): the model does not use the zone.
- How you would detect it if it were not, in order of urgency: (1) a flag-rate ratio < 0.8 persistently in large samples (which is why 08-01 measures it every month on accumulated data, not on each batch); (2) FNR/FPR differences between zones (a zone where more returns slip through or that is bothered more), even if the flag rates match; (3) non-zero coefficients or importances for the zone, or for its proxies. The simulated scenario shows that last point: if zone C had many more new customers, the flag rate would rise to 0.475 and the ratio would fall to 0.50 even though the model still does not look at the zone, because
new_customeracts as a proxy. Then the question is no longer technical but the one from 02-04: is it legitimate for new customers to be reviewed more (yes: they really do return more, 33 % versus 9 %) and are we sure the concentration in C is not down to something else (income, origin)? The possible answers: a wider human review band for the affected group, per-group thresholds (with legal advice), removing the proxy if its effect is not justified, and always documenting it in the model card.
Feedback
- Typical mistake: concluding "there is bias" (or "there is none") from a single small sample; the impact ratio has variance and needs an interval and a sample size. Another one: comparing flag rates without looking at the actual return rates per group (flag parity and error parity are not the same thing; which one is required depends on the case).
- The zone having a coefficient ≈ 0 does not guarantee parity if there are proxies; parity holding does not guarantee that decisions are fair one by one. Group metrics are necessary, not sufficient.
- Variants: compute the cost with a review band (€3 per review and 100 % reviewer accuracy) for bands 0.15-0.5 and 0.2-0.5; repeat the analysis by
categoryand bypayment_method(cash on delivery is a candidate socio-economic proxy); writeparity_ok(rates, minimum=0.8)from 02-04 and add it as a test innovamarket_ai/tests/.
Common Mistakes and Tips
- Comparing figures from different validations. Everything you compare must be measured on the same partition or the same cross-validation; and small differences are compared fold by fold (paired), not by loose means.
- Confusing technical and business metrics. The AUC can go up without the cost going down (practice 2) and the optimal threshold changes with the costs (practice 5). Always report both.
- Shuffled time series and lags from the future.
TimeSeriesSplitandshift(k)with k > 0; check by hand that the row for week t contains only data from weeks < t. - Scaling (or not) without thinking. k-means and logistic regression need scaling; trees do not. And the scaler is fitted on training data only (inside the
Pipeline). - Small samples and big conclusions. 750 orders are too few to measure parity; 48 weeks are too few to fit lags; one split is too little to choose features. When the conclusion matters, ask for more data or more repetitions.
- Tip: every practice should end in a sentence for Diego: "the current model stays; the new feature does not save euros", "the linear model with calendar, MAE 20, retrain every quarter", "three segments and one action per segment", "threshold 0.2 with a band, at parity across zones with n = 2,999". If you cannot write it, the practice is not finished.
Conclusion
Five practices and five decisions: a batch of 3,003 rows ends up at 2,999 after removing 3 duplicates and a €99,999 amount, with the nulls imputed in the pipeline and the leak out; an interaction feature raises the AUC from 0.8365 to 0.8409 consistently but does not save euros at threshold 0.2, so the production model does not change; in demand, the linear model with calendar (MAE 22.6 over four quarters) beats the baselines (27-30), the lags (which add nothing with independent noise and destabilise the first block) and gradient boosting (40.4, which neither extrapolates the trend nor isolates Black Friday), and removing the Black Friday variable costs 214 units of error in a single week; k-means finds three segments (silhouette 0.572) with a business action for each; and the optimal threshold depends on the costs (0.15 with 5/30, 0.2 with 8/20, 0.1 with 5/50) while parity across zones, ambiguous with 750 orders (ratio 0.81, interval 0.58-0.93), is clear with 2,999 (0.99), although a proxy could break it without the model looking at the zone.
In 09-03, Neural Network Projects, you will go up a level of complexity with PyTorch: the returns MLP with an architecture and learning-rate search, the photo CNN with a third class and data augmentation, a GRU on the demand series that you will compare with this lesson's linear model, and a review classifier with embeddings trained from scratch versus the bag of words. The underlying question will be the one from 05-03: when does the network pay off against a well-built classical model?
Fundamentals of Artificial Intelligence (AI)
Module 1: Introduction to Artificial Intelligence
Module 2: Basic Principles of AI
- Fundamental Concepts: Agents, Environments and Rationality
- Types of Artificial Intelligence
- Data as the Raw Material of AI
- Ethics and Considerations in AI
Module 3: Algorithms in AI
- Introduction to Algorithms
- Search Algorithms
- Adversarial Search: Games and Minimax
- Optimization Algorithms
Module 4: Machine Learning
- Basic Concepts of Machine Learning
- Types of Machine Learning
- Data Preparation and Feature Engineering
- Machine Learning Algorithms
- Model Evaluation and Validation
- Overfitting, Regularization and Hyperparameter Tuning
Module 5: Neural Networks and Deep Learning
- Introduction to Neural Networks
- Neural Network Architecture
- How a Network Learns: Gradient Descent and Backpropagation
- Deep Learning and Its Applications
- Transformers, Large Language Models and Generative AI
Module 6: Logic and Expert Systems
- Logic in AI
- Expert Systems
- Reasoning under Uncertainty: Probability and Bayesian Networks
- Applications of Expert Systems
Module 7: Tools and Programming Languages in AI
- Programming Languages for AI
- Scientific Python: NumPy, pandas and Matplotlib
- Popular Tools and Libraries
- Development Environments
Module 8: Projects and Case Studies
Module 9: Exercises and Practice
- Algorithm Exercises
- Machine Learning Practice
- Neural Network Projects
- Capstone Project: from Idea to Prototype