Fourth project and a new enemy: extreme class imbalance. You are going to build a fraudulent card transaction detector where only 1% of the cases are fraud — the scenario where accuracy deceives, thresholds are decided in euros and the metrics from module 6 earn their keep. The problem will feel familiar twice over: it is the spam detector from Bayes' theorem in 02-05 with money on the line, and the supervised cousin of the order anomalies you hunted with DBSCAN in 05-04 — except here we do have historical fraud labels, so we can train classifiers. We will use a synthetic dataset generated right in the code (real fraud datasets are confidential by nature; the classic public one on Kaggle comes anonymized through PCA), which also gives us full control over the experiment.

1. Problem definition, costs and target metric
2. Generating the synthetic dataset
3. EDA with imbalanced classes
4. Why accuracy lies at 99%
5. Strategies against imbalance
6. Modeling and comparison with CV
7. The precision-recall curve and AUC-PR
8. The optimal threshold, in euros
9. Final evaluation on the test set
10. Conclusions and the link to production

• Problem: binary classification — is this transaction fraudulent (1) or legitimate (0)?
• The asymmetry that governs everything: the two errors do not cost the same. An undetected fraud (false negative) costs the defrauded amount plus handling, say €150 on average; blocking a legitimate transaction (false positive) costs the manual review plus customer friction, say €5. This cost matrix, as you learned in 06-04 "ROC curve and AUC", will decide the threshold.
• Metrics: precision, recall and F1 for the fraud class (06-02 "Evaluation metrics"), the precision-recall curve with its AUC-PR as the model-comparison metric, and the total cost in euros as the final business metric. Accuracy is banned, and you are about to see why.

make_classification generates a classification problem with controlled structure; with weights=[0.99, 0.01] we impose the imbalance. Then we dress the abstract features with realistic names and scales so we can reason like analysts:

What we are simulating and what we are not: the generated columns are abstract Gaussian combinations that we name so we can reason with them — the structure of the problem (few informative features, 1% positives, some label noise) is faithful to reality; the values are not literal hours or euros. In a real case, features like dist_home_km or ratio_avg_amount would come out of the feature engineering from 03-06 applied to the cardholder's history — exactly the way you built MercaFresh's RFM.

We split with stratification, essential with 1% positives (06-01):

With imbalance, global histograms are useless: the frauds vanish under the legitimate mass. The right tool is comparing distributions per class:

The density=True is the key: it normalizes each histogram so you can overlay 370 frauds on 37,000 legitimate transactions. Look for the features where the two bells separate (the informative ones) and those that overlap completely (the ones that will barely contribute). This very plot, comparing "today" against "the historical record", will be your drift detector in production — we come back to it at the end.

The experiment that proves it, with DummyClassifier playing the role of metrics con artist:

A model that says "everything is legitimate" is right 99% of the time and does not catch a single fraud: total cost = every fraud paid in full. It is the canonical example from 06-02: when classes are imbalanced, accuracy mostly measures the share of the majority class. From here on, precision/recall/F1 for the fraud class, and euros.

Three families, from simplest to most invasive:

1. class_weight="balanced": the model weights every fraud example ~99 times more in its loss function. It does not touch the data, it is one line, and it is almost always the first thing to try.
2. Moving the threshold: train normally, but do not cut at 0.5 — cut wherever the cost matrix says (06-04). It is the most honest strategy because it separates the model (probabilities) from the decision (business threshold).
3. Resampling: changing the training set itself. Oversample the minority (duplicate frauds, or generate interpolated synthetics with SMOTE, available in the imbalanced-learn library) or undersample the majority. In code we illustrate it with sklearn's resample:

Golden rule: resampling is applied to the training set only, never to the test set (we want to evaluate against the 1% reality) and, if combined with CV, inside each fold — resampling before splitting duplicates the same frauds across train and validation: the leak from 06-01, imbalanced edition.

Three models from the course with class_weight where it exists, compared with stratified CV (06-03) on AUC-PR (average_precision), recall and precision:

Ballpark results:

Notice that the @0.5 columns are almost anecdotal: they depend on a threshold we have not decided yet. AUC-PR, by contrast, evaluates the quality of the probability ranking across all thresholds at once — that is why it is the comparison metric. And a note on ROC-AUC: with 1% positives it usually comes out sky-high (0.95+) for everyone, because the abundant true negatives inflate it; the precision-recall curve, which ignores true negatives, is far more demanding and realistic here. It is the final nuance that was left as a note in 06-04.

With HistGradientBoosting chosen, we draw its curve using CV probabilities (without touching the test set):

The gray line is humbling and necessary: a random classifier has precision = 1% on this problem. Everything above that line is the model's merit. The curve is a menu of trade-offs: each point is a possible threshold — more recall (catching more frauds) paid for with precision (more false alarms). Which one to pick? The one that minimizes euros.

We apply the cost matrix to every candidate threshold, as in 06-04 but with currency:

With a false negative 30 times more expensive than a false positive, the optimal threshold falls well below 0.5 (typically between 0.05 and 0.15): it pays to review plenty of legitimate transactions in order to catch more frauds. Compare the optimal threshold's cost against two references: the "default" 0.5 threshold and the "block nothing" policy (= every fraud paid). The difference, in euros, is the argument this project defends itself with in front of the business — far more persuasive than any F1.

We train on the full training set, apply the chosen threshold, and open the test set exactly once:

The final report must tell three numbers: fraud recall (what fraction of the fraud we catch), precision (what fraction of the alerts are real — that is the review team's workload) and the savings in euros versus doing nothing. If they match the CV estimates reasonably well, the protocol was clean.

• Splitting without stratifying. With 1% positives, a random split can leave the test set almost fraud-free and the metrics turn into noise. stratify=y, always.
• Resampling before splitting. Duplicating frauds and then dealing out train/test puts copies of the same fraud on both sides: inflated, illusory metrics. Resample only within the training set, and inside each fold if there is CV.
• Showing off ROC-AUC. A 0.95 ROC-AUC with 1% positives can coexist with 8% precision. Report AUC-PR and the precision-recall curve.
• Setting the threshold with the test set. The threshold is a hyperparameter of the decision: it is chosen with CV on the training set. Picking it by looking at the test set is overfitting the final decision.
• Treating the cost matrix as eternal. The €150/€5 are business estimates that change (average amounts, cost of the review team). Document the figures and recompute the threshold when they change.

1. IsolationForest, the unsupervised cousin. Train sklearn.ensemble.IsolationForest ignoring the labels and check how many real frauds show up among its anomalies, connecting with the approach from 05-04 "DBSCAN clustering". Hint: use contamination=0.01; it is the tool for the cold start, when there are no historical labels yet.
2. Sensitivity to the cost matrix. Recompute the optimal threshold for several scenarios (FN = €50, €300; FP = €1, €20) and plot how it moves. Hint: if the optimal threshold barely changes across plausible scenarios, your decision is robust; if it dances around, the business must refine its estimates before deploying.
3. Temporal drift. Simulate fraud evolving: generate a second dataset with a different random_state and slightly more class_sep or an offset added to two features, and evaluate the old model on it. Hint: measure how much recall and AUC-PR drop and which per-class histograms give the change away — you are reproducing the cycle from 08-03 in miniature.

This project has taught you how to work when the class that matters is a needle in a haystack: stratify everything, distrust accuracy (and even ROC-AUC), compare models by AUC-PR, and above all separate the model from the decision — the algorithm supplies the probabilities, the euros supply the threshold, exactly the method from 06-04 taken to its logical conclusion. You have also seen the arsenal against imbalance (class weights, threshold, resampling) and its traps. And there remains the final warning, which connects with 08-02 "Deploying models to production" and 08-03 "Model maintenance and monitoring": fraud is the most drift-prone problem in existence, because on the other side there are adversaries adapting to your detector — the model that saves thousands of euros today degrades within months, and histogram monitoring plus periodic retraining are not optional, they are part of the product. With this you close the block of supervised projects; in the last project we go back home, to MercaFresh, to close the circle with unsupervised learning: segmenting customers from start to finish.