The MercaFresh dataset is now clean and gap-free, but "correct" is not the same as "useful". Customer total spend has a very long right tail (remember the histograms from module 2), signup dates are just calendar labels no model knows how to interpret, and the richest information — individual orders — sits at order level when the churn model needs one row per customer. In this lesson you'll learn to reshape variables: mathematical transformations that correct skewness, discretization into bins, decomposition of dates into meaningful components, and aggregations that turn thousands of orders into a per-customer profile. We'll close with scikit-learn pipelines, the tool that organizes all these steps into a reproducible chain.

1. Why transform: when the shape of the data gets in the way
2. Mathematical transformations: log, root and powers
3. Box-Cox and Yeo-Johnson: the transformation that tunes itself
4. Binning: discretizing continuous variables
5. Dates: from dead text to useful components
6. Aggregations per entity: from orders to customers
7. Pipelines: chaining transformations in order

Many models perform better — or outright assume — that variables have reasonably symmetric distributions and roughly linear relationships. But business variables rarely come that way:

• Amounts and spending: most MercaFresh customers spend little and a few spend enormously. That's the positive skew you saw in 02-01 (mean > median) and a distribution close to the exponential from 02-02.
• Counts: number of orders, of visits, of incidents — long right tails.
• Dates: a datetime says nothing by itself; what predicts is what the date implies (weekend? how long ago?).
• Wrong granularity: we have 80,000 orders, but churn is predicted per customer.

Transforming means changing the representation of the data without changing its meaning, so the patterns are easier for the model to see.

The logarithm compresses large values far more than small ones, so it "tucks in" the right tail of a skewed distribution:

Details that matter:

• We use np.log1p(x) (which computes log(1 + x)) instead of np.log(x) because the logarithm of 0 is minus infinity, and MercaFresh has customers with a spend of 0. log1p handles the 0 gracefully (log1p(0) = 0).
• skew() measures asymmetry numerically: ~0 is symmetric, > 1 is strong right skew. It's the numeric translation of what the histogram shows by eye.
• After the logarithm, the distribution looks like the normal from 02-02, and tools that assume normality (z-scores included) work well again.

The logarithm isn't the only option; there's a "ladder" ordered by aggressiveness:

Practical rule: try from least to most aggressive and check with skew() and the histogram which one leaves the distribution most symmetric without contorting it.

Instead of choosing by hand between root and logarithm, Box-Cox automatically finds the optimal exponent (called λ, lambda) that makes the variable as close to normal as possible. λ = 0 is equivalent to the logarithm; λ = 0.5, to the square root; λ = 1, to doing nothing.

Limitation: Box-Cox only accepts positive values. For variables with zeros or negatives (a points balance that can go negative, a change in spending), there is Yeo-Johnson, its generalization. In scikit-learn both live in PowerTransformer, with the advantage of the fit/transform pattern you already know from the previous lesson (it learns λ on train, applies it to new data):

(The reshape(-1, 1) turns the vector into a one-column matrix: scikit-learn transformers always expect 2D data, rows × columns.)

A readability warning: after transforming, the values are no longer in euros. A coefficient on log(spend) is interpreted in multiplicative terms, not additive ones. We gain model performance at the cost of less direct interpretation; it's wise to always keep the original variable alongside.

Discretizing (or binning) means converting a continuous variable into brackets: exact age into age groups, spend into low/medium/high levels. What for?

• Capturing simple non-linear relationships ("under-30s and over-65s buy differently").
• Robustness against outliers: the €15,400 customer falls into the "high" bracket and stops distorting things.
• Communication with the business: "high-spend customers" is easier to grasp than a coefficient.

The price: information is lost (everyone in the bracket becomes indistinguishable). It's a trade-off, not a free trick.

cut slices where you say: the boundaries carry business meaning (legal adulthood, retirement). Each interval includes its right edge by default: (18, 30], (30, 45], etc.

qcut slices by percentiles (the ones from 02-01): each bracket holds the same number of observations, even if its boundaries in euros come out "ugly". It's the natural choice for skewed variables: with equal-width cut, 90% of customers would land in the first bracket.

strategy accepts "uniform" (equal widths, like automatic cut), "quantile" (like qcut) and "kmeans". Its added value is the usual one: the boundaries are learned in fit and reapplied identically to new data, preventing the brackets from drifting every month.

In 03-01 we converted dates from text to datetime. Now we squeeze them: a model can't use "2024-06-05", but it can use the signals that date hides.

Why does this work? Because to predict demand or churn at MercaFresh, what matters is not June 5 itself, but that it was a Monday, that it was a holiday, or that it happened 80 days ago. Every extracted component turns calendar knowledge into a column the model can use.

Two nuances:

• dayofweek and month are numeric but cyclical: Sunday (6) is "next to" Monday (0), and December next to January. Many models handle them fine as categories (lesson 03-04); trigonometric sine/cosine encodings exist for order-sensitive models, and for now it's enough to know that nuance is there.
• The recency "reference date" must be a fixed date of the analysis (the dataset's cutoff date), not Timestamp.now(): otherwise the same code gives different results every day.

The churn model needs one row per customer, but the living information sits in the orders table. The solution is to aggregate: summarize each customer's history into statistics.

The syntax agg(new_name=("column", "function")) (named aggregation) produces columns with clean names in a single step. Look at what we've achieved: customer 101, with three orders spread between May and August, is now described by their frequency, their average ticket and — pure gold for churn — how many days they've gone without buying.

The last step is joining this profile to the clean customer table from the previous lessons:

With how="left" we also keep the customers with no orders (their aggregates will come out as NaN... which you already know how to handle from the previous lesson: here a 0 in num_orders and a no_orders indicator would be imputations with full business meaning). These aggregations are the prelude to the RFM analysis we'll complete in 03-06.

We've already chained several steps: impute, transform with logarithms, discretize... each with its fit on train and its transform on the rest. Doing it by hand, in order, without forgetting any step and without leaks, is fragile. scikit-learn offers two pieces to assemble it:

A Pipeline is a list of named steps executed in order: each step's output feeds the next. Calling fit_transform makes each step fit and transform in a chain; calling transform later on new data makes them all apply what they learned without relearning anything. A single object encapsulates the entire preprocessing.

Not every column needs the same treatment: numeric ones want imputation and logarithms; categorical ones, other techniques (those of the next lesson). ColumnTransformer applies its own treatment to each group of columns and stitches the results together:

Each tuple is (name, transformer, list_of_columns); "passthrough" lets columns through untouched. For now, keep the introductory idea: the complete preprocessing can live in a single object that is fitted with fit on training data and applied with transform to any future data. In the coming lessons we'll fill in the missing branches, and in module 6 the pipeline will prove its definitive value when combined with validation.

• Applying np.log to values with zeros. It produces silent -inf values that blow up the model later. Use np.log1p, and check beforehand that there are no negatives.
• Transforming and forgetting the back-transformation. If the model predicts log(spend), the prediction in euros is np.expm1(prediction). Document which variables are transformed.
• Choosing cut brackets that leave empty groups. Always check value_counts() after discretizing; a bracket with 3 observations contributes nothing. With skewed data, qcut is usually the better starting point.
• Using Timestamp.now() as the recency reference. The dataset must have a fixed, documented cutoff date, or the results change every day you run the code.
• Aggregating with mean on columns with outliers. Customer 107's average spend is dominated by their €15,400 order; consider also adding the per-customer median.
• Fitting transformers on the whole dataset. PowerTransformer and KBinsDiscretizer learn parameters: same leakage risk as imputation (03-02). Inside a Pipeline, this mistake becomes almost impossible to make — one more reason to use them.
• Tip: before and after every transformation, plot the histogram and look at skew(). Two lines of code that keep you from transforming blindly.

MercaFresh's delivery_incidents variable (number of delivery incidents per customer) takes the values [0, 0, 1, 0, 2, 0, 1, 9, 0, 3]. Compute its skewness with skew(), apply the ladder transformation you consider appropriate (justify which one, and why the plain logarithm would cause trouble) and recompute the skewness.

Split the order amounts [12, 18, 22, 25, 31, 38, 45, 60, 85, 240] into 4 levels with qcut and into 4 equal-width brackets with cut. Compare the value_counts() of both and explain which is more useful here and why.

Starting from the lesson's orders DataFrame, build the per-customer profile with a single aggregation: number of orders, median order amount and days since the last order (reference: 2026-08-24). Name the columns n_orders, median_amount and recency_days.

Exercise 1

With small counts and abundant zeros, sqrt or log1p are the natural choices; the pure logarithm is ruled out by the zeros.

Exercise 2

With cut, the 240 outlier stretches the range and leaves two brackets empty: useless. qcut distributes the observations by percentiles and produces balanced levels: it's the right choice for this skewed distribution.

Exercise 3

The median amount shields customer 107's profile from their giant order, and recency_days — the days without buying — will be one of the star variables of the churn model.

You've learned to reshape data so it reveals its patterns: correct skewness with logarithms, roots and the automatic Box-Cox/Yeo-Johnson transformations; discretize with cut, qcut and KBinsDiscretizer knowing what is gained and what is lost; turn dates into components with predictive power (weekday, holiday, recency); summarize the orders table into per-customer profiles with groupby().agg(); and chain it all in Pipeline and ColumnTransformer so the preprocessing is reproducible and leak-proof.

You'll have noticed that the ColumnTransformer left one branch empty: the categorical columns. The customer's city, the product category and the MercaFresh subscription plan are still text, and no model knows how to multiply by "Barcelona". In the next lesson we solve exactly that: encoding categorical variables.