The MercaFresh churn dataset is now clean, complete, transformed, encoded and scaled. In the previous lessons we repaired and adapted what was already there; in this final lesson of the module we take the leap that usually delivers the most performance: creating new information. Feature engineering means building variables that condense business knowledge — how long since a customer last bought, how much they're worth, whether their activity is growing or fading — so the model doesn't have to discover it on its own. There's a much-quoted saying in the profession: the data and the features set the ceiling of what can be achieved; algorithms merely get closer to or further from that ceiling. Here you'll learn to build business features (with the classic RFM), interactions and polynomials, and to select which ones to keep. We'll close by assembling the definitive churn dataset and the checklist for the whole module.

1. What feature engineering is and why it matters so much
2. Business features: RFM analysis on MercaFresh
3. Ratios and trends: squeezing what you already have
4. Interactions and polynomials
5. Feature selection: keeping what earns its place
6. The final MercaFresh churn dataset
7. Checklist for the complete preprocessing

A feature is each column the model receives as input. Feature engineering is the art of manufacturing new columns from the existing ones, incorporating domain knowledge. Why does it usually deliver more than switching algorithms?

• Models see little on their own. A logistic regression (04-02) only combines its inputs linearly: if the churn signal lives in the spend/order ratio and you give it spend and orders separately, it may never find it. Give it the ratio already computed, and the signal is served on a plate.
• Business knowledge isn't in the raw data. Any MercaFresh account manager knows that "a customer who used to order weekly and has gone 45 days without buying is at risk"; the orders table, on its own, doesn't say it — it has to be distilled into a column.
• It's the most profitable lever. Moving from a decent algorithm to a sophisticated one usually gains a few points; one good new feature can transform the problem. That's why experienced teams spend more time on features than on models.

We already have the raw material from the previous lessons: the per-customer aggregations from 03-03 were the first step. Now we turn them into a system.

The RFM framework is a classic of customer analytics that fits churn like a glove:

The business intuition: a customer who bought a lot, often and recently is healthy; one whose recency grows while their frequency drops is on their way out — exactly what the churn model must learn. Let's build it from the orders table:

Three tricks of the trade:

• The fixed cutoff date (not now()) makes the calculation reproducible, as we saw in 03-03.
• The clip(lower=1) avoids divisions by zero for one-day-old customers: new features can also manufacture impossible values, and the validation rules from 03-01 apply to them just the same.
• Every RFM feature has a direct business reading: when the model says "recency is what weighs most", the retention team will know exactly what to do. Good features are also a communication channel with the business.

A very common variant is converting R, F and M into scores from 1 to 5 with qcut (the binning from 03-03!) and talking about "5-5-5 customers" or "1-1-3"; the fine-grained segmentation of these profiles is something we'll do with clustering in module 5 and in project 09-05.

A ratio relates two magnitudes and often says more than either one alone:

inactivity_ratio is a gem for churn: 60 days without buying is alarming for a weekly customer (high ratio) and normal for a quarterly one (low ratio). The ratio encodes that context; the loose columns don't.

Churn is a process, not an instant: before leaving, the customer cools off. A trend feature captures it by comparing their recent activity with the previous period:

A customer with trend = 0.25 (from 8 orders down to 1) is a textbook churn candidate even if their historical spend is high. Notice how the earlier lessons resurface: fillna(0) is an imputation with business meaning (03-02) and the +1 is the same stability trick as log1p (03-03).

Sometimes the signal lives in the combination of two variables: the effect of spend on churn may depend on the plan (losing €50/month from a basic customer is not the same as from a premium one). An interaction is simply the product of two features; a polynomial also adds powers (x², x³) to capture curvature.

PolynomialFeatures generates them mechanically:

From 2 columns come 5: the originals, their squares and their product. Two warnings and one preference:

• Combinatorial explosion: with 20 features and degree=2 you get 230 columns; with degree=3, more than 1,700. interaction_only=True restricts it to products, but it still grows fast.
• More mechanically manufactured columns = more risk of the model memorizing noise (the overfitting we'll formalize in 06-05) and more need for selection (next section) or regularization (07-01).
• A craftsman's preference: rather than polynomials in bulk, a few interactions chosen with your head. recency_days * orders_per_month ("how much silence a habitually active customer is accumulating") is worth more than twenty blind products.

Creating features is easy; the danger is ending up with 80 columns of which 50 are noise, redundancy or dead weight. More features is not always better: they make training more expensive, hinder interpretation and give the model more rope to fit chance. Three introductory filters, from simplest to most formal:

A column that barely varies can't distinguish anyone. If is_active_customer is 1 for 99.5% of the rows, it contributes nothing:

(Careful: the threshold depends on the scale, so it's applied to comparable data — another point for the scaling from 03-05.)

Two uses of the correlation matrix from 02-03, now as a selection tool:

• Against the target: features with ~0 correlation with churn are candidates to go (with the caveat from 02-03: Pearson only sees linear relationships).
• Among themselves: two features with a 0.95 correlation between them (as total_spend and num_orders usually have) are nearly the same information twice; keeping one simplifies without losing signal.

It's literally the ranking of correlations with churn we built at the end of 02-03, now applied to deciding which columns live.

SelectKBest scores each feature with a statistical test against the target and keeps the k best:

f_classif is, in essence, a hypothesis test like those in 02-04: does this feature's mean differ between the customers who leave and those who stay? High score = clear difference = promising feature. It's a filter method (it evaluates each feature in isolation, ignoring combinations): fast and useful as an initial sieve, not as a final verdict. And like everything that learns from the target, it's fitted on training data only — at this point in the module, the reflex should be automatic.

More sophisticated families exist (wrapper selection, PCA as dimensionality reduction in 05-03, Lasso regularization as implicit selection in 07-01); to close out preprocessing, these three filters suffice.

Time to assemble the whole module's work into the definitive table, one row per customer:

And the complete preprocessing, as a single executable object — the ColumnTransformer we've been filling in lesson by lesson, now with all its branches:

This object is the executable summary of the module: a single fit on training data learns medians, lambdas, quartiles and categories; a single transform prepares any future customer. This dataset — these features, this preprocessor — is the one we'll use conceptually for the rest of the course whenever we talk about MercaFresh churn.

The module's checklist, in order of application:

1. Audit (03-01): info(), describe(), nunique(); understand what each row represents.
2. Clean (03-01): exact and logical duplicates; validity rules for impossible values; normalize text and categories; fix types (to_numeric, to_datetime); diagnose outliers (error or reality?).
3. Handle nulls (03-02): diagnose the mechanism (MCAR/MAR/MNAR); drop only when safe; impute (median by default, by groups if there's structure); missingness indicators if the absence is informative.
4. Transform (03-03): correct skewness (log1p, Yeo-Johnson); discretize when useful; decompose dates; aggregate per entity to the problem's granularity.
5. Create features (03-06): RFM, ratios, trends, interactions chosen with business judgment.
6. Encode (03-04): ordinal with explicit order for real ordinals; one-hot for nominals; extreme care with target encoding.
7. Scale (03-05): according to the algorithm; RobustScaler if there are legitimate outliers.
8. Select (03-06): near-zero variance, redundancy by correlation, SelectKBest as a sieve.
9. Cross-cutting: every step that learns parameters runs fit on training data only; encapsulate in Pipeline/ColumnTransformer; document every decision.

(The 5→6→7 order is no accident: features are created on readable values, then encoded, and scaling goes last because every new column needs a scale too.)

• Creating features using the future. If churn is measured 90 days ahead, no feature may be computed with data past the cutoff date: it's data leakage in its temporal version, the subtlest of all.
• Manufacturing hundreds of mechanical features and not one business feature. PolynomialFeatures(degree=3) is no substitute for a conversation with MercaFresh's retention team. The best features are born from business questions.
• Not validating the new features. A ratio can generate infinities (division by zero) or absurd values: run every new feature through the same audit as in 03-01 (describe(), histogram).
• Keeping twin features. total_spend, num_orders and avg_order_spend together are partially redundant: review the correlation matrix between features, not just against the target.
• Selecting features on the whole dataset. SelectKBest looks at the target: if it's fitted with data that will later be test, the evaluation is contaminated. fit on train, always.
• Not documenting the data dictionary. Three months from now nobody will remember how trend was computed. Keep a table of feature → definition → formula → lesson/date.
• Tip: every time a feature occurs to you, first write in one sentence the business story it tells ("how much silence an active customer is accumulating"). If you can't formulate the sentence, it's probably not a good feature either.

Propose (without code) three new features for the MercaFresh demand problem (predicting how many units of each product will sell tomorrow), inspired by the techniques of this lesson. For each one, write the business sentence it tells.

With the lesson's rfm table, create the feature value_at_risk = avg_order_spend * orders_per_month * inactivity_ratio and explain what it measures in business terms. Then compute its Spearman correlation with recency_days and reason whether you expected that sign.

You have 12 numeric features and the target churn. Write the code that: (a) removes those with variance < 0.01; (b) from the remainder, selects the 6 best with SelectKBest and f_classif; (c) prints which ones survived. State which subset of the data you would run the fit calls on.

Exercise 1

Guideline answers (many are valid):

• avg_sales_last_4_weeks (temporal aggregation): "how much has sold lately is the best starting point for tomorrow".
• is_pre_holiday (calendar component, 03-03): "on the eve of a holiday, fresh-produce purchases spike".
• price_vs_category_ratio (ratio): "a product temporarily cheaper than its category steals sales from its substitutes".

What matters: each feature has a clear formula and a one-sentence business story.

Exercise 2

What it measures: the customer's usual monthly spend (ticket × frequency) weighted by their degree of inactivity — in other words, how many euros per month are at risk of being lost. A valuable, very inactive customer scores high; a cheap or fully active one, low. It's a prioritization feature: whom to call first. The correlation with recency_days comes out positive, as expected: recency enters the numerator of inactivity_ratio, so the more days without buying, the more value at risk — Spearman (02-03) is the right choice because the relationship is monotonic but not linear.

Exercise 3

Additional reflection exercise: why would SelectKBest on the whole dataset contaminate the evaluation? Because it scores each feature by its relationship with the target; if it sees the test labels, the resulting selection is optimized for test too, and the subsequent evaluation no longer measures real generalization.

You've closed the module with its most creative lesson: what feature engineering is and why it sets the model's ceiling, how to build business features with the RFM framework (recency, frequency, monetary value), ratios and trends that capture a customer cooling off, interactions and polynomials in moderation, and three selection filters — near-zero variance, correlation and SelectKBest — to keep what earns its place. And above all, you've assembled the result of the six lessons into a concrete product: the final MercaFresh churn dataset and its executable preprocessor, with the nine-step checklist as the map of the whole module.

That dataset is no longer a dirty table exported from a CRM: it's a numeric matrix, complete, scaled and loaded with business knowledge, ready for training. In module 4 the moment we've spent three modules preparing finally arrives: supervised learning algorithms, starting with linear regression — the simplest model and the best place to truly understand what it means for a machine to learn from MercaFresh's data.