In the last two lessons we analyzed each MercaFresh variable on its own: its shape, its center, its spread. But the interesting business questions are almost always about relationships: do customers who wait longest for their deliveries churn more? Do large orders use card payment more often? Covariance and correlation are the tools that quantify whether two variables move together, in which direction, and how strongly. Mastering them is key in Machine Learning: they guide feature selection, warn about redundancy, and act as the first radar for churn signals. You will also learn their most important limitation: correlation does not imply causation.

1. Covariance: the idea and its limitation
2. Pearson correlation: interpretation and ranges
3. Spearman correlation: when to use it
4. Correlation matrices and heatmaps with pandas
5. Correlation does not imply causation
6. Relevance to Machine Learning
7. MercaFresh case: which variables relate to churn?

The covariance between two variables X and Y measures whether they tend to deviate from their means in the same direction:

cov(X, Y) = Σ (xᵢ − x̄)(yᵢ − ȳ) / (n − 1)

The intuition behind the product (xᵢ − x̄)(yᵢ − ȳ):

• If when X is above its mean, Y is too (and vice versa), the products are positive → positive covariance.
• If when X goes up, Y tends to go down, the products are negative → negative covariance.
• If there is no pattern, positives and negatives cancel out → covariance close to 0.

• We generate the order amount as €3.2 per item plus normal noise: there is a real relationship by construction.
• pandas' .cov() computes the sample covariance (dividing by n−1, consistent with what we saw in 02-01).

The scale limitation. The covariance between items and amount comes out at ≈ 44... 44 what? Its units are "items × euros". If we expressed the amount in cents, the covariance would be multiplied by 100 without the relationship changing at all. The magnitude of the covariance is not interpretable on its own: only its sign is. We need a unit-free version: the correlation.

The Pearson correlation normalizes the covariance by dividing it by the product of the standard deviations:

r = cov(X, Y) / (sₓ · s_y)

The result is a unit-free number, always between −1 and +1, invariant to changes of scale:

• r = +1: perfect positive linear relationship (all points on an ascending line).
• r = −1: perfect negative linear relationship.
• r = 0: no linear relationship (some other kind of relationship may exist).

| Approximate |r| | Usual interpretation | |---|---| | 0.0 – 0.2 | Very weak or none | | 0.2 – 0.4 | Weak | | 0.4 – 0.6 | Moderate | | 0.6 – 0.8 | Strong | | 0.8 – 1.0 | Very strong |

(These thresholds are rough guides and depend on the domain: in marketing an r = 0.3 can be gold; from a physical sensor, disappointing.)

Two fundamental caveats about Pearson:

1. It only detects linear relationships. A perfect U-shaped relationship (for example, spending vs. age, peaking at middle age) can yield r ≈ 0.
2. It is sensitive to outliers. A single €900 hospitality order can inflate or sink the correlation. Remember the boxplot from lesson 02-01: inspect before computing.

Hence the mantra: always draw the scatter plot (plt.scatter(x, y) or df.plot.scatter(...)) before trusting a coefficient.

The Spearman correlation applies Pearson's formula to the ranks of the data (the position of each value once sorted) rather than to the raw values. It measures whether the relationship is monotonic (as X grows, Y consistently grows — or shrinks), even if it is not linear.

Use it when:

• The relationship looks monotonic but curved (e.g., spending grows with tenure, but with diminishing returns).
• There are strong outliers: ranks tame them (the €900 order simply becomes "the first one").
• One of the variables is ordinal (satisfaction low/medium/high): Pearson is not justified, Spearman is.

With a gentle curve both come out high, but Spearman better captures the monotonicity; with outliers or sharper curves, the gap between them widens, and that discrepancy is itself a diagnostic: if Spearman ≫ Pearson, suspect non-linearity or outliers.

With many variables, computing pairwise correlations by hand is unworkable. The correlation matrix computes them all at once:

Details of the code:

• df.corr() returns a symmetric matrix with ones on the diagonal (every variable correlates 1 with itself). It accepts method="pearson" (the default) or "spearman".
• imshow with cmap="coolwarm" and limits vmin=-1, vmax=1 paints positives red, negatives blue, and zeros white; fixing the limits matters so the colors stay comparable across analyses.
• The double loop writes the numeric value into each cell: a heatmap without numbers forces you to guess.
• If you have the seaborn library installed, sns.heatmap(matrix, annot=True, cmap="coolwarm", vmin=-1, vmax=1) does the same in one line.

How to read it: look for (1) intense cells off the diagonal — strong relationships — and (2) blocks of variables highly correlated with each other, which indicate redundant information.

The fact that X and Y move together does not prove that X causes Y. There are at least four possible explanations:

Examples:

• Common cause (confounder). At MercaFresh, ice cream sales correlate with gazpacho sales. Neither causes the other: summer causes both. If the demand ML model does not include the season, it will learn spurious relationships.
• Reversed direction. "Customers who contact support churn more." Does contacting support cause the churn? More likely the other way around: the problems that lead to churn also cause the support contact.
• Pure coincidence. With hundreds of variables, some will correlate strongly by sheer chance (there are whole collections of famous "spurious correlations", such as swimming pool drownings vs. films featuring a particular actor). The more pairs you look at, the more false discoveries: lesson 02-04 will provide tools to judge whether an observed relationship is statistically credible.

For predictive work, a stable correlation can be enough (if contacted-support predicts churn, it is useful even if it is not the cause). But to intervene ("will churn drop if we improve X?") you need causal reasoning or controlled experiments, such as the A/B test we will see in 02-04.

• Feature selection. The correlations of each variable with the target are a cheap first screening of which features look promising. Systematic feature construction and selection is developed in lesson 03-06.
• Redundancy detection (multicollinearity). If two input variables correlate at 0.97 (e.g., "amount with VAT" and "amount without VAT"), they carry almost the same information; keeping both destabilizes some linear models and muddies coefficient interpretation. For now, hold on to the idea: red blocks in the heatmap = pruning candidates; the details of their effects will come up when we cover regression and regularization (modules 4 and 7).
• Domain understanding. Before modeling, the correlation matrix is the fastest X-ray of the story your data tell, and it also catches information leakage (a variable "suspiciously" correlated with the target at 0.99 is usually a copy of it in disguise).

The team wants a first list of churn signals. Churn is binary (0/1); correlating a binary variable with a numerical one via Pearson is known as the point-biserial correlation, and pandas computes it with the same .corr():

Approximate output:

Reading:

• days_since_last_purchase is the strongest signal: the longer without ordering, the more likely the churn. A moderate positive correlation, very valuable in churn work.
• delivery_incidents adds signal: delivery problems push customers toward leaving.
• orders_per_month correlates negatively: frequency protects.
• avg_delivery_minutes barely correlates with churn directly... but it does cause incidents, which do correlate: an example of a causal chain that simple correlation cannot untangle.
• avg_order_spend contributes almost nothing: a candidate to drop as a churn predictor (though not from the business).

This table is exactly the kind of exploratory analysis that will feed the logistic regression in module 4 to predict churn; here we limit ourselves to measuring relationships.

• Interpreting the magnitude of the covariance. Only its sign is interpretable; for magnitudes, always use the correlation.
• Concluding causation from correlation. Before saying "X causes Y", rule out the reverse direction, confounders, and chance; ideally, design an experiment.
• Trusting r without looking at the scatter plot. Anscombe's quartet (four datasets with the same r and radically different shapes) is the classic reminder: draw before you conclude.
• Using Pearson with ordinal variables or extreme outliers. Spearman is the robust choice in both cases.
• Confusing r ≈ 0 with independence. Pearson only measures linear relationships; a perfect U gives r ≈ 0. If you suspect curves, look at the plot and try Spearman.
• Mass-mining correlations without skepticism. With 50 variables there are 1,225 pairs: by pure chance several will look "strong". Prioritize the ones that make business sense and validate with the techniques of the next lesson.
• Tip: sort the correlations with the target by absolute value (sort_values(key=abs)), as we did with churn: the sign matters for interpretation, but the strength is what sets priorities.

Without code: the covariance between MercaFresh's delivery_minutes and incidents is 3.1. (a) Can you claim the relationship is strong? (b) If we convert the minutes to seconds, what happens to the covariance? And to the Pearson correlation?

Generate two variables with a purely quadratic relationship: x = np.linspace(-3, 3, 200) and y = x**2 + normal noise (σ=0.5). Compute Pearson and Spearman between x and y. Explain the results and the practical lesson you draw.

With the customers DataFrame from the worked case (including the churn column), compute the Spearman correlation matrix and compare it with Pearson's for the pair (days_since_last_purchase, churn). Does it change much? Why might Spearman be a good idea with days_since_last_purchase?

Solution 1

• (a) No. Covariance has no interpretable scale: 3.1 "minutes × incidents" could correspond to a strong or a weak relationship depending on each variable's spread. You would need the correlation.
• (b) Converting minutes to seconds (×60) multiplies the covariance by 60. The Pearson correlation does not change at all: it is invariant to linear changes of scale, which is exactly its reason for existing.

Solution 2

Both come out close to 0, even though y depends entirely on x. Pearson fails because the relationship is not linear; Spearman does too, because it is not monotonic either (y goes down and then up). Lesson: no correlation coefficient replaces the scatter plot; r ≈ 0 means "no linear/monotonic relationship", not "no relationship".

Solution 3

The values will be similar (the simulated relationship is roughly monotonic). Even so, Spearman is defensible here because days_since_last_purchase comes from an exponential with a right tail (lesson 02-02): its outliers (customers inactive for 60+ days) influence the ranks less than the raw values, making the measure more robust.

You now know how to measure relationships between variables: covariance gives the sign, Pearson correlation adds a universal scale between −1 and +1 for linear relationships, and Spearman covers monotonic relationships, ordinal variables, and data with outliers. You have built correlation matrices and heatmaps with pandas, identified the variables most related to MercaFresh's churn and, above all, internalized that correlation does not imply causation. But one uncomfortable doubt remains: that r = 0.35 between inactivity and churn — is it a real signal, or could it be an artifact of this particular sample's randomness? Answering that question rigorously — quantifying the uncertainty of what we measure in a sample — is the job of statistical inference, the star of the next lesson.