The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Statistical inference

Next ►
►

The previous lesson ended with an uncomfortable question: is that 0.35 correlation between inactivity and churn real, or a mirage of the sample? Statistical inference is the discipline that answers that kind of question: how to go from what we see in a sample to reliable statements about the population (a distinction we defined in 02-01), quantifying the uncertainty at every step. In this lesson you will learn to estimate with confidence intervals, understand the central limit theorem through simulation, and run hypothesis tests (t-test and chi-square) with scipy, applying them to the A/B test of MercaFresh's new page. These tools will stay with you for the whole course: comparing two ML models is, at heart, comparing two samples of results.

Contents

  1. Point and interval estimation
  2. The central limit theorem (with a simulation)
  3. Hypothesis testing: the logic
  4. The p-value and type I and type II errors
  5. The t-test in practice: MercaFresh's A/B test
  6. The chi-square test for categorical variables
  7. Connection to Machine Learning

Point and interval estimation

A point estimate summarizes a population parameter with a single number computed from the sample: "the average spend per order is €47.20". Useful, but deceptively precise: another sample would have given 46.80 or 47.90.

An interval estimate adds the uncertainty: "the average spend is between €45.90 and €48.50 with 95% confidence". That range is the confidence interval (CI).

The key ingredient is the standard error (SE): the standard deviation of the estimator, not of the data. For the sample mean:

SE = s / √n

where s is the sample standard deviation and n the sample size. Note the √n: to halve the error you need four times as much data. The 95% CI for the mean (with reasonably large n) is approximately:

x̄ ± 1.96 · SE

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

# Sample: amounts of 400 MercaFresh orders
amounts = rng.gamma(shape=4.0, scale=12.0, size=400)

mean = amounts.mean()
se = amounts.std(ddof=1) / np.sqrt(len(amounts))

# 95% CI using Student's t distribution (correct for any n)
ci = stats.t.interval(0.95, df=len(amounts) - 1, loc=mean, scale=se)
print(f"Sample mean: {mean:.2f} EUR")
print(f"95% CI: ({ci[0]:.2f}, {ci[1]:.2f}) EUR")

Approximate output:

Sample mean: 48.06 EUR
95% CI: (45.72, 50.41) EUR
  • std(ddof=1) forces the sample deviation (n−1) because amounts is a NumPy array (remember the difference in defaults covered in 02-01).
  • stats.t.interval uses Student's t distribution, a bell with slightly heavier tails than the normal that corrects for the extra uncertainty of estimating s; with n = 400 it is nearly identical to using ±1.96.

Correct interpretation of the 95%: if we repeated the sampling many times and computed a CI each time, 95% of those intervals would contain the true population mean. It is not exactly "there is a 95% probability that μ lies in this interval" (μ is fixed; the interval is what's random), even though in practice people read it that way informally.

The central limit theorem (with a simulation)

Why can we use bell curves (normal or t) for the mean, if MercaFresh's amounts are skewed with a right tail? Because of the most important result in statistics:

Central limit theorem (CLT): the distribution of the mean of n independent observations approaches a normal as n grows, whatever the original distribution of the data (as long as it has finite variance). Its center is μ and its deviation is σ/√n.

Instead of proving it, let's watch it happen:

import matplotlib.pyplot as plt

rng = np.random.default_rng(0)

# HIGHLY skewed population: exponential amounts with mean 40 EUR
population = lambda size: rng.exponential(scale=40, size=size)

fig, axes = plt.subplots(1, 4, figsize=(14, 3.2), sharey=False)

# Panel 0: the original distribution (no bell whatsoever)
axes[0].hist(population(10_000), bins=60, color="gray")
axes[0].set_title("Original data\n(exponential)")

# Panels 1-3: distribution of the MEAN of samples of size n
for ax, n in zip(axes[1:], [5, 30, 200]):
    means = np.array([population(n).mean() for _ in range(5_000)])
    ax.hist(means, bins=60, color="steelblue")
    ax.set_title(f"Sample means\nn = {n}")

plt.tight_layout()
plt.show()

What the code does: it draws 5,000 samples of size n from an exponential population (thoroughly skewed) and plots the histogram of their 5,000 means. The result:

  • With n = 5, the histogram of means still inherits skewness.
  • With n = 30, it is already a fairly decent bell.
  • With n = 200, it is a textbook normal, much narrower (the σ/√n effect).

This is the foundation that legitimizes the confidence intervals and tests of this lesson even with non-normal data, as long as n is not tiny. It also explains the starring role of the normal distribution announced in 02-02.

Hypothesis testing: the logic

A hypothesis test is a procedure for deciding, with explicit rules, whether the data provide enough evidence against a default claim.

  • Null hypothesis (H₀): the "boring", no-effect claim. "The new product page does not change the average spend."
  • Alternative hypothesis (H₁): what we suspect. "The new page changes the average spend."

The logic is analogous to a trial: H₀ is innocent until proven guilty. We compute how unusual our data would be if H₀ were true; if they are unusual enough, we reject H₀.

graph TD
    A["Formulate H0 and H1"] --> B["Choose test and alpha level (e.g. 0.05)"]
    B --> C["Collect data and compute statistic"]
    C --> D["Compute p-value"]
    D --> E{"p-value < alpha?"}
    E -- "Yes" --> F["Reject H0:<br>statistically significant effect"]
    E -- "No" --> G["Fail to reject H0:<br>insufficient evidence"]

Important: "failing to reject H₀" is not "proving H₀". Absence of evidence is not evidence of absence; the effect may exist but the sample may be too small.

The p-value and type I and type II errors

The p-value is the probability of observing data as extreme as ours (or more so) assuming H₀ is true. A p-value of 0.03 means: "if the new page had no effect at all, only 3% of experiments would show a difference like this by pure chance".

What the p-value is not:

  • It is not the probability that H₀ is true.
  • It does not measure the size of the effect: with a huge n, a laughable difference of €0.02 can yield p < 0.001.

The threshold α (typically 0.05) sets how much false-alarm risk we accept, and from it follow the two possible errors:

H₀ is true (no effect) H₀ is false (there is an effect)
We reject H₀ Type I error (false alarm), prob. α Correct call (power of the test)
We fail to reject H₀ Correct call Type II error (missed effect), prob. β

At MercaFresh: a type I error would be redesigning the whole website over an improvement that never existed; a type II error, discarding a page that really did sell more. Lowering α reduces false alarms but increases missed effects: it is a trade-off, not a revealed truth. (This same tension will reappear in module 6 as the balance between a classifier's false positives and false negatives.)

The t-test in practice: MercaFresh's A/B test

Case: MercaFresh launches a new product page and wants to know whether it improved the average spend per order. For two weeks, half the visitors see the old page (group A, control) and the other half the new one (group B, treatment), assigned at random. Random assignment is the key that makes it legitimate to talk about causality, closing the loop on "correlation is not causation" from 02-03.

The two-sample independent t-test tests H₀: "the means of both groups are equal".

rng = np.random.default_rng(7)

# We simulate the experiment: the new page raises the average spend by ~2 EUR
group_A = rng.gamma(shape=4.0, scale=12.0, size=980)          # mean ~48 EUR
group_B = rng.gamma(shape=4.0, scale=12.5, size=1020)         # mean ~50 EUR

print(f"Mean A: {group_A.mean():.2f} | Mean B: {group_B.mean():.2f}")
print(f"Observed difference: {group_B.mean() - group_A.mean():.2f} EUR")

# Welch's t-test (equal_var=False): does not assume equal variances
t_stat, p_value = stats.ttest_ind(group_B, group_A, equal_var=False)
print(f"t = {t_stat:.2f}, p-value = {p_value:.4f}")

Approximate output:

Mean A: 47.66 | Mean B: 50.16
Observed difference: 2.50 EUR
t = 2.32, p-value = 0.0203

Step-by-step interpretation:

  1. The observed difference is about €2.50 per order in favor of the new page.
  2. The p-value ≈ 0.02 < 0.05: if the new page had no effect, a difference like this would only appear in ~2% of experiments. We reject H₀: the improvement is statistically significant.
  3. Business decision: significant and relevant (€2.50 × thousands of orders/month justifies the change). Always check both: statistical significance and effect size.

Technical notes:

  • stats.ttest_ind implements the t-test for independent samples; equal_var=False (Welch's test) is the safe default because it does not require equal variances between groups.
  • Skewed data like these? The CLT protects us: with ~1,000 observations per group, the means behave like normals.
  • There is also stats.ttest_rel for paired samples (the same customer measured before and after) and stats.ttest_1samp for comparing one sample against a fixed value ("does the average delivery time exceed the promised 40 min?").

The chi-square test for categorical variables

The t-test compares means of numerical variables. When both variables are categorical, the appropriate test is the chi-square test of independence: it tests H₀: "the two variables are independent", based on their contingency table.

MercaFresh case: does the conversion rate (buying or not) depend on the page version?

import pandas as pd

# Observed contingency table from the A/B test
#                 purchase   no_purchase
table = pd.DataFrame({"purchase": [312, 368],
                      "no_purchase": [2688, 2632]},
                     index=["page_A", "page_B"])
print(table)
print(f"Conversion A: {312/3000:.1%} | Conversion B: {368/3000:.1%}")

chi2, p_value, dof, expected = stats.chi2_contingency(table)
print(f"chi2 = {chi2:.2f}, p-value = {p_value:.4f}, degrees of freedom = {dof}")

Output:

        purchase  no_purchase
page_A       312         2688
page_B       368         2632
Conversion A: 10.4% | Conversion B: 12.3%
chi2 = 5.15, p-value = 0.0232, degrees of freedom = 1

How it works under the hood (intuition): the test computes the table we would expect under independence (same proportions in both rows: expected returns it) and measures how far the observed counts deviate from the expected ones. Large deviation → large chi² → small p-value.

Here p ≈ 0.023 < 0.05: conversion does depend on the page; version B converts significantly more (12.3% vs. 10.4%). The chi-square also serves questions like "does the payment method depend on the province?" or "does churn depend on the loyalty tier?".

Question Variables Test
Does the average spend differ between two groups? Numerical vs. binary Two-sample t-test
Did the mean change after an intervention (same subjects)? Numerical, paired Paired t-test
Does the mean exceed a reference value? Numerical vs. constant One-sample t-test
Are two categoricals independent? Categorical vs. categorical Chi-square

Connection to Machine Learning

Inference permeates ML practice more than it seems:

  • Comparing models. "The new model scores 84.1% and the old one 83.6%": real improvement or evaluation-sample noise? The difference between two accuracies is a difference between two sample proportions — exactly the territory of this lesson. In module 6 we will see how cross-validation (06-03) yields several measurements per model, allowing us to reason about performance variability instead of trusting a single number.
  • A/B testing models in production. Deploying the new model to 50% of the traffic and comparing business metrics is literally this lesson's experiment; we will pick it up again in module 8.
  • Quantified skepticism. The mental habit of the p-value — "could this be chance?" — is the best vaccine against hasty conclusions when exploring data, including the "promising" correlations of the previous lesson.

Common Mistakes and Tips

  • Interpreting the p-value as P(H₀ true). It is P(data this extreme | H₀ true): it conditions in the opposite direction. This distinction connects directly to the next lesson (Bayes).
  • Confusing significance with relevance. With an enormous n, everything is "significant". Always report the effect size (the difference in euros, in conversion points) alongside the p-value.
  • Checking the results every day and stopping the test when p < 0.05. This "peeking" drastically inflates false positives: fix the duration or the sample size before you start.
  • Running many tests and keeping the ones that come out. With 20 tests at α = 0.05, expect ~1 false positive. If you make multiple comparisons, adjust for it (e.g., Bonferroni correction: use α/20) or at least disclose it.
  • Using the t-test with tiny, heavily skewed samples. The CLT needs a sufficient n; with n < 20-30 and heavy tails, be wary (non-parametric tests like Mann-Whitney, stats.mannwhitneyu, exist as an alternative).
  • Forgetting random assignment in an A/B test. If group B is "the users of the new app", you are no longer comparing pages, you are comparing user types: the confounder from 02-03 in action.
  • Tip: before launching an experiment, write down H₀, H₁, α, and the sample size in a document. Experiments defined after the fact find whatever they want to find.

Exercises

Exercise 1

With a sample of 100 MercaFresh delivery times (sample mean 41.3 min, sample standard deviation 9.8 min), compute by hand the standard error and an approximate 95% CI for the mean (use ±1.96·SE). MercaFresh promises "average delivery below 40 min": is the interval compatible with that promise?

Exercise 2

Use NumPy to simulate the CLT for a uniform distribution between 0 and 10: plot the histogram of the means of 5,000 samples of size n = 2 and n = 50. What do you observe in the shape and the width? What theoretical width does the CLT predict for n = 50? (σ of a uniform(0,10) ≈ 2.89.)

Exercise 3

MercaFresh tries two email subject lines to reactivate inactive customers. Subject A: 1,500 sends, 129 reactivations. Subject B: 1,500 sends, 168 reactivations. State H₀ and H₁, build the contingency table, run the chi-square with scipy, and conclude at α = 0.05.

Solutions

Solution 1

  • SE = 9.8 / √100 = 0.98 min.
  • 95% CI ≈ 41.3 ± 1.96 · 0.98 = 41.3 ± 1.92 → (39.4, 43.2) minutes.
  • The interval contains values below 40, so we cannot rule out that the true mean keeps the promise... but most of the interval sits above 40 and so does the point estimate. Formally: a one-sample t-test against 40 would not reject H₀ at 5% (the evidence of a broken promise is inconclusive), though operationally MercaFresh would do well to keep an eye on its deliveries.

Solution 2

import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(3)
fig, axes = plt.subplots(1, 2, figsize=(9, 3))
for ax, n in zip(axes, [2, 50]):
    means = rng.uniform(0, 10, size=(5_000, n)).mean(axis=1)
    ax.hist(means, bins=50, color="steelblue")
    ax.set_title(f"n = {n}  (std = {means.std():.2f})")
plt.tight_layout(); plt.show()

With n = 2 the distribution of means is triangular (not normal yet); with n = 50 it is a clear, much narrower bell. The CLT predicts a deviation of σ/√n = 2.89/√50 ≈ 0.41, which will match the empirical std of the right panel. Code trick: generating a 5,000×n matrix and averaging over rows (mean(axis=1)) avoids the loop.

Solution 3

  • H₀: the reactivation rate is independent of the subject line (same rate for A and B). H₁: the rates differ.
import pandas as pd
from scipy import stats
table = pd.DataFrame({"reactivated": [129, 168],
                      "not_reactivated": [1371, 1332]},
                     index=["subject_A", "subject_B"])
chi2, p, dof, exp = stats.chi2_contingency(table)
print(f"Rate A: {129/1500:.1%} | Rate B: {168/1500:.1%}")
print(f"chi2 = {chi2:.2f}, p = {p:.4f}")

Result: rates of 8.6% vs. 11.2%, chi² ≈ 5.4, p ≈ 0.020 < 0.05. We reject H₀: subject B reactivates significantly more customers. With an effect size of +2.6 percentage points over 1,500 sends, it is not just significant but actionable: MercaFresh should adopt subject B (ideally confirming it without "peeking" in a second wave).

Conclusion

You have covered the core of statistical inference: estimating with confidence intervals instead of bare numbers, understanding why the central limit theorem makes means behave like normals, and running complete hypothesis tests — the t-test for means and the chi-square for categoricals — interpreting p-values and watching for type I and type II errors. MercaFresh's A/B test showed the full circuit, from hypothesis to business decision, and you noted the connection to ML: comparing models is comparing samples. But classical inference leaves one question unanswered head-on: how do we update what we already believed when new evidence arrives? That is exactly the specialty of Bayes' theorem, with which we will close the module in the next lesson.

◄ Previous
◄
Next ►
►

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources

The Project | About Us | Contribute | Donations | License
© Copyright 2026. All rights reserved