The Project | About Us | Contribute | Donations | License

HOME

At the end of the previous module we left the Machine Learning project workflow at the doorstep of the exploration phase: before training any model, you need to understand your data. Descriptive statistics is exactly that: the toolkit that condenses thousands of rows into a handful of meaningful numbers and plots. In this lesson you will learn to tell a population from a sample, to classify variable types, and to compute and interpret measures of central tendency, dispersion, and position, applying all of it to MercaFresh's order data with pandas and matplotlib.

Population and sample
Types of variables
Measures of central tendency: mean, median, and mode
Measures of dispersion: range, variance, standard deviation, and IQR
Measures of position: percentiles and quartiles
Basic visualizations: histogram and boxplot
Full worked case: MercaFresh orders

Population and sample

Population: the complete set of elements we want to study. For MercaFresh, all the orders ever placed in its history (or all that ever will be).
Sample: a subset of the population that we actually have on hand. For instance, the 10,000 orders from the last quarter.

In practice we almost never work with the full population: either it is unmanageably large, or it includes future data that does not exist yet. That is why we compute statistics on the sample and use them as an approximation of the population's parameters.

Concept	Population	Sample
Size symbol	N	n
Mean	μ (mu)	x̄ (x bar)
Standard deviation	σ (sigma)	s
Known in practice?	Almost never	Yes, computed from the data

This distinction may look like a technicality right now, but it is the foundation of the statistical inference we will cover in lesson 02-04: there we will learn to quantify how much we can trust a sample when talking about the population.

In Machine Learning the same idea reappears under a different name: the training set is a sample of all possible data, and we want the model to generalize to the population (future data).

Types of variables

Every column in a dataset is a variable, and its type determines which operations and plots make sense.

graph TD
    V[Variable] --> N[Numerical]
    V --> C[Categorical]
    N --> NC["Continuous<br>(order amount: EUR 47.35)"]
    N --> ND["Discrete<br>(number of items: 1, 2, 3...)"]
    C --> CN["Nominal<br>(province: Barcelona, Valencia...)"]
    C --> CO["Ordinal<br>(satisfaction: low < medium < high)"]

Numerical continuous: can take any value within a range. MercaFresh example: order amount (€47.35), delivery time (38.2 minutes).
Numerical discrete: only countable integer values. Example: number of items per order, number of orders per customer per month.
Categorical nominal: categories without an order. Example: delivery province, payment method (card, paypal, cash_on_delivery).
Categorical ordinal: categories with an order but no defined distance between them. Example: customer rating (low < medium < high), preferred delivery time slot.

A common trap: the fact that something is encoded with digits does not make it numerical. The postal code 08001 is a nominal variable; computing its mean means nothing. In module 3 (lesson 03-04) we will see how to convert categorical variables into numbers properly for our models.

Measures of central tendency: mean, median, and mode

They answer the question: what is the "typical" value?

Arithmetic mean (x̄): the sum of the values divided by n. It uses all the information, but it is sensitive to extreme values.
Median: the middle value once the data are sorted (if n is even, the mean of the two middle ones). Robust against extremes.
Mode: the most frequent value. It is the only one of the three that makes sense for categorical variables.

Let's see it with simulated MercaFresh data:

import numpy as np
import pandas as pd

# Seed so the results are reproducible
rng = np.random.default_rng(42)

# We simulate 1,000 orders: most between EUR 20 and 80,
# plus 15 huge orders from business customers (hospitality)
amounts = np.concatenate([
    rng.gamma(shape=4.0, scale=12.0, size=985),  # household orders
    rng.uniform(400, 900, size=15)               # hospitality orders
])

orders = pd.DataFrame({"order_amount": amounts.round(2)})

print("Mean:   ", orders["order_amount"].mean().round(2))
print("Median: ", orders["order_amount"].median().round(2))

Approximate output:

Mean:    54.79
Median:  44.87

Detailed walkthrough of the code:

np.random.default_rng(42) creates a random number generator with a fixed seed: every run produces the same data, which is essential for reproducible analysis.
rng.gamma(...) generates amounts with a realistic shape (lots of mid-sized orders, a tail of large ones); you don't need to understand that distribution yet — we will cover it in lesson 02-02.
np.concatenate joins the household orders with 15 outlying business orders.
.mean() and .median() are pandas methods that compute the mean and median, ignoring null values if there were any.

Look closely at the result: the mean (≈€55) is clearly higher than the median (≈€45). Those 15 hospitality orders "pull" the mean upward, but barely move the median. If MercaFresh claimed "our typical order is €55" it would be exaggerating: half of the orders don't even reach €45.

Measure	Advantage	Drawback	When to prefer it
Mean	Uses all the data; nice mathematical properties	Sensitive to extreme values	Symmetric data with no outliers
Median	Robust to extremes	Ignores the magnitude of the values	Skewed data (amounts, salaries, times)
Mode	Works for categoricals	May not be unique, or may not exist	Categorical variables; discrete data

The mode with a categorical variable:

payment_methods = pd.Series(["card"] * 620 + ["paypal"] * 290 + ["cash_on_delivery"] * 90)
print(payment_methods.mode()[0])   # card  -> the most frequent payment method

Measures of dispersion: range, variance, standard deviation, and IQR

Two supermarkets can share the same average order amount and behave in completely different ways: one with orders always hovering around €50, another with orders of €5 and €500. Dispersion measures that variability.

Range: maximum − minimum. Simple but fragile: it depends on just two values.
Variance (s²): the mean of the squared deviations from the mean. Its units are the original units squared (euros²!), which makes interpretation awkward.
Standard deviation (s): the square root of the variance. It brings us back to the original units: "amounts typically deviate about €30 from the mean".
Interquartile range (IQR): Q3 − Q1, the width of the middle 50% of the data. Robust to extreme values, just like the median.

series = orders["order_amount"]

print("Range:              ", (series.max() - series.min()).round(2))
print("Variance:           ", series.var().round(2))
print("Standard deviation: ", series.std().round(2))

q1 = series.quantile(0.25)
q3 = series.quantile(0.75)
print("IQR:                ", (q3 - q1).round(2))

Approximate output:

Range:               885.05
Variance:            5090.71
Standard deviation:  71.35
IQR:                 35.09

Key points about the code:

pandas' .var() and .std() use the sample version by default (they divide by n−1, parameter ddof=1). NumPy (np.var, np.std) defaults to the population version (divides by n, ddof=0). With 1,000 data points the difference is tiny, but it is worth knowing it exists.
.quantile(0.25) returns the value below which 25% of the data fall (the first quartile, Q1).

Notice the contrast: the standard deviation (≈€71) is inflated by the hospitality orders, while the IQR (≈€35) is a better description of the variability of the typical household order. Rule of thumb: mean + standard deviation for symmetric data, median + IQR for skewed data or data with outliers.

Measures of position: percentiles and quartiles

The pth percentile is the value below which p% of the data fall. The quartiles are three special percentiles: Q1 (25th percentile), Q2 (50th percentile = median), and Q3 (75th percentile).

An example with MercaFresh delivery times:

# Delivery times in minutes: most hover around 35-40 min,
# with a few slow deliveries due to traffic
deliveries = pd.Series(rng.gamma(shape=9.0, scale=4.5, size=1000).round(1),
                       name="minutes")

percentiles = deliveries.quantile([0.25, 0.50, 0.75, 0.90, 0.95, 0.99])
print(percentiles.round(1))

Approximate output:

0.25    30.7
0.50    38.5
0.75    47.9
0.90    57.9
0.95    63.7
0.99    75.7

The business reading is immediate: half of the orders arrive in under 39 minutes, but the slowest 5% take more than 64. If MercaFresh promises "delivery in under 60 minutes", the 90th percentile (≈58 min) says it is cutting it close: roughly 1 in 10 deliveries grazes or breaks the promise. High percentiles (p95, p99) are the standard in logistics and engineering for monitoring the worst cases, not the average case.

Basic visualizations: histogram and boxplot

Numbers summarize; plots reveal the shape of the data.

Histogram

It splits the range of values into intervals (bins) and counts how many observations fall into each one.

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(8, 4))
ax.hist(orders["order_amount"], bins=50, color="steelblue", edgecolor="white")
ax.axvline(orders["order_amount"].mean(), color="red", linestyle="--", label="Mean")
ax.axvline(orders["order_amount"].median(), color="green", linestyle="--", label="Median")
ax.set_xlabel("Order amount (EUR)")
ax.set_ylabel("Number of orders")
ax.set_title("Distribution of order amounts at MercaFresh")
ax.legend()
plt.tight_layout()
plt.show()

bins=50 controls the granularity: too few bins hide details, too many create noise. Try several values.
axvline draws vertical lines to compare the mean and median visually: in a right-tailed distribution like this one, the mean lands to the right of the median.

The histogram will show a main mass of orders between €20 and €100 with a long right tail (the hospitality orders). That skewness is the visual explanation of why the mean and the median differ.

Boxplot

It condenses the median, the quartiles, and the atypical values into a single drawing:

The box runs from Q1 to Q3 (its height is the IQR), with a line at the median.
The whiskers extend to the last data point within 1.5 × IQR from the box.
Points beyond the whiskers are flagged as outliers (candidates for atypical values, not automatic culprits).

fig, ax = plt.subplots(figsize=(8, 3))
ax.boxplot(orders["order_amount"], vert=False)
ax.set_xlabel("Order amount (EUR)")
ax.set_title("Boxplot of order amounts: hospitality orders show up as outliers")
plt.tight_layout()
plt.show()

In this boxplot the 15 hospitality orders will appear as isolated points on the right. Spotting them is the first step; deciding what to do with them (are they errors? legitimate customers from another segment?) belongs to the data cleaning of module 3 (lesson 03-01) and to the anomaly detection we will cover later on.

Full worked case: MercaFresh orders

Let's put it all together in the typical flow of a first exploration:

orders_full = pd.DataFrame({
    "order_amount": orders["order_amount"],
    "delivery_minutes": deliveries,
    "num_items": rng.poisson(lam=12, size=1000),          # discrete
    "payment_method": rng.choice(["card", "paypal", "cash_on_delivery"],
                                 size=1000, p=[0.62, 0.29, 0.09])  # nominal
})

# describe() computes in one shot: n, mean, std, minimum, quartiles, and maximum
print(orders_full[["order_amount", "delivery_minutes", "num_items"]].describe().round(2))

# For the categorical variable: frequencies, not means
print(orders_full["payment_method"].value_counts(normalize=True).round(3))

describe() is the Swiss army knife of exploration: one table with almost every measure from this lesson.
value_counts(normalize=True) yields proportions instead of counts; it is the natural summary of a nominal variable.

With this table, the MercaFresh team can already answer business questions ("what does the typical order look like?", "are we keeping our delivery promise?") without having trained a single model.

Common Mistakes and Tips

Using the mean with skewed data or outliers. Amounts, times, and salaries almost always have a right tail: report the median too, or the median alone.
Computing means of categorical variables encoded as numbers (postal codes, product IDs). Check the meaning of the column, not just its dtype.
Mixing up variance and standard deviation when interpreting. The variance is in squared units; when communicating results, always use the standard deviation.
Forgetting the ddof parameter: pandas uses n−1 and NumPy uses n by default. If you mix libraries and the numbers "don't add up", this is usually why.
Removing outliers blindly. A point beyond the boxplot's whiskers is a candidate for review, not a guaranteed error: MercaFresh's hospitality orders are real money.
Tip: always draw a histogram before deciding which statistics to report. Two datasets can share the same mean and standard deviation and have radically different shapes.

Exercises

Exercise 1

The delivery times (in minutes) of 9 orders from one afternoon are: [32, 35, 29, 41, 38, 33, 36, 95, 34]. Compute the mean and the median by hand (and check with pandas). Which one better represents the "typical" delivery, and why?

Exercise 2

Classify these MercaFresh variables by type (numerical continuous/discrete, categorical nominal/ordinal): (a) total order weight in kg, (b) number of fresh products in the order, (c) customer type (individual, business), (d) loyalty program tier (bronze, silver, gold), (e) delivery postal code.

Exercise 3

Using the orders_full DataFrame from the worked case, compute the IQR of delivery_minutes and determine the outlier limits according to the 1.5 × IQR rule (lower: Q1 − 1.5·IQR; upper: Q3 + 1.5·IQR). How many orders fall outside them?

Solutions

Solution 1

import pandas as pd
times = pd.Series([32, 35, 29, 41, 38, 33, 36, 95, 34])
print(times.mean().round(2))   # 41.44
print(times.median())          # 35.0

The mean (41.4 min) is distorted by the 95-minute delivery (perhaps a traffic incident). The median (35 min) better represents the typical delivery: 8 of the 9 orders arrived in under 41 minutes. With few data points and one extreme value, the median is the robust choice.

Solution 2

(a) Weight in kg → numerical continuous (it can be 7.35 kg).
(b) Number of fresh products → numerical discrete (0, 1, 2...).
(c) Customer type → categorical nominal (there is no order between individual and business).
(d) Loyalty tier → categorical ordinal (bronze < silver < gold).
(e) Postal code → categorical nominal, even though it is written with digits: neither the ordering nor arithmetic between codes has any meaning.

Solution 3

q1 = orders_full["delivery_minutes"].quantile(0.25)
q3 = orders_full["delivery_minutes"].quantile(0.75)
iqr = q3 - q1
lower_lim = q1 - 1.5 * iqr
upper_lim = q3 + 1.5 * iqr
outliers = orders_full[(orders_full["delivery_minutes"] < lower_lim) |
                       (orders_full["delivery_minutes"] > upper_lim)]
print(f"Limits: [{lower_lim:.1f}, {upper_lim:.1f}] -> {len(outliers)} outliers")

With the simulated data you will get around 5-15 outliers, all above the upper limit (abnormally slow deliveries): exactly the points the boxplot draws beyond the right whisker. The lower limit will come out positive but below the actual minimum, so there will be no outliers on the low side.

Conclusion

In this lesson you have built the basic vocabulary of data exploration: the difference between population and sample, the types of variables, and the three families of measures (central tendency, dispersion, and position), together with the histogram and the boxplot for seeing the shape of the data. You have verified with MercaFresh's orders that choosing between mean/standard deviation and median/IQR is not cosmetic: it changes the business conclusions. And that "shape" of the data revealed by the histogram has a proper mathematical name: in the next lesson we will study probability distributions, the theoretical models (Bernoulli, binomial, Poisson, normal...) that describe how data like the amounts and times we just explored are generated.

Basic statistics concepts

Contents

Population and sample

Types of variables

Measures of central tendency: mean, median, and mode

Measures of dispersion: range, variance, standard deviation, and IQR

Measures of position: percentiles and quartiles

Basic visualizations: histogram and boxplot

Histogram

Boxplot

Full worked case: MercaFresh orders

Common Mistakes and Tips

Exercises

Exercise 1

Exercise 2

Exercise 3

Solutions

Conclusion

Machine Learning Course

Module 1: Introduction to Machine Learning

Module 2: Foundations of Statistics and Probability

Module 3: Data Preprocessing

Module 4: Supervised Machine Learning Algorithms

Module 5: Unsupervised Machine Learning Algorithms

Module 6: Model Evaluation and Validation

Module 7: Advanced Techniques and Optimization

Module 8: Model Implementation and Deployment

Module 9: Hands-On Projects

Module 10: Additional Resources