The Project | About Us | Contribute | Donations | License

HOME

In 07-01 we decided that Marta's workshop is built on Python and checked with a benchmark that its speed comes from delegating computation to native libraries. The first three of those libraries, the ones that hold up everything else, are the ones we used in module 4 without stopping: NumPy, which provides the numerical array and vectorised operations; pandas, which provides the table with column names, dates and groupings; and Matplotlib, which draws. generate_orders_ml was NumPy plus pandas; groupby("category")["returned"].mean() was pandas; the learning curves of 04-06 were Matplotlib. This lesson explains them systematically and with one continuous example: the NovaMarket orders and weekly demand you already know. We will see the ndarray, its shape and type, indexing, broadcasting and boolean masks; Series and DataFrame, reading and writing CSV, selection with loc/iloc, missing values, groupby, merge, dates and resampling; and figures with histograms, bars, lines and subplots, which we will describe in text. We will finish with an "operation → how to do it" table that serves as a cheat sheet for module 8. It matters because any AI project spends more time preparing and looking at data with these three libraries than calling fit. What scikit-learn and PyTorch do with those arrays is the business of 07-03.

NumPy: the ndarray, shape and dtype
Creation, indexing and slicing
Vectorised operations and broadcasting
Aggregations by axis, boolean masks and randomness
Basic linear algebra and why NumPy is the foundation of everything
pandas: Series and DataFrame, reading, writing and inspection
Selection and filtering: loc, iloc and masks
Missing values, groupby and aggregations
Joining tables with merge
Dates: to_datetime, .dt, resample and rolling
apply in moderation
Matplotlib: figure, axes and the four basic plots
"Operation → how to do it" table
Common Mistakes and Tips
Exercises
Conclusion

NumPy: the `ndarray`, shape and `dtype`

A Python list can contain anything ([1, "hello", 2.5]), and that is why each element is a separate object with its type, its reference counter and its memory address: flexible and slow. NumPy's ndarray is the opposite: a contiguous block of memory in which every element has the same type (dtype) and a shape that says how many dimensions it has and how many elements per dimension. That is the secret of the 07-01 benchmark: the CPU walks the block without asking anything.

import numpy as np

amounts = np.array([129.90, 45.50, 899.00, 19.99, 249.00])   # 1 dimension: a vector
print(amounts, amounts.dtype, amounts.shape, amounts.ndim)
# [129.9   45.5  899.    19.99 249.  ] float64 (5,) 1

table = np.array([[129.90, 2, 3],        # 2 dimensions: a matrix of 4 rows x 3 columns
                  [45.50, 1, 5],         # (amount, num_items, delivery_days)
                  [899.00, 1, 2],
                  [19.99, 3, 7]])
print(table.shape, table.dtype)         # (4, 3) float64  <- all float, even though there are integers
print(np.array([1, 2, 3]).dtype)        # int64
print(np.array([1, 2.5]).dtype)         # float64: NumPy promotes to the most general type

shape is a tuple: (5,) is a vector of 5; (4, 3) a 4 × 3 matrix; (3000, 21) was the feature matrix of 04-03 (3,000 orders, 21 columns after the ColumnTransformer); a 16 × 16 image in 05-04 was (16, 16) and a batch of them (n, 1, 16, 16).
dtype is the common type: float64 (8 bytes, the usual one), float32 (4 bytes, the one PyTorch uses by default), int64, bool. An array of a million float64 takes exactly 8 MB; an equivalent Python list, about four times more.
ndim is the number of dimensions (axes). In AI vocabulary, a 1-dimensional array is a vector, a 2-dimensional one a matrix and anything beyond a tensor.

Creation, indexing and slicing

Ways of creating arrays that you will see all the time:

np.zeros(3)                 # [0. 0. 0.]
np.ones((2, 3))             # 2x3 matrix of ones
np.full(3, 7.5)             # [7.5 7.5 7.5]
np.arange(0, 10, 2)         # [0 2 4 6 8]         like range, but returns an array
np.linspace(0, 1, 5)        # [0.   0.25 0.5  0.75 1.  ]  5 evenly spaced points
np.eye(3)                   # 3x3 identity matrix

Indexing and slicing work as in lists, but with one position per axis separated by commas, and with the rule that a slice is a view, not a copy:

table[0]          # [129.9   2.    3. ]   first row
table[0, 0]       # 129.9                 row 0, column 0
table[:, 0]       # [129.9   45.5  899.    19.99]   ALL rows, column 0 (the amounts column)
table[1:3]        # rows 1 and 2 (3 is not included), all columns
table[-1, -1]     # 7.0                   last row, last column

view = table[:, 0]        # view of the amounts column
view[1] = 0.0             # modifies table[1, 0] too!
copy = table[:, 0].copy()    # if you want independence, copy explicitly

A slice being a view is a design decision for efficiency (megabytes are not copied when slicing), and a classic source of surprises: if you are going to modify the slice, use .copy().

Vectorised operations and broadcasting

Arithmetic operators act element by element over the whole array, with no loop:

amounts = np.array([129.90, 45.50, 899.00, 19.99, 249.00])
np.round(amounts * 1.21, 2)      # [ 157.18   55.06 1087.79   24.19  301.29]  with VAT
amounts - 5                      # [124.9   40.5  894.    14.99 244.  ]        shipping costs taken out
discounts = np.array([0.10, 0.0, 0.15, 0.0, 0.05])
np.round(amounts * (1 - discounts), 2)     # [116.91  45.5  764.15  19.99 236.55]  array by array
np.log(amounts).round(3)         # [4.867 3.818 6.801 2.995 5.517]  "universal" functions (ufuncs)

amounts * 1.21 combines an array of shape (5,) with a scalar: NumPy "stretches" the scalar to the array's shape. That generalised rule is called broadcasting: two arrays can operate together if, comparing their shapes from right to left, each pair of dimensions is equal or one of them is 1 (or missing). Numerical example, the most common operation in ML, standardising columns (subtracting each column's mean and dividing by its standard deviation; it is what StandardScaler does in 04-03):

table = np.array([[129.90, 2, 3], [45.50, 1, 5], [899.00, 1, 2], [19.99, 3, 7]])   # (4, 3)
means = table.mean(axis=0)     # (3,)  [273.5975   1.75     4.25  ]  one mean per column
stds = table.std(axis=0)       # (3,)  [363.36     0.829    1.920]
z = (table - means) / stds     # (4,3) - (3,) -> the (3,) row is subtracted from EACH row of table
print(z.round(3))
# [[-0.395  0.302 -0.651]
#  [-0.628 -0.905  0.391]
#  [ 1.721 -0.905 -1.172]
#  [-0.698  1.508  1.432]]

col = np.array([[1.0], [2.0], [3.0], [4.0]])   # (4, 1)
row = np.array([10.0, 20.0, 30.0])             # (3,)
print(col + row)                               # (4,1) + (3,) -> (4,3): addition table
# [[11. 21. 31.]
#  [12. 22. 32.]
#  [13. 23. 33.]
#  [14. 24. 34.]]

np.array([1, 2, 3]) + np.array([1, 2])
# ValueError: operands could not be broadcast together with shapes (3,) (2,)

Checking the rule: (4, 3) with (3,): comparing from the right, 3 = 3 and the other dimension is missing, compatible. (4, 1) with (3,): 1 against 3 (fine, one of them is 1), 4 against nothing: result (4, 3). (3,) with (2,): 3 ≠ 2 and neither is 1: error. When a NumPy or PyTorch computation gives you a shape error, this is the game you have to play.

Aggregations by axis, boolean masks and randomness

Aggregating is reducing an axis: sum, mean, std, min, max, argmax (position of the maximum). Without axis they reduce everything; with axis=0 they walk the rows and return one value per column; with axis=1, one value per row:

table.sum()            # 1118.39                     everything
table.sum(axis=0)      # [1094.39    7.     17.  ]    per column: total amount, items, days
table.sum(axis=1)      # [134.9   51.5  902.    29.99]  per row (meaningless here, but it illustrates the axis)
table.max(axis=0)      # [899.   3.   7.]
table.argmax(axis=0)   # [2 3 3]                     in which row the maximum of each column sits

Mnemonic: axis is the axis that disappears. With axis=0 the rows disappear and one figure per column remains.

Boolean masks are the vectorised way to filter and count, and we have used them without naming them since module 4 (rng.random(n) < 0.30 to decide which customers are new):

amount = np.array([129.90, 45.50, 899.00, 19.99, 249.00, 75.0])
returned = np.array([0, 0, 1, 0, 1, 0])
amount > 100                         # [ True False  True False  True False]  bool array
amount[amount > 100]                 # [129.9 899.  249. ]   filters
(amount > 100).sum()                 # 3      True counts as 1: how many satisfy it
(amount > 100).mean()                # 0.5    proportion that satisfy it
returned[amount > 100].mean()        # 0.667  return rate of the expensive orders
returned[amount <= 100].mean()       # 0.0    and of the cheap ones
np.where(amount > 100, "high", "low")     # ['high' 'low' 'high' 'low' 'high' 'low']  vectorised conditional
amount[(amount > 40) & (returned == 0)]   # [129.9  45.5  75. ]  combine with & (and), | (or), ~ (not)

Watch the parentheses: & has higher precedence than >, so amount > 40 & returned == 0 without parentheses fails.

Reproducible randomness: since NumPy 1.17 the recommended way is to create a seeded generator, np.random.default_rng(seed), and ask it for numbers. It is what generate_orders_ml does and why the 3,000 orders are the same in every module:

rng = np.random.default_rng(42)
rng.random(3)                     # [0.774 0.439 0.859]   uniform in [0, 1)
rng.integers(1, 6, size=5)        # [1 4 2 1 3]           integers from 1 to 5 (6 not included)
rng.normal(0, 25, 3).round(1)     # [ 3.2 -7.9 -0.4]      normals with mean 0 and std 25 (the demand noise)
rng.choice(["A", "B", "C"], size=6, p=[0.4, 0.35, 0.25])   # ['B' 'A' 'C' 'B' 'C' 'B']  zones
np.random.default_rng(42).random(3)   # [0.774 0.439 0.859]  same seed, same numbers

Basic linear algebra and why NumPy is the foundation of everything

The @ operator is the matrix product, and with it you write in one line what the logistic regression of 04-04 and every nn.Linear layer of 05-02 do inside: z = X @ w + b:

X = np.array([[1.0, 129.90, 2],      # 3 orders x 3 features (the first is the bias)
              [1.0, 45.50, 1],
              [1.0, 899.00, 1]])
w = np.array([-3.4, 0.01, 0.35])     # weights (made up, in the style of the "hidden truth" of 04-01)
z = X @ w                            # (3,3) @ (3,) -> (3,)   one logit per order
print(z, 1 / (1 + np.exp(-z)))
# [-1.401 -2.595  5.94 ]  [0.198 0.069 0.997]   sigmoid -> probability of return
X.T                                  # transpose, (3,3) here; of a (3000,21) it would be (21,3000)
A = np.array([[2.0, 1.0], [1.0, 3.0]]); b = np.array([3.0, 5.0])
np.linalg.solve(A, b)                # [0.8 1.4]   solves A x = b (better than inv(A) @ b)

np.linalg also has inv, det, eig (eigenvalues, the basis of the PCA of 04-02), svd and norm. You do not need to master the algebra to follow the course; you do need to know that every model is in the end a sequence of @ and element-wise functions over arrays.

That is why NumPy is the foundation: scikit-learn receives and returns arrays (or DataFrames it converts into arrays), Matplotlib draws arrays, pandas stores each column in an array, and the PyTorch tensors of 05-03 are "arrays with a gradient": same concept of shape, dtype and broadcasting, plus requires_grad so that autograd records the operations, and torch.tensor(array) / tensor.numpy() to move from one to the other without copying when the type allows it. Whoever understands NumPy has 80 % of PyTorch.

pandas: `Series` and `DataFrame`, reading, writing and inspection

NumPy does not know that column 0 is "amount", nor does it understand dates or missing values in integers. pandas puts two labelled structures on top of the arrays:

Series: a one-dimensional array with an index (row labels) and a name.
DataFrame: a table of columns, each one a Series (possibly of a different dtype), sharing an index. It is what generate_orders_ml returns.

import pandas as pd
from novamarket_ml import generate_orders_ml, generate_weekly_demand   # generators from module 4

s = pd.Series([129.90, 45.50, 899.00], index=["P1001", "P1002", "P1003"], name="amount")
print(s["P1002"], s.mean(), (s > 100).sum())      # 45.5 358.13 2   access by label; NumPy underneath

orders = generate_orders_ml(3000, 42)              # DataFrame of 3000 x 7
print(orders.shape)                                # (3000, 7)
print(orders.head())                               # first 5 rows
orders.info()                                      # columns, non-nulls, dtypes, memory
print(orders.describe().round(2))                  # statistics of the numeric columns
print(orders["category"].value_counts())           # count per value

Output (abridged) of info() and describe():

RangeIndex: 3000 entries, 0 to 2999
Data columns (total 7 columns):
 #   Column          Non-Null Count  Dtype
 0   amount          3000 non-null   float64
 1   num_items       3000 non-null   int64
 2   delivery_days   3000 non-null   int64
 3   new_customer    3000 non-null   int64
 4   category        3000 non-null   str
 5   postcode_zone   3000 non-null   str
 6   returned        3000 non-null   int64
memory usage: 164.2 KB

        amount  num_items  delivery_days  new_customer  returned
count  3000.00    3000.00        3000.00       3000.00   3000.00
mean    125.29       2.98           3.99          0.30      0.16
std      84.33       1.41           2.00          0.46      0.37
min       5.88       1.00           1.00          0.00      0.00
50%     108.12       3.00           4.00          0.00      0.00
max     632.61       5.00           7.00          1.00      1.00

category
electronics    1038
home            908
computing       615
accessories     439

You already recognise the figures: 30 % new customers, return rate 0.16 (16.4 %), mean amount 125 € with median 108 (long tail: the generator's gamma). Text columns appear as str in recent versions of pandas (object in earlier ones). head(), info(), describe() and value_counts() are the first thing to run on any new file, before thinking about models.

CSV round trip, which is how orders.csv and company will circulate around NovaMarket:

orders.to_csv("orders.csv", index=False)       # index=False: do not save the 0..2999 as a column
o2 = pd.read_csv("orders.csv")                 # infers types; parse_dates=[...] for date columns
print(o2.shape)                                # (3000, 7)
# First lines of the file:
# amount,num_items,delivery_days,new_customer,category,postcode_zone,returned
# 130.51,4,4,1,home,B,0

read_csv accepts sep=";" (the Spanish CSVs from Excel), decimal=",", encoding="latin-1", usecols, nrows; and there are read_excel, read_sql (for the 07-01 query straight into a DataFrame), read_parquet (compressed columnar format, much faster than CSV for large tables) and read_json.

Selection and filtering: `loc`, `iloc` and masks

Three ways of selecting, and it pays to tell them apart:

orders["amount"]                            # one column -> Series
orders[["amount", "category"]]              # several columns -> DataFrame (list inside brackets)
orders.loc[0]                               # row with LABEL 0 -> Series with one entry per column
orders.loc[0:2, ["amount", "returned"]]     # labels 0,1,2 (loc INCLUDES the end) and two columns
orders.iloc[0:2, 0:3]                       # POSITIONS: rows 0-1, columns 0-2 (iloc excludes the end)
orders.iloc[-1, 0]                          # 94.06   last order, first column

expensive = orders[orders["amount"] > 300]                          # boolean mask, as in NumPy
print(len(expensive), expensive["returned"].mean().round(3))        # 134 0.612  <- 61 % of the expensive ones are returned
sel = orders[(orders["category"] == "electronics") & (orders["new_customer"] == 1)]
print(len(sel), sel["returned"].mean().round(3))                    # 316 0.415  <- the row from exercise 3 of 07-01
orders.query("amount > 300 and category == 'home'").shape           # (37, 7)   the same idea, as text
orders.sort_values("amount", ascending=False).head(3)               # the three most expensive orders (632, 608, 578 €; all three returned)

loc goes by labels (those of the index and the column names) and includes the endpoint; iloc goes by integer positions and excludes it, like Python. As long as the index is the RangeIndex 0..n-1 they coincide, which is why the confusion goes unnoticed until you filter or reorder and the labels stop being positions.
To assign over a selection use df.loc[mask, "column"] = value; assigning over df[mask]["column"] may modify a copy and not the original (the famous SettingWithCopyWarning).

Creating new columns is assigning, and pd.cut discretises into bands (remember 04-03):

orders["amount_per_item"] = (orders["amount"] / orders["num_items"]).round(2)
orders["band"] = pd.cut(orders["amount"], bins=[0, 50, 150, 400, np.inf],
                        labels=["<50", "50-150", "150-400", ">400"])
print(orders.groupby("band", observed=True)["returned"].mean().round(3))
# <50 0.049   50-150 0.115   150-400 0.300   >400 0.743

The return rate multiplies by 15 between the cheapest band and the most expensive: the generator's +0.010 * (amount - 100) seen from the data.

Missing values, `groupby` and aggregations

In 04-03 we dirtied the orders with NaN and imputed them inside the pipeline. The pandas tools to see and treat them:

rng = np.random.default_rng(7)
dirty = orders.copy()
dirty.loc[rng.random(len(dirty)) < 0.05, "delivery_days"] = np.nan   # 5 % of deliveries with no data
dirty.loc[rng.random(len(dirty)) < 0.02, "amount"] = np.nan
print(dirty.isna().sum())                 # amount 73, delivery_days 142, rest 0
print(dirty["amount"].mean().round(2))    # 124.79  aggregations IGNORE NaN by default
print(dirty.dropna().shape)               # (2787, 7)  drop rows with any NaN: 213 are lost
filled = dirty.fillna({"delivery_days": dirty["delivery_days"].median(),
                       "amount": dirty["amount"].median()})   # impute by median
print(filled.isna().sum().sum())          # 0
print(dirty["delivery_days"].dtype)       # float64: putting NaN in an integer column makes pandas cast it to float

isna()/notna(), dropna(), fillna(), and in models the SimpleImputer of 04-03 (which also remembers the training median to apply it in production, something a manual fillna does not do).

groupby is the central operation of analysis: split into groups, apply an aggregation to each and combine. It is the SQL GROUP BY of 07-01, and with it we answer Diego's questions:

orders.groupby("category")["returned"].mean().round(3)
# accessories 0.130  computing 0.159  electronics 0.207  home 0.135   (same figure as the SQL query)

rate = orders.groupby("category")["returned"].agg(orders="count", returned_orders="sum", rate="mean").round(3)
print(rate.sort_values("rate", ascending=False))
#              orders  returned_orders   rate
# electronics    1038              215  0.207
# computing       615               98  0.159
# home            908              123  0.135
# accessories     439               57  0.130

# Two keys -> hierarchical index; unstack() moves the second key to columns
orders.groupby(["category", "postcode_zone"])["returned"].mean().round(3).unstack()
# postcode_zone      A      B      C
# accessories    0.128  0.123  0.140
# computing      0.139  0.178  0.165
# electronics    0.223  0.187  0.207
# home           0.121  0.162  0.117

orders.groupby("new_customer").agg(mean_amount=("amount", "mean"),
                                   return_rate=("returned", "mean"),
                                   orders=("amount", "size")).round(3)
#               mean_amount  return_rate  orders
# new_customer
# 0                 124.813        0.091    2094
# 1                 126.397        0.333     906

Readings: by zone the rate does not vary systematically (0.12-0.22 with no pattern per column): consistent with postcode_zone having no influence in the generator, and with the model in 04-04 giving it almost zero weights. New customers spend almost the same on average (126 € versus 125 €) but return 3.7 times more (33 % versus 9 %). The agg(name=("column", "function")) syntax (named aggregation) is the most readable; pd.crosstab(orders["category"], orders["returned"], normalize="index") gives the same table of rates in contingency format.

Joining tables with `merge`

NovaMarket's data is spread out: orders.csv does not say how much handling a return costs or from which warehouse it ships. That is in products.csv. We build a small table per category (in reality it would be per product) and join it:

products = pd.DataFrame({
    "category": ["electronics", "home", "computing", "accessories"],
    "flagship_product": ["NovaSound headphones", "NovaClean robot vacuum",
                         "NovaBook laptop", "NovaCase sleeve"],
    "return_cost": [12.0, 18.0, 25.0, 4.0],          # euros per return handled
    "warehouse": ["Getafe", "Zaragoza", "Getafe", "Zaragoza"],
})
merged = orders.merge(products, on="category", how="left")   # key: category; left: keeps every order
print(merged.shape)                                          # (3000, 10): 7 columns + 3 new ones
merged["cost_incurred"] = merged["returned"] * merged["return_cost"]
print(merged.groupby("warehouse").agg(orders=("amount", "size"), returned_orders=("returned", "sum"),
                                      cost=("cost_incurred", "sum")))
#           orders  returned_orders    cost
# Getafe      1653              313  5030.0
# Zaragoza    1347              180  2442.0

on is the key column (if it has a different name in each table, left_on/right_on); how decides what happens with unmatched keys: left keeps every row of the left table (orders) and puts NaN where there is no product; inner only those matching in both; outer all of them. If you remove "accessories" from products, inner leaves 2,561 orders and left leaves 3,000 with 439 warehouse set to NaN.
join is the same by index; pd.concat([df1, df2]) stacks tables (rows or columns) with no key.
Diego has his figure: returns cost about 7,500 € in these 3,000 orders, two thirds in Getafe (electronics and computing). It is the kind of data that turns "AUC 0.844" into "euros saved".

Dates: `to_datetime`, `.dt`, `resample` and `rolling`

The weekly demand for the NovaClean (generate_weekly_demand, 04-02) comes with a date column of type datetime64. pandas converts text to dates with pd.to_datetime and exposes the parts with .dt:

demand = generate_weekly_demand(104, 42)        # 104 weeks: 2024 and 2025, Mondays
print(demand.dtypes)                             # week int64, date datetime64, units int64
pd.to_datetime("24/11/2025", dayfirst=True)     # Timestamp('2025-11-24')  careful with the day-first (Spanish) format
demand["date"].dt.year.value_counts().sort_index()     # 2024: 53 weeks, 2025: 51
demand["month"] = demand["date"].dt.month        # also .dt.day, .dt.dayofweek, .dt.day_name(), .dt.quarter
demand[demand["date"] >= "2025-11-01"].head(4)         # comparing with text works
#     week       date  units
# 96    97 2025-11-03    573
# 97    98 2025-11-10    578
# 98    99 2025-11-17    610
# 99   100 2025-11-24    825   <- Black Friday

# Resample from weekly to monthly: the index must be the date
monthly = demand.set_index("date")["units"].resample("MS").sum()   # MS = month start
print(monthly.head(4)); print(len(monthly), monthly.idxmax(), monthly.max())
# 2024-01-01 1714 / 2024-02-01 1459 / 2024-03-01 1656 / 2024-04-01 2261 ... 24 months; maximum June 2025: 3268

# Moving average: smooths the weekly noise
demand["moving_avg_4"] = demand["units"].rolling(4).mean()   # NaN in the first 3

resample accepts "W", "MS", "QS" (quarter), "YS", and any aggregation (sum, mean, max); rolling(k) computes sliding windows (center=True to centre them). Beware the classic monthly trap: a month with five Mondays adds up five weeks and another four; to compare months seriously you have to normalise by weeks or work with days.

`apply` in moderation

apply runs a Python function row by row or value by value. It is convenient and it is a Python loop in disguise (07-01): use it when there is no vectorised alternative, and prefer map for dictionaries, .str for text and pd.cut/np.where for bands and conditionals:

def band(amt):
    return "low" if amt < 50 else ("medium" if amt < 150 else "high")
a = orders["amount"].apply(band)                          # works, but it is a Python loop
b = pd.cut(orders["amount"], [0, 50, 150, np.inf], labels=["low", "medium", "high"], right=False)   # vectorised
# both: {'medium': 1596, 'high': 898, 'low': 506}; with 3,000 rows there is no difference; with 3 million there is
orders["category"].map({"electronics": "ELEC", "home": "HOME", "computing": "COMP", "accessories": "ACC"})
orders["category"].str.upper(); orders["category"].str.len(); orders["category"].str.contains("elec")

Matplotlib: figure, axes and the four basic plots

Matplotlib has two layers: the quick one (plt.plot(...)) and the object-oriented one, which is the one worth learning: a figure (fig, the canvas) contains one or more axes (ax, each plot), and you draw and label on the axes. Since the course environment does not display images, we save to file and describe what is seen.

import matplotlib.pyplot as plt

# 1) Histogram of amounts
fig, ax = plt.subplots(figsize=(7, 4))                       # one figure with one axes
ax.hist(orders["amount"], bins=40, color="steelblue", edgecolor="white")
ax.axvline(orders["amount"].median(), color="darkred", linestyle="--",
           label=f"median {orders['amount'].median():.0f} EUR")   # vertical line
ax.set_xlabel("Order amount (EUR)"); ax.set_ylabel("Number of orders")
ax.set_title("NovaMarket: distribution of amounts (3,000 orders)")
ax.legend()
fig.tight_layout(); fig.savefig("amounts_hist.png", dpi=120)   # also .pdf, .svg

# 2) Bars: return rate by category
rate = orders.groupby("category")["returned"].mean().sort_values(ascending=False)
fig, ax = plt.subplots(figsize=(6, 4))
bars = ax.bar(rate.index, rate.values * 100, color=["firebrick", "darkorange", "goldenrod", "seagreen"])
ax.bar_label(bars, fmt="%.1f %%")                            # the value above each bar
ax.axhline(orders["returned"].mean() * 100, color="gray", linestyle=":", label="overall mean")
ax.set_ylabel("Return rate (%)"); ax.set_title("Return rate by category"); ax.legend()
fig.tight_layout(); fig.savefig("rate_bars.png", dpi=120)

# 3) Line: weekly demand and moving average
demand["moving_avg_8"] = demand["units"].rolling(8, center=True).mean()
fig, ax = plt.subplots(figsize=(9, 4))
ax.plot(demand["date"], demand["units"], color="lightgray", marker=".", linewidth=1, label="weekly units")
ax.plot(demand["date"], demand["moving_avg_8"], color="navy", linewidth=2, label="moving average (8 weeks)")
peak = demand.loc[demand["units"].idxmax()]
ax.annotate("Black Friday", xy=(peak["date"], peak["units"]),
            xytext=(peak["date"], peak["units"] + 60), arrowprops=dict(arrowstyle="->"), ha="center")
ax.set_xlabel("Week"); ax.set_ylabel("Units sold")
ax.set_title("Weekly demand for the NovaClean robot vacuum"); ax.legend(loc="upper left"); ax.grid(alpha=0.3)
fig.tight_layout(); fig.savefig("demand_moving_avg.png", dpi=120)

# 4) Subplots: scatter and line, side by side
fig, axes = plt.subplots(1, 2, figsize=(10, 4))              # 1 row x 2 columns -> array of axes
axes[0].scatter(orders["amount"], orders["delivery_days"], c=orders["returned"], cmap="coolwarm", s=8, alpha=0.6)
axes[0].set_xlabel("Amount (EUR)"); axes[0].set_ylabel("Delivery days"); axes[0].set_title("Orders (red = returned)")
by_day = orders.groupby("delivery_days")["returned"].mean() * 100
axes[1].plot(by_day.index, by_day.values, marker="o", color="darkred")
axes[1].set_xlabel("Delivery days"); axes[1].set_ylabel("Return rate (%)"); axes[1].set_title("Rate by delivery days")
fig.suptitle("Amount, delivery and returns"); fig.tight_layout(); fig.savefig("subplots.png", dpi=120)

What is seen in each file:

amounts_hist.png: an asymmetric bell with the peak in the 69-84 € bar (277 orders), the dashed median line at 108 € to the right of the peak and a tail stretching out to 630 € with a few dozen orders above 400. It is the generator's gamma distribution and the reason we talked about scaling and outliers in 04-03.
rate_bars.png: four descending bars, electronics 20.7 %, computing 15.9 %, home 13.5 %, accessories 13.0 %, with the dotted overall-mean line (16.4 %) crossing between the first two and the last two.
demand_moving_avg.png: grey dots oscillating each week between 308 and 825 units, and over them a smooth blue line rising from about 350 units at the start of 2024 to about 650 in mid-2025 with two yearly "humps" (maximum around July, minimum around January) and two isolated grey spikes that the moving average almost ignores: the Black Friday week of 2024 (724 units) and that of 2025 (825), the latter with the "Black Friday" arrow. Trend + seasonality + event, exactly what we said was there in 04-02.
subplots.png: on the left, a cloud of points with seven horizontal bands (delivery days 1-7), almost all blue (not returned) and with red points concentrated to the right (high amounts) and towards the top (5-7 day deliveries); on the right, a line rising almost straight from 8.6 % returns with 1-day delivery to 26 % with 7 days. It is the relationship hidden in the generator's +0.35 * (delivery_days - 4).

Save the figures with fig.savefig; in a notebook (07-04) they display on their own. When the plot is statistical and you want it to come out right first time, seaborn (on top of Matplotlib: sns.histplot, sns.barplot, sns.heatmap for correlation and confusion matrices) saves code; for interactive plots on the web, plotly. Both use DataFrames directly. pandas itself has df.plot() as a shortcut over Matplotlib.

"Operation → how to do it" table

Operation (what we did in module 4)	NumPy	pandas
Create reproducible data	`rng = np.random.default_rng(42)`; `rng.normal`, `rng.integers`, `rng.choice`	`pd.DataFrame({...})` with the arrays
Filter rows by condition	`a[a > 100]`, `a[(c1) & (c2)]`	`df[df["amount"] > 100]`, `df.query("...")`
Vectorised conditional	`np.where(cond, x, y)`	`np.where` over columns, `pd.cut` for bands
Count / proportion satisfying	`(a > 100).sum()` / `.mean()`	`(df["x"] > 100).sum()`, `df["returned"].mean()`
Mean/std per column	`a.mean(axis=0)`, `a.std(axis=0)`	`df.mean(numeric_only=True)`, `df.describe()`
Standardise	`(a - a.mean(0)) / a.std(0)` (broadcasting)	the same, or `StandardScaler` (07-03)
Rate per group	(by hand with masks per value)	`df.groupby("cat")["returned"].mean()`
Several named aggregations	—	`df.groupby("k").agg(n=("x","size"), m=("y","mean"))`
Join two tables by key	—	`df1.merge(df2, on="key", how="left")`
New derived columns	`a[:,0] / a[:,1]`	`df["c"] = df["a"] / df["b"]`
Missing values	`np.isnan(a)`, `np.nanmean(a)`	`df.isna().sum()`, `df.dropna()`, `df.fillna({...})`
Dates	`np.datetime64` (basic)	`pd.to_datetime`, `.dt.month`, `resample("MS").sum()`, `rolling(4).mean()`
Matrix product / linear layer	`X @ w + b`	`df.values @ w`
Read/write	`np.loadtxt`, `np.save`	`pd.read_csv`, `df.to_csv(index=False)`, `read_sql`, `read_parquet`
Convert to scikit-learn / PyTorch	already an array	`df.values` / `df.to_numpy()`; `torch.tensor(df.values, dtype=torch.float32)`

Common Mistakes and Tips

for loops over the rows of a DataFrame (for i in range(len(df)), iterrows). It is the number one mistake of people arriving from other languages; there is almost always a vectorised operation, a groupby or a merge that does it in one line and a hundred times faster.
Modifying a view believing it is a copy (NumPy) or a copy believing it is the view (pandas with df[mask]["col"] = ...). Rules: .copy() when you want independence; df.loc[mask, "col"] = ... to assign.
Confusing loc and iloc after filtering or sorting: df.loc[0] is the row with label 0, which may not exist; df.iloc[0] is the first row. reset_index(drop=True) after a filter gives you labels 0..n-1 back if you need them.
Incompatible shapes: read the message could not be broadcast together with shapes (a,) (b,) and apply the right-to-left rule; reshape(-1, 1) turns a vector (n,) into a column (n, 1), which scikit-learn demands when there is a single feature.
Dates read as text: after read_csv, df.dtypes should say datetime64; if it says object/str, apply pd.to_datetime(..., dayfirst=True) or parse_dates when reading. And in a Spanish CSV check sep=";" and decimal=",".
Silent NaN: mean() ignores them and you may not notice that data is missing. info() and isna().sum() always at the start.
Plots without labels or units: an ax.set_xlabel takes a second and stops Diego asking "are these euros or units?". And always save with tight_layout() so the text is not cut off.

Exercises

Exercise 1. With pure NumPy (no pandas), starting from orders["amount"].to_numpy() and orders["returned"].to_numpy(): (a) compute the return rate of the orders above and below the median amount using masks; (b) standardise the amount with broadcasting and check that the resulting mean is ≈ 0 and the standard deviation ≈ 1; (c) build with np.where an "expensive"/"cheap" label and count how many there are of each with a mask.

Exercise 2. With pandas: compute the mean amount and the return rate by category and by new/returning customer in a single table with groupby and named aggregation, and take it to wide format with unstack so that the columns are new/returning. In which category is the difference in rate between new and returning customers largest? Then join with the products table of section 9 and compute the total cost of returns by warehouse and customer type.

Exercise 3. With the weekly demand: (a) resample to quarters ("QS") summing units and identify the quarter with the most sales; (b) compute the 12-week moving average and the week-on-week percentage change of the smoothed series (pct_change), and locate the week with the largest relative rise; (c) plot (saving to file) the units per year as two overlaid lines (day of the year on the x axis, one line per year; hint: .dt.year and .dt.dayofyear) and describe what you see.

Solutions

Solution 1.

amount = orders["amount"].to_numpy(); returned = orders["returned"].to_numpy()
median = np.median(amount)
print(returned[amount > median].mean().round(3), returned[amount <= median].mean().round(3))
# 0.254 0.075   -> expensive orders are returned more than 3 times as often
z = (amount - amount.mean()) / amount.std()
print(z.mean().round(10), z.std().round(6))      # 0.0  1.0   (rounding: the mean is of the order of 1e-16)
label = np.where(amount > median, "expensive", "cheap")
print((label == "expensive").sum(), (label == "cheap").sum())   # 1500 1500

With 3,000 values and the median at the central position, exactly half falls on each side. Notice that we did not even need pandas: this is what scikit-learn does underneath with the numeric columns.

Solution 2.

t = orders.groupby(["category", "new_customer"]).agg(mean_amount=("amount", "mean"),
                                                     rate=("returned", "mean")).round(3)
wide = t["rate"].unstack().rename(columns={0: "returning", 1: "new"})
wide["difference"] = wide["new"] - wide["returning"]
print(wide.sort_values("difference", ascending=False))
#              returning    new  difference
# electronics      0.116  0.415       0.299
# home             0.069  0.297       0.228
# computing        0.099  0.304       0.205
# accessories      0.068  0.259       0.191
merged = orders.merge(products, on="category", how="left")
merged["cost_incurred"] = merged["returned"] * merged["return_cost"]
print(merged.groupby(["warehouse", "new_customer"])["cost_incurred"].sum().unstack())
# new_customer        0       1
# warehouse
# Getafe         2083.0  2947.0
# Zaragoza        872.0  1570.0

The largest difference is in electronics (30 points): the generator combines the category effect with the new-customer effect and the new × amount cross term, and electronics has high amounts. In cost, new customers (30 % of the orders) generate 60 % of the returns cost in both warehouses (2,947 € of 5,030 in Getafe; 1,570 € of 2,442 in Zaragoza): a quantified argument for Diego to prioritise use case 3 on new customers.

Solution 3.

series = demand.set_index("date")["units"]
quarterly = series.resample("QS").sum()
print(quarterly.idxmax().date(), quarterly.max())      # 2025-07-01 8404  (third quarter of 2025)
smooth = series.rolling(12).mean()
change = smooth.pct_change() * 100
print(change.idxmax().date(), change.max().round(2))   # 2024-04-22 3.4 %  (the week the spring rise kicks in)
demand["year"] = demand["date"].dt.year; demand["day_of_year"] = demand["date"].dt.dayofyear
fig, ax = plt.subplots(figsize=(9, 4))
for year, g in demand.groupby("year"):                   # one group (and one line) per year
    ax.plot(g["day_of_year"], g["units"], marker=".", label=str(year))
ax.set_xlabel("Day of the year"); ax.set_ylabel("Units"); ax.set_title("NovaClean: weekly demand, one line per year")
ax.legend(); fig.tight_layout(); fig.savefig("demand_by_year.png", dpi=120)

You see two lines with the same shape (rising until the summer, falling until the winter, with an isolated spike at the end of November, Black Friday), the 2025 one shifted about 130 units above the 2024 one along its whole length (yearly means: 471 and 597 units): the trend (+2.5 units per week × 52 weeks) and the seasonality separated at a glance. That is the plot Marta would show Diego before talking about forecasting models (use case 2).

Conclusion

We have opened the basic toolbox of scientific Python with NovaMarket's data. NumPy provides the ndarray (contiguous block, shape and dtype), creation and indexing with views, vectorised operations and broadcasting (with which we standardised columns in one line), aggregations by axis, boolean masks to filter and count, the default_rng generator that makes our 3,000 orders reproducible and the @ product with which any linear layer is written; and it is the foundation of everything else, including PyTorch tensors, which are arrays with a gradient. pandas adds labels: Series and DataFrame, read_csv/to_csv, the compulsory inspection with head/info/describe, selection with loc/iloc and masks, the treatment of NaN, the groupby that answers Diego's questions (electronics 20.7 % returns; new customers 3.7 times more), the merge with the products table that turns returns into euros per warehouse, and dates with .dt, resample (weekly to monthly) and rolling. Matplotlib draws with figure and axes: histogram of amounts, rate bars, the demand line with its moving average and Black Friday marked, and subplots. The table in section 13 summarises how to do each operation that in module 4 we took for granted.

With arrays and tables under control, the next step in Marta's workshop is the map of the libraries built on top: scikit-learn and its fit/predict API, PyTorch versus TensorFlow/Keras, Hugging Face for language, OpenCV for vision, the LLM SDKs, experta and pgmpy for rules and probability, OR-Tools for optimisation, and the tools for saving, serving and tracking models. That is 07-03, Popular Tools and Libraries, where we will also save and reload the pipeline of 04-03 and the MLP of 05-02 and check that they still predict the same.

Scientific Python: NumPy, pandas and Matplotlib

Contents

NumPy: the `ndarray`, shape and `dtype`

Creation, indexing and slicing

Vectorised operations and broadcasting

Aggregations by axis, boolean masks and randomness

Basic linear algebra and why NumPy is the foundation of everything

pandas: `Series` and `DataFrame`, reading, writing and inspection

Selection and filtering: `loc`, `iloc` and masks

Missing values, `groupby` and aggregations

Joining tables with `merge`

Dates: `to_datetime`, `.dt`, `resample` and `rolling`

`apply` in moderation

Matplotlib: figure, axes and the four basic plots

"Operation → how to do it" table

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Fundamentals of Artificial Intelligence (AI)

Module 1: Introduction to Artificial Intelligence

Module 2: Basic Principles of AI

Module 3: Algorithms in AI

Module 4: Machine Learning

Module 5: Neural Networks and Deep Learning

Module 6: Logic and Expert Systems

Module 7: Tools and Programming Languages in AI

Module 8: Projects and Case Studies

Module 9: Exercises and Practice

Module 10: Additional Resources

Scientific Python: NumPy, pandas and Matplotlib

Contents

NumPy: the ndarray, shape and dtype

Creation, indexing and slicing

Vectorised operations and broadcasting

Aggregations by axis, boolean masks and randomness

Basic linear algebra and why NumPy is the foundation of everything

pandas: Series and DataFrame, reading, writing and inspection

Selection and filtering: loc, iloc and masks

Missing values, groupby and aggregations

Joining tables with merge

Dates: to_datetime, .dt, resample and rolling

apply in moderation

Matplotlib: figure, axes and the four basic plots

"Operation → how to do it" table

Common Mistakes and Tips

Exercises

Solutions

Conclusion

Fundamentals of Artificial Intelligence (AI)

Module 1: Introduction to Artificial Intelligence

Module 2: Basic Principles of AI

Module 3: Algorithms in AI

Module 4: Machine Learning

Module 5: Neural Networks and Deep Learning

Module 6: Logic and Expert Systems

Module 7: Tools and Programming Languages in AI

Module 8: Projects and Case Studies

Module 9: Exercises and Practice

Module 10: Additional Resources

NumPy: the `ndarray`, shape and `dtype`

pandas: `Series` and `DataFrame`, reading, writing and inspection

Selection and filtering: `loc`, `iloc` and masks

Missing values, `groupby` and aggregations

Joining tables with `merge`

Dates: `to_datetime`, `.dt`, `resample` and `rolling`

`apply` in moderation