When we closed module 3 we left a promise hanging: instead of programming the algorithm that solves the problem ourselves, we would hand the system historical data and let it learn the rule on its own. This lesson keeps that promise and opens module 4. We will see exactly what "learning from data" means (using Tom Mitchell's classic definition), fix the vocabulary we will use throughout the module (dataset, feature, label, model, parameters, loss, generalisation...), draw the complete workflow of a machine learning project as a map of the lessons to come, and discuss when ML is worth using and when a fixed rule is better. We will finish by training the first "real" model of the course with scikit-learn: a NovaMarket returns predictor that we will compare with Diego's manual rule ("risk if the amount exceeds €300") and with the threshold we learned by hand in 01-02. This matters because everything that follows (types of learning, data preparation, algorithms, evaluation, neural networks) rests on the ideas and vocabulary of this lesson.

1. What learning from data means: Mitchell's definition
2. From learn_threshold to machine learning: nothing new under the sun
3. Essential vocabulary
4. The workflow of an ML project: map of the module
5. When to use ML and when not to
6. Python example: NovaMarket's data and the first model with scikit-learn
7. Common Mistakes and Tips
8. Exercises
9. Conclusion

In 01-02 we saw that a traditional program receives data + rules and produces results, whereas machine learning receives data + results and produces rules (a model). That picture is useful, but we need an operational definition, one that tells us when a program "has learned". The most widely quoted is Tom Mitchell's (1997):

A program learns from experience E with respect to a task T and a performance measure P if its performance on T, as measured by P, improves with E.

Applied to NovaMarket's use case 3 (predicting whether an order will be returned):

The definition has two practical consequences worth internalising from day one:

• No performance measure, no learning. "The model seems to work" means nothing; P has to be defined before you start. It is the same demand as the performance measure of the rational agent in 02-01.
• The improvement must be measured on new data. A system that memorises the history and gets 100 % of the orders it has already seen right has learned nothing useful: what matters is what it does with tomorrow's order. We will call that ability generalisation, and it will be the thread running through 04-05 and 04-06.

Notice that Mitchell says nothing about "intelligence", nor about how learning happens. It is a functional definition: if performance improves with experience, there is learning. This fits the "acting rationally" view we adopted in 01-02.

You already did machine learning in 01-02 without calling it that. Remember the function learn_threshold(data): it walked through the 13 historical orders, tried each amount as a candidate threshold for the rule amount > threshold, counted the hits of each one and kept the best (€120, which got 11 of 13 right, against 7 of 13 for Diego's rule). In Mitchell's terms:

• T: decide whether an order is at risk of being returned.
• E: the 13 orders with their actual outcome.
• P: number of hits.

And in module 3 terms, learn_threshold was an exhaustive search (03-01) over a space of 13 candidates with an objective function (the hits) to be maximised (03-04). That is exactly what any machine learning algorithm does, with three differences of degree:

1. The family of candidate rules is far richer than "one threshold on one column": combinations of many columns, trees of questions, weighted sums, networks of neurons.
2. The search space is so large (often infinite) that it cannot be walked in full; the strategies of 03-04 are used, such as climbing along the gradient, instead of brute force.
3. The performance measure is computed on held-out data the algorithm has not seen, to measure generalisation rather than memory.

That is why we closed 03-04 with the sentence "learning is optimising": training a model means searching, in an enormous space of possible models, for the one that minimises an error function on the training data, and trusting that this model will also work on new data. The whole of module 4 is the grown-up version of learn_threshold.

These are the terms that will appear in every lesson of the module. The examples are taken from the returns predictor we will build in section 6.

Two clarifications that head off frequent confusions:

• Parameters versus hyperparameters. In learn_threshold, the threshold was the parameter (the data chose it); had we decided "we will only try thresholds that are multiples of 10", that would have been a hyperparameter (we chose it). Parameters are learned; hyperparameters are tuned by trying and validating (04-06).
• Loss versus performance measure. The loss is what the algorithm minimises internally during training; the performance measure (P) is what we care about in the end (hits, euros saved). Sometimes they coincide and sometimes they do not: a mathematically convenient loss is minimised and the business metric is used for evaluation (04-05).

One last notational convention you will see across the literature and in scikit-learn: the table of features is called X (upper case, because it is a matrix of rows × columns) and the vector of labels is called y (lower case, because it is a single column).

Training the model is only one step, and not even the longest, of a machine learning project. The complete workflow looks a lot like the agent cycle of 02-01, but applied to building the model:

Each step has its lesson in this module or elsewhere in the course:

Two remarks about the diagram:

• It is a cycle, not a line. You almost never get it right first time: evaluation reveals that features are missing, that there is data leakage or that the model overfits, and you go back. Marta must plan for iterations.
• Training is the small part. In a real project, step 3 takes most of the time, and step 7 is the one most often forgotten and the most expensive to forget. Lesson 08-01 walks through this same workflow as a complete project, with roles, deliverables and decisions.

Diego, quite reasonably, asks why NovaMarket needs a model when it already has a rule that "works". Machine learning is not always the answer; sometimes a fixed rule (or an SQL query) is better. These are the signals:

Applied to the nine NovaMarket use cases we catalogued in 01-03: returns prediction (3), demand forecasting (2), recommendation (1) and review classification (4) are clear ML candidates (many examples, patterns that are hard to write, changing over time). Routes (5) and assignment (6) we solved in module 3 with optimisation, without learning anything. The returns and warranty rules (8) are, precisely, fixed rules: they are handled with logic and expert systems (module 6). Choosing the tool well is the first decision of step 1 of the workflow.

A practical golden rule: always start from the simplest possible rule as a baseline (Diego's, for example) and demand that the model beat it measurably. We will do exactly that right now.

As in the rest of the course, we do not have the real orders.csv, so we will generate fictional orders with code. What matters is that the generating function is reproducible (same seed, same data) and that it encodes a realistic but noisy relationship, as happens in reality: expensive orders, from new customers, with slow deliveries and in electronics are returned more often, but no rule is right every time. We will reuse this function in every lesson of the module, so save it in a file novamarket_ml.py.

Output:

Line-by-line explanation:

• np.random.default_rng(seed) creates a random number generator with a fixed seed: every time you call the function with the same seed you will get exactly the same orders. This is essential for your results and those in this lesson to match (barring small differences between NumPy versions).
• Each column is generated with a sensible distribution: amounts follow a gamma distribution (many small orders and a tail of expensive ones, mean around €125), items and delivery days are uniform integers, 30 % of customers are new, and category and zone are drawn with fixed probabilities.
• The variable z is the hidden truth: a risk score that combines the columns with weights we have chosen. In the real world that formula does not exist, or nobody knows it; here we write it so that we can check afterwards whether the model discovers it. Notice that postcode_zone does not appear in z: the zone does not influence the return. We do this on purpose, remembering the postcode bias of 02-04; in 04-03 we will check that a good model gives it practically zero importance.
• prob = 1 / (1 + np.exp(-z)) converts the score into a probability between 0 and 1 (this function, the sigmoid, reappears in 04-04 with logistic regression and in module 5).
• returned is decided by a random draw according to that probability. This introduces noise: two identical orders can end up one returned and the other not. No model, however good, will get 100 % right; it is worth knowing before chasing the impossible.
• The result is a pandas DataFrame (a table), with 16.4 % of orders returned. In 04-05 we will see that this imbalance (many more "no" than "yes") shapes how evaluation must be done.

We are going to train a decision tree with scikit-learn. We will not go into how it works internally (that is 04-04); for now we care about the usage pattern, which is identical for every model in the library. We will use only the numeric columns; the text ones (category, postcode_zone) need a prior transformation that we will see in 04-03.

Output:

Step by step:

1. train_test_split shuffles the orders and splits them into training (2,250) and test (750). random_state=42 fixes the shuffle so that it is reproducible; stratify=y guarantees that both parts have the same proportion of returns (16.4 %). The test set is the "future experience" with which we will measure generalisation; it is not touched until the end. The underlying reason is developed in 04-05.
2. DecisionTreeClassifier(max_depth=3) creates the still-empty model. max_depth=3 is a hyperparameter: at most three chained questions. We choose it; in 04-06 we will learn to choose it with data.
3. model.fit(X_train, y_train) is the training: the search for the parameters (which column to ask about at each node and with what threshold) that best separate returned orders from non-returned ones. It is the sophisticated version of learn_threshold, and in 04-04 we will see exactly what it optimises.
4. model.score(X, y) computes the proportion of hits (accuracy). On training it gets 87.3 % right and on test 86.8 %: similar figures, a sign that the model generalises rather than memorises (in 04-06 we will see what happens when that is not the case).
5. predict returns the class (0 or 1) of each new order, and predict_proba the estimated probability of each class. The €350 order from a new customer has a 95 % probability of being returned; the €180 one is almost at 50 %, a clear case for the human review band of 02-04.

This pattern (fit → predict/predict_proba → score) is the same for a logistic regression, a random forest or a support vector machine: the algorithm changes, the interface does not. It is one of the reasons for scikit-learn's success, which is presented as a tool in 07-03.

We evaluate the earlier rules on the same test set, to compare on equal terms:

Output:

Reading the results:

• Diego's rule gets 84.3 % right... but the rule "no order is ever returned" gets 83.6 %. Since only 16 % of orders are returned, getting a lot right is easy: just always say no. Diego barely improves on that triviality. This accuracy trap with imbalanced classes is the starting point of 04-05; for now, hold on to the idea that the accuracy figure, on its own, misleads.
• The rule from 01-02 (threshold €120) does far worse here (62.7 %). It is not that that learning was wrong: it is that it was learned from 13 carefully chosen examples and does not generalise to a realistic history of 3,000 orders, in which most orders above €120 are not returned. It is the lesson of section 1: little data, unreliable model.
• The tree gets 86.8 % right, the best figure, and above all it does something qualitatively different: it combines the amount with whether the customer is new and with the delivery days. It flags 52 orders and is right on 38 (73 %), whereas Diego flags 37 and is right on 21 (57 %). It detects almost twice as many returns with fewer false alarms. Even so, 85 of the 123 returns slip past it: the improvement is real, but a lot of work remains, and that work is the rest of the module (more features in 04-03, better algorithms in 04-04, a threshold tuned to cost in 04-05, hyperparameters in 04-06).

Diego, sceptical, will ask the right question: "and what exactly has the tree learned?". You can see it with from sklearn.tree import export_text; print(export_text(model, feature_names=columns)). You will see rules of the kind "if amount > €195 and new customer → returned; if amount > €358 → returned". In other words, the tree has discovered that Diego was not far off (very expensive orders are indeed returned), but that for new customers the risk threshold drops to about €195. In 04-04 we will read trees in detail.

• Evaluating on the same data used for training. It is the number one mistake of beginners. A model that memorises gets 100 % of what it has already seen right and fails spectacularly on anything new. Always hold out a test set before training and do not look at it until the end.
• Confusing parameters with hyperparameters. Parameters are fitted by fit; hyperparameters you fix when creating the model (max_depth=3). If you find yourself "trying values by hand" for something, it is a hyperparameter, and 04-06 will give you the method to do it properly.
• Forgetting the baseline. Always compare with the dumbest reasonable rule ("never returned", "last week's average") and with the manual rule currently in force. A model that does not clearly beat them does not deserve to be deployed.
• Trusting the accuracy figure. With imbalanced classes, the 84 % of Diego's rule and the 83.6 % of "never" are almost the same thing. Wait until 04-05 before drawing strong conclusions from a single figure.
• Jumping straight to the algorithm. The workflow of section 4 starts by defining T, E and P and by getting to know the data. Marta will spend far more time preparing the data (04-03) than calling fit.
• Fixing the seed and forgetting about it. random_state=42 makes your experiments reproducible, but it does not mean the result is "the right one": try several seeds before claiming that one model is better than another. The outputs in this lesson may vary slightly depending on the version of NumPy or scikit-learn.

Exercise 1. Formulate, using Mitchell's definition (T, E, P), two NovaMarket use cases other than returns: demand forecasting (case 2) and review classification (case 4). For each one, state what the features, the label and an instance would be.

Exercise 2. Change the tree's max_depth hyperparameter to 1 and retrain. Display the rules with export_text and compare the test accuracy with the "never returned" rule. What threshold has the single-question tree chosen? Why does a single threshold on amount fail to beat "never" on this dataset, when in 01-02 it did beat Diego?

Exercise 3. Generate a second dataset with another seed (generate_orders_ml(3000, 7)) and evaluate the already-trained tree on it (without calling fit again), as if it were the following week's orders. Does the accuracy hold up? What does that tell you about generalisation? Then repeat, evaluating Diego's rule on those same data.

Solution 1.

• Demand forecasting. T: predict how many units of a product will be sold next week. E: the weekly sales history (orders.csv aggregated by product and week), with calendar and promotions. P: mean error in units between forecast and actual sales (in 04-05 we will call it MAE). An instance is "product X, week Y"; the features, the week of the year, the sales of the previous weeks, whether there is a promotion; the label, the units sold that week (a number, not a category: it is a regression problem, 04-02).
• Review classification. T: decide whether a review in reviews.csv is positive, neutral or negative (or whether it mentions a shipping problem). E: past reviews labelled by hand. P: percentage of correctly classified reviews, or better, per-class measures (04-05). An instance is a review; the features, the text (turned into numbers, 04-03) and the star rating; the label, the assigned class.

Solution 2. With max_depth=1 the tree chooses the question amount <= 195.65 and in both branches predicts class 0, so its test accuracy is 0.836, identical to "never returned". Even above €195 most orders are not returned (the rate there is around 40 %), so if only one question can be asked and the aim is to maximise hits, the best thing is to always say "no". In 01-02 the history of 13 orders was balanced (almost half returned), and that is why a threshold did win. Moral: the same family of models behaves differently depending on the class proportions; and to detect returns you need to combine variables (depth 3 already manages it) and use better measures than accuracy (04-05).

Solution 3. With new_orders = generate_orders_ml(3000, 7) and model.score(new_orders[columns], new_orders["returned"]) you will get around 0.88, even slightly above the 0.868 on test: the model generalises to orders it has never seen because the new data follow the same "hidden truth". Diego's rule again stays close to 0.85 (and "never" around 0.84). If in reality the conditions changed (a campaign with many new customers, a new supplier with more defects), generalisation would no longer be guaranteed: that is why step 7 of the workflow, monitoring, exists.

In this lesson we have defined machine learning with Mitchell's formula (task, experience, performance), we have checked that learn_threshold from 01-02 already met that definition and that "learning is optimising" (03-04) is literally what fit does, we have fixed the module's vocabulary (dataset, instance, feature, label, model, parameters versus hyperparameters, training, inference, loss function, generalisation), we have drawn the complete workflow as a map of the lessons to come and we have discussed when ML pays off compared with a fixed rule. In the code we have created the function generate_orders_ml, which will accompany us throughout the module, and we have trained the first scikit-learn model with the fit/predict/score pattern, which beats Diego's rule by combining several columns, although it still lets many returns slip through and has taught us that the accuracy figure, on its own, misleads.

In the next lesson, Types of Machine Learning, we will classify the problems ML can solve: the returns predictor is an example of supervised classification; demand forecasting will be supervised regression; customer segmentation for the recommender will be unsupervised; and we will also look at reinforcement learning, which connects with the agents of 02-01 and with AlphaGo. Marta and Diego will thus have a map to decide, for each of the nine use cases, what type of learning they need.