In the previous lesson we trained the first model of the course: a tree that learned, from orders labelled as returned or not, to predict the label of new orders. That is only one of the kinds of problem machine learning knows how to solve. When Marta goes over the list of NovaMarket's nine use cases she discovers that they do not all have the same shape: in some there is a known correct answer for every historical example (was it returned? how many units were sold?), in others there is no label at all and what is sought is structure (what kinds of customers do we have?), and in others there is not even a fixed dataset, but a system that acts and receives rewards or penalties (which coupon should be offered?). This lesson presents the four major paradigms of machine learning (supervised, unsupervised, semi- and self-supervised, and reinforcement), compares them in a table and illustrates them with code on NovaMarket's data. It matters because the first decision in any project is identifying what type of problem we have: the data to gather, the applicable algorithms and the way to measure success all depend on it.

1. The four paradigms at a glance
2. Supervised learning: classification and regression
3. Unsupervised learning: clustering, dimensionality reduction and anomalies
4. Semi-supervised and self-supervised
5. Reinforcement learning
6. Comparison table of paradigms
7. Python examples: classification, regression, clustering and a multi-armed bandit
8. Common Mistakes and Tips
9. Exercises
10. Conclusion

The most useful way to classify ML problems is to ask what "correct answer" information we have in the data:

In the vocabulary of 04-01: the paradigm depends on whether we have the label (y), on whether we have it for all instances or only some, or on whether the "label" is a reward that arrives after acting.

It is the most widely used paradigm in business and the one that will take up most of the module. Supervised means that every training example carries its correct answer, as if a supervisor had marked the exercise: the model learns the function that goes from the features to the label and then applies it to new examples.

It splits in two according to the type of label:

Nuances worth being clear about:

• Binary vs multiclass. "Returned yes/no" is binary classification; "review positive/neutral/negative" is multiclass (three mutually exclusive classes). There is also multilabel classification, in which an example can have several labels at once (an incident can be "delay" and "damaged product").
• The boundary is the question, not the data. The amount of an order is a number; predicting it is regression. But "will the order exceed €100?" is classification. The same data admits both formulations, and choosing well depends on the business decision that is going to be taken.
• Demand forecasting is regression on time series. It has a peculiarity, the ordering in time, which forces a special kind of validation (04-05) and offers features of its own (week of the year, previous week's sales: 04-03).

Here there is no label. We only have the features of each example, and what we ask of the algorithm is that it discover structure: groups, principal directions, unusual cases. It is the most frequent situation in practice (labelling costs money: someone has to read and classify each review) and the one that demands the most care when interpreting, because there is no correct answer to check against.

Three main tasks:

A warning Marta will repeat to Diego: a clustering algorithm always returns groups, even if the data have no structure. That k-means says there are three segments does not prove they exist; you have to look at them, give them business names and check that they behave differently (we will do it in section 7). In 04-04 we will see how k-means works internally and how to choose the number of groups.

Between the two extremes there are very common intermediate situations:

• Semi-supervised. We have labels for a small part of the data and many unlabelled examples. It is the situation of incidents.csv at NovaMarket: the support team has classified by hand half of last quarter's tickets (delay, damaged product, order error, warranty...) and the other half has no category. Instead of throwing away the unlabelled half, semi-supervised methods use it to learn the structure of the text better and propagate the labels to similar examples (for instance, train on the labelled ones, predict the rest, keep the most confident predictions as "pseudo-labels" and retrain).
• Self-supervised. There are no human labels, but the data itself provides the supervision signal: part of the example is hidden and the model is asked to reconstruct it. This is how large language models learn from unlabelled text (predicting the next word or the masked word), and how many image models are pretrained. It is the reason generative AI has taken off: it turns oceans of unlabelled data into useful experience. It is developed in 05-05.

Both appear here only so that you recognise the situation when you meet it; in this course we will mostly work with supervised and unsupervised learning.

The fourth paradigm is different from the previous ones: there is no fixed dataset. There is an agent (02-01) that acts in an environment, observes the state, chooses an action and receives a reward (positive or negative), and its goal is to learn a policy (which action to take in each state) that maximises the accumulated reward in the long run.

The elements, with the conceptual example of an agent deciding which coupon to offer each customer at checkout:

Two ideas specific to reinforcement:

• Exploration versus exploitation. The agent only learns from the actions it tries. If from the start it sticks with the coupon that seems best (exploit), it will never know whether another would have worked better; if it tries at random all the time (explore), it wastes reward. A balance is needed: the simplest strategy, epsilon-greedy, explores a fraction ε of the time and exploits the rest. We will see it in code.
• Delayed reward. The prize often arrives long after the action that caused it (in chess, victory comes after forty moves). Assigning the credit to the right actions (credit assignment) is the hard part, and it is solved by algorithms such as Q-learning or policy-gradient methods, beyond the scope of this course.

Connections with what we have already seen: the agent-environment-reward framework is the PEAS of 02-01 with the performance measure turned into a learning signal; and AlphaGo (01-01, 03-03) combined the adversarial search we studied with minimax with neural networks trained by reinforcement, playing millions of games against itself. It is also applied in robotics, systems control (data-centre cooling, traffic) and, more recently, in fine-tuning large language models from human preferences (05-05).

The four examples are deliberately short: the goal is to recognise the shape of each type of problem. The algorithms are explained in 04-04 and proper evaluation in 04-05; here we will use provisional measures. We assume that the function generate_orders_ml from 04-01 is saved in novamarket_ml.py.

We repeat the problem from 04-01 with another algorithm (logistic regression) to check that the interface is identical:

Output:

The label is a category (0/1) and the output, a class plus a probability per class: the signature of classification. Logistic regression gets slightly more right than the tree of 04-01 (88.3 % versus 86.8 %), but remember the warning about accuracy with imbalanced classes: draw no conclusions until 04-05.

We need a sales series. The function generate_weekly_demand creates the units sold per week of a fictional product (the NovaClean robot vacuum) with a growing trend, yearly seasonality, the Black Friday spike and noise. Add it to novamarket_ml.py as well:

And the model, a linear regression that uses only the week number as a feature:

Output:

Explanation:

• The label (units) is a number: regression. LinearRegression learns a straight line units ≈ 391 + 2.74 × week; the slope 2.74 is close to the true trend of 2.5 units per week that we hid in the generator.
• We split by time, not at random: we train on the past (weeks 1-78) and test on the future (79-104). Shuffling weeks would mix future and past, and the result would be deceptively good. This is the temporal validation of 04-05.
• The mean error of 47 units is large because the straight line ignores seasonality and Black Friday. In 04-03 and 04-04 we will add features (week of the year, previous week's sales) to capture them. What matters now is the shape of the problem: numeric input, numeric output.

Now there is no label. We generate fictional customers with three behavioural features (annual spend, number of orders, tenure in months) and ask k-means to find three groups. Add the function to novamarket_ml.py:

Output:

Explanation:

• Notice that fit_predict receives only X: there is no label to learn. The algorithm assigns each customer to the group whose centre is closest (04-04 explains the detail).
• StandardScaler is necessary because k-means measures distances: without scaling, annual spend (hundreds or thousands of euros) would completely dominate the number of orders (single units). Scaling and distances are covered in 04-03.
• The interpretation is supplied by the business, not the algorithm. Looking at the summary, Marta christens the segments: 0 are occasional (351 customers, spending €186 a year in 2-3 orders, with us for less than a year), 2 are regular (197 customers, about €700 in 9 orders, more than two years) and 1 are big spenders (52 customers, €2,200 in 20 orders). For the recommender (case 1) this means, for example, that occasional customers should be shown entry-level products and big spenders new releases and high-end accessories. The segment numbers (0, 1, 2) are arbitrary: if you change the seed they may be permuted.
• The groups k-means has found match the three "hidden kinds" of the generator; on a real dataset we would not have that confirmation, and the segments would have to be validated with the sales team and with the elbow method (04-04).

The "multi-armed bandit" is the simplest reinforcement problem: several slot machines (arms) with unknown payout probabilities, and you have to decide which one to pull each round to win as much as possible. There is no state (it is always the same situation) and the reward is immediate, so only the exploration-exploitation dilemma remains. We program it in pure Python for NovaMarket's four coupons; the reward is 1 if the customer buys again in the following month:

Output:

Explanation:

• The agent keeps, for each coupon, an estimate of its success rate (successes / tries). With probability epsilon it picks a coupon at random (explores); otherwise it picks the one with the best estimate (exploits). The estimate is updated with every reward: that is the learning.
• With epsilon=0 (exploit only) the agent tries "no coupon" in the first round, obtains a positive estimate before the others and never tries anything else again: 294 repeat purchases. It is the trap of not exploring, the reinforcement version of the local optimum of 03-04.
• With epsilon=0.1 it soon discovers that free shipping works best (11 %) and uses it 4,091 times: 510 repeat purchases, 73 % more. With epsilon=0.5 it explores too much and loses a little (505). The value of ε is a hyperparameter and tuning it is a classic problem.
• A real system would add the state (the optimal coupon depends on the cart and the customer: a "contextual bandit") and the cost of the coupon in the reward. But the essence is already here: learning by acting, with no prior dataset, balancing exploration and exploitation.

• Confusing classification with regression because of the data type. The question defines the problem: "how much" is regression, "which/whether" is classification. Predicting the amount is regression; predicting whether it exceeds a threshold is classification.
• Using clustering expecting "the answer". k-means always finds groups; their value depends on their being interpretable and actionable. Name them, compare them and validate with the business.
• Shuffling time series. For demand, train on the past and evaluate on the future; never train_test_split with shuffling (04-05 formalises it with TimeSeriesSplit).
• Forgetting to scale before clustering. Without StandardScaler, the column with the largest magnitude decides the groups.
• Thinking reinforcement is "better" because it sounds advanced. It needs many interactions and an environment where making mistakes is cheap (or can be simulated). For most NovaMarket cases, supervised and unsupervised are the answer; reinforcement makes sense for repeated decisions with fast feedback (coupons, prices).
• Ignoring the unlabelled data. If half of incidents.csv has no category, do not discard it without thinking: it can serve for semi-supervised learning or, at the very least, to understand the distribution.

Exercise 1. Classify each of NovaMarket's nine use cases (recommendation, demand forecasting, fraud/returns, review classification, routes, order assignment, assistant, returns/warranty rules, incident diagnosis) according to the main paradigm or technique you would apply: supervised (classification or regression), unsupervised, reinforcement, or "not ML" (search/optimisation from module 3, logic from module 6, generative AI from module 5). Justify each one in a single line.

Exercise 2. In the regression of section 7.2, change the temporal cut-off to train only on the first year (week <= 52) and evaluate on the second. Does the mean error get worse or better? Then, deliberately make the mistake of shuffling: replace the model with KNeighborsRegressor(n_neighbors=3) (from sklearn.neighbors, which predicts with the mean of the 3 most similar weeks) and compare it with the temporal split (weeks 1-78 / 79-104) and with a shuffled split train_test_split(demand[["week"]], demand["units"], test_size=0.25, random_state=42). Why does the error come out lower with the shuffle, and why is that misleading?

Exercise 3. Modify simulate_coupons so that exploration changes over time: during the first 400 rounds always choose at random (ε = 1) and from then on exploit almost always (ε = 0.02). Compare the total repeat purchases with epsilon=0.1 for several seeds (seed=1..5). Also try an ε that decays quickly, epsilon = max(0.01, 1 / (round_ + 1) ** 0.5). Which strategy works best and why?

Solution 1. (1) Recommendation: unsupervised (segmentation) and supervised (predicting the probability of purchase), sometimes with reinforcement to choose what to show. (2) Demand forecasting: supervised, regression on a time series. (3) Fraud/returns: supervised, classification; anomaly detection (unsupervised) when there are no fraud labels. (4) Review classification: supervised, multiclass classification on text (with self-supervised pretrained models, module 5). (5) Routes: not ML, it is search/optimisation (03-02, 03-04). (6) Order assignment: optimisation (03-04); ML could be added to forecast times. (7) Assistant: generative AI (LLM, 05-05), not classical ML trained by Marta. (8) Returns/warranty rules: logic and expert systems (module 6), not ML. (9) Incident diagnosis: supervised (ticket classification), with semi-supervised if labels are missing; also expert systems for the rules part (06-04).

Solution 2. Training only on the first year, the line is fitted with fewer points and the estimated trend is worse, so the error on the second year grows (about 59 units versus 47). With KNeighborsRegressor(n_neighbors=3), the temporal split gives a mean error of about 36 units and the shuffled one about 29: the shuffle looks better. It is misleading because, when shuffling, the "3 most similar weeks" of a test week are its immediate neighbours, which the model has seen in training; in production, to forecast week 105 you do not have the sales of week 104, let alone those of week 106. The temporal split reproduces the real situation (predicting the future from the past) and that is why it is the only honest one; in 04-05 we will formalise it with TimeSeriesSplit.

Solution 3. The strategy "explore 400 rounds and then exploit" matches or beats ε = 0.1 for most seeds (in our tests, 512/551/517/493/534 repeat purchases versus 510/552/461/409/508): after 400 spread-out tries (about 100 per coupon) the estimate of each rate is already reasonable and it is not worth continuing to spend 10 % of the rounds exploring. By contrast, the ε that decays as 1/√round works worse (about 380 repeat purchases with seed 1): it drops so fast that after a few dozen rounds it hardly explores any more, and gets "stuck" on the coupon that was lucky at the start, with estimates based on very few tries. The lesson is twofold: you have to explore enough before committing (the equivalent of the random restarts of 03-04) and, if the environment changes (a coupon stops working), a near-zero ε takes a long time to notice; in practice a minimum level of exploration is always kept.

In this lesson we have organised machine learning into its major paradigms: supervised (classification when the label is a category, regression when it is a number), unsupervised (clustering, dimensionality reduction, anomaly detection, where there is no label and the interpretation is supplied by the business), the intermediate semi-supervised and self-supervised, and reinforcement learning, in which an agent learns by acting and receiving rewards, with the exploration-exploitation dilemma as its distinguishing feature. We have compared them in a table and seen them in code on NovaMarket: logistic regression for returns, linear regression for NovaClean demand (with the new function generate_weekly_demand), k-means to segment customers (with generate_customers_ml) into occasional, regular and big spenders, and an epsilon-greedy bandit to choose coupons.

We now know what type of problem each use case is. But, before training anything serious, we have to do the work that in 04-01 we said takes most of the time: in the next lesson, Data Preparation and Features, we will take a "dirty" version of NovaMarket's orders (missing values, an impossible amount, categories as text, dates) and build step by step the pipeline that turns them into the numeric matrix the algorithms need, avoiding along the way the data leakage that ruins so many projects.