We closed module 7 with the workshop set up: a src/novamarket/ package with data.py, train.py and predict.py, tests that pass, a reproducible environment and a saved model with its metrics JSON. What was missing, we said, was a method: how to take a use case from the idea to a system that works in production and keeps working months later. This lesson provides it. We pick up the workflow diagram we presented in 04-01 as a map (problem → data → preparation → training → evaluation → deployment → monitoring) and develop it as a project, with the questions to answer in each phase, the roles involved and the mistakes that sink whole projects. And we do it on the case that has been with us since module 4: Marta and Diego take the returns predictor (use case 3) from idea to production inside novamarket_ai/, with a definition document, a decision to deploy in batches with a human review band, a model card generated from train.py, a predict_batch.py that sorts each night's orders into three queues and a drift check that warns when it is time to retrain. It matters because most AI projects do not fail because of the algorithm, but because one of these phases was skipped; and because in 09-04 you will make the same journey with your own project: here you have the method and the worked example.

1. From map to method: the life cycle and the CRISP-DM and CRISP-ML(Q) frameworks
2. Phase 1: define the business problem and success
3. Phase 2: data (sources, availability, leakage, temporal split, governance)
4. Phase 3: exploration and analysis
5. Phase 4: quick prototype (baseline → simple model → iterate)
6. Phase 5: evaluation with business criteria and with the stakeholders
7. Phase 6: deployment
8. Phase 7: monitoring and maintenance
9. Phase 8: documentation and communicating results
10. Team roles and typical project mistakes
11. Worked example: the returns predictor in novamarket_ai/
12. Common Mistakes and Tips
13. Exercises
14. Conclusion

In 04-01 we drew the machine learning workflow as a sequence with loops back. Here we redraw it as the life cycle of a project, adding what we left open then: the business definition at the start, deployment and vigilance at the end, and the fact that the arrows back are not exceptions but the norm.

This cycle is not an invention of the course: it is the shape taken by two frameworks widely used in industry.

• CRISP-DM (Cross-Industry Standard Process for Data Mining, late 1990s) defines six phases: business understanding, data understanding, data preparation, modelling, evaluation and deployment. Its great insight was to put business understanding before the data and to draw the arrows back.
• CRISP-ML(Q) is an update designed for modern machine learning: it adds monitoring and maintenance as a phase in its own right (a deployed model degrades) and a cross-cutting quality assurance (Q): in each phase, requirements, risks and checks are defined. Our eight-step cycle is essentially CRISP-ML(Q) with documentation made explicit as phase 8.

One idea worth fixing from now on: the time of an AI project is not split into equal parts. In real projects it is common for definition and data to consume more than half the effort, modelling a small fraction, and deployment and monitoring the rest. Anyone planning "two weeks of data and two months of model" is planning the project backwards.

It is the phase that saves the most projects and is skipped the most. It consists of answering these questions in writing, on one page, before touching a single row of data:

1. What question does the system answer? Not "predict returns", but "what is the probability that this order will be returned, known at the moment it is being prepared?".
2. What decision changes? If the prediction changes no action, the project is worthless however good the metric. At NovaMarket the decision is: approve the order as usual, pass it to a person, or prepare the reverse logistics in advance (reusable packaging, return label, do not close the stock replenishment).
3. What is the business metric and what is the technical one? The business metric is expressed in the unit of the person who decides: euros (cost of false positives and false negatives, 04-05), hours of work, customers retained. The technical one (AUC, F1, MAE) is a means. Both must be written down, together with the relationship between them.
4. What is the baseline? What is done today: nothing, a rule of Diego's ("> €300"), a manual process. Without a baseline there is no way to say whether the model adds anything.
5. What constraints are there? Legal (GDPR: legal basis, automated decisions; AI Act: risk level, 02-04), ethical (affected groups, proxies), operational (how many orders can one person review per day?), technical (how quickly is the answer needed?).
6. When will we consider it has worked? A verifiable success criterion: "reduce the cost of unanticipated returns by at least 25 % against the baseline within three months, without exceeding 300 manual reviews a day".

This is where the nine-point ethical checklist of 02-04 applies and, if the system decides about people, the impact assessment. Not as a formality afterwards: its answers condition which data can be used (point 2), which model (explainability, point 4) and what human review is needed (point 5).

The output of the phase is a definition document: a one-page table signed off by business and data. We will see it done for the returns predictor in section 11.

With the problem defined, it is time to inventory the data. The questions:

• Sources: which systems hold them? (orders.csv and customers.csv from the ERP, incidents.csv from the CRM, web logs). How often are they updated? Who owns them? Do they need joining (SQL, 07-01) and on which key?
• Availability at prediction time: the most important and most forgotten question. Every feature must exist when the model is going to be used, not when it is trained. return_reason exists in the history, but only after the return: it is leakage (04-03). The actual delivery_days is not known either when the order is prepared; the promised one is. And a customer's previous_orders must be computed against their history up to that day, not against the whole file.
• Labels: how is the truth obtained? A return is recorded weeks after the purchase: the model is trained on "mature" orders and in production the label arrives late, which affects how it is measured (phase 7).
• Temporal split: in a problem with dates, the test set must be later than the training set (TimeSeriesSplit, 04-05); a random split lets the model "see the future" and gives optimistic metrics.
• Volume and quality: how many positive cases there are (16.4 % of 3,000 orders is 492 returns: enough for a logistic regression, little for a deep network), NaN, duplicates, outliers (04-03).
• Governance: legal basis and minimisation (GDPR), who accesses what, where the raw data is kept (never in Git, 07-04), what is anonymised, how long it is retained. And a datasheet: origin, period, columns, known biases, restrictions on use.

A rule that saves grief: build the training dataset with the same query you will use in production. If training comes out of a manual Excel and production out of a SQL query, the columns end up meaning different things (the so-called training-serving skew). We will see it with days_since_start in section 11.

Before modelling, look. Exploration (07-02) has three goals in a project:

1. Confirm that the problem exists in the data: does the return rate really vary with amount, category, type of customer? If nothing varies, there is no signal to learn.
2. Detect quality and leakage problems: columns with too many NaN, impossible values (the €99,999 amount), columns that predict "too well" (suspected leakage).
3. Feed the definition: exploration often changes the question. If 60 % of returns are electronics, perhaps the first deployment should be limited to that category.

The deliverable is an exploration notebook (Jupyter, 07-04) with charts and a list of findings and of the decisions they trigger. It is the right phase for the notebook; whatever matures moves to src/.

The classic mistake is to start with the most powerful model. The right order:

1. Baseline without a model: "never returned" (€3,690 on the 750-order test set), Diego's rule "> €300" (€3,195). Any model has to beat that in euros.
2. Simple, interpretable model, with the full pipeline: logistic regression inside a Pipeline that imputes, scales and encodes (04-03, 04-04). Make it work end to end before improving anything.
3. Iterate with a budget: each iteration changes one thing (a new feature, an algorithm, a hyperparameter, 04-06) and is recorded with its metric. Marta gave herself a two-week budget for the prototype: the logistic regression gave AUC 0.844; the tuned forest 0.838; the 21→16→8→1 MLP, 0.834. None improved significantly on the logistic regression, which moreover explains its decisions. She stops: the time budget protects against "just one more try".

Practical rule: the prototype ends when two consecutive iterations do not move the business metric, not when the ideas run out.

Technical evaluation (04-05) is necessary but not sufficient. In a project you evaluate with the people who are going to live with the system:

• Business metric on the held-out test set: cost €2,485 at threshold 0.5 → €1,610 at threshold 0.2 (35 % less), against €3,690 for doing nothing.
• Operational capacity: at threshold 0.2, 200 orders out of 750 are flagged (27 %); Diego says his team cannot review that many. Hence the human review band 0.2-0.5: 60 automatic orders, 140 to review, 550 approved.
• Fairness: parity of flag rates across postcode zones (02-04) on the test set.
• Concrete cases: show Diego ten orders from each queue with the explanation (the coefficients of the logistic regression) so he can judge whether the model "makes sense". This step uncovers leakage and errors that no metric shows.
• Risks and plan B: what happens if the model fails (it is switched off and the manual process resumes; see rollback in phase 6).

The output is an explicit decision: proceed to deployment, go back to data, or stop. Stopping is a valid outcome of a well-run project; what is not valid is deploying because a lot has already been invested.

Deploying is integrating the model into a real flow. The decisions:

Two notes. Shadow mode means running the model in production without its outputs affecting anything, only to record them and compare later with reality: it is the cheapest way to discover that production data is not like the training data. And the A/B test (part of the orders with the model, part without, comparing the cost) is the only way to measure causal impact; at a conceptual level it is enough to know that it exists and that it requires assigning the groups at random and waiting until there are enough cases.

A deployed model starts going stale on day one. Three things must be watched:

1. Technical health: did the batch run? how many orders? how long did it take? were there errors or missing columns?
2. Data drift: the distribution of the inputs changes (higher amounts at Christmas, more new customers after a campaign, a new category). It is detected by comparing the recent distribution with the training one: difference of means, an index like the PSI (Population Stability Index) by bins, or simply the model's flag rate (if it suddenly flags 50 % of orders, something has changed).
3. Concept drift: the relationship between inputs and output changes (a new free-returns policy makes the same orders get returned more). It can only be detected when the actual labels arrive, late: AUC and cost must be recomputed on the orders from a few weeks ago whose returns are already known.

And act: alerts with thresholds (warning / alert), a retraining calendar (periodic or drift-triggered), model versioning (v1.0.0, v1.1.0, each with its metrics JSON) and a decision log (which model scored which batch, how many orders went to each queue, what the reviewer decided): it serves to audit, to explain to a customer and to retrain with human labels. The specific tools of this terrain (MLflow, drift dashboards) were mentioned in 07-03; here it is enough to understand what is measured and to program it by hand.

Two short documents and a habit:

• Model card: what it is, what it can and cannot be used for, what data it was trained on, how it performs (technical and business metrics, per group where relevant), known limitations, owner, version and date. It was born as an academic proposal and today it is standard practice and, for high-risk systems, part of what regulation demands (02-04). We will generate it automatically from train.py.
• Datasheet: origin, period, columns, how it was labelled, known biases, legal basis.
• Communication: you do not bring Diego an AUC; you bring him "€1,610 against €3,690 per 750 orders, 140 reviews per 750, and a list of cases you can look at". To management, three figures and one risk. To the technical team, the repository and the model card.

Even in a company of 180 people, an AI project needs four hats (sometimes on few heads):

And the mistakes that sink the most projects:

1. Starting with the model ("let's do deep learning") with no decision and no business metric.
2. No baseline: an AUC of 0.84 means nothing if Diego's rule already achieved almost the same.
3. Wrong metric: optimising accuracy with imbalanced classes (04-05), or a technical metric that does not move in euros.
4. Data that will not exist in production: leakage and columns computed differently at serving time.
5. Not closing the loop: deploying without monitoring, without retraining, without a log; or staying in a notebook that never reaches production.
6. Not involving the decision-maker: Diego finds out about the system the day 641 orders land on him for review.

We walk through the eight phases on the project we already have in novamarket_ai/. All the code in this section has been run in the course environment; the new files are predict_batch.py, drift.py, the model card functions in train.py and a test.

These phases are already done in modules 4 and 7 and we do not repeat them: data.py generates and prepares the orders (leakage out, 21-column Pipeline), and the logistic regression won the prototype with AUC 0.844 and cost €1,610 at threshold 0.2. Only one correction that shows up when preparing the deployment, typical of phase 2: prepare_orders computed days_since_start relative to the first order in the file. In training that is 1 September 2025; in a nightly batch of 1 December it would be... 1 December, and the column would be 0 instead of 91. It is a textbook case of training-serving skew. The fix: a start_date parameter that in production is passed explicitly.

We also move the business_cost(matrix, cost_fp=5, cost_fn=30) function of 04-05 into data.py, because the model card needs it.

We extend train.py so that, besides the model and its versions JSON, it writes the model card in two formats (JSON for machines, Markdown for people) and a drift reference that monitoring will use. train() now returns a third value with the context (splits and test probabilities).

Explanation of the new pieces:

• The card gathers in one place what an auditor, a new colleague or Diego needs: intended and out-of-scope use (the phrase "not for denying purchases" is an ethical decision turned into a document), data, technical and business metrics, honest limitations and version.
• The drift reference stores what the amount looked like in training (mean and the ten bins that leave 10 % of orders in each), the proportion of new customers and what fraction of orders the model flagged on the test set. It is what monitoring will compare against every week.
• The outer edges are stored as None (infinity): that way an order more expensive than any in training falls into the last bin instead of getting lost.

Actual output of python -m novamarket.train in the course environment:

And models/model_card.md looks like this (excerpt):

The figures are exactly those of 04-05 (€2,485 → €1,610, 60 and 200 flagged): the model card invents nothing, it documents what the code computes. models/ now contains returns_model.joblib, returns_model.json, model_card.json, model_card.md and drift_reference.json; the three JSON files and the Markdown do go into Git (07-04).

The nightly batch receives a CSV with the day's orders already prepared by the same query that feeds training (order_id plus the 15 columns of prepare_orders), scores them with the saved model and produces the queues.

Explanation:

• assign_queue is the human review band of 04-05 and 06-04 turned into code: three zones and two thresholds, defined as named constants so that a change is made in a single place (and stays in Git).
• score_batch does not touch the model: the saved Pipeline already imputes, scales and encodes (which is why we insisted in 04-03 on putting the preparation inside). It sorts by descending risk: if the team only gets through 300 reviews, it reviews the 300 riskiest.
• log_decision is the decision log of phase 7: one line per batch with the model version (read from the JSON that save left), the date and the counts. It is the minimum needed to audit and to plot the evolution week by week.
• The output is a CSV that the warehouse tool already knows how to load: integrate into the existing flow instead of forcing Diego to call an API.

To test it without real data, we simulate the file that the SQL query would produce for 1 December 2025 (3,000 orders, one NovaMarket day), without the returned label, passing the same start_date as in training:

Actual output of the batch for the normal day:

And the first rows of queues_2025-12-01.csv (sorted by risk) and the log line:

Notice the number Diego sees first: 641 orders to review in one day. On the 750-order test set the band gave 140 reviews (19 %); at the scale of 3,000 orders a day that is more than 600, twice what his team can absorb (300, according to the definition document). This does not invalidate the model: it is exactly the kind of finding that shadow mode exists for, and it is resolved in the evaluation with the stakeholders: raise the lower threshold (0.25 or 0.3) at the price of letting more returns slip through, or keep 0.2 and review in order of risk until capacity runs out. You will quantify it in exercise 1.

Every Monday the latest batch (or the union of the seven batches of the week) is compared with the reference saved at training time. We measure four things: the PSI of the amount, the mean amount, the new-customer rate and the model's flag rate (proportion of orders that do not go to "approve").

Explanation:

• The PSI compares two histograms with the same bins: for each bin, the difference in frequencies multiplied by the logarithm of their ratio, summed. It is 0 if they are identical and grows with the difference; the industry convention (inherited from credit scoring) is that below 0.1 nothing is happening, between 0.1 and 0.25 you should watch, and above 0.25 you should act. The bins are the training deciles (10 % of orders in each), so the reference is simply ten times 0.1.
• The flag rate is the cheapest and most useful indicator: it needs no labels and reflects any change in the inputs that affects the model. If it suddenly flags 51 % instead of 27 %, the review team is swamped and something has changed.
• The warning and alert thresholds are decisions, not laws: they are agreed with Diego and written into the document.

Actual output of the two runs:

The first batch looks like the training data and everything stays green. In the second (we simulate a Christmas campaign: orders 40 % more expensive and 45 % new customers), the mean amount has risen to €175, the PSI is in the watch zone and the model flags 51 % of the orders: 1,530 orders in the review and flag queues in a single day. The alert fires before any actual return is known; that is data drift. If a month later, with the labels mature, the AUC on those orders had dropped, it would also be concept drift, and the response would be to retrain with the campaign data included (python -m novamarket.train on the extended history, version 1.1.0, new model card, compare in shadow, replace).

The project tests are extended with two for the batch (tests/test_predict_batch.py: the boundaries of assign_queue with 0.19/0.20/0.49/0.50, and that score_batch returns one row per order with probabilities between 0 and 1) and pytest passes 7 tests.

With this the project has closed the loop: definition, data, prototype, evaluation, batch deployment with a review band and rollback, monitoring with alerts, and documentation. In 09-04 you will walk this same path with a case of your own.

• Starting with the model. Write the definition document (one table) before opening the notebook. If you cannot fill in "decision that changes" and "baseline", the project does not exist yet.
• Technical metric without translation. An AUC on its own convinces no one; always accompany it with the cost in euros and the number of cases that reach people.
• Data that will not exist or will change at serving time. Check column by column that it exists at prediction time and is computed the same way (days_since_start was a silent example). Use the same query for training and serving.
• Random split in temporal problems. Test set later than training; otherwise, optimistic metrics.
• Deploying all at once. Shadow → pilot in one warehouse → everything, always with a rollback switch.
• Fully automating decisions about people. Human review band proportional to the impact (02-04) and a log of what the person decides.
• Not monitoring because "the model already works". Program the flag rate and a PSI from day one; it is fifteen lines.
• Alert thresholds taken from a manual. 0.1/0.25 for the PSI and ±5 points of flag rate are starting points; adjust them with the real variability of your weeks and write them down.
• Documenting at the end. The model card generated from the code costs nothing to maintain; the one written by hand at the end never gets written.
• Tip: save every model version with its JSON and its card, and never overwrite the previous one until the new one has been through shadow.

Diego can only review 300 orders a day. With the file queues_2025-12-01.csv (or generating it with simulate_day), compute how many orders would go to "review" if the lower threshold were 0.25 and 0.30 (keeping 0.5 as the upper one). Propose a reasoned decision. Hint: score_batch already returns return_prob; it is enough to count ((p >= u) & (p < 0.5)).sum().

Describe (without code) a change at NovaMarket that would make drift.py raise no alert and yet make the model genuinely worse. Which indicator of the monitoring calendar would detect it, and with how much delay? What would you add to the weekly check to reduce that delay?

Fill in the definition table of section 11.1 for NovaMarket's product recommender (use case 1). Pay special attention to "decision that changes", "business metric", "baseline" and "deployment" (batch or real time? shadow mode or A/B?).

Solution 1. Run on queues_2025-12-01.csv (2,999 orders), counting ((p >= u) & (p < 0.5)).sum() for each u: with lower threshold 0.2 there are 641 orders in "review"; with 0.25, 484 remain; with 0.3, 355 remain (the "flag" queue does not change: 308). None of the three fits in 300, but with 0.3 the excess is small (55 orders). The price shows on the 04-05 test set (750 orders, review at €3 and a reviewer who gets it right): with the 0.2-0.5 band the cost was €1,555 and 35 returns slipped through; with 0.25-0.5 it rises to €1,795 (47 slip through) and with 0.3-0.5 to €1,966 (56 slip through). That is, raising the lower threshold to 0.3 saves about 290 reviews a day but costs about €400 more per 750 orders: capacity versus cost, the kind of trade-off Diego decides, not the model. A reasonable decision: lower threshold 0.3 in the pilot, queue sorted by risk (the 55 that do not get reviewed are the lowest-probability ones, between 0.30 and 0.32, and are approved), measure the actual cost after a month with mature labels and, if it pays off, request extra review capacity to go back to 0.2. An equally valid alternative: keep 0.2, review the 300 riskiest and approve the rest; since the queue is sorted, the result is practically the same as raising the threshold, but it leaves a record of how many "medium-risk" orders go unreviewed each day. What matters is not the number but the process: the band is an operational decision taken with the people who bear it, not a model parameter.

Solution 2. Example: NovaMarket launches "free, no-questions-asked returns for 60 days". The incoming orders are the same (same amounts, same customers, same model flag rate), but now they get returned more and for different reasons: the relationship between inputs and output has changed (concept drift). drift.py would see nothing. It would be detected by the monthly review with mature labels (AUC and actual cost per threshold), with a delay of 30 to 60 days (the time returns take to be recorded). To shorten it: (a) add to the weekly check the actual return rate of orders from 3-4 weeks ago compared with the training one (16.4 %), even with partial labels; (b) use the review queue as a sensor: the proportion of orders the reviewer confirms as risky changes before the overall figures do; (c) subscribe the project to changes in commercial policy (a row in the decision log: "policy change on day X"), because concept drift almost always has a cause known to the business.

Solution 3. One proposal: Question: which products are most likely to interest this customer now. Decision that changes: which 6 products are shown in the "You may also like" block on the product page and in the weekly email (today: the category's best sellers). Business metric: additional revenue per session and click/purchase rate of the block; technical: precision in the top 6 (proportion of recommendations that end in a click or a purchase) measured on past sessions. Baseline: "category best sellers" (what exists today) and "the same products the customer bought last time". Success criterion: +X % of block revenue in a 4-week A/B without raising the return rate. Constraints: GDPR (profiling: information and the possibility to object), no recommending by postcode, no pushing products with high returns (link with use case 3), minimal risk under the AI Act. Data: orders.csv, customers.csv, products.csv, web browsing; typical leakage: using purchases made after the session. Deployment: the list of recommendations per customer can be computed in a nightly batch (enough for the email and for most product pages) and served from a table; the website reads it through an API; A/B is mandatory because the effect can only be measured causally. Monitoring: coverage (how many customers get a recommendation?), diversity, clicks, and that the same five products do not always appear. Owners: marketing (business), Marta, systems.

This lesson has turned the map of 04-01 into a method. The life cycle of an AI project (definition of the problem and of success, data, exploration, prototype, evaluation, deployment, monitoring and documentation) is the shape taken by CRISP-DM and CRISP-ML(Q), with the arrows back as the norm and quality as the cross-cutting thread. We have seen which questions each phase answers (the decision that changes, the metric in euros versus the technical one, the baseline, the availability of each piece of data at prediction time, the temporal split, the prototype budget, the evaluation with the affected people, batch versus real time, shadow and A/B, rollback, data and concept drift, model card and datasheet), who takes part (business, data, engineering, legal) and which mistakes sink projects (starting with the model, having no baseline, wrong metric, data that will not exist in production, not closing the loop). And we have actually done it with the returns predictor inside novamarket_ai/: a one-page definition document, the fix of a training-serving skew in days_since_start, a model card in JSON and Markdown generated from train.py with the usual figures (AUC 0.844; €2,485 → €1,610), a predict_batch.py that sorts one night's 2,999 orders into 2,050 approved, 641 to review and 308 flagged and leaves a record in a log, a drift.py that gives green on a normal day and an alert on a campaign day (PSI 0.224, flag rate of 51 %), and a monitoring calendar with owners and thresholds.

Marta and Diego now have a method and a project that follows it. But a method is also learned by looking at how others have applied it, well or badly: what the recommendation platforms did, the medical image diagnosis systems, AlphaGo and AlphaFold, the recruitment tools that discriminated, the assistants that made up policies and the celebrated projects that failed. That is the next lesson, 08-02, AI Case Studies, where each case ends with the same question: what does NovaMarket learn from this.