We ended the previous lesson noting that most of the systems NovaMarket is going to build are sub-symbolic: they learn from data. This lesson is devoted to that raw material. Marta has been saying it in every meeting for weeks: "before we talk about models, let's talk about what data we have and what state they are in". She is right, and in this lesson we will see why. We will explain why data are the fuel of AI, what types of data exist and how they are organised, what the life cycle of data is in an organisation, how their quality is measured and what concrete problems NovaMarket's files hide, where the biases come from that later turn into unfair decisions, what conceptual obligations personal-data protection imposes and why quality matters more than quantity. We will close with a Python program that reads a sample orders.csv and produces a small quality report, exactly the kind of check that should precede any AI project.

1. Why data are the fuel of AI
2. Types of data
3. Labelled and unlabelled data
4. The data life cycle: collection, storage, quality, governance
5. Dimensions of data quality, with NovaMarket's real problems
6. Bias in data: the origin of algorithmic unfairness
7. Personal data and the GDPR: minimisation, anonymisation and pseudonymisation
8. Quantity versus quality; synthetic data
9. Example in Python: a quality report for orders.csv

Remember the scheme from 01-02: in traditional programming, data + program → results; in machine learning, data + results → model, and then the model produces results for new data. The model contains no more knowledge than there was in the data it was built from. Three consequences follow that are worth committing to memory:

• No data, no model. The recommender in 01-03 could not recommend anything for the TV until there were orders containing TVs (the "cold start").
• Bad data, bad models. The classic expression is garbage in, garbage out: if 50% of the "cause" column in incidents.csv is empty, no algorithm is going to diagnose incidents well.
• The model inherits everything in the data, including the errors, the gaps and the biases. If NovaMarket historically reviewed orders from certain postcodes more, a model trained on that history will learn to be suspicious of those postcodes (we will develop this in section 6 and in 02-04).

That is why, in a real AI project, between 60% and 80% of the effort goes into obtaining, understanding, cleaning and preparing data, not into training models. It is the first thing Marta wanted Diego to understand: the "cost of AI" is largely the cost of putting the data in order, and that cost is amortised across every subsequent project.

Data are classified in several complementary ways. Knowing them helps you know which technique can be applied and what preparation they will need.

Most of a company's data volume is unstructured, and most of the historical value has been extracted from structured data. Deep learning (module 5) changed that by making text and images exploitable.

A single file mixes several: orders.csv has identifiers, a date, a numerical amount, a categorical city and a binary categorical returned column. Knowing what each column is is the first step of any analysis, and it is surprising how many problems come from treating an identifier as a number (adding up postcodes) or a number as text (sorting amounts alphabetically: "1000" before "200").

A decisive distinction for machine learning:

• Labelled data: each example comes with the correct answer we want the model to learn to produce. In orders.csv, the returned column is the label if we want to predict returns. In reviews.csv, a "sentiment" column (positive/negative) filled in by a person would be the label.
• Unlabelled data: we only have the examples, with no answer. Thousands of unclassified reviews, browsing history without knowing what the customer ended up buying.

With labelled data you can do supervised learning (learning the relationship between the features and the label); with unlabelled data, unsupervised learning (discovering structure: groups of similar customers, products bought together). Both are developed in 04-02; here it is enough to know that labels are expensive (they often have to be applied by hand) and valuable, and that their quality conditions everything.

An example of how expensive and delicate a label is: incidents.csv has around 400 records and half of the cause column empty. That column is the label that use case 9 (incident diagnosis) would need to learn. With 200 examples labelled unevenly (operators filled in the cause when they had time, that is, more on quiet days and less at peaks), the system would learn from a biased sample. Diego proposed a non-technical but correct solution: make the cause mandatory on the form from now on and organise a session to label the 200 missing ones retrospectively.

Data do not appear ready to use; they go through a cycle, and each stage can introduce or correct problems:

• Collection: how the data come in. Each channel has its vices: the web returns form allows fields to be left empty; the warehouse terminal stores weight in grams and the website in kilos; the external courier sends dates in DD/MM/YYYY format while the internal system uses YYYY-MM-DD. The golden rule is validate at source: it is far cheaper to require the incident cause to be filled in at the time than to reconstruct it months later.
• Storage: where and how they are kept. Transactional databases (the ones that run the store), data warehouses for analysis, flat files like the CSVs we work with in the course. Here traceability matters (where does each datum come from? when was it loaded?) as does the schema (what does each column mean, in what units).
• Quality: checking and correcting (section 5). Ideally it is a continuous process, not a one-off fix before each project.
• Use: analysis, reports, model training, decisions. It is what justifies everything above.
• Governance: the set of rules and responsibilities over the data: who owns each dataset (Marta proposed that each area own its own: operations owns incidents.csv, marketing owns reviews.csv), who can access what, how long data are retained, how they are documented (a data dictionary with the definition of each column) and how the legislation is complied with (section 7). Governance closes the cycle because its rules improve the next collection.

Without governance, every AI project rediscovers the same problems. With it, NovaMarket's second project (demand forecasting) reuses the clean data from the first (recommendation).

"Quality data" is not a vague judgement: it breaks down into measurable dimensions. The five most used, with the problems Marta found when she opened NovaMarket's CSVs for the first time:

Two important remarks:

• Quality is measured, not assumed. Marta did what we will do in section 9: count blanks, duplicates and formats per column and put numbers on them. Discovering that 58% of the rows in a sample had some problem was what convinced Diego to devote time to cleaning before models.
• Quality depends on the use. An empty postcode is irrelevant for classifying reviews and serious for planning routes. There is no such thing as "the" quality level: there is enough for each use case.

How these problems are cleaned up to prepare a model (value imputation, category encoding, scaling, handling outliers) is the content of 04-03; here we stop at detecting and measuring them.

A bias in the data is a systematic deviation between what the data represent and the reality they should represent. It is not a problem of "dirty data" (the data can be spotless and biased) but of what was collected, from whom and how it was labelled. Since the model inherits what is in the data, the bias turns into systematically skewed decisions, and when those decisions affect people, into algorithmic unfairness. The three main origins:

Other frequent biases: survivorship bias (we only see the data that "survived": we analyse the customers who keep buying and forget why those who left did so), measurement bias (a sensor or form captures one group worse: the returns form in the app is more awkward than on the website, so mobile customers fill in less) and confirmation bias in the analysis (looking in the data for what we already believed, like Diego's €300 rule).

The essential point in this lesson: bias enters through the data and shows up in the decisions. How to assess its ethical consequences, how to measure the disparity between groups and what to do about it is the content of the next lesson, 02-04.

A good part of NovaMarket's data are personal data: they identify or make it possible to identify a person (name, email, address, phone, purchase history, even a combination of postcode, date of birth and sex). In the European Union their processing is regulated by the General Data Protection Regulation (GDPR). We are not going to study the regulation here (it is picked up again in the regulatory framework of 02-04) but rather the principles that directly affect working with data for AI:

• Purpose and legal basis: data are collected for a specific purpose and with a legal basis (contract, consent, legitimate interest...). Using purchase history to recommend products may fit; using it for something unforeseen (selling it, profiling for third-party purposes) does not.
• Minimisation: collect and use only the data necessary for the purpose. If the date, the product, the city and the units are enough to forecast demand, there is no need to load the customer's name and email into the training set. Fewer personal data means less risk and fewer obligations.
• Storage limitation: do not keep personal data longer than necessary. This clashes with the instinct to "keep everything in case it is useful for training something"; retention periods must be defined.
• Individuals' rights: access, rectification, erasure ("to be forgotten"), objection to profiling and to automated decisions with significant effects. A model trained on the data of a customer who requests erasure raises practical questions the project must anticipate.
• Security: protecting the data from improper access, especially when they are copied to analysis environments or sent to cloud services (remember the cloud/device dichotomy from 02-02).

Two techniques that reduce risk and will come up constantly:

Marta decided that the datasets used to train models at NovaMarket will always be, at a minimum, pseudonymised, and that demand forecasting will work with aggregated (anonymous) data.

Important notice: everything above is a conceptual description for training purposes, not legal advice. Any project that processes personal data must be reviewed with the data protection officer or a legal professional, who will assess the legal basis, the need for an impact assessment and the specific measures.

It is common to hear that "the more data, the better". It is half true:

• More data help when they are varied and representative: they allow finer patterns to be learned and reduce the danger of the model memorising specific cases (overfitting, the subject of 04-06). The great advances of deep learning rested on enormous datasets.
• More bad data do not help: multiplying a biased set by ten gives a model ten times more confident in its bias. Two hundred well-labelled examples of incident causes are worth more than two thousand labelled any old how.
• The data must be relevant: for NovaMarket's recommender, the orders of another store in another sector contribute little.

Practical rule: first quality and representativeness; then quantity. And always measure whether more data actually improve the result.

Synthetic data are artificially generated data (by rules, simulation or generative models) that imitate the statistical properties of real ones. They are used to: fill out rare classes (there are very few real frauds to learn from), test systems without exposing personal data, or simulate scenarios that have not yet happened (a Christmas campaign for a new product). They have an obvious limit: they only contain what whoever generated them knew or assumed; they discover nothing new about the world, and if they are generated from biased data, they inherit the bias. They are a complement, not a substitute.

We are going to do what Marta did on the first day: read an extract of orders.csv, count the problems per dimension and produce a small report. To keep the example self-contained, the CSV is written inside the program as a text string; in practice you would read it with open("orders.csv"). We only use the standard library: csv, io, re and collections.

Output:

Explanation: csv.DictReader reads the first line as the header and turns each following line into a dictionary, which lets us access values by name (row["amount"]). Notice that everything is read as text: '89.90' is a string, not a number; converting it will be our responsibility. If you look at the data carefully you will already see the planted problems: an empty amount, an empty customer_id, an empty returned, an empty postcode, three date formats, a date with month 13, an amount of 15990 that smells of cents, "zaragoza" in lowercase and order 48202 twice.

Output:

Explanation: for each column we walk through the rows and count those whose cell is empty after stripping whitespace (strip()); a blank space is also a blank. The expression sum(1 for r in rows if ...) is a compact way of counting. With the real incidents.csv, this report is the one that would show "cause: 50.0%".

Explanation: Counter builds a dictionary value → number of occurrences; we keep those that appear more than once. Here the duplicate is exact (the same row loaded twice), the easy case. The hard duplicates are those in customers.csv: the same customer with the email in upper case or with a trailing space. For those, the key would have to be normalised before counting: r["email"].strip().lower(). Try it as a variant.

Explanation: each pattern is a regular expression: ^\d{4}-\d{2}-\d{2}$ means "exactly four digits, hyphen, two digits, hyphen, two digits". classify_date returns the name of the first pattern that matches. The final Counter tells us how many dates there are in each format: three rows do not follow the standard. Sorting or comparing those dates as text would give absurd results, and any program expecting YYYY-MM-DD would fail on them.

Explanation: accuracy cannot be fully checked without a source of truth, but we can detect impossible values (month 13) and values outside the reasonable range (a €15,990 order in a store whose average basket is around €100 smells of an amount in cents: €159.90). The €5,000 threshold is a business rule Diego set. Notice that the suspicious amount appears twice because the row is duplicated: quality problems pile up.

Output:

Explanation: counting the values as they are there are seven cities; normalising (no whitespace, lowercase) there are six. The difference is exactly the number of inconsistencies. This trick (comparing the number of distinct values before and after normalising) works for any categorical column.

Output:

Explanation: the report brings together the previous checks and adds a global metric: the percentage of rows with at least one problem. 58% problematic rows in a small, exaggerated sample; in NovaMarket's real file the figure was lower but enough to justify a preliminary cleaning project. The "quality score" is a simplification (all dimensions weigh the same), but it turns a feeling into a number that can be tracked over time: Marta's goal is for it to rise month by month. As a mental exercise, think about what you would add to measure timeliness (you would need a column with the last-updated date) and what you would need to measure accuracy for real (a reference source to compare against).

• Starting with the model. The natural impulse is to train something as soon as possible. Always start with a quality report like the one in section 9; an hour of checks saves weeks of debugging why the model "does strange things".
• Trusting that a clean file is not biased. Quality and representativeness are different things. Always ask who is missing from the data and why.
• Treating a blank as zero, or as just another category, without thinking. An empty amount is not an amount of €0; an empty cause is not "no cause". Before deciding what to do with blanks you have to understand why they are missing (is it random, or is more missing on peak days?).
• Ignoring units. Cents versus euros, grams versus kilos, local time versus UTC. Document the units in the data dictionary and validate the ranges.
• Keeping everything "just in case". It clashes with the GDPR's minimisation and storage limitation and increases risk. Collect and keep what is necessary for a defined purpose.
• Anonymising by removing only the name. Postcode + date of birth + sex identify a large part of the population. If you need to anonymise for real, aggregate or generalise, and ask for expert review.
• Tip: create the data dictionary from day one (column, meaning, type, unit, allowed values, who is responsible). It is the most useful and least glamorous document of an AI project.

For each of the five files (customers.csv, orders.csv, products.csv, reviews.csv, incidents.csv), state: (a) whether it is structured, semi-structured or unstructured (or mixed: which columns of each type); (b) which column could be a label and for which use case from the list in 01-03; (c) whether it contains personal data and which measure (pseudonymisation, anonymisation, minimisation) you would apply before using it for training.

NovaMarket wants to train a model that predicts whether an order will be returned, using the history of orders.csv from the last three years. During that period: (i) returns from the mobile app were recorded in another system and are not in the file; (ii) Diego's team manually reviewed and cancelled many "doubtful" returns on orders over €300 (his old rule); (iii) the return_reason column was filled in freely by customers. Identify which type of bias each circumstance introduces and what effect it would have on the model.

Extend the program in section 9 with two checks: (a) the returned field only admits the values yes and no (anything else, including blank, is a consistency problem); (b) the postcode, when not empty, must have exactly 5 digits and its first two digits must be consistent with the city for at least these cases: Zaragoza → 50, Madrid → 28, Getafe → 28 (an accuracy problem). Add the count of both to the report.

Solution 1.

Solution 2.

• (i) App returns missing: sampling bias (or measurement bias). The model will underestimate returns from customers who buy on mobile; if the channel is correlated with age or product type, the model will be systematically wrong for those groups.
• (ii) Manual cancellations of doubtful returns over €300: historical bias (the data reflect a past policy). The history contains fewer "completed" returns on high-value orders than would occur without Diego's intervention; the model may learn that expensive orders are rarely returned, exactly the opposite of what motivated the rule, and it also absorbs the subjective criterion of what was "doubtful". If the model is used to decide what to review, a feedback loop is created.
• (iii) Reason written freely by the customer: labelling bias (and a consistency problem). "I don't like it", "it wasn't what I expected", "size" and "poor quality" may or may not be the same thing; the model learns the words each customer uses, not the real cause. The reasons would have to be normalised to a closed list before using them as a label.

Solution 3.

To incorporate it into the report, add two lines inside quality_report with len(returned_issues(rows)) and len(postcode_issues(rows)) and include those identifiers in the "row with issue" condition. With the sample data the postcode check finds no errors; add a row 48212,C1009,2026-03-06,20.00,Madrid,50003,no and check that it detects it.

In this lesson we have treated data as what they are in modern AI: the raw material from which the model inherits everything, the knowledge as well as the errors and biases. We have classified data by structure (structured, semi-structured, unstructured), by nature (numerical, categorical, text, image, time series) and by the presence of labels, anticipating the distinction between supervised and unsupervised learning. We have gone through the data life cycle (collection, storage, quality, use, governance) and broken quality down into five measurable dimensions (completeness, accuracy, consistency, timeliness, uniqueness) illustrated with the real problems of NovaMarket's CSVs: half of the cause column empty, duplicate customers, dates in three formats and prices in cents. We have seen that bias enters through the data (sampling, historical, labelling) and turns into skewed decisions, we have presented the GDPR principles that affect working with data (purpose, minimisation, retention, rights, security) together with pseudonymisation and anonymisation, and we have qualified the myth that "more data is always better". Finally, we have written a program that reads a CSV with the standard library and produces a quality report: the first tool that should be run in any AI project.

With the data now on the table, the next lesson, Ethics and Considerations in AI, tackles what happens when those data, with their biases, feed decisions that affect people: which ethical principles should guide NovaMarket, which concrete problems (discrimination, privacy, opacity, accountability, impact on jobs, disinformation, sustainability) must be anticipated, what the European regulatory framework requires and which practical tools (checklist, impact assessment, human review, fairness metrics) make it possible to move from good intentions to facts. There we will pick up the return-risk model and the postcode again to measure, with numbers, whether a model treats all groups the same.