In the previous lesson we saw that data carry biases and that the model inherits them. Now comes the uncomfortable question Diego raised at the end of that meeting: "If the return-risk model is more suspicious of one neighbourhood than another, who answers when a customer from that neighbourhood complains?". This lesson answers that kind of question. Ethics in AI is not an appendix for the end of the project nor a matter exclusively for lawyers: it is part of the design, just like the performance measure of 02-01 or the data quality of 02-03. We will see why it is not optional, which principles guide it, which concrete problems appear (bias and discrimination, privacy and surveillance, opacity, accountability, impact on jobs, disinformation, sustainability), what the European regulatory framework requires at an introductory level (GDPR and AI Act) and, above all, which practical tools make action possible: a project checklist, impact assessment, human review and fairness metrics. We will finish with code that measures, with numbers, whether NovaMarket's risk model treats two zones the same and adds a human-review rule. With this we will close module 2.

1. Why ethics is not optional
2. Ethical principles for AI
3. Bias and discrimination: a numerical example
4. Privacy and surveillance
5. Transparency and explainability: black box versus interpretable model
6. Responsibility and accountability
7. Impact on jobs, disinformation and sustainability
8. Introductory regulatory framework: GDPR and the European AI Act
9. Practical tools: checklist, impact assessment, human-in-the-loop and fairness metrics
10. Example in Python: measuring the disparity between zones and adding human review

There are three reasons, and it is worth keeping all of them in mind because they convince different audiences:

• Reason of principle: AI systems make or influence decisions that affect people (what is recommended to them, whether their return is reviewed, whether their review is published, how long their parcel takes). Any system that decides about people must do so fairly and respectfully; automating it does not remove that obligation, it scales it: a biased person reviews a hundred orders a day; a biased model reviews three thousand without tiring.
• Legal reason: in the European Union, the GDPR and the AI Act impose concrete obligations with very heavy penalties. Ignoring them is not a business option.
• Business reason (the one that convinced Diego): an unfair or opaque system generates complaints, claims, bad press and loss of trust, which in an e-commerce business with a 4% return rate and tight margins translates into money. And a system nobody understands cannot be corrected when it fails.

The practical consequence is that ethics enters the project from the start (when setting the performance measure, when choosing the data), not as a final review. Remember the anecdote from 02-01: the incentive for "conversations closed per hour" produced conversations closed without being resolved. A model that optimises "returns avoided" with no constraint whatsoever will produce exactly the kind of behaviour this lesson is about.

Most ethical frameworks (those of the European Commission, the OECD, UNESCO, many companies) converge on a small set of principles, largely inherited from bioethics:

The principles are useful as a compass, but abstract: "be fair" does not say how. The rest of the lesson makes them concrete in problems, obligations and tools.

This is the most studied problem and the one that connects most directly with 02-03. A model discriminates when it systematically treats a group of people worse because of a characteristic that should not matter (sex, origin, age, disability, religion...) or because of a stand-in for it (a proxy): the postcode, the name, the browser language or the type of phone often correlate with origin or income level.

Let us see it with NovaMarket's return-risk model (case 3). Suppose the model assigns each order a risk score between 0 and 1 and that orders with a score greater than or equal to 0.40 are flagged for manual review (the review delays the refund by several days and may end in denial). We compare two delivery zones, A and B, with customers who in every relevant respect (amount, order history, previous returns) are the same:

Customers in B return exactly as much as those in A (30%), but the model flags 90% of them versus 20% in A. How did that happen? Through what we saw in 02-03: in the history, the team reviewed orders from zone B more, so more fraudulent returns were "found" there, and the model learned that zone B (or its postcode) is an indicator of risk. The model is "accurate" with respect to a biased past and systematically unfair with respect to the present reality. Moreover, if its decisions feed the future history (B gets reviewed more, more gets found in B), the bias reinforces itself.

Notice two things:

• The unfairness is not in anyone's intention. Marta did not include "origin" or "income" in the model; she included the postcode because "it was useful for the routes". The bias came in through a proxy and through the history.
• It is detected by measuring: comparing rates per group, as we will do in section 10. Without that comparison, the model appears to work well "on average".

The groups that must be compared depend on the context: zones, age brackets, sex, channel (web/app), language. In Spain and the EU there are characteristics legally protected against discrimination, and even if the model does not use them directly, it is the team's responsibility to check that it does not discriminate through proxies.

AI systems feed on personal data and, in addition, create new information about people: a recommender infers interests (sometimes sensitive ones: health, pregnancy, financial situation) from purchases; a routing model knows at what time someone is home at each address; an assistant stores intimate transcripts. Three concrete risks:

• Unwanted inference: the system deduces things the customer has not revealed and would not want known. Recommending baby products to someone who has bought a pregnancy test can be useful or a serious intrusion.
• Reuse of data for purposes other than those for which they were collected (the purpose and minimisation principles of 02-03).
• Surveillance: applying AI to employees themselves (delivery drivers' productivity, emotion analysis on the customer-service team) or to customers in a disproportionate way. It is an especially sensitive area: the AI Act prohibits some of these practices (section 8) and the rest require labour safeguards.

Measures: minimise, pseudonymise or anonymise (02-03), limit who accesses the inferences, do not use sensitive categories or their proxies for decisions that affect the customer, and always ask "how would the customer feel if they knew exactly what we do with their data?". Diego, who started out sceptical about AI, was the most categorical here: "the drivers are not a data point".

Transparency means it is known that there is an AI system, what it does and with what data. Explainability means being able to give reasons for a specific decision. Both collide with a technical reality: the most powerful models (deep neural networks, module 5) are black boxes: millions of numerical parameters with no individual meaning. Simple models (rules, small decision trees, regressions with few variables) are interpretable: you can read why they decide.

The decision is not "interpretable always" nor "the most powerful always": it depends on the consequences. Practical rule: the greater the impact of the decision on a person, the more weight explainability must carry. For NovaMarket:

• Ordering the "also bought" box: can be a black box; the cost of a bad recommendation is low.
• Flagging a return for review or denying it: requires an explanation (to the customer and to the auditor). Interpretable or hybrid model, and in any case the ability to give the reason.
• Drafting the assistant's replies: black box (LLM) acceptable if there are controls and the customer knows it is an automated assistant.

There are techniques for explaining black boxes (showing which variables weighed most in a prediction, finding the minimal change that would flip the decision: "if your amount were €20 lower it would not have been flagged"), which are mentioned in module 4; here it is enough to know that they exist, that they are approximate and that they do not replace the design decision to use an interpretable model when the consequences demand it.

Back to Diego's question: who answers? The short answer is: always a person or an organisation, never "the algorithm". An AI system has no legal personality or consciousness (02-02); responsibility rests with whoever designs, deploys and operates it. For that to be real and not a slogan, you need:

• System owner: a named person responsible for each AI system in production (just like the data owner of 02-03). At NovaMarket, Marta for the recommender and forecasting; Diego for the route planner and the return rules.
• Traceability: recording which version of the model made which decision, with what data, when. Without a record there is no audit and no possibility of redress.
• Complaint channels: the affected customer must be able to turn to a person who reviews the decision (linking with the human-in-the-loop of section 9 and with GDPR rights).
• Clear split with vendors: if the assistant is supplied by a third party, the contract must set out who answers for what. "The vendor does it" does not exempt whoever deploys it towards the customer.

When the system fails (and it will: environments are stochastic, as we saw in 02-01), accountability consists of being able to answer three questions: what happened, why it happened and what has been done so it does not happen again.

Three considerations that go beyond the individual system but that a professional must know:

• Impact on jobs. AI automates tasks, not whole occupations, but it changes the content of many roles. At NovaMarket, the assistant does not eliminate the customer-service team but it does shift its work towards the difficult cases; the planner changes the drivers' autonomy; incident diagnosis changes the operator's job. Good practices: involve the affected teams in the design (Diego insisted that the drivers test the planner), train for the new tasks, measure the impact and be transparent. Automation done against employees usually fails operationally as well as ethically (remember how the 2015 chatbot was received).
• Disinformation and deepfakes. Generative AI produces text, images, voice and video indistinguishable from real ones, which makes impersonation easier (a fake audio of the managing director requesting a transfer), as well as fake reviews (which directly affect NovaMarket's case 4: a review classifier must allow for some of them being generated) and large-scale manipulation. As an organisation: do not generate misleading content, label generated content (the AI Act requires it in several cases), and protect channels against impersonation.
• Energy sustainability. Training and running large models consumes significant energy and cooling water. For most projects (including NovaMarket's) the consumption is modest, but the question "do I need a huge model in the cloud for this or is a small one enough?" has a cost dimension (which Diego likes) and an environmental one. Choosing the smallest model that solves the problem is good engineering and good ethics.

Prior notice: what follows is a conceptual introduction for training purposes, not legal advice. The regulations are complex, have staggered application deadlines and their interpretation evolves; any real project must be reviewed with a legal or compliance professional and, where applicable, with the data protection officer.

We already saw in 02-03 its principles on personal data (purpose, minimisation, retention, security, rights). For AI it is worth underlining the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects on the person (barring exceptions with safeguards), which includes the right to obtain human intervention, to express one's point of view and to contest the decision. Automatically denying a return or a warranty claim is an example of a decision that can have significant effects: that is why NovaMarket's case 8 and case 3 need human review. It also requires, when the processing entails a high risk to individuals, a data protection impact assessment before starting.

It is the world's first horizontal regulation on AI, adopted in 2024 with staggered application over the following years. Its central idea is a risk-based approach: the greater the risk of an AI use to safety, health or fundamental rights, the more obligations it carries:

In addition, general-purpose AI models (large language models) carry their own obligations for their providers (documentation, copyright, and more for those with systemic risk), which indirectly affect whoever integrates them.

Approximate and indicative classification of NovaMarket's cases (to be confirmed by a professional):

Reading: none of NovaMarket's cases is of unacceptable risk, and only one falls into the "limited" level of the AI Act (the assistant, with transparency obligations). But note two nuances that are often overlooked: (a) the GDPR continues to apply on top of the AI Act to anything that decides about people (cases 3 and 8), and (b) "minimal" systems can become "high risk" if they are repurposed to manage workers (cases 5 and 6). The risk level depends on the use, not on the technique.

From principles to facts. Four tools that fit in any project:

Marta added it to the start of every AI project at NovaMarket. It is short on purpose: if it does not fit on one page it does not get used.

1. Purpose: which decision or action are we automating and whom does it affect? What is the performance measure and what undesirable behaviour could maximise it?
2. Data: which data do we use, with what legal basis and minimisation? What sampling, historical or labelling biases do they have? Do they contain protected characteristics or proxies?
3. Fairness: which groups will we compare and with which metric? What difference will we consider unacceptable?
4. Explainability: what level of explanation does the customer, the operator and the auditor need? Does the chosen model allow it?
5. Human oversight: which decisions go through a person? How does the affected party appeal?
6. Responsibility: who is the owner? What is logged? What about the vendor?
7. Impact: on employees? on vulnerable customers? risk of misuse? proportionate energy cost?
8. Regulation: approximate risk level (AI Act)? automated decision (GDPR)? reviewed by legal?
9. Monitoring: how will we detect that the system degrades or discriminates once in production? Who looks at it and how often?

For systems that decide about people, the checklist is expanded into an impact assessment (algorithmic and, if there are high-risk personal data, data protection): a document describing the system, those affected, the risks identified for each group, the mitigation measures and the monitoring plan. It is done before deploying and reviewed periodically. It is the AI equivalent of an occupational risk assessment.

Placing a person in the decision loop, in proportion to the impact:

Human review is not free (Diego reminds us) nor infallible (the person may simply confirm what the machine says, the so-called automation bias). It must be sized, reviewers must be trained, and it must be measured whether they really correct the AI or merely rubber-stamp it.

To go from "we believe it is fair" to "we have measured", outcomes are compared between groups. The simplest idea is rate parity: the proportion of people who receive a certain outcome (being flagged, being approved, receiving an offer) should be similar across all groups, unless there is a legitimate and demonstrable reason for it to differ. A widely used practical rule (the "four-fifths rule", of US employment-law origin) considers a difference suspicious when the rate of the less favoured group is below 80% of that of the most favoured.

There are refinements: comparing not the raw rate but the error rate per group (does the model make more mistakes, flagging someone who was not going to return, in one zone than in another?), or comparing only among people with the same real risk. These variants, and the fact that not all of them can be satisfied at once, are covered in module 4 together with the evaluation metrics. Here the intuition is enough: compute the same figure for each group and see whether it is similar. That is what the following code does.

We reproduce the numerical example from section 3 with fictional data. Each row is an order with its zone (A or B), the risk score the model gave it, whether the model flagged it (score ≥ 0.40) and whether the order was actually returned. Customers in both zones are identical in amount and history; the only difference is that the model systematically adds a few tenths to zone B (the bias inherited from the history).

Output:

Explanation: rate_by_group is a generic function: for each value of the group column (zone) it counts how many rows there are and how many have the positive outcome (yes) in the given column, and returns the proportion. We call it twice: once with the model's decision (flagged_risk) and once with reality (actually_returned). The contrast is the heart of the problem: same reality (30% and 30%), very different decisions (20% versus 90%). If we had only looked at the overall flag rate (11 of 20, 55%) we would have seen nothing.

Output:

Explanation:

• The impact ratio applies the four-fifths rule: 0.20 / 0.90 = 0.22, far below 0.80. A clear alarm.
• The false-positive rate per group measures the concrete harm: of the customers in zone A who were not going to return anything, the model bothered (delayed the refund, placed under suspicion) 14%; of those in zone B, 86%. It is the same information as before, but expressed as "how many innocents we harm in each group", which is how a customer, a judge or Diego understands it.

These two figures are the ones that should appear on the system's monitoring dashboard (point 9 of the checklist), computed every week on the real data.

Correcting the model at the root requires going back to the data (removing the postcode and its proxies, rebalancing the history, retraining: module 4). In the meantime, and as a permanent safeguard, we add human review in the grey zone and record the reason:

Output:

and, for order 11:

Explanation:

• decide_with_review replaces the binary decision (flag / do not flag) with three outputs. The grey zone (score between 0.40 and 0.60) is no longer flagged automatically: a person reviews it, sees the data and decides. In our example, 6 of the 9 zone-B orders the model was flagging go to human review; a reviewer who checks that a customer with 5 previous orders and 0 returns gives no grounds for suspicion will approve the refund. The rule does not remove the model's bias, but it prevents it from automatically turning into harm, and it also generates data (the human decisions in the grey zone) that help diagnose and correct the model.
• explain builds the minimal explanation that section 5 demands: which score, which decision and which data were used. Notice that on reading it the anomaly jumps out: a risk of 0.46 for a loyal customer with €140 is hard to justify, and that discomfort is exactly what explainability is supposed to produce. Note also that the explanation does not mention the zone: if the model uses it and it does not appear in the explanation, the explanation is misleading; if it does appear, the problem is obvious. In both cases, the conclusion is that the zone should not be in the model.
• The cost of human review (8 reviews out of 20 orders in the example) is real and Diego will quantify it; but it is lower than that of complaints and reputational damage, and it will fall once the model is corrected.

This small program contains, in miniature, the complete ethical practice: measure by group, compare against a disparity threshold, put a person where the model is doubtful and explain each decision.

• Leaving ethics for the end. Once the model is trained and deployed, correcting the bias costs ten times more. Use the checklist in the project's first meeting.
• "We don't use sensitive data, so we don't discriminate". Proxies (postcode, name, device, language) discriminate just the same. The only way to know is to measure by group.
• Looking only at the global metric. A model with 90% overall accuracy can have 60% on a minority group. Always break down by relevant groups.
• Confusing explanation with justification. That the system can say "because the postcode is 28XXX" does not make the decision acceptable; explainability exists precisely to detect unacceptable decisions.
• Decorative human review. If the reviewer has 20 seconds per case and sees the machine's recommendation in large type, they will approve 99%. Design the review so that the person can disagree and measure how often they do.
• Blaming the vendor or the algorithm. Responsibility towards the customer lies with whoever deploys. Demand documentation, per-group metrics and explanation capability from the vendor before signing.
• Treating regulation as a list of distant prohibitions. The risk level depends on the use: a harmless route planner becomes a workforce-management system if it is used to penalise. Review each new use with legal.
• Tip: for any system that decides about people, ask yourself three questions in this order: who is harmed by an error? will we know? will that person be able to complain to someone? If any answer is "I don't know", the project is not ready.

NovaMarket wants a system (case 4) that automatically classifies reviews as positive/negative and hides those it detects as fake or abusive. Go through the nine points of the checklist in section 9.1 and write, for each one, one or two sentences with the specific risks and measures for this case.

Diego proposes using the route planner's data (times per stop, kilometres, stops per hour) to compute an "efficiency score" for each driver and to use it to decide shifts and contract renewals. Analyse the proposal from: (a) the approximate risk level under the AI Act; (b) the obligations it would trigger; (c) the likely data biases (think of urban versus outlying zones, traffic, type of parcel); (d) an alternative that keeps the operational benefit without those risks.

With the data from section 10: (a) compute the false-negative rate per zone (among orders that were indeed returned, the proportion the model did not flag); (b) write a function parity_ok(rates, minimum=0.8) that returns True if the impact ratio of any dictionary of rates satisfies the four-fifths rule; (c) test what happens to the impact ratio of flagging if we raise the threshold to 0.50: is the disparity corrected? Explain why or why not.

Solution 1. One example answer (there are many valid ones):

1. Purpose: classify and moderate reviews; it affects the customers who write them (their voice can be hidden) and those who read them (biased information if negative ones are hidden). Performance measure with a catch: "maximise the visible average rating" would lead to hiding legitimate criticism; use accuracy in detecting fake/abusive reviews measured against human labels.
2. Data: reviews with customer_id (pseudonymise); sampling bias (mostly the extremes write); labelling of "fake" and "abusive" is subjective, needs guidelines and several labellers.
3. Fairness: compare the hiding rate by review language, length, customer tenure and, if possible, zone; the model could hide more reviews written with spelling mistakes or in another language.
4. Explainability: the customer must be told why their review is not published; the model must give the reason (abusive, suspected fake) and not a bare "rejected".
5. Human oversight: hiding for "fake" always goes through human review (impact on the customer's reputation); sorting by sentiment can be automatic with auditing.
6. Responsibility: system owner in marketing; log of every hidden review, reason and reviewer.
7. Impact: on customers who criticise legitimately; risk of misuse (hiding criticism of high-margin products); low energy cost.
8. Regulation: minimal level under the AI Act in principle, but moderation of user content carries transparency and appeal obligations under digital-services rules; consult legal.
9. Monitoring: every month, hiding rate per group, rate of appeals accepted (how many hidings human review reverses), audited random sample.

Solution 2.

• (a) By being used to decide shifts and contract renewals, it enters the domain of "employment and worker management": approximately high risk, very different from the minimal level of the original planner. It is a textbook case of how use changes the level.
• (b) Likely obligations: documented risk management, data quality and representativeness, technical documentation, effective human oversight, transparency towards workers, logging; and, under the GDPR, an impact assessment, information and rights of employees, in addition to labour law and the participation of workers' representatives.
• (c) Biases: drivers in outlying zones or dense traffic make fewer stops per hour for reasons beyond their control (measurement and assignment bias); those carrying bulky parcels or signature deliveries take longer; the historical data reflect the routes they were assigned, not their effort. The "efficiency" would largely measure the route, not the person.
• (d) Alternative: use the planner's data to improve routes and assignment (which is their purpose), detect structural problems (zones where nobody arrives on time) and give each driver their own information for their own use; keep decisions about people in a human-resources process with agreed criteria, where route data are, at most, one contextualised element reviewed by people.

Solution 3.

• (a) The false negatives are also disparate, but the other way round: in zone A the model lets 67% of the real returns through and in B none. It is the other side of the coin: by suspecting almost everyone in B, it "gets" all their returns at the cost of harming those who do not return. A fair model should have similar error rates in both groups.
• (b) parity_ok returns False for the current flagging.
• (c) Raising the threshold to 0.50 reduces flagging in both zones (10% and 60%) but does not correct the disparity (ratio 0.17, even worse). It is logical: the bias is in the scores (zone B systematically has a few tenths more), and moving the threshold shifts the cut-off for everyone equally without touching the difference between groups. The solution lies not in the threshold but in the model and its data: remove the zone and its proxies, rebalance the history and retrain (module 4), keeping human review and periodic measurement in the meantime.

In this lesson we have seen that ethics in AI is part of the design, mandatory by principle, by law and by business, and we have made it concrete. We started from the principles (beneficence, non-maleficence, autonomy, justice, explainability) and brought them down to recognisable problems: discrimination through proxies (with the numerical example of the risk model that flags 90% of one zone and 20% of another with the same real return rate), privacy and unwanted inferences, the tension between black boxes and interpretable models, the responsibility that always rests with people and organisations, the impact on jobs, disinformation and sustainability. We have presented the European regulatory framework at an introductory level (GDPR, with the right against automated decisions, and the AI Act with its four risk levels) and classified NovaMarket's cases approximately, confirming that the level depends on the use and not on the technique. And we have gathered tools that fit in any project: the nine-point checklist, the impact assessment, human review proportional to the impact and rate parity between groups, which we have programmed in Python to measure the disparity, apply the four-fifths rule and add a grey zone of human review with a logged explanation.

With this we close module 2. You now have the conceptual foundations: you know how to describe any system as a rational agent in its environment (02-01), place it on the map of types of AI (02-02), understand that its decisions are born from the data and their quality (02-03) and evaluate its ethical and regulatory consequences before putting it into production (02-04). In module 3, Algorithms in AI, we will start building: we will pick up the problem formulation from 02-01 (states, actions, goal, cost) and see how an agent finds its own way to its goal with search algorithms, how it decides against an adversary and how it optimises when the solution space is too large to traverse in full. NovaMarket's route planner and the assignment of orders to warehouses will be our first real problems.