We close the foundations module with the most influential probability result in Machine Learning. The previous lesson ended with a pending question: how do we rationally update what we already believe when new evidence arrives? Bayes' theorem is exactly that: a rule for revising probabilities in the light of data. Besides underpinning a classic classifier we will meet in module 4, Bayesian reasoning is indispensable for correctly interpreting detection systems — for fraud, disease, spam — where human intuition fails spectacularly. We will build it step by step from conditional probability and apply it to MercaFresh's fraudulent-order detector.

1. Conditional probability
2. Bayes' theorem: derivation and interpretation
3. Priors, likelihood, and posterior
4. Full numerical example: fraud in MercaFresh orders
5. The false positive trap (the prosecutor's fallacy)
6. Frequentism vs. Bayesianism (briefly)
7. Bayes in Machine Learning

The conditional probability P(A|B) — read "probability of A given B" — is the probability of A knowing that B has occurred:

P(A|B) = P(A ∩ B) / P(B)

where P(A ∩ B) is the probability of both occurring. Conditioning means restricting the universe: we no longer consider all possible cases, only those where B holds.

MercaFresh example: if 3% of all orders are cash on delivery and also fraudulent, and 9% of orders are cash on delivery, then:

P(fraud | cash_on_delivery) = 0.03 / 0.09 = 0.33

Among cash-on-delivery orders, one third are fraudulent, even though overall fraud is much lower.

The crucial point — the source of nearly every intuition failure — is that P(A|B) ≠ P(B|A):

• P(wet | raining) ≈ 1, but P(raining | wet) is lower (a sprinkler may have caught you).
• At MercaFresh: P(alert | fraud) — what proportion of frauds triggers the alert — is not the same as P(fraud | alert) — what proportion of alerts are real fraud. The second is the one that matters to the team reviewing alerts, and it is usually surprisingly low. This lesson revolves around how to get from one to the other.

The derivation fits in three lines. The joint probability can be written two ways:

1. P(A ∩ B) = P(A|B) · P(B)
2. P(A ∩ B) = P(B|A) · P(A)

Setting them equal and solving for P(A|B):

P(A|B) = P(B|A) · P(A) / P(B)

That is the whole theorem: a direct consequence of the definition of conditional probability. Its power lies in the reading: it lets us invert the conditioning, computing the direction we care about — P(hypothesis | evidence) — from the direction we usually know — P(evidence | hypothesis).

When the denominator P(B) is not known directly, it is computed with the law of total probability, summing over all the ways B can occur. For a hypothesis H and its negation ¬H:

P(B) = P(B|H) · P(H) + P(B|¬H) · P(¬H)

This "expanded" version is the one we will use in the numerical example.

In the inferential use of the theorem, each piece has its own name:

P(H|E) = P(E|H) · P(H) / P(E)

The dotted arrow is the most elegant idea of the whole approach: learning is iterative. Each new piece of evidence turns the posterior into the prior of the next calculation. In a sense, "learning from data" — the very definition of ML from lesson 01-01 — is applying Bayes over and over.

MercaFresh installs an automatic fraudulent-order detector. The system's numbers:

• Prior: 1% of orders are fraudulent. P(F) = 0.01.
• Sensitivity (likelihood): if an order is fraudulent, the detector flags it 95% of the time. P(alert|F) = 0.95.
• False positive rate: if an order is legitimate, the detector gets it wrong and flags it 4% of the time. P(alert|¬F) = 0.04.

Question: an alert comes in. What is the probability that the order really is fraudulent? The typical intuition: "the detector is 95% accurate, so ~95%". Let's see.

Step 1 — Total probability of an alert (law of total probability):

P(alert) = P(alert|F)·P(F) + P(alert|¬F)·P(¬F) P(alert) = 0.95 · 0.01 + 0.04 · 0.99 = 0.0095 + 0.0396 = 0.0491

Step 2 — Bayes:

P(F|alert) = P(alert|F) · P(F) / P(alert) = 0.0095 / 0.0491 ≈ 0.193

Only 19.3% of alerts correspond to real fraud. Four out of five alerts are false alarms — with a "95%" detector.

The clearest way to see it is with natural frequencies over 10,000 orders:

Of the 491 alerts, only 95 are fraud: 95/491 ≈ 19.3%. The culprit is the prior: fraud is so rare that even a small error rate over the enormous mass of legitimate orders (4% of 9,900 = 396) drowns out the true positives. This table is, in essence, a confusion table of probabilities; in module 6 (lesson 06-02) we will meet it again as a classifier's confusion matrix, with metrics such as precision (which here would be exactly that 19.3%).

Let's verify it with code, twice: by formula and by simulation.

• np.where(condition, a, b) assigns each order its alert probability depending on whether it is fraud or not; comparing rng.random(n) against that probability simulates the outcome (the Bernoulli trick from lesson 02-02).
• is_fraud[alert] filters the flagged orders, and its .mean() is the fraction of them that are fraud: the empirical posterior. Formula and simulation agree.

Iterating the prior: suppose the flagged order is also cash on delivery, and we know that P(cod|F) = 0.30 versus P(cod|¬F) = 0.08. We use the previous posterior (0.193) as the new prior:

P(F | alert, cod) = (0.30 · 0.193) / (0.30 · 0.193 + 0.08 · 0.807) ≈ 0.47

Two moderate pieces of evidence have taken the probability from 1% to 47%. That is how a Bayesian reasoner accumulates evidence. (Careful: chaining like this assumes the pieces of evidence are independent given the fraud; that "naive" assumption has a name of its own, and we will meet it shortly.)

Confusing P(evidence|innocent) with P(innocent|evidence) is known as the prosecutor's fallacy, after real court cases where it was argued that "the probability of this match in an innocent person is 1%, therefore they are 99% guilty". As you have just seen, if the hypothesis is rare a priori, the posterior can be tiny even when the test looks excellent.

Practical rules that follow from the example:

• With rare events, the prior rules. No detector shines with a 1% base rate: always ask "how many real cases are there for every possible false alarm?".
• Think in natural frequencies ("out of every 10,000 orders...") instead of conditional percentages: the count table makes obvious what the formulas hide.
• The operational cost of false positives is real: with 491 daily alerts and only 95 frauds, MercaFresh's review team spends 80% of its time on legitimate orders (and may annoy good customers by blocking them). Raising the detector's threshold reduces alerts but lets frauds slip through: it is the same type I / type II trade-off from lesson 02-04, now in probabilistic form.

The two great schools of statistics differ over what "probability" means:

In modern ML practice the two coexist without drama: we evaluate models with frequentist methods (tests, intervals) and use Bayesian ideas in classifiers, regularization, or uncertainty estimation. They are complementary toolboxes, not warring religions.

• Naive Bayes (lesson 04-06). The evidence chaining we did with "alert + cash on delivery" is literally the mechanism of the Naive Bayes classifier: it combines many features assuming they are independent given the class (hence naive) and picks the class with the highest posterior. In module 4 we will build it for classification; here you already have all of its mathematics.
• Interpreting classifiers. Any detector (churn, fraud, spam) that outputs probabilities must be read in a Bayesian key: the usefulness of its alerts depends on the base rate, not just on its "accuracy".
• Update thinking. Systems that learn incrementally — adjusting beliefs order by order — follow the prior → evidence → posterior cycle you have just practiced.

• Inverting the conditioning. P(A|B) ≠ P(B|A) is the mistake. Faced with any conditional percentage, ask yourself: conditioned on what?
• Ignoring the base rate. A 95% test on a 1% event produces mostly false alarms. Always compute the posterior before trusting an alert.
• Forgetting the denominator. P(E) must include all the ways of observing the evidence (detected frauds + false alarms). Omitting the second term inflates the posterior.
• Chaining correlated evidence as if it were independent. If two signals are nearly the same thing (a high amount and a high item count), multiplying their contributions counts the same evidence twice and exaggerates the posterior; remember the correlations from 02-03.
• Treating the prior as a whim. In real applications the prior comes from historical data (the observed fraud rate), not from a hunch. Document it like any other piece of data.
• Tip: whenever a Bayesian calculation confuses you, translate it into a frequency table over 10,000 cases, like the one in this lesson. It never fails to restore intuition.

At MercaFresh, 20% of customers belong to the "large family" segment. Among them, 60% buy the store brand; among the remaining customers, only 25% do. An order comes in with store-brand products: what is the probability that it belongs to a large family? State the prior, the likelihood, and the evidence before computing.

MercaFresh's fraud detector is recalibrated: it keeps its sensitivity at 0.95 but reduces false positives from 0.04 to 0.01. With the same 1% prior, recompute P(fraud|alert) with Bayes' formula and rebuild the 10,000-order table. How much does the ratio of useful alerts improve?

Adapt the NumPy simulation from the example to verify your Exercise 2 result with one million orders. Then answer: if instead of improving the detector, the fraud rate rose from 1% to 5% (original detector: 0.95 / 0.04), what posterior would you get? What does comparing the two scenarios teach you?

Solution 1

• Prior: P(LF) = 0.20. Likelihood: P(SB|LF) = 0.60. Also, P(SB|¬LF) = 0.25.
• Evidence: P(SB) = 0.60·0.20 + 0.25·0.80 = 0.12 + 0.20 = 0.32.
• Posterior: P(LF|SB) = 0.12 / 0.32 = 0.375.

Seeing store-brand products nearly doubles the probability of a large family (from 20% to 37.5%), but it remains more likely that the order is not one: evidence updates, it does not pronounce a verdict.

Solution 2

• P(alert) = 0.95·0.01 + 0.01·0.99 = 0.0095 + 0.0099 = 0.0194.
• P(F|alert) = 0.0095 / 0.0194 ≈ 0.49.

Table over 10,000 orders: 95 true positives and 99 false positives → 194 alerts, of which nearly half are real fraud. Cutting false positives from 4% to 1% (without touching sensitivity) lifts useful alerts from 19% to 49% and reduces daily alerts from 491 to 194: for rare events, lowering the false positive rate pays off far more than raising sensitivity.

Solution 3

By formula, the second scenario: P(alert) = 0.95·0.05 + 0.04·0.95 = 0.0855, posterior = 0.0475/0.0855 ≈ 0.556. The moral: the posterior depends on the detector and on the prior in equal measure. The very same detector goes from generating 80% false alarms to being mostly reliable just because the phenomenon is more frequent. This is why an ML model validated in one environment (or one era) with a given base rate can degrade when that rate shifts: a drift phenomenon we will revisit when discussing monitoring in module 8.

You have closed the module with the crown jewel of applied probability: starting from conditional probability you derived Bayes' theorem, learned the prior–likelihood–posterior vocabulary, and applied it to MercaFresh's fraud detector, discovering that a "95%" detector produces four false alarms for every real fraud when the event is rare, and that evidence accumulates by iterating the theorem. You also placed the frequentist and Bayesian schools as complementary tools, and noted that the Naive Bayes classifier of module 4 is nothing more than this theorem turned into an algorithm. This concludes the foundations of statistics and probability: you can now describe data, model its randomness, measure relationships, quantify uncertainty, and update beliefs. In module 3 we will put these tools to work on real, imperfect data: we will start with data cleaning, the step where MercaFresh's datasets — with their duplicates, errors, and impossible values — get ready to feed the models.