The confidence intervals of the previous lesson left us with claims of the form "the store vs online gap is between 0.4 and 1 point". In this lesson we learn the other great machinery of inference: hypothesis tests, the formal procedure for deciding between two competing claims — "the change has had no effect" versus "yes, it has" — with a known and controlled risk of error. We will see the logic of the test (surprisingly similar to a trial), the role of the p-value (the most cited and worst understood number in all of statistics), and we will apply it to three real NovaMarket decisions: whether the new app design raises the average transaction value, whether logistics is breaching its delivery SLA, and whether online satisfaction really is worse than in-store.

1. The logic of the test: presumption of innocence
2. Null and alternative hypotheses; one-tailed and two-tailed tests
3. The five-step procedure: test statistic, α and decision rule
4. Case 1 — Does the new app design raise the average transaction value? (z test for one mean)
5. The p-value: correct interpretation and famous misreadings
6. Case 2 — Is the delivery SLA being breached? (test for a proportion)
7. The t test for one mean and the CI ↔ test connection
8. Case 3 — Store vs online (test for two means)
9. Statistical significance vs practical relevance

A hypothesis test works exactly like a criminal trial:

Three key ideas from the analogy:

• The burden of proof falls on the alternative. We start from the premise that the change does not work, and we only accept it if the data back it up strongly. It is a deliberately conservative design: it protects the company from chasing mirages.
• We never "accept" \( H_0 \). A "not guilty" verdict does not certify innocence; it only says the evidence was insufficient. Likewise, "failing to reject \( H_0 \)" means "the data are not enough to claim the effect", not "we have proved there is no effect".
• "Beyond reasonable doubt" needs a numerical threshold. That threshold will be \( \alpha \), the significance level.

• \( H_0 \) (null): the status quo claim, always carrying the equals sign: \( \mu = \mu_0 \), \( p = p_0 \). It is the one put on trial.
• \( H_1 \) (alternative): what we want to detect. Its form defines the type of test:

Golden rule: the direction of the test is decided before looking at the data, based on the business question — never afterward, by looking at which way the sample came out (that artificially inflates results, as we will see when we discuss p-hacking in the next lesson).

Every test follows the same liturgy:

1. State \( H_0 \) and \( H_1 \) (and decide whether it is one- or two-tailed).
2. Set the significance level \( \alpha \): the maximum probability we accept of rejecting \( H_0 \) when it is true (convicting an innocent). Usual conventions: 0.05 (standard), 0.01 (demanding), 0.10 (exploratory). It is set before seeing the data.
3. Compute the test statistic: a measure of the discrepancy between what was observed and what \( H_0 \) predicts, in units of standard error. For a mean it is the Module 2 z-score applied to \( \bar{x} \): \[ z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} \qquad \text{or} \qquad t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \] Read it as: "how many standard errors away is my sample from what the null claims?".
4. Decide, by either of these two equivalent routes:
   • Critical region: reject if the statistic exceeds the table's critical value (1.645 for one-tailed at 5%; 1.96 for two-tailed at 5%…).
   • p-value: compute the probability of a result as extreme as the one observed, assuming \( H_0 \) is true, and reject if \( p < \alpha \).
5. Conclude in business language, not jargon: what was decided, at what risk, and what it implies.

The product team redesigned the app's checkout screen with personalized recommendations. NovaMarket's historical average transaction value is \( \mu_0 = \text{€}32.40 \) with \( \sigma = \text{€}21.50 \) (Module 2), and the redesign is assumed not to change the dispersion. After the launch, a random sample of \( n = 225 \) app receipts is drawn: \( \bar{x} = \text{€}34.60 \).

Step 1. \( H_0: \mu = 32.40 \) (the redesign changes nothing) versus \( H_1: \mu > 32.40 \) (one-tailed: we only care about an increase).

Step 2. \( \alpha = 0.05 \).

Step 3. Standard error: \( 21.50/\sqrt{225} = 21.50/15 = \text{€}1.433 \). Statistic:

\[ z = \frac{34.60 - 32.40}{1.433} = \frac{2.20}{1.433} \approx 1.53 \]

The sample mean sits 1.53 standard errors above what the null predicts.

Step 4. One-tailed critical value at 5%: 1.645. Since \( 1.53 < 1.645 \), we fail to reject \( H_0 \). Via the p-value route: \( p = P(Z \ge 1.53) = 1 - 0.9370 = 0.063 \) (Module 4 Z table). Since \( 0.063 > 0.05 \), same decision.

Step 5. Business conclusion: "Receipts in the sample are up €2.20 on average, but with 225 receipts that increase is compatible with sampling chance (p = 0.063): we cannot claim at the 5% level that the redesign raises the average transaction value. A promising but inconclusive result; we recommend enlarging the sample before rolling out the design". Careful: we have not proved that the redesign does not work — only that the evidence is not yet sufficient. The difference between "not detected" and "does not exist" is the heart of the next lesson.

Correct definition: the p-value is the probability of obtaining a result as extreme as or more extreme than the one observed, assuming \( H_0 \) is true. It is a measure of surprise: small p = "if the null were true, seeing this would be very odd" → we grow suspicious of the null.

In case 1: if the redesign did nothing, a sample with mean ≥ €34.60 would show up 6.3% of the time by pure chance. Odd, but not extraordinary.

Misinterpretations of the p-value are so frequent that the American Statistical Association (ASA) published an official statement in 2016 to fight them. The four most famous:

Professional tip: in reports, always accompany the p-value with the effect size (the €2.20 increase) and, if you can, its confidence interval. The p says "how sure"; the effect says "how much it matters".

NovaMarket Online's SLA promises that at most 10% of deliveries exceed 48 hours (recall: F(48) = 0.899 in the Module 4 model, about 10% late). Operations suspects the new logistics operator is breaching it. An audit covers \( n = 300 \) deliveries: 39 late, that is, \( \hat{p} = 0.13 \).

Setup. \( H_0: p = 0.10 \) (the SLA is being met) versus \( H_1: p > 0.10 \) (it is being breached). \( \alpha = 0.05 \).

Statistic. With proportions, the standard error is computed with the null's value (\( p_0 \)), because the test is run "as if \( H_0 \) were true":

\[ z = \frac{\hat{p} - p_0}{\sqrt{\dfrac{p_0(1-p_0)}{n}}} = \frac{0.13 - 0.10}{\sqrt{\dfrac{0.10 \times 0.90}{300}}} = \frac{0.03}{0.01732} \approx 1.73 \]

(Validity: \( np_0 = 30 \) and \( n(1-p_0) = 270 \), both ≥ 10, so the Module 4 normal approximation is legitimate.)

Decision. \( 1.73 > 1.645 \): we reject \( H_0 \). p-value: \( P(Z \ge 1.73) = 1 - 0.9582 = 0.042 < 0.05 \).

Business conclusion: "With 13% of deliveries late out of 300 audited, there is significant evidence at the 5% level that the operator is breaching the 10% SLA (p = 0.042). We recommend triggering the contract's review clause and repeating the audit next month". Note the caution: p = 0.042 is significant but not overwhelming; a second audit protects the decision.

When \( \sigma \) is unknown (the usual case), the statistic uses \( s \) and is compared against the t table with \( n-1 \) degrees of freedom, exactly as with intervals.

Example: Club Nova spend against the target. Years ago, Finance set a planning assumption: "the average member spends €120 a month". Do our data contradict it (\( n = 60 \), \( \bar{x} = 126.40 \), \( s = 38.50 \), SE = 4.97)?

• \( H_0: \mu = 120 \) versus \( H_1: \mu \neq 120 \) (two-tailed: falling short would matter as much as overshooting). \( \alpha = 0.05 \).
• \( t = \dfrac{126.40 - 120}{4.97} = \dfrac{6.40}{4.97} \approx 1.29 \)
• Two-tailed critical value: \( t_{0.025,,59} = 2.001 \). Since \( |1.29| < 2.001 \), we fail to reject. (The two-tailed p is around 0.20.)
• Conclusion: "the data are compatible with the €120 assumption; there is no basis for revising the financial plan".

The connection with intervals. In the previous lesson we obtained the 95% CI for spend: [116.46; 136.34]. Notice: it contains 120, and the two-tailed test at 5% does not reject \( \mu = 120 \). That is no coincidence — it is a miniature theorem:

A two-tailed test at level \( \alpha \) rejects \( H_0: \mu = \mu_0 \) exactly when \( \mu_0 \) falls outside the CI at \( (1-\alpha) \) confidence.

The interval is, literally, "the set of all the nulls that would survive the test". That is why many analysts prefer to report the CI: it contains the same decision and also the effect size.

We close the case left open in the previous lesson: in-store satisfaction \( \bar{x}_1 = 8.3 \) (\( n_1 = 450 \), \( s_1 = 1.5 \)) versus online \( \bar{x}_2 = 7.6 \) (\( n_2 = 190 \), \( s_2 = 1.9 \)). Marta asks: "is online really worse, or is it noise?".

• \( H_0: \mu_1 - \mu_2 = 0 \) versus \( H_1: \mu_1 - \mu_2 \neq 0 \). \( \alpha = 0.05 \). (We state it two-tailed: before seeing the data, the difference could have gone either way.)
• Statistic (large, independent samples; the standard error of the difference, 0.155, we already computed):

\[ z = \frac{(8.3 - 7.6) - 0}{\sqrt{\dfrac{1.5^2}{450} + \dfrac{1.9^2}{190}}} = \frac{0.70}{0.155} \approx 4.52 \]

• Decision: \( 4.52 \gg 1.96 \); the two-tailed p-value is below 0.0001 (off the Z table's range: 4.52 standard errors is "practically impossible by chance" territory).
• Consistency with the CI: [0.40; 1.00] did not contain zero — same conclusion, as it had to be.

Business conclusion: "The 0.7-point difference between store and online is statistically unequivocal (p < 0.0001). The online channel has a real satisfaction problem; we propose investigating its causes — the delivery times from case 2 are the prime suspect".

The lesson's final idea, and perhaps the most important one for a professional: significant does not mean important. "Significant" only means "hard to attribute to chance". With enormous samples, tiny, irrelevant effects come out significant; with small samples, large, valuable effects may not.

The professional protocol: always look at both things — the p-value (is it real?) and the effect size with its CI (how big is it and how much does it matter?) — and translate both into the language of the business (euros, points, customers). Deciding how much sample is needed for a relevant effect to also be detectable is exactly the problem the next lesson tackles.

• "Accepting \( H_0 \)". Failing to reject is not accepting. Write "there is insufficient evidence that…", never "it is proven that there is no…". The app case is the canonical example.
• Choosing the direction of the test after seeing the data. Turning a two-tailed test into a one-tailed one after the fact halves the p by sleight of hand. The hypothesis is fixed before looking.
• Interpreting p as P(H₀ true). It is P(data this extreme | H₀), and conditionals do not flip for free (Bayes, Module 3).
• Using \( \hat{p} \) instead of \( p_0 \) in the standard error of a proportion test. In the test, the standard error is computed under the null (\( p_0 \)); in the interval, with \( \hat{p} \). It is the operational difference between the two tools.
• Treating 0.05 as a sacred boundary. p = 0.049 and p = 0.051 say almost the same thing. Report the exact p and the effect, and let the decision weigh costs and context.
• Ignoring the effect size. A significance asterisk with no euros behind it should not move any decision.

Exercise 1. Fruit trays labeled "500 g". A sample of \( n = 49 \) trays gives \( \bar{x} = 493 \) g with \( \sigma = 21 \) g (known from the packing plant's history). Test at the 5% level whether the mean weight differs from the label (think first about whether it is one- or two-tailed) and conclude in business language.

Exercise 2. Marketing claims that "more than half" of Club Nova members are active. With the sample from lesson 05-01 (\( n = 150 \), 96 active, \( \hat{p} = 0.64 \)), test \( H_0: p = 0.50 \) versus \( H_1: p > 0.50 \) at the 1% level. Compute the statistic, decide, and give the approximate p-value.

Exercise 3. An analyst writes: "p = 0.20 in the spend-vs-€120 test, so there is an 80% probability that average spend differs from €120… no, wait, that it equals €120". Explain why both readings are wrong and give the correct one. Then, without computing anything new, say what would have happened to the test if the 95% CI had been [121.3; 141.2].

Solution 1. Two-tailed (\( H_1: \mu \neq 500 \)): overweight costs margin and underweight is a legal problem — both deviations matter. SE \( = 21/\sqrt{49} = 3 \) g. \( z = (493-500)/3 = -2.33 \). Since \( |-2.33| > 1.96 \), we reject (two-tailed p \( = 2 \times P(Z \le -2.33) = 2 \times 0.0099 \approx 0.02 \)). Conclusion: "the trays weigh significantly less than labeled on average (about 7 g, p ≈ 0.02); the packing line must be recalibrated and the risk of complaints reviewed". Common mistake: stating it left one-tailed because the sample came out low — the direction is decided by the question, not by the data.

Solution 2. SE under the null: \( \sqrt{0.50 \times 0.50/150} = \sqrt{0.001667} = 0.0408 \). \( z = (0.64 - 0.50)/0.0408 \approx 3.43 \). One-tailed critical value at 1%: 2.326. Since \( 3.43 > 2.326 \), we reject; \( p = P(Z \ge 3.43) \approx 0.0003 \). Conclusion: very strong evidence that more than half of the members are active, even at a demanding α. Common mistake: using \( \hat{p} = 0.64 \) in the standard error (it would give 0.0392); in the test we use \( p_0 = 0.50 \).

Solution 3. The p-value is not the probability of any hypothesis: p = 0.20 means that, if average spend were exactly €120, a deviation like the one observed (or larger) would appear 20% of the time by chance — too frequent to discard the null. Neither "80% that it differs" nor "20% that it is equal". With a 95% CI of [121.3; 141.2], the 120 would fall outside the interval, and by the CI ↔ two-tailed test equivalence, the test at 5% would have rejected \( H_0: \mu = 120 \). Common mistake: forgetting that this equivalence requires the level (95% ↔ 5%) and the two-tailed character to match.

You now command statistics' decision-making machinery: stating \( H_0 \) and \( H_1 \) (with the burden of proof on the alternative), setting \( \alpha \) before looking, measuring the discrepancy in standard errors (z or t, for means and proportions), translating it into a properly interpreted p-value, and concluding in business language — knowing that the CI and the two-tailed test are two sides of the same coin, and that statistical significance never substitutes for practical relevance.

But in this lesson we have lived with two discomforts: in rejecting \( H_0 \) we may be convicting an innocent, and in failing to reject it (as with the app) we may be letting a culprit walk free. What are those two errors called, what do they cost, how are they balanced, and — the million-euro question — how much sample do I need for my test to have real power? All of that, including the promise outstanding from Module 1 about the famous "around 400 respondents for a ±5%", is resolved in Errors, Power and Sample Size.