The previous lesson left us with two unresolved discomforts: in rejecting \( H_0 \) we may "convict an innocent", and in failing to reject it — as happened with the app redesign — we may "let a culprit walk free". This lesson gives those two errors a name, a number and a price (Type I and Type II), introduces the power of a test, and answers the most practical question in all of inference: how much sample do I need? With it, we will settle the promise we left in Module 1 (that "around 400 respondents give a ±5%") and size a real project: the coupon campaign pilot Marta wants to run before launching to all 380,000 Club Nova members. We close with two warnings of professional maturity: the danger of multiple comparisons (p-hacking) and the ethics of statistical honesty.

1. The two possible errors: Type I (α) and Type II (β)
2. Power (1 − β) and what increases it
3. The trade-off between α and β
4. Sample size for estimating a mean with a given margin of error
5. Sample size for estimating a proportion: closing the promise of the ~400
6. Sizing the coupon pilot: the minimum relevant effect
7. Multiple comparisons and p-hacking
8. The ethics of statistical honesty

Every test ends in a decision (reject or fail to reject \( H_0 \)) made without knowing the truth. Crossing decision and reality yields four cells:

The classic analogy is a smoke detector:

• Type I error (false alarm): the detector goes off with no fire. At NovaMarket: concluding that the app redesign raises the average transaction value when in fact it does nothing → a useless change is rolled out across the chain, with its development and training costs.
• Type II error (undetected fire): there is a fire and the detector stays silent. At NovaMarket: the redesign did work, but our sample of 225 receipts failed to detect it (sound familiar? p = 0.063) → an improvement worth money gets shelved.

Two important nuances:

• \( \alpha \) is something you choose when setting the significance level: it is the false-alarm rate your decision rule tolerates. That is why it is set before looking at the data.
• \( \beta \) is not a single number: it depends on how big the real effect is. If the redesign raises the transaction value by €5, detecting it is easy and \( \beta \) is small; if it raises it by €0.50, it is exceedingly hard and \( \beta \) is enormous. That is why β is always computed "for an assumed effect".

Power is the probability of detecting an effect when it really exists: the sensitivity of the smoke detector. A test with power 0.80 (the usual standard in practice) finds the effect 4 out of every 5 times it is there; with power 0.30, most real effects slip past it — and the study is, almost literally, money down the drain.

Four levers increase power:

Let us reread the app case through these eyes: with \( n = 225 \), a real increase of €1 (an effect of 1/1.433 ≈ 0.7 standard errors) would have had very low power — the study was born nearly incapable of detecting modest effects. The "not significant" of the previous lesson was, in part, a design problem, not a reality problem. Moral: power is decided before collecting the data, not lamented afterward.

With the sample fixed, α and β are the two pans of a scale: if you make the criterion more demanding (lowering α from 0.05 to 0.01 to guard against false alarms), the bar for rejection rises and more real effects escape you (β goes up, power goes down). And vice versa. It is the smoke detector's sensitivity dial: very sensitive = many false alarms; very insensitive = fires going undetected.

Where to set the balance? It depends on the relative cost of each error, which is a business decision, not a statistical one:

The only way to lower both errors at once is more information: increasing \( n \). Which brings us to the star question.

For estimation, the setup is direct: I fix the margin of error \( E \) I can tolerate and solve for \( n \) in the interval formula. From \( E = z_{\alpha/2} \cdot \sigma/\sqrt{n} \):

\[ n = \left( \frac{z_{\alpha/2} \cdot \sigma}{E} \right)^{2} \]

Example: Club Nova spend to within ±€5. The CI from lesson 05-02 came out [116.46; 136.34] — a margin of ±€9.94, which Finance finds too wide. They ask for ±€5 at 95%. Using as \( \sigma \) the best information available (the \( s = 38.50 \) from the earlier study):

\[ n = \left( \frac{1.96 \times 38.50}{5} \right)^{2} = \left( \frac{75.46}{5} \right)^{2} = (15.09)^{2} \approx 227.8 ;\Rightarrow; n = 228 \]

(Always round up: 227 members do not guarantee the margin.) Roughly quadrupling the sample (from 60 to 228) to halve the margin — the square-root law once again. And if they asked for ±€2.50, it would come out at \( n \approx 912 \): every doubling of precision quadruples the cost. This calculation turns a vague conversation ("we want more precision") into a budgetable decision ("do we pay for 228 or 912 verifications?").

For a proportion, solving from its margin of error:

\[ n = \frac{z_{\alpha/2}^{2} \cdot p(1-p)}{E^{2}} \]

Problem: we need \( p \)… which is precisely what we want to estimate. Two ways out: use a prior estimate if one exists, or assume the worst case, \( p = 0.5 \), which maximizes \( p(1-p) = 0.25 \) and guarantees the margin no matter what.

The Module 1 promise. When we designed the satisfaction survey, we said that "around 400 respondents give a margin of ±5%". Now we can prove it. With 95% confidence, worst case and \( E = 0.05 \):

\[ n = \frac{1.96^{2} \times 0.25}{0.05^{2}} = \frac{0.9604}{0.0025} = 384.2 ;\Rightarrow; n = 385 \approx 400 \]

There it is — the famous ~400 of every election poll and market study. The full worst-case table, useful for budgeting any survey:

Two valuable readings: precision gets quadratically more expensive (from ±5% to ±1% the sample multiplies by 25), and — a habitual surprise — the size of the population barely matters: 385 responses give ±5% just the same for the 9,500 members in Cuenca as for the 380,000 in Club Nova (the finite-population correction only helps when you sample a large fraction of the population, which is rare in practice). You taste the soup with a spoon, whether the pot is large or enormous.

For tests (not just estimating, but detecting an effect), the sample size depends on four ingredients that must be agreed before the study: \( \alpha \), the desired power, the dispersion \( \sigma \) and — the most business-driven decision of all — the minimum relevant effect (MRE): the smallest effect that would be worth detecting.

The case. Marketing wants to send personalized coupons to all 380,000 Club Nova members, but first demands a pilot: coupon group vs control group, comparing the average transaction value (\( \mu_0 = \text{€}32.40 \), \( \sigma = \text{€}21.50 \)). Finance calculates that the campaign is only profitable if it lifts the transaction value by at least €1: that is the MRE — smaller lifts do not pay for the cost of the discounts.

For comparing two groups with a two-tailed test, the working formula (which we will use as a recipe, without deriving it) is:

\[ n_{\text{per group}} \approx \frac{2,(z_{\alpha/2} + z_{\beta})^{2},\sigma^{2}}{\Delta^{2}} \]

where \( \Delta \) is the MRE and \( z_{\beta} \) the z value of the desired power (power 0.80 → \( z_{\beta} = 0.84 \), the normal's P80; power 0.90 → 1.282, our old acquaintance P90).

With \( \alpha = 0.05 \) (two-tailed, 1.96), power 0.80 and \( \Delta = \text{€}1 \):

\[ n_{\text{per group}} \approx \frac{2 \times (1.96 + 0.84)^{2} \times 21.50^{2}}{1^{2}} = \frac{2 \times 7.84 \times 462.25}{1} \approx 7{,}248 ;\Rightarrow; \text{about } 7{,}250 \text{ members per group} \]

Business reading: the pilot needs about 14,500 members (7,250 with coupons, 7,250 as control) — perfectly feasible out of 380,000, but a very long way from the "500 and we'll see" Marketing was proposing. With 500 per group, the power to detect +€1 would hover around 10%: nine times out of ten the pilot would say "not significant" even if the campaign worked, condemning a good idea (the exact design error we made with the app in the previous lesson). And if the MRE were €2, the sample would drop to \( 7{,}248/4 \approx 1{,}812 \) per group: detecting fine effects is very expensive; detecting coarse ones, cheap. The MRE — how much the transaction value has to rise before we care — is the conversation the analyst must force with Finance before collecting a single data point.

A silent danger: every test at the 5% level is a lottery with a 5% false-alarm rate. If you run many tests, the false alarms pile up.

A homegrown example: comparing the satisfaction of each of NovaMarket's 42 stores against the chain average, at the 5% level. Even if no store were genuinely different, we would expect \( 42 \times 0.05 \approx 2 \) "significant" stores by pure chance. The probability of at least one false alarm would be \( 1 - 0.95^{42} \approx 0.88 \): 88%. The headline "Sevilla-Nervión stands out significantly" would be almost guaranteed… and probably empty.

The deliberate version of this phenomenon is called p-hacking: trying many variants (segments, metrics, time windows, dropping "outliers") until something crosses the 0.05 line, and publishing only that. No bad faith is needed — the enthusiasm of "keep digging, something will turn up" is enough. Basic defenses:

• Preregister the hypothesis, the metric and the analysis before looking at the data (just as we fixed the direction of the test in the previous lesson).
• Distinguish confirmatory analysis (the agreed hypothesis) from exploratory analysis (everything else: legitimate, but labeled as hypothesis-generating and reconfirmed with fresh data).
• If you run many comparisons, adjust the criterion (the simplest fix, the Bonferroni correction, splits α across the tests: with 42 stores, demand p < 0.05/42 ≈ 0.0012). The serious treatment of comparing many groups at once arrives with ANOVA in the lesson Analysis of Variance.

We close the module with what turns technique into a profession. Whoever masters inference can, with the same data, manufacture whatever headline they want: choose the test after the fact, cut the sample where it suits, stay silent about the analyses that did not pan out. An honest analyst binds themselves to the opposite:

• Decide before looking: hypothesis, α, metric and sample size are fixed in advance; changes made along the way are disclosed.
• Report everything: the effect with its interval (not just the p), the analyses that did not come out significant, the number of comparisons attempted and the limitations of the sample (no p-value cures the biases of Module 1).
• Do not mislead the decision-maker: translate into euros and points, distinguish "not detected" from "does not exist", and resist the pressure of "find me a p < 0.05 for the committee".

The analytics team's credibility is its only real asset: it is built report by report and lost with a single inflated headline. As Marta puts it: "I'd rather have a 'we don't know yet, we need 7,250 per group' than a 'it works!' that collapses at rollout".

• Confusing the error types. Mnemonic: Type I = false alarm (you reject without cause; its probability is the α you set); Type II = undetected fire (you fail to reject when you should; probability β).
• Believing β is fixed. Power is always "power to detect an effect of such-and-such size"; make a habit of asking "power against what effect?" of any study put in front of you.
• Interpreting "not significant" without checking the power. With low power, a p > 0.05 says almost nothing — the study could not have detected the effect even if it existed. Before shelving an idea, check what power it was tested with.
• Rounding n down, or forgetting the worst case for proportions. The computed n is a minimum: always round up. And with no prior estimate of p, use 0.5.
• Sizing for the expected effect instead of the minimum relevant one. The pilot is not designed for the effect you dream of (+€3) but for the minimum that would justify acting (+€1); otherwise a "not significant" result will not let you tell "it falls short of the threshold" from "we couldn't see it".
• Looking at twenty slices and reporting the one that came out. If you have tried 20 segments, one will almost surely be significant by chance. Disclose how many comparisons you made, or adjust the criterion.

Exercise 1. NovaMarket's quality control tests every batch of fruit (\( H_0 \): the batch meets the standard) with α = 0.05. (a) Describe in business terms the Type I and Type II errors of this test and who "pays" for each one. (b) If the supplier is new and there are serious doubts about their reliability, would you move α toward 0.01 or toward 0.10? Reason it out with the α↔β scale.

Exercise 2. HR wants to estimate the proportion of employees interested in the new shift plan with a margin of ±4% at 95%, with no prior estimate. (a) Compute the n needed. (b) An executive objects: "we are 6,000 employees, not Club Nova's 380,000; we'll need far fewer". Is he right?

Exercise 3. For the coupon pilot, Finance changes the minimum relevant effect from €1 to €1.50, keeping α = 0.05 (two-tailed), power 0.80 and σ = €21.50. Recompute the n per group and explain why the result is not "two thirds" of the original.

Solution 1. (a) Type I: rejecting a batch that actually met the standard — a false alarm; the company pays (wasted fruit, conflict with the supplier). Type II: accepting a defective batch — an undetected fire; the customer and the brand pay (complaints, health risk). (b) Toward 0.10: with a doubtful supplier, the expensive error is Type II, so a more sensitive detector is warranted — more false alarms in exchange for less bad fruit slipping through; α = 0.01 would do exactly the opposite. Common mistake: thinking that "more demanding" (a lower α) always protects more; it protects against Type I, but exposes you to Type II.

Solution 2. (a) \( n = \dfrac{1.96^{2} \times 0.25}{0.04^{2}} = \dfrac{0.9604}{0.0016} = 600.25 \Rightarrow n = 601 \). (b) No: n depends on the margin, the confidence and \( p(1-p) \), not on the population size — 601 responses give ±4% just the same for 6,000 as for 380,000 (the finite-population correction would barely lower the figure here, and it only matters when sampling a large fraction of the population). Common mistake: scaling n in proportion to population size; the precision comes from the spoon, not the pot.

Solution 3. \( n = \dfrac{2 \times (1.96 + 0.84)^{2} \times 21.50^{2}}{1.50^{2}} = \dfrac{7{,}248.1}{2.25} \approx 3{,}221 \Rightarrow \) about 3,225 members per group (~6,450 in total). It is not two thirds of 7,250 but less than half, because \( \Delta \) enters the denominator squared: relaxing the detectable effect by 50% divides the sample by \( 1.5^{2} = 2.25 \). Common mistake: scaling linearly with \( \Delta \); the relationship is quadratic, and it is the reason fine effects are so expensive to prove.

The inference module is complete. We know how to estimate parameters and measure the precision of the estimate (05-01), wrap it in intervals with guarantees (05-02), put claims to the test with a properly understood p-value (05-03) and, in this lesson, manage the two possible errors — false alarm (α) and undetected fire (β) —, demand power from our studies and compute the sample needed: 228 members for spend to ±€5, the 385 that explain the mythical "~400 for ±5%" promised in Module 1, and 7,250 per group so the coupon pilot can detect the euro that makes it profitable. And, above all, we have learned that statistical honesty — deciding before looking, reporting everything, not torturing the data — is worth more than any formula.

Up to here we have made inferences about one variable at a time: a mean, a proportion, a difference. But NovaMarket's juiciest questions relate variables to one another: do stores with more foot traffic sell more? Does satisfaction rise when delivery times fall? Measuring the strength of those relationships — and not confusing correlation with causation — is the starting point of Module 6, which opens with Correlation Analysis.