How Many of Your Banner's Significant Results Are Noise?

A companion to Sampling Weights in Significance Tests, which ends by observing that a mis-specified standard error turns ten spurious findings into sixteen. This article deals with the ten.

Part 1 — The arithmetic nobody runs

A standard market research banner is a table of perhaps 25 row variables crossed with 8 column subgroups, with significance letters against every cell. Nobody thinks of that as a multiple comparisons problem. It is one, and the arithmetic is not subtle.

If you run $m$ independent tests at $\alpha =0.05$ and every null hypothesis is true, the number of false positives is $\text{Binomial}(m,0.05)$. Its expectation is $0.05m$. The probability of at least one is $1-0.95^m$, which reaches 99% by $m=90$.

I simulated this directly rather than relying on the closed form, because real banner tests are on discrete binomial data with finite bases and the exact distribution matters. Two independent samples of $n=300$ per subgroup, true proportion 0.35 in both, 200 tests, 20,000 replications:

Ten. Every time. On a table where, by construction, nothing whatsoever is happening.

This is not a statement about bad data or a bad sample. It is what $\alpha =0.05$ means, applied 200 times. The 5% is a per-test error rate, and a banner is not one test.

The practical consequence is that a debrief deck sourced from an all-null banner would contain roughly ten "findings", each one carrying a significance letter, each one defensible on its own terms, and every one of them noise. In the worst run of twenty thousand, it would contain twenty-six.

Why "but my tests aren't independent" doesn't save you

The standard objection is that banner tests are correlated — the same respondents appear across the row variables, subgroups are nested, and so the independence assumption is wrong.

True, and it doesn't help. Correlation between tests changes the variance of the false-positive count, not its mean. Expected false positives is $0.05m$ by linearity of expectation, which holds regardless of dependence. Correlation makes the count clumpier — more runs with zero, more runs with twenty — but the average is unmoved. You cannot correlate your way out of the expectation.

Part 2 — The corrections, and what they cost

Bonferroni

Test each hypothesis at $\alpha /m$ instead of $\alpha$. This controls the family-wise error rate (FWER) — the probability of any false positive — at $\alpha$, and it does so under arbitrary dependence, which is why it survives despite being crude.

$$p_i<\frac{\alpha }{m}$$

The simulation confirms it works exactly as advertised: 0.049 expected false positives across the whole 200-test banner.

Benjamini–Hochberg

Bonferroni controls the probability of any error. BH controls the false discovery rate — the expected proportion of your significant findings that are wrong. That is usually the quantity you actually care about: "of the fourteen things I am about to put in the deck, how many are junk?"

Sort the p-values ascending, $p_{(1)}\le \cdots \le p_{(m)}$. Find the largest $k$ with:

$$p_{(k)}\le \frac{k}{m}\alpha$$

and reject everything up to and including $p_{(k)}$.

Under the global null, FDR control coincides with FWER control, which is why BH also returns ~0.05 expected false positives in the table above. The two methods diverge only when some effects are real.

The part that is usually left out

So run BH and move on? Not quite. Here is the same banner with 20 of the 200 comparisons carrying a genuine 8 percentage point difference, 6,000 replications:

Read that power row again. Correcting for multiplicity takes a test that finds real effects half the time and reduces it to finding them one time in twenty. Bonferroni at $\alpha =0.05/200=0.00025$ demands a z-statistic beyond 3.66; an 8pp difference on bases of 300 simply does not produce that except by luck.

BH does better than Bonferroni — 8.3% against 4.8%, a genuine improvement of about 70% in relative terms — but it is still nowhere near usable. BH's advantage grows with the proportion of true effects and with per-test power; here both are modest, so it has little to work with.

This is the honest position, and it is uncomfortable: on a typical banner, you can control the error rate or you can detect the effects, and you cannot do both. A correction applied to a 200-test banner does not rescue the analysis. It converts a table full of false positives into a table full of nothing.

Part 3 — What to actually do

If correction is not the answer, and doing nothing is not the answer, what is left? Four things, in rough order of value.

1. Stop running 200 tests

The multiplicity problem is created at the point where you decide to test every cell of a banner against every other. Almost nobody needs that. A study has perhaps three to eight questions it was commissioned to answer.

Pre-specify those, test them, and correct across that small family. At $m=6$, Bonferroni costs you very little: the threshold is 0.0083, needing a z of 2.64 against 1.96 — a real but survivable loss. At $m=200$ it is fatal. The cost of correction scales with how undisciplined you were about what you were testing.

This is the same discipline as pre-registration in clinical work, and it is unpopular for the same reason: it stops you claiming credit for things you found by looking.

2. Separate confirmatory from exploratory, and label them

The rest of the banner is not worthless — it is exploratory. The correct treatment is to report it without significance flags, or with flags explicitly marked as uncorrected and exploratory, and to describe patterns rather than assert them.

A finding that emerges from an uncorrected 200-cell scan is a hypothesis, not a result. It is worth saying so in the deck. "Among over-55s in the North, consideration is 11 points higher; this was not a pre-specified comparison and would need confirming in a follow-up wave" is an honest and genuinely useful sentence. Putting a significance letter next to it is not.

3. Use effect sizes and intervals, not flags

A significance letter compresses an estimate and its uncertainty into one bit, and it is the wrong bit. It answers "can we reject zero?" when the client's question is nearly always "how big is it, and how sure are we?"

Reporting a difference of +8.4pp, 95% CI [−1.2, +18.0] conveys everything the flag does and considerably more: the direction, the plausible magnitude, and the fact that the data are consistent with anything from a small negative to a large positive. Nobody reads that and writes a strategy deck around it. Whereas "significant at 95%" invites exactly that.

This also sidesteps the multiplicity problem in a useful way. Interval estimates do not make a dichotomous claim, so they do not accumulate dichotomous errors the same way. (Simultaneous coverage across many intervals is its own problem, but a reader who sees twenty overlapping intervals draws the right conclusion without needing the correction.)

4. If you must flag a whole banner, use BH and state it

When the deliverable is a flagged banner and that is non-negotiable — which in tracker and syndicated work it often is — Benjamini–Hochberg across the table is the least-bad option, and the technical note should say so:

Significance flags control the false discovery rate at 5% across all comparisons in the table (Benjamini–Hochberg). Flags identify differences unlikely to be artefacts of multiple testing; the absence of a flag does not establish equivalence.

That second clause matters. With power at 8%, absence of a flag is nearly uninformative, and a reader who interprets unflagged cells as "no difference" will be wrong most of the time.

Part 4 — Implementation

R — p.adjust covers both methods:

p <- c(0.001, 0.008, 0.02, 0.04, 0.13, 0.31)

p.adjust(p, method = "bonferroni")
p.adjust(p, method = "BH")          # Benjamini-Hochberg

# Adjusted p-values are directly comparable to alpha
sum(p.adjust(p, method = "BH") < 0.05)

For survey data, generate the p-values with the design-aware machinery first — svyby plus svycontrast, per the companion article — then adjust. Correcting naive p-values that were wrong to begin with fixes nothing.

Python — statsmodels:

from statsmodels.stats.multitest import multipletests

reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method="fdr_bh")
# method="bonferroni" for FWER control

Implementing BH directly, since it is six lines and worth understanding:

import numpy as np

def bh_reject(p, alpha=0.05):
    """Return a boolean mask of hypotheses rejected under Benjamini-Hochberg."""
    p = np.asarray(p)
    m = len(p)
    order = np.argsort(p)
    thresholds = alpha * np.arange(1, m + 1) / m
    below = p[order] <= thresholds
    out = np.zeros(m, dtype=bool)
    if below.any():
        k = np.flatnonzero(below).max() + 1   # largest k satisfying the condition
        out[order[:k]] = True                 # reject the k smallest p-values
    return out

Note the last step: BH rejects all hypotheses up to rank $k$, including any whose individual p-value exceeds its own threshold. That step-up behaviour is what makes BH more powerful than a naive per-test comparison, and it is the part people get wrong when they implement it by hand.

SPSS — the base crosstabs procedure applies a Bonferroni option to column proportion tests (CTABLES with COMPARETEST), scoped to comparisons within a row. That is narrower than the whole table, so it under-corrects relative to a table-wide family. If the family you care about is the whole banner, export the p-values and adjust them yourself.

Part 5 — Checklist

1. Count your tests before you run them. Rows × columns × comparisons. If the number is over about 20, uncorrected flags are not reportable as findings.
2. Pre-specify the confirmatory family. Three to eight comparisons the study exists to answer. Correct within that family; Bonferroni is fine at that size.
3. Label everything else exploratory and strip the flags, or mark them explicitly as uncorrected.
4. Prefer intervals to flags wherever the deck format allows it.
5. State the method in the technical note, including the fact that an unflagged cell is not evidence of no difference.
6. Correct the standard errors before you correct the p-values. On weighted data, a multiplicity correction applied to naive p-values is a rounding error on top of a much larger mistake — see the companion article.
7. Be suspicious of a banner with far more flags than $0.05m$. If a 200-test table returns 45 flags, either something real is going on or the standard errors are too small. Both are worth knowing about, and they are distinguishable: rerun with correct design-based SEs and see how many survive.

Reproducing the simulations

Both tables come from the same script. The global-null case:

import numpy as np
from scipy import stats

rng = np.random.default_rng(20260812)
n_tests, n_per_group, reps, p_true = 200, 300, 20000, 0.35
flags = []

for _ in range(reps):
    a = rng.binomial(n_per_group, p_true, n_tests)
    b = rng.binomial(n_per_group, p_true, n_tests)
    pa, pb = a / n_per_group, b / n_per_group
    pooled = (a + b) / (2 * n_per_group)
    se = np.sqrt(pooled * (1 - pooled) * 2 / n_per_group)
    se = np.where(se == 0, np.nan, se)
    p = np.nan_to_num(2 * stats.norm.sf(np.abs((pa - pb) / se)), nan=1.0)
    flags.append((p < 0.05).sum())

print(np.mean(flags), np.percentile(flags, 90), np.max(flags))
# 10.06  14.0  26

For the power comparison, add a fixed 8pp effect to a random 20 of the 200 tests and count detections among those 20 under each correction.

Sources and further reading

• Benjamini, Y. & Hochberg, Y. (1995) "Controlling the false discovery rate: a practical and powerful approach to multiple testing", Journal of the Royal Statistical Society B 57(1) — the original, and unusually readable.
• Benjamini, Y. & Yekutieli, D. (2001) "The control of the false discovery rate in multiple testing under dependency", Annals of Statistics 29(4) — the variant to reach for when tests may be negatively correlated.
• Gelman, A., Hill, J. & Yajima, M. (2012) "Why we (usually) don't have to worry about multiple comparisons", Journal of Research on Educational Effectiveness 5(2) — the strongest argument on the other side, and worth reading precisely because it disagrees.
• Rothman, K. (1990) "No adjustments are needed for multiple comparisons", Epidemiology 1(1) — the classic dissent.