The "±3%" On Your Report Applies To One Number

Near the front of most survey reports sits a sentence like:

Based on a sample of 1,000 adults, the margin of error is ±3.1% at the 95% confidence level.

It is arithmetically correct and almost universally misread. That number is the margin of error for one specific quantity: a proportion of exactly 50%, on the full unweighted base of 1,000, under simple random sampling.

Essentially nothing in the report is that quantity. Readers apply it to everything anyway — subgroup figures, low-incidence items, differences between groups — and it is wrong for all of them, sometimes by a factor of two or more.

Part 1 — What the number actually depends on

$$\text{MoE}=z_{0.975}\sqrt{\frac{\text{deff}\cdot p(1-p)}{n}}$$

Three things move it, and the headline figure fixes all three at their most flattering values.

The proportion itself

$p(1-p)$ is maximised at $p=0.5$. Quoting the margin of error at 50% is quoting the widest interval on the base — which sounds conservative and is, for that base. Everything else on the same base has a narrower interval:

So on this dimension alone the headline is genuinely conservative. If it were the only factor, over-quoting would be harmless. It isn't.

(And at low incidence the symmetric interval is the wrong shape entirely — see confidence intervals for small bases. The table above uses the Wald form to make the comparison like-for-like; the honest interval near 2% is asymmetric.)

The base

Almost nothing in a report is quoted on the full sample. The interesting findings are subgroups, and MoE scales with $1/\sqrt{n}$:

A finding about 18–24 year-olds, on a base of 120, has a margin of error nearly three times the one printed at the front of the report.

The design effect

The headline assumes simple random sampling, which almost no real survey is. Weighting and clustering both inflate the variance — see clustering design effects for where deff comes from:

Part 2 — Compounding

Individually these look like modest adjustments. They multiply.

Take an ordinary finding: 8% of a 120-respondent subgroup, from a weighted clustered design with deff 1.8.

$$\text{MoE}=1.96\sqrt{\frac{1.8\times 0.08\times 0.92}{120}}=6.51\text{pp}$$

Against a quoted headline of 3.10pp, that is 2.1× wider. The reader has been told ±3.1 and the true figure is ±6.5 — meaning an 8% result is consistent with anything from about 1.5% to 14.5%.

The direction of the error matters. Base and design effect both push the interval wider, and both apply to exactly the findings people care about most: subgroups, and low-incidence behaviours within subgroups. The one factor that pushes narrower — low $p$ — is the weakest of the three. The headline is systematically optimistic precisely where readers lean on it hardest.

And it does not apply to differences at all

The most common use of the number is the one it was never computed for. Readers see two subgroups at 34% and 41%, note that the 7-point gap "exceeds the ±3% margin of error", and conclude the difference is real.

The margin of error for a single estimate is not the margin of error for a difference. For two independent groups:

$${\text{MoE}}_{\text{diff}}=z\sqrt{\text{Var}({\hat{p}}_1)+\text{Var}({\hat{p}}_2)}$$

which for two equal subgroups of size $n$ is roughly $\sqrt{2}$ times the individual MoE — before accounting for the fact that subgroup bases are smaller than the total. On two subgroups of 250 each, the difference MoE at $p\approx 0.4$ is about 8.6pp, so a 7-point gap is not significant. The reader using the headline concluded the opposite.

Part 3 — What to publish instead

Report intervals per figure, not a margin once. Any tabulation tool can attach a confidence interval to every cell. If the deliverable is a deck rather than a table, put intervals on the numbers that carry the argument.

If you must quote one number, quote it honestly. Something like:

Margins of error vary by base size, by the percentage being estimated, and by the survey design. On the full sample (n=1,000, design effect 1.8) the margin of error is ±4.2 percentage points for an estimate near 50%. Subgroup figures have substantially wider margins — for a base of 120, around ±9 points. Differences between subgroups require a wider margin still and should not be assessed against these figures.

Longer, and it prevents the specific errors readers actually make.

Put the effective base in the table. If the base is 400 and the design effect is 1.8, the effective base is 222. Showing "n=400 (effective 222)" tells a reader more about precision than a margin quoted once at the front.

Set a minimum reporting base and enforce it. Below roughly 50–100, do not publish a percentage. The margin of error exceeds the range of plausible findings, and a number with a ±14 point interval invites a conclusion the data cannot support.

Part 4 — Implementation

import numpy as np
from scipy import stats

Z95 = stats.norm.ppf(0.975)

def moe(p, n, deff=1.0, conf=0.95):
    """Margin of error in percentage points for a single proportion."""
    z = stats.norm.ppf(0.5 + conf / 2)
    return z * np.sqrt(deff * p * (1 - p) / n) * 100

def moe_difference(p1, n1, p2, n2, deff=1.0, conf=0.95):
    """Margin of error for a difference between two independent groups."""
    z = stats.norm.ppf(0.5 + conf / 2)
    v = deff * (p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
    return z * np.sqrt(v) * 100

def effective_base(n, deff):
    """The base that matters for precision."""
    return n / deff

# The headline, and what it hides
print(moe(0.50, 1000, 1.0))              # 3.10  <- what gets printed
print(moe(0.08,  120, 1.8))              # 6.51  <- a real subgroup finding
print(moe_difference(0.34, 250, 0.41, 250, 1.0))   # 8.57 <- a 7pp gap is NOT significant

For weighted survey data, get the design effect from the data rather than assuming one:

library(survey)
des <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = dat)

est <- svymean(~outcome, des, deff = TRUE)
est                      # estimate, SE and deff
confint(est)             # the interval that should be reported
deff(est)

# Per-subgroup, which is where the headline breaks down
svyby(~outcome, ~agegroup, des, svymean, deff = TRUE, vartype = c("se", "ci"))

Part 5 — Checklist

1. Do not quote a single margin of error as though it governs the report. If your template does, change the template.
2. State the assumptions with the number — the proportion, the base, and the design effect it was computed at.
3. Attach intervals to individual figures, especially subgroup and low-incidence ones.
4. Never let a headline MoE be used to test a difference. It is the wrong quantity and it is too small.
5. Publish effective bases alongside nominal ones wherever the design effect is above about 1.2.
6. Enforce a minimum reporting base, and report counts rather than percentages below it.
7. Use an interval method that behaves at the boundaries — Wilson, not Wald — since low-incidence subgroup items are exactly where symmetric intervals fail. See the small bases article.

Reproducing the tables

Every figure here is analytic — no simulation required:

import numpy as np
from scipy import stats

z = stats.norm.ppf(0.975)
moe = lambda p, n, deff=1.0: z * np.sqrt(deff * p * (1 - p) / n) * 100

headline = moe(0.50, 1000, 1.0)                  # 3.10

for p in (0.50, 0.30, 0.20, 0.10, 0.05, 0.02):
    print(p, round(moe(p, 1000), 2), round(moe(p, 1000)/headline, 2))

for n in (1000, 500, 300, 200, 100, 50):
    print(n, round(moe(0.5, n), 2), round(moe(0.5, n)/headline, 2))

print(round(moe(0.08, 120, 1.8), 2))             # 6.51 = 2.1x headline

Sources and further reading

• AAPOR (2023) Margin of Sampling Error and Credibility Interval — the professional association's own guidance, including the point that a single MoE does not apply to subgroups or differences.
• Kish, L. (1965) Survey Sampling, Chapter 8 — design effects and their consequences for reported precision.
• Gelman, A. & Hill, J. (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models, Chapter 2 — on why interval estimates beat point estimates plus a footnote.