5,000 Interviews Across 120 Sampling Points Is Not 5,000 Interviews

Clustered sampling exists for cost reasons. Sending interviewers to 120 postcode sectors and completing 40 interviews in each is far cheaper than 5,000 addresses scattered at random across the country.

The cost is paid in precision, and it is paid at a rate most people underestimate. People living in the same neighbourhood resemble each other — on income, tenure, ethnicity, political attitudes, broadband availability, almost anything geographic. Each additional interview within a cluster therefore tells you less than a fresh interview elsewhere would.

Part 1 — The formula

For equal clusters of size $m$ and intra-cluster correlation $\rho$, the design effect is:

$$\text{deff}=1+(m-1)\rho$$

and the effective sample size is $n_{\text{eff}}=n/\text{deff}$.

The multiplier on $\rho$ is $m-1$, and $m$ is often large. That is the entire problem: a correlation small enough to be dismissed gets multiplied by the cluster size.

At $m=41$, an intra-cluster correlation of 0.012 — which sounds like nothing — gives $\text{deff}=1+40\times 0.012=1.48$. You have lost a third of your sample.

The ICC you need is the one for your variable

This is where the formula gets misapplied. $\rho$ is the intra-cluster correlation of the variable being estimated, not a general property of the design. It differs by variable, sometimes by an order of magnitude:

• Strongly geographic measures — tenure, deprivation, broadband speed, local service satisfaction — have high ICCs.
• Measures with little geographic structure — most attitudes, personality-adjacent items, brand preferences in a national market — have low ones.

A single design effect applied across a whole survey is an approximation, and the direction of its error varies by question.

There is a related trap worth flagging because it caught me while building the simulation below. If you generate clustered binary data by thresholding a correlated latent variable, the correlation of the resulting 0/1 variable is much lower than the latent correlation — thresholding attenuates it. A latent correlation of 0.10 produced a binary ICC of 0.063 in the runs below. So when someone quotes an ICC, it matters whether it is the ICC of the response or of some underlying construct. Use the ICC of the thing you are actually estimating.

Part 2 — Simulated, against the formula

Clustered binary data, outcome around 40%, 4,000 replications per row. I measured the binary ICC directly with the ANOVA estimator and compared the realised design effect against the formula evaluated at that measured ICC.

The formula tracks the realised design effect closely throughout, which is the point of showing it: this is not an obscure correction, it is a well-behaved and predictable one.

The row to sit with is the third. A measured intra-cluster correlation of 0.012 — a number most people would round to zero — turns 5,000 interviews into an effective 3,222. You paid for 5,000 and, for inferential purposes, you have roughly 3,200. The margin of error is 1.69pp rather than the 1.36pp that a simple-random-sampling calculation reports.

Note also how the penalty depends on cluster size rather than cluster count. At the same ICC of about 0.012, the 41-per-cluster design carries deff 1.55 while the two 20-per-cluster designs carry about 1.23. Fewer interviews per point, more points, same total — materially better precision. That is the design lever, and it is a cost conversation, not a statistical one.

Part 3 — Degrees of freedom, the second penalty

The variance inflation is the visible cost. The less visible one is the collapse in degrees of freedom.

For a stratified clustered design:

$$\text{df}=(\text{number of PSUs})-(\text{number of strata})$$

A survey of 5,000 people across 120 sampling points in 30 strata has 90 degrees of freedom, not 4,999. This matters in two ways:

1. Confidence intervals should use $t_{90}$ rather than the normal. The difference is small at 90 df (2.00 versus 1.96) but grows quickly for small subgroups confined to few clusters.
2. Any test relying on asymptotics is on thinner ice than the raw sample size suggests.

Software that ignores the design and uses $n-1$ produces intervals that are too narrow twice over — once through the variance, once through the reference distribution.

Part 4 — Implementation

Do not compute this by hand. Declare the design and let the software linearise — the same machinery as in the sampling weights article, where the ultimate-cluster variance formula is given in Section 2.2.

R:

library(survey)

des <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w,
                 data = dat, nest = TRUE)

# deff = TRUE reports the design effect alongside the estimate
svymean(~outcome, des, deff = TRUE)

degf(des)                              # design degrees of freedom, not n-1

# ICC for a specific variable, if you want it for planning
library(ICC)
ICCbare(factor(psu), outcome, data = dat)

Stata:

svyset psu [pweight = w], strata(stratum)
svy: mean outcome
estat effects              // deff and deft
estat size                 // effective sample size

Python — measuring the ICC yourself, which is worth doing for design planning:

import numpy as np

def anova_icc(y_by_cluster):
    """ANOVA estimator of the ICC. y_by_cluster: (clusters x cluster_size)."""
    k, m = y_by_cluster.shape
    grand = y_by_cluster.mean()
    cl_means = y_by_cluster.mean(axis=1)
    msb = m * ((cl_means - grand) ** 2).sum() / (k - 1)
    msw = ((y_by_cluster - cl_means[:, None]) ** 2).sum() / (k * (m - 1))
    return (msb - msw) / (msb + (m - 1) * msw)

def deff_from_icc(icc, cluster_size):
    return 1 + (cluster_size - 1) * icc

def plan_sample(target_moe_pp, p, icc, cluster_size, z=1.96):
    """Interviews needed to hit a target MoE under clustering."""
    deff = deff_from_icc(icc, cluster_size)
    n_srs = (z ** 2) * p * (1 - p) / (target_moe_pp / 100) ** 2
    return {"n_srs": int(np.ceil(n_srs)), "deff": deff,
            "n_clustered": int(np.ceil(n_srs * deff))}

# To hit +/-1.5pp on a 40% measure with ICC 0.012 and 40 per point:
print(plan_sample(1.5, 0.40, 0.012, 40))
# {'n_srs': 4098, 'deff': 1.468, 'n_clustered': 6016}

That last calculation is the one to run before fieldwork. Budgeting 4,100 interviews for a ±1.5pp target and then clustering them 40-to-a-point delivers about ±1.8pp instead.

Part 5 — Checklist

1. Declare the design, always. svydesign(ids = ~psu, strata = ~stratum, ...) — weights alone are not the design.
2. Plan with the design effect, not without it. Sample size calculations that assume simple random sampling under-deliver on precision by exactly the deff.
3. Prefer more clusters with fewer interviews each. The penalty scales with cluster size, not cluster count. This is the main design lever available.
4. Use the ICC of the variable you care about. A design effect computed on tenure will mislead about brand attitudes, and vice versa.
5. Report the effective sample size next to the nominal one wherever deff exceeds about 1.2.
6. Use design degrees of freedom — PSUs minus strata — not $n-1$.
7. Watch subgroups confined to few clusters. A subgroup living in 12 sampling points has roughly 12 clusters' worth of information regardless of how many interviews it contains.
8. Do not subset the data frame to analyse a subgroup — subset the design object, or you will discard strata and PSUs that legitimately contribute.

Reproducing the simulation

import numpy as np
from scipy import stats
rng = np.random.default_rng(2718)

def run(n_total, k, rho_latent, reps=4000):
    """Clustered binary data; returns measured ICC and realised deff."""
    m = n_total // k
    ests, iccs = [], []
    for _ in range(reps):
        ce = rng.normal(0, np.sqrt(rho_latent), k)             # cluster effect
        iv = rng.normal(0, np.sqrt(1 - rho_latent), (k, m))    # individual
        y = ((ce[:, None] + iv) < stats.norm.ppf(0.40)).astype(float)
        ests.append(y.mean())
        iccs.append(anova_icc(y))
    ests = np.array(ests)
    p_bar = ests.mean()
    realised_deff = ests.var(ddof=1) / (p_bar * (1 - p_bar) / n_total)
    return float(np.mean(iccs)), float(realised_deff)

icc, deff = run(5000, 120, 0.02)
print(icc, deff, 5000 / deff)      # 0.01237  1.552  3222

Note the latent correlation passed in is 0.02 but the measured binary ICC is 0.0124 — the attenuation described in Part 1.

Sources and further reading

• Kish, L. (1965) Survey Sampling, Chapters 5–6 — cluster sampling and the origin of the design effect.
• Lumley, T. (2010) Complex Surveys: A Guide to Analysis Using R, Chapters 2–4 — declaring designs and what the software does with them.
• Gabler, S., Häder, S. & Lahiri, P. (1999) "A model based justification of Kish's formula for design effects for weighting and clustering", Survey Methodology 25(1) — how the weighting and clustering components combine.
• Park, I. & Lee, H. (2004) "Design effects for the weighted mean and total estimators under complex survey sampling", Survey Methodology 30(2) — the joint treatment when both are present.