Your Response Rate Says Almost Nothing About Your Bias

Response rate is the number everyone asks for. It goes in the technical annex, it gets compared against industry benchmarks, and a low one is treated as prima facie evidence that the survey is unreliable.

It is a reasonable proxy for effort. It is a poor proxy for bias, and the relationship between them is well understood and rarely acted on.

Part 1 — The identity

Let ${\rho }_i$ be respondent $i$'s response propensity — the probability that this person, if sampled, takes part. Let $y_i$ be their answer. The mean over respondents estimates:

$$\mathbb{E}\left[{\bar{y}}_r\right]\approx \frac{\mathbb{E}[\rho y]}{\mathbb{E}[\rho ]}$$

so the bias relative to the population mean $\bar{Y}=\mathbb{E}[y]$ is:

$$\text{Bias}=\frac{\mathbb{E}[\rho y]}{\mathbb{E}[\rho ]}-\mathbb{E}[y]=\frac{\mathbb{E}[\rho y]-\mathbb{E}[\rho ]\mathbb{E}[y]}{\mathbb{E}[\rho ]}$$

which is:

$$\boxed{\text{Bias}\left({\bar{y}}_r\right)=\frac{\mathrm{Cov}(\rho ,y)}{\bar{\rho }}}$$

This is the standard result (Bethlehem; Groves). Read it carefully, because it says something specific:

• The numerator is the covariance between propensity and outcome. If people who respond differ systematically on the thing you are measuring, there is bias.
• The denominator $\bar{\rho }$ is the response rate.

So the response rate enters only as a divisor. Raising it shrinks whatever bias exists, but a high response rate over a large covariance is still a large bias, and a low response rate over zero covariance is still zero bias.

The response rate cannot tell you about the numerator, and the numerator is where the bias comes from.

Part 2 — Confirming it

I simulated a population of 60,000 with an outcome driven by a latent factor, and a response propensity driven by the same factor with adjustable strength. The response rate was tuned by solving for the propensity intercept. 500 replications per cell.

Every row compares the bias predicted by the identity with the bias actually realised:

The identity predicts the realised bias to two decimal places in all sixteen cells. That is a strong check on the reasoning rather than a discovery — but it means the conclusions below rest on arithmetic, not on intuition.

The comparison that matters is across rows rather than down them:

A 70% response rate with a strong propensity–outcome link carries a 5.82 point bias. A 15% response rate with no link carries 0.01 points.

The 70% survey is worse — by a factor of several hundred — and the response rate reports it as the better one.

Note also the pattern down each block: at a fixed link strength, halving the response rate roughly doubles the bias. So the rate does matter. It is a multiplier on a quantity you have not measured, which is a different thing from being a measure of quality.

Part 3 — What to do instead

The identity is not just diagnostic; it tells you where to spend effort.

Measure the numerator, not the denominator

$\mathrm{Cov}(\rho ,y)$ is estimable whenever you know something about non-respondents. Sources, in rough order of availability:

• Sample frame variables. Anything you sampled on — region, age band, firmographics, list source — is known for respondents and non-respondents alike. Compare the two groups on those.
• Paradata. Number of contact attempts, mode, day of week, time to respond. Respondents who took six attempts resemble non-respondents more than the ones who answered immediately. Comparing early to late responders is a genuinely useful proxy for the propensity gradient, and it costs nothing but a field to log.
• External benchmarks. Population totals from official statistics for anything you also measured.
• A non-response follow-up. Expensive and decisive: a short questionnaire pushed hard at a subsample of non-respondents.

Understand that weighting only helps through the same identity

Weighting on auxiliary variables $x$ removes the part of the covariance that runs through $x$. It does nothing about the part that doesn't. Formally, weighting reduces the bias to a residual covariance conditional on $x$:

$${\text{Bias}}_{\text{weighted}}\approx \frac{\mathbb{E}\left[\mathrm{Cov}(\rho ,y\mid x)\right]}{\bar{\rho }}$$

So the value of a weighting scheme depends entirely on whether $x$ predicts both response and the outcome. Weighting on age and gender is close to useless if response propensity is driven by interest in the topic and interest in the topic drives the answers. This is the single most common failure in practice: the weighting variables are the ones that were available, not the ones that were relevant.

It is also the connection back to weight trimming. Weights help exactly when $\mathrm{corr}(w,y)$ is non-zero — which is the same statement as "the weighting variables predict the outcome". If that correlation is zero, your weights are not fixing non-response bias, and trimming them costs nothing because they were doing nothing.

Report something more useful than a rate

Three numbers that actually inform a reader:

1. Response rate, defined to a standard (AAPOR RR3 or similar), because the convention exists and comparisons need it.
2. Respondent vs frame comparison on every auxiliary variable you have, pre-weighting. This is the visible part of the covariance.
3. What weighting changed. If the weighted and unweighted estimates differ by 4 points, the weights found real non-response bias on those variables — which is evidence both that bias existed and that you removed some of it. If they differ by 0.1 points, the weights did nothing, and the honest reading is that either there was no bias on those dimensions or your variables cannot see it.

That third number is the most informative and the least often reported.

Part 4 — Implementation

import numpy as np

def nonresponse_bias_estimate(propensity, y):
    """Groves/Bethlehem bias: Cov(rho, y) / mean(rho).

    Needs propensity for the FULL sample, so it is used with modelled
    propensities or in simulation - not directly on respondent-only data.
    """
    rho, y = np.asarray(propensity, float), np.asarray(y, float)
    return float(np.mean(rho * y) - rho.mean() * y.mean()) / rho.mean()


def early_vs_late_check(y, contact_attempts, quantile=0.5):
    """Cheap proxy for the propensity gradient using paradata.

    A gradient across contact effort is evidence that Cov(rho, y) != 0,
    because late responders sit closer to non-respondents on propensity.
    """
    y, a = np.asarray(y, float), np.asarray(contact_attempts, float)
    cut = np.quantile(a, quantile)
    early, late = y[a <= cut], y[a > cut]
    return {
        "early_mean": float(early.mean()),
        "late_mean": float(late.mean()),
        "gradient_pp": float(late.mean() - early.mean()) * 100,
        "n_early": len(early), "n_late": len(late),
    }

R — the two comparisons worth running on every project:

library(survey)

# 1. Do respondents differ from the frame on what you know?
frame_compare <- function(resp, frame, vars) {
  sapply(vars, function(v)
    c(respondents = mean(resp[[v]]), frame = mean(frame[[v]]),
      gap = mean(resp[[v]]) - mean(frame[[v]])))
}

# 2. What did weighting actually change?
des_unw <- svydesign(ids = ~1, weights = ~1, data = dat)
des_w   <- svydesign(ids = ~psu, strata = ~stratum, weights = ~w, data = dat)

cbind(unweighted = coef(svymean(~outcome, des_unw)),
      weighted   = coef(svymean(~outcome, des_w)))

Part 5 — Checklist

1. Stop treating the response rate as a quality score. It is the denominator of the bias, not the bias.
2. Compare respondents with the frame on every variable you have for both. This is the cheapest evidence about the numerator that exists.
3. Log paradata and use it. Contact attempts and response timing give you a propensity gradient for free. An early-versus-late split on a key outcome is a five-minute check with real diagnostic value.
4. Choose weighting variables that predict the outcome, not the ones that happen to be on the frame. A weighting scheme built from demographics alone will not fix a topic-interest bias.
5. Report the weighted-versus-unweighted difference. It is direct evidence of how much bias the weights found.
6. Do not accept "response rate was 62%" as evidence of representativeness — from a supplier or from yourself. Ask what is known about the non-respondents.
7. Do not dismiss a low-response survey on the rate alone. Ask the same question in reverse: is there reason to think propensity relates to the outcome here?

Reproducing the simulation

import numpy as np
rng = np.random.default_rng(90210)

def solve_intercept(slope, target_rate, n=200000):
    """Find the propensity intercept that hits a target response rate."""
    lo, hi = -12.0, 12.0
    x = rng.normal(0, 1, n)
    for _ in range(60):
        mid = (lo + hi) / 2
        if (1/(1+np.exp(-(mid + slope*x)))).mean() < target_rate: lo = mid
        else: hi = mid
    return (lo + hi) / 2

n_pop, slope, target = 60000, 1.5, 0.70
b = solve_intercept(slope, target)

x = rng.normal(0, 1, n_pop)
y = (rng.uniform(size=n_pop) < 1/(1+np.exp(-(0.2 + 0.8*x)))).astype(float)
rho = 1/(1+np.exp(-(b + slope*x)))
resp = rng.uniform(size=n_pop) < rho

predicted = (np.mean(rho*y) - rho.mean()*y.mean()) / rho.mean()
realised  = y[resp].mean() - y.mean()
print(predicted*100, realised*100)     # 5.82  5.82

Sources and further reading

• Groves, R.M. (2006) "Nonresponse rates and nonresponse bias in household surveys", Public Opinion Quarterly 70(5) — the meta-analysis showing empirically that response rate and bias are weakly related. The single most useful citation on this topic.
• Bethlehem, J. (2002) "Weighting nonresponse adjustments based on auxiliary information", in Survey Nonresponse — the derivation of the bias identity used here.
• Groves, R.M. & Peytcheva, E. (2008) "The impact of nonresponse rates on nonresponse bias", Public Opinion Quarterly 72(2) — 959 bias estimates across 59 studies; the correlation with response rate is weak.
• Peytchev, A. (2013) "Consequences of survey nonresponse", Annals of the American Academy of Political and Social Science 645(1) — practical treatment of what to do about it.