Your Tracker Is Throwing Away Half Its Power

Trackers re-interview. Some designs re-contact the same panel every wave; most refresh part of the sample and carry the rest over. Either way, wave 2 is not a fresh independent sample of wave 1.

Almost every wave-on-wave significance test treats it as one:

$$\text{SE}({\hat{p}}_2-{\hat{p}}_1)=\sqrt{\frac{{\hat{p}}_1(1-{\hat{p}}_1)}{n_1}+\frac{{\hat{p}}_2(1-{\hat{p}}_2)}{n_2}}$$

That formula assumes $\text{Cov}({\hat{p}}_1,{\hat{p}}_2)=0$. When respondents are shared between waves and individual responses are stable over time — which is the entire premise of tracking — the covariance is positive and substantial, and the formula overstates the standard error.

This is the rare methodological error that is conservative. You will not publish false movements because of it. You will fail to publish real ones, which in a tracker is the more expensive failure: the whole point of the instrument is detecting change.

Part 1 — The correct variance

For a difference of two means or proportions:

$$\text{Var}({\hat{p}}_2-{\hat{p}}_1)=\text{Var}({\hat{p}}_1)+\text{Var}({\hat{p}}_2)-2\text{Cov}({\hat{p}}_1,{\hat{p}}_2)$$

Only respondents present in both waves contribute to the covariance — fresh respondents are independent of everything in the previous wave. With $n$ respondents per wave and $n_{\text{over}}$ carried over, and writing $c$ for the per-respondent covariance $\text{Cov}(y_{i1},y_{i2})$ among those retained:

$$\text{Cov}({\hat{p}}_1,{\hat{p}}_2)=\frac{n_{\text{over}}c}{n^2}$$

so:

$$\text{Var}({\hat{p}}_2-{\hat{p}}_1)=\frac{{\hat{p}}_1(1-{\hat{p}}_1)}{n}+\frac{{\hat{p}}_2(1-{\hat{p}}_2)}{n}-\frac{2n_{\text{over}}c}{n^2}$$

The scaling matters and is easy to get wrong. The covariance term carries a factor $n_{\text{over}}/n^2$, not $1/n$ — the correction is proportional to the share of the sample that overlaps. (I got this wrong on the first pass of the simulation below, which produced 8.6% type I error at 50% overlap and agreed with the correct version only at 100% overlap. The failure is invisible in a full-panel design and shows up immediately in a partial one.)

$c$ is estimated directly from the matched respondents:

$$\hat{c}=\frac{1}{n_{\text{over}}-1}\sum _{i\in \text{matched}}\left(y_{i1}-{\bar{y}}_1^{\text{m}}\right)\left(y_{i2}-{\bar{y}}_2^{\text{m}}\right)$$

For binary data this is just the sample covariance of the two 0/1 vectors over the matched panel, and it can be read off a 2×2 turnover table: $\hat{c}=p_{11}-p_{1\cdot }{p}_{\cdot 1}$, where $p_{11}$ is the proportion answering yes in both waves.

Part 2 — What it costs, simulated

Two waves of $n=800$, true proportion 40% in both — so the null is true and any rejection is a false positive. Respondent-level stability is controlled by a latent correlation $\rho$. 8,000 replications per cell. Nominal type I error 5%.

With no overlap the two tests coincide exactly, as they must. As overlap and stability rise, the independent test becomes progressively more conservative: at full panel and $\rho =0.6$ it rejects 1.11% of the time against a nominal 5%. The overlap-aware test holds ~5% throughout.

That conservatism is not free. Here is the same design with a real 3, 4 or 5 point movement, 50% overlap, $\rho =0.6$:

At a 4-point movement — a size a client would certainly want to know about — the conventional test finds it 36% of the time and the correct test 45%. Roughly a quarter of the detectable movements are being discarded by using the wrong formula, on data already collected and already paid for.

For a tracker running twelve waves a year across thirty measures, that is a lot of unreported real change.

Part 3 — Practical complications

You need respondent identifiers linked across waves. If the deliverable is wave-level tables with no matching key, the covariance is not recoverable and the independent-samples test is the only option available. This is worth specifying at the design stage rather than discovering at analysis.

Partial overlap needs care with the matching. Compute $\hat{c}$ only over genuinely matched respondents, and use the actual matched count for $n_{\text{over}}$ — not the nominal recontact target, which is always optimistic after attrition.

Weights change between waves. Each wave is calibrated separately, so a retained respondent has two different weights. The covariance must be computed on the weighted contributions, which brings this back to the linearisation machinery in the companion article: write the wave-on-wave difference as a contrast, form the influence values for each wave, and take the variance of the summed weighted influence values across the design. Section 2.7 there gives the pattern; the only change is that the two terms come from different waves rather than different subgroups.

Attrition is not random. Respondents who drop out differ systematically from those who stay, so the matched panel is not a random subsample. The covariance estimate is fine — it is a property of the people you matched — but the panel's level may drift from the population. This is a separate problem from the variance calculation and needs its own treatment, usually attrition weighting.

Conservative is not the same as safe. It is tempting to treat the independent-samples test as the cautious choice. In a tracker it is not: the cost of a missed real movement is a client acting on the belief that nothing changed. That is a decision error with the same consequences as a false positive, and it is arguably more likely to matter.

Part 4 — Implementation

import numpy as np
from scipy import stats

def wave_on_wave(y1, y2, matched_idx1, matched_idx2, n1=None, n2=None):
    """Test a wave-on-wave difference accounting for respondent overlap.

    y1, y2          : 0/1 (or numeric) responses for each full wave
    matched_idx1/2  : index arrays selecting the SAME respondents in each wave,
                      in the same order
    """
    y1, y2 = np.asarray(y1, float), np.asarray(y2, float)
    n1 = n1 or len(y1)
    n2 = n2 or len(y2)
    p1, p2 = y1.mean(), y2.mean()
    diff = p2 - p1

    var_indep = p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2

    n_over = len(matched_idx1)
    if n_over > 1:
        c = np.cov(y1[matched_idx1], y2[matched_idx2])[0, 1]  # per respondent
        var = var_indep - 2 * c * n_over / (n1 * n2)
        var = max(var, 1e-12)
    else:
        var = var_indep

    se = np.sqrt(var)
    z = diff / se
    return {
        "diff": diff,
        "se": se,
        "se_if_treated_independent": np.sqrt(var_indep),
        "p": 2 * stats.norm.sf(abs(z)),
        "n_overlap": n_over,
    }

R, on a long-format dataset with a respondent key:

library(survey)

# Matched panel: one row per respondent, one column per wave
wide <- reshape(long[long$id %in% panel_ids, ],
                idvar = "id", timevar = "wave", direction = "wide")

des <- svydesign(ids = ~1, weights = ~w.2, data = wide)

# svycontrast handles the covariance between the two wave estimates
est <- svymean(~ outcome.1 + outcome.2, des)
svycontrast(est, quote(outcome.2 - outcome.1))

# Equivalently, for a fully matched panel:
svyttest(I(outcome.2 - outcome.1) ~ 1, des)

svymean on both waves at once returns the full covariance matrix, and svycontrast uses it — which is exactly the point. Computing the two means in separate calls throws the covariance away and puts you back where you started.

Part 5 — Checklist

1. Establish whether waves share respondents, and get the matching key into the analysis dataset. Without it, none of this is available.
2. Use the contrast, not the difference of two independent estimates. In survey, that means one svymean over both waves plus svycontrast — not two separate calls.
3. Get the covariance scaling right. The term is $2n_{\text{over}}c/n^2$. Check it by confirming the correction vanishes at zero overlap and that type I error holds at ~5% in a null simulation of your own design.
4. Use the achieved matched count, not the recontact target.
5. Handle the weights properly. Two waves means two weight variables; linearise the contrast rather than differencing weighted percentages.
6. State the method. "Wave-on-wave tests account for the panel overlap between waves (matched n = 412 of 800)" tells a reader something real about the precision of the tracker.
7. Treat attrition separately. The covariance adjustment fixes the variance; it does nothing about differential dropout.

Reproducing the simulation

import numpy as np
from scipy import stats

rng = np.random.default_rng(20260812)

def null_rejection_rate(overlap, rho, n=800, reps=8000):
    """Type I error of both tests when the truth is 'no change'."""
    n_over, n_fresh = int(n * overlap), n - int(n * overlap)
    rej_i = rej_c = 0
    for _ in range(reps):
        l1o = rng.normal(0, 1, n_over)
        l2o = rho * l1o + np.sqrt(1 - rho**2) * rng.normal(0, 1, n_over)
        l1f, l2f = rng.normal(0, 1, n_fresh), rng.normal(0, 1, n_fresh)
        thr = stats.norm.ppf(0.40)                    # 40% in both waves
        w1 = np.concatenate([l1o < thr, l1f < thr]).astype(float)
        w2 = np.concatenate([l2o < thr, l2f < thr]).astype(float)

        p1, p2 = w1.mean(), w2.mean()
        v_indep = p1*(1-p1)/n + p2*(1-p2)/n
        if n_over > 1:
            c = np.cov(w1[:n_over], w2[:n_over])[0, 1]
            v_corr = max(v_indep - 2*c*n_over/n**2, 1e-12)
        else:
            v_corr = v_indep

        if abs((p2-p1)/np.sqrt(v_indep)) > 1.96: rej_i += 1
        if abs((p2-p1)/np.sqrt(v_corr))  > 1.96: rej_c += 1
    return rej_i/reps, rej_c/reps

print(null_rejection_rate(1.0, 0.6))   # (0.0111, 0.0486)
print(null_rejection_rate(0.5, 0.6))   # (0.0305, 0.0521)

Sources and further reading

• Kish, L. (1965) Survey Sampling, Chapter 12 — panel and repeated-survey designs, including the variance of change.
• Lumley, T. (2010) Complex Surveys: A Guide to Analysis Using R, Chapter 8 — svycontrast and estimating differences with their covariance.
• Duncan, G.J. & Kalton, G. (1987) "Issues of design and analysis of surveys across time", International Statistical Review 55(1) — the standard treatment of the trade-offs between panel, rotating panel and repeated cross-section designs.
• Binder, D.A. (1983) "On the variances of asymptotically normal estimators from complex surveys", International Statistical Review 51(3) — the linearisation result underlying the weighted version of this contrast.