How to Validate an LLM That Codes Your Open Ends

Coding open-ended responses is the most tedious job in survey research, and the one large language models are most obviously suited to. The work is now routine: send the verbatim and the codeframe, get a category back.

The question that decides whether the output is usable is not "is the model good?" It is "how would I know?" — and the answer involves two measurement problems that are easy to get wrong in opposite directions.

The first is that raw agreement with a human coder is a misleading statistic, and it flatters the model exactly when the codeframe is most unbalanced. The second is that the validation exercise is itself a sample, and most validation samples are far too small to support the conclusion drawn from them.

Part 1 — Raw agreement is not a measure of quality

Suppose your machine coder agrees with a human on 92.7% of items. Good?

It depends entirely on the shape of the codeframe, and not at all on anything you would call model quality. I simulated two independent coders — one "human", one "machine" — each with the same per-item accuracy, coding into a four-category frame, and varied only how balanced those categories are. 2,000 items, 400 replications.

Read across the top four rows. Raw agreement is 92.6–92.7% in every one of them. Kappa runs from 0.902 down to 0.647. Identical coders, identical accuracy, identical agreement — and a chance-corrected statistic that says "excellent" in one case and "moderate" in another.

The reason is straightforward: when one category holds 92% of the mass, two coders who both guess randomly still agree most of the time. Chance agreement is high, so observed agreement has to clear a much higher bar to mean anything.

This matters practically because real codeframes are unbalanced. A brand-reasons question where 60% of answers are "price" and the remaining 40% spread over eleven codes is the normal case, not the exception. Reporting raw agreement on that frame overstates performance, and the overstatement is largest for the frames where the rare codes are the interesting ones.

Kappa or alpha?

For two coders and nominal categories with no missing data they agree to three decimal places, as the table shows — the small divergence in the last row is Monte Carlo noise. Use either.

Krippendorff's α is the more general instrument: it handles more than two coders, missing values, and ordinal or interval codeframes with appropriate difference functions. If your validation involves three coders, or partial coverage, or an ordered frame where confusing "very positive" with "positive" should count less than confusing it with "negative", α is the one that extends. Cohen's κ is fine for the standard two-coder nominal case and is more widely recognised.

The benchmark is human–human, not perfection

The most useful comparison is not the model against ground truth — for genuinely ambiguous verbatims there is no ground truth. It is the model against a human, benchmarked against how well two humans agree with each other.

If two experienced coders achieve κ = 0.75 on your frame, a model achieving κ = 0.72 against one of them is performing at close to human level, and the residual disagreement is mostly irreducible ambiguity in the codeframe. Judging that same model against an implicit standard of 0.95 would reject a coder better than the humans you are comparing it with.

Always double-code a subsample by hand. Without the human–human figure you have no scale to read the machine–human figure against.

Part 2 — How many items do you need to check?

Validation is a sampling exercise, and κ estimated on a handful of items is very imprecise. Same setup — mildly skewed four-category frame, coder accuracy 0.88, true κ near 0.737 — varying only how many items are validated. 3,000 replications each.

At the 50-item validation that people actually run, the 95% interval spans 0.57 to 0.88. That range covers "moderate, needs work" and "excellent, ship it" simultaneously. The exercise cannot distinguish between them, and a point estimate of 0.73 from 50 items carries essentially no information about which side of a 0.70 threshold the coder really sits on.

300–400 items is the point where the interval becomes narrow enough to act on (±0.06 or so). Below 200, you are not validating; you are generating a number.

That is not an expensive requirement. Four hundred items is a few hours of one coder's time, against a codeframe that will be applied to tens of thousands of verbatims.

Part 3 — A validation protocol

1. Draw a random validation sample of at least 300 items. Random, not the first 300 and not a convenience slice — early responses differ systematically from late ones.

2. Over-sample the rare codes deliberately, and weight back. With a dominant category, a simple random sample of 400 might contain four instances of the code you most care about. Stratify by the machine's predicted code, sample more heavily from the rare strata, and weight when computing overall agreement. This gives usable precision on the rare codes without inflating the total.

3. Double-code a subsample by two humans to establish the human–human ceiling. 100–150 items is usually enough for a benchmark.

4. Have the human coder work blind. If they can see the machine's suggestion, they will anchor on it, agreement will rise, and the statistic becomes meaningless. This is the single easiest way to invalidate the whole exercise.

5. Report κ (or α) with a confidence interval, plus the human–human benchmark, plus raw agreement for context. Never raw agreement alone.

6. Look at the confusion matrix, not just the summary. A κ of 0.75 that comes from uniform low-level noise is a different problem from one driven by two categories the model systematically conflates. The second is often fixable by rewriting one codeframe definition.

7. Check per-category recall on the codes that matter. Overall κ is dominated by the common categories. If the finding depends on a code held by 3% of responses, validate that code specifically.

8. Re-validate when anything changes — model version, prompt, codeframe, or survey wave. Prompt changes that look cosmetic can shift boundary behaviour, and model providers update models underneath you.

Part 4 — Implementation

import numpy as np
from scipy import stats

def cohens_kappa(a, b, k):
    """Cohen's kappa for two coders over k nominal categories."""
    a, b = np.asarray(a), np.asarray(b)
    n = len(a)
    po = float(np.mean(a == b))
    pa = np.bincount(a, minlength=k) / n
    pb = np.bincount(b, minlength=k) / n
    pe = float(np.sum(pa * pb))
    return (po - pe) / (1 - pe) if pe < 1 else 1.0

def krippendorff_alpha_nominal(a, b, k):
    """Krippendorff's alpha, nominal, two coders, no missing values."""
    o = np.zeros((k, k))
    for c1, c2 in zip(a, b):
        o[c1, c2] += 1
        o[c2, c1] += 1
    n_c = o.sum(axis=1)
    n_tot = n_c.sum()
    d_o = o.sum() - np.trace(o)
    d_e = (n_c.sum() ** 2 - np.sum(n_c ** 2)) / (n_tot - 1)
    return 1 - d_o / d_e if d_e > 0 else 1.0

def kappa_ci_bootstrap(a, b, k, reps=2000, conf=0.95, seed=0):
    """Bootstrap CI for kappa. Prefer this to the analytic SE, which
    behaves poorly on unbalanced frames."""
    rng = np.random.default_rng(seed)
    a, b = np.asarray(a), np.asarray(b)
    n = len(a)
    draws = [cohens_kappa(a[i], b[i], k)
             for i in (rng.integers(0, n, n) for _ in range(reps))]
    lo, hi = np.percentile(draws, [(1-conf)/2*100, (1+conf)/2*100])
    return cohens_kappa(a, b, k), float(lo), float(hi)

def confusion(a, b, k, labels=None):
    """Where the disagreement actually is."""
    m = np.zeros((k, k), int)
    for c1, c2 in zip(a, b):
        m[c1, c2] += 1
    return m

def per_category_recall(human, machine, k):
    """Recall for each code - overall kappa hides failures on rare codes."""
    human, machine = np.asarray(human), np.asarray(machine)
    return {c: float(np.mean(machine[human == c] == c))
            for c in range(k) if (human == c).sum() > 0}

Reporting template — the four numbers that belong in a technical annex:

kappa, lo, hi = kappa_ci_bootstrap(human, machine, k)
print(f"Machine-human agreement: kappa = {kappa:.3f} (95% CI {lo:.3f}-{hi:.3f})")
print(f"Human-human benchmark:   kappa = {human_human:.3f}")
print(f"Raw agreement:           {np.mean(human == machine):.1%}")
print(f"Validation sample:       n = {len(human)}")

Part 5 — Checklist

1. Never report raw agreement alone. It is inflated by codeframe imbalance and says nothing about skill.
2. Validate on 300+ items. Below 200 the confidence interval is wider than the decision you are making.
3. Establish the human–human ceiling. Without it you have no scale.
4. Blind the human coder to the machine's output.
5. Report a confidence interval on κ, bootstrapped rather than analytic.
6. Read the confusion matrix, and check recall on the codes that carry the finding.
7. Stratify the validation sample to get precision on rare codes.
8. Re-validate on every change — model, prompt, codeframe, wave.
9. Say so in the annex. "Open ends were coded by an LLM, validated against a blind human coder on a stratified sample of 400 responses (κ = 0.78, 95% CI 0.72–0.83; human–human benchmark κ = 0.81)" is a sentence that survives review. "Coded using AI" is not.

Reproducing the simulations

import numpy as np
rng = np.random.default_rng(160235)

def simulate(prev, accuracy, n_items, reps=400):
    """Two independent coders of equal accuracy over a given category balance."""
    prev = np.asarray(prev); k = len(prev)
    po_all, kappa_all = [], []
    for _ in range(reps):
        truth = rng.choice(k, n_items, p=prev)
        human, machine = truth.copy(), truth.copy()
        for coder in (human, machine):
            err = rng.uniform(size=n_items) > accuracy
            coder[err] = rng.choice(k, err.sum())
        po_all.append(np.mean(human == machine))
        kappa_all.append(cohens_kappa(human, machine, k))
    return np.mean(po_all), np.mean(kappa_all)

# Same accuracy, same raw agreement, very different kappa
print(simulate([.25,.25,.25,.25], 0.95, 2000))   # (0.926, 0.902)
print(simulate([.92,.04,.03,.01], 0.95, 2000))   # (0.927, 0.647)

Sources and further reading

• Krippendorff, K. (2018) Content Analysis: An Introduction to Its Methodology, 4th edn, Chapters 11–12 — α, its variants, and the reasoning behind chance correction.
• Cohen, J. (1960) "A coefficient of agreement for nominal scales", Educational and Psychological Measurement 20(1) — the original κ.
• Feinstein, A.R. & Cicchetti, D.V. (1990) "High agreement but low kappa: I. The problems of two paradoxes", Journal of Clinical Epidemiology 43(6) — the definitive treatment of why high agreement and low κ coexist, which is Part 1 of this article stated properly.
• Landis, J.R. & Koch, G.G. (1977) "The measurement of observer agreement for categorical data", Biometrics 33(1) — the origin of the conventional κ bands. Widely used, and worth knowing that the authors described them as arbitrary.
• Gwet, K. (2014) Handbook of Inter-Rater Reliability, 4th edn — alternatives such as AC1 that behave better under high prevalence, if the κ paradox is biting hard.