Agreement Isn't Quality: A Better Test for AI Coding
Qualitati Research Team · 2026-08-05 · 7 min read
Agreement is not the same thing as quality. A July 2026 study found that LLM qualitative coding agreed with human coders far less than humans agreed with each other — yet when a blind domain expert judged the codes themselves, they preferred human codes only 51.5% of the time. Low agreement did not mean low quality.
What did the study test?
The study asked whether inter-coder agreement with humans is a valid way to decide if an LLM is good enough to automate qualitative coding. According to Liu, Esbenshade, Xiao, Tian, Zhang, He and Sun (2026), in "Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth" (arXiv:2607.28890, 30 July 2026), the design was deliberately symmetric — humans and machines were held to the same standard, and neither was assumed correct in advance.
- Corpus. 2,560 educator messages.
- Codebook. A single 72-item codebook, applied identically by every coder.
- Coders. Five LLM systems and three trained human coders.
- Verification. An independent domain expert blindly judged 855 pairwise comparisons of coding decisions, without knowing which came from a human and which from a model.
- Ranking. A Bradley–Terry model converted the pairwise preferences into an overall ordering of all eight coders.
How much did AI and human coders disagree?
Substantially — by the usual reliability standards, badly enough to fail. Human–LLM agreement reached a Jaccard index of 0.30, against 0.52 for human–human agreement (Liu et al., 2026). On a conventional reliability screen, that gap is the end of the conversation: the model is judged unreliable and the automation decision is made against it.
What did the blind expert actually prefer?
Almost nothing distinguished them. Across the 855 blind pairwise comparisons, the expert preferred the human coding 51.5% of the time and the LLM coding 48.5% of the time — a difference that did not reach statistical significance (p = 0.537). The Bradley–Terry ranking went further: two of the five LLM systems placed above two of the three trained human coders.
| Measure | Result | What it implies |
|---|
| Human–human agreement (Jaccard) | 0.52 | The conventional benchmark ceiling |
| Human–LLM agreement (Jaccard) | 0.30 | Would normally read as "not good enough" |
| Blind expert preference | 51.5% human / 48.5% LLM (p = 0.537) | No detectable quality difference |
| Bradley–Terry ranking | 2 LLMs above 2 of 3 human coders | Coder identity does not predict rank |
Why would agreement and quality come apart?
Because disagreement has more than one cause. Two coders can diverge because one is wrong — or because both are defensible readings of an ambiguous passage. A 72-item codebook applied to short, context-poor messages generates a great deal of the second kind. The authors also report cases in which human consensus reflected a shared bias, one the blind verifier rejected in favour of the LLM's interpretation. When that happens, treating human consensus as ground truth does not merely understate model quality; it encodes the humans' blind spot into the benchmark itself.
This is the quiet problem with the standard evaluation. Agreement with humans measures similarity to a particular group of coders. It only measures quality if that group is right — which is exactly the thing the metric cannot check.
What this means for researchers
Do not stop computing reliability — but stop treating it as a pass/fail gate on its own. Three practical shifts follow from the paper:
- Read low agreement as a question, not a verdict. Pull the disagreement cases and look at them. If the model's codes are defensible, the disagreement is telling you the codebook is ambiguous, not that the model failed.
- Add a blind verification pass. Strip the source labels from a sample of coded segments, have a domain expert judge them head-to-head, and see whether provenance actually predicts preference. This is cheap relative to a full double-coding round.
- Audit your human baseline. High human–human agreement can mean shared training, shared assumptions, or shared bias. Agreement between coders drawn from the same background is weaker evidence than it looks.
If you are running AI-assisted coding in practice, the workflow implication is that your tooling has to make disagreements inspectable — you need to see which segments split, and why. That is the design principle behind ThemeLens, which keeps every code traceable to its source excerpt so a reviewer can adjudicate rather than accept a similarity score. The same logic applies upstream: cleaner, better-probed transcripts from an AI Interviewer reduce the share of genuinely ambiguous passages that drive agreement down in the first place.
Limits worth noting
This is one corpus, one domain, and one blind verifier. A 72-item codebook is unusually granular, and granularity mechanically depresses agreement scores — the effect might be smaller with a compact codebook. And a single expert, however independent, is still a single perspective. The paper's contribution is the protocol as much as the numbers: a transferable way to check whether your agreement metric is measuring what you think it measures.
FAQ
Does this mean inter-coder reliability is useless for AI coding?
No. It remains a useful signal of consistency and a good early warning that a codebook is ambiguous. What the study challenges is using it alone to decide whether an LLM's coding is good enough.
Is a Jaccard of 0.30 acceptable, then?
Not automatically. The finding is that a score that low does not by itself prove poor quality — you have to look at the disagreements. In this study they turned out to be largely defensible; in another study they might not.
What is blind expert verification?
An independent expert compares coding decisions from different sources without knowing which source produced which, then states a preference. Because provenance is hidden, the judgement cannot be shaped by expectations about humans or machines.
Can human coders share a bias?
Yes — and the paper documents it. Coders trained together, on the same codebook, in the same field, can converge on a reading that an outside expert rejects. High agreement between them looks like reliability but can be shared error.
Last updated: 5 August 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the authors. For the full method and results, read the original paper.