Can a Second LLM Catch the First One's Coding Errors?

Qualitati Research Team · 2026-08-28 · 6 min read

A high F1 score does not mean an AI coder is accurate. In a 2026 study by Dunivin, Noori, Frey and Atkinson, an LLM coding 3,000 emails hit F1 scores between 0.74 and 1.00 while producing false-positive rates of 8% to 54% on the same codes. Adding a second LLM to re-read and critique each positive label recovered much of that lost precision.

What did the study test?

The researchers tested a two-stage coding workflow instead of the usual one-pass approach. In stage one, an LLM applies a human-designed codebook to text and is tuned to favour recall — catch everything that might belong to a code. In stage two, a separate LLM critic re-reads the source passage behind every positive label and decides whether the code genuinely applies.

The corpus was 3,000 high-content emails drawn from Apache Software Foundation project evaluation discussions, coded against six codes. The authors also ran a human audit of 360 positive annotations (60 passages across the six codes) to understand what the machine was getting wrong.

How accurate was the first-pass AI coding?

Accurate on aggregate metrics, unreliable in the labels a researcher would actually read. According to Dunivin and colleagues (2026), first-pass F1 ranged from 0.74 to 1.00 across the six codes, but false-positive rates on the same codes ranged from 8% to 54%. For the worst codes, roughly half of everything the model flagged did not belong there.

That gap is the practical point. F1 blends precision and recall into one number and can stay respectable while precision quietly collapses on a specific code. If you then build a theme from the passages the model tagged, you are reading a pile in which a large minority of the evidence is noise.

Did the second LLM actually help?

Yes, and most on the codes that needed it. The critique stage improved F1 by 0.04 to 0.25 depending on the code. The two weakest codes moved from 0.52 to 0.69 and from 0.55 to 0.79 — the difference between a code you would discard and one you could work with.

Stage	What it optimises	Reported result
Stage 1 — LLM coder	Recall (catch every candidate)	F1 0.74–1.00; false positives 8%–54%
Stage 2 — LLM critic	Precision (re-read and reject)	F1 gain of 0.04–0.25; worst codes 0.52→0.69 and 0.55→0.79

The design is deliberately cheap. The critic only inspects passages already flagged as positive, so the second pass touches a fraction of the corpus rather than re-processing everything.

Why does the first pass over-code?

The human audit of 360 annotations surfaced two recurring error patterns, both of them familiar to anyone who has trained a junior coder:

Misinterpretation of code definitions. The model read the codebook label more loosely (or more literally) than the researcher intended, and stretched the code to passages that share surface vocabulary.
Confusion between meta-discussion and actual decision criteria. The model coded people talking about a criterion the same way it coded people applying one. In interview data the equivalent is coding a participant's hypothetical — "I suppose some people care about price" — as though it were their own stated reason.

Neither failure is fixed by a better model alone. Both are definition problems, which is why a critic that is forced back to the source text catches them: it re-tests the definition against the passage instead of against its own earlier judgement.

What this means for researchers

Three things follow for anyone running AI-assisted coding on real data.

Stop treating a single agreement statistic as a quality check. Report precision per code, not just an overall F1 or kappa. A code with a 54% false-positive rate is broken even inside an otherwise healthy-looking run.

Audit at the code level, not the corpus level. The study's spread — 8% to 54% — was across codes in the same run with the same model. Codes that require inference about intent degrade; concrete, surface-detectable codes hold up. Sampling a handful of passages at random across all codes would have hidden this.

Separate the recall pass from the precision pass. Asking one model to be simultaneously exhaustive and strict pushes it toward a mediocre middle. Splitting the job lets you tune each half. If you are running thematic analysis in ThemeLens, the practical version is: generate codes broadly, then re-read the tagged excerpts against the definition before you promote anything to a theme.

The study also reinforces a point about codebooks. Two of its error patterns trace back to definitions that were clear to the researchers and ambiguous to the model. Writing inclusion and exclusion criteria into each code — what does not count — costs ten minutes and removes a large share of the over-coding a critic would otherwise have to clean up.

Limits of this evidence

The corpus is mailing-list text from open-source project evaluations, not interview transcripts, and the codes are organisational rather than experiential. Whether the same critique gains transfer to semi-structured interview data, where meaning is carried by hesitation and context, is an open question the study does not answer. The result is best read as a warning about metric choice and a workflow worth testing, not a validated pipeline for qualitative interview analysis.

FAQ

Does a second LLM pass fix AI coding errors?

It fixes one class of error — false positives from over-application of a code. In this study the critique stage raised F1 by 0.04 to 0.25. It does nothing for codes the first pass missed entirely, since the critic only reviews passages that were already flagged.

Is F1 a good measure of AI qualitative coding quality?

Not on its own. This study found F1 between 0.74 and 1.00 alongside false-positive rates up to 54% on the same codes. Report precision and recall separately, per code.

Which codes are most at risk of AI over-coding?

Codes that require judging intent or distinguishing discussion of a topic from enactment of it. The audit found the model repeatedly conflated meta-discussion with actual decision criteria.

Where can I read the original paper?

The preprint is Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique by Dunivin, Noori, Frey and Atkinson (arXiv, 2026).

Last updated: 28 August 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the authors.