AI Found 2x More Codes Than Humans: A 2026 Study
Qualitati Research Team · 2026-08-07 · 7 min read
A four-stage LLM pipeline coded 28 interview transcripts in about 45 minutes and produced 12 higher-level and 73 lower-level concepts — roughly double the 8 and 31 that five human researchers had produced manually on the same data. The experts rated the output a median 4 out of 5 across every quality dimension, but flagged the extra granularity as the main thing needing human repair.
What did the study test?
The study tested whether qualitative content analysis can be decomposed into discrete, auditable machine steps rather than handed to a chatbot as one prompt. Forster and colleagues (2026), publishing in Studies in Health Technology and Informatics (dHealth 2026), built a pipeline around GPT-4.1 and ran it over 28 semi-structured interviews on attitudes to health data donation — a dataset five researchers had already analysed by hand. That manual analysis became the comparison point.
The design matters more than the model. Each stage does one job, and only three of the four stages involve an LLM at all:
- Meaning-unit segmentation. The transcript is split into coherent thought units, with positions tracked so the original text can be reconstructed exactly by concatenating the units.
- Initial coding against a progressive codebook. Each unit receives one to three codes. The first transcript seeds the codebook; every later transcript must try to match an existing code before minting a new one.
- Hierarchical concept development. Two clustering passes group codes into lower-level concepts, then synthesise those into higher-level concepts, with a review step that sweeps up codes left ungrouped.
- Deterministic quote extraction. No LLM. A Python routine matches concepts back to source units by exact code matching, so a quote cannot be invented.
Stages 1 to 3 used one-shot prompting — methodology instructions plus a single worked example — with prompts stored in editable templates rather than buried in code.
How accurate was the AI analysis?
Accurate enough to be usable, and consistent enough to be trusted with a codebook. According to Forster et al. (2026), 92.6% of codes reused an existing label rather than creating a new one, with reuse exceeding 90% from the third interview onward. That is the number to watch: it means the codebook stabilised early instead of drifting into a long tail of near-duplicate codes — the classic failure mode when an LLM codes each transcript in isolation.
Agreement with the human analysis was looser at the code level than at the concept level. Three evaluators who had done the original coding estimated 41–60% thematic overlap at code level and 61–100% at concept level. In other words, the machine and the humans carved the text differently but arrived at broadly the same themes. One evaluator summarised it as differences of granularity and emphasis rather than contradictory coding.
Expert ratings
The five researchers scored the output using QUEST, an evaluation framework adapted from healthcare LLM assessment, on 5-point scales where 1 means fundamental problems and 5 means publication-ready.
| QUEST dimension | Median | Range |
|---|
| Quality of information | 4 | no variance |
| Understanding and reasoning | 4 | 3–4 |
| Expression style | 4 | 2–4 |
| Safety and harm | 4 | 4–5 |
| Trust and confidence | 4 | 3–4 |
Four of the five evaluators recommended minor revisions; one recommended major revisions. All five accepted the output for scientific use, conditional on researchers refining and contextualising it.
Why did the AI produce twice as many concepts?
Because a pipeline has no reason to consolidate, and a research team does. Human analysts collapse overlapping ideas through discussion — someone argues that two codes are really one, and the codebook shrinks. The pipeline in this study had a single-coder architecture with no consensus step, so distinctions survived that a team would have merged. The evaluators named the symptom directly: concepts that were "too similar" to each other, and umbrella terms broad enough to need splitting.
This is worth sitting with, because more concepts is not automatically worse. Finer granularity can surface distinctions a tired team would have flattened away. But it shifts the work rather than removing it: instead of generating themes, the researcher now audits and consolidates them. Any honest estimate of time saved has to price that consolidation pass in — the 45 minutes of compute is not the whole cost.
What this means for researchers
The transferable lesson is architectural, not about GPT-4.1 specifically. Three design choices in this pipeline are worth copying regardless of which model you use:
- Carry the codebook forward. Forcing each new transcript to match existing codes before creating new ones is what produced the 92.6% reuse rate. Coding transcripts independently and merging afterwards does not.
- Make quote retrieval deterministic. Extracting quotes with code, not with the model, removes hallucinated evidence as a category of error rather than as something to spot-check. Every claim traces back through concept to code to meaning unit to transcript position.
- Expect to consolidate. Budget a human pass for merging near-duplicate concepts and splitting umbrella terms. Treat the machine output as a first-pass codebook, not a finished framework.
If you are building this into your own workflow, a tool with a persistent codebook and traceable quotes does most of this for you — ThemeLens applies the same carry-forward coding and quote-provenance logic across a transcript set, and the AI Interviewer keeps interviews, codes, and source segments linked so the audit trail survives into write-up.
How much should you trust this result?
Treat it as a well-instrumented proof of concept, not as validation. The authors are explicit about the limits: only one-shot prompting was tested, only one model provider was used, and there was no formal inter-rater reliability test with paired annotations — the overlap figures are retrospective estimates by evaluators, not computed agreement statistics. The dataset is a single health-domain corpus of 28 interviews. And because the pipeline ran on a commercial API, the authors note that GDPR compliance for health data would require data processing agreements or locally hosted models.
What the study does establish is that a staged, transparent pipeline with expert oversight can produce output that the people who did the manual analysis are willing to accept — a meaningfully higher bar than a benchmark score.
FAQ
Can an LLM replace manual qualitative content analysis?
Not on this evidence. Every evaluator in the study conditioned their acceptance on researcher refinement, and one recommended major revisions. The pipeline produced a defensible first-pass structure, not a finished analysis.
What is a progressive codebook?
A codebook that accumulates across transcripts: the first transcript seeds it, and each subsequent transcript must attempt to match existing codes before creating new ones. It is what keeps code proliferation under control across a corpus.
Why does AI coding produce more codes than humans?
Because consolidation is a social process. Human teams merge overlapping codes through discussion; a single-pass pipeline has no equivalent step, so fine distinctions survive that a team would have collapsed.
Does the pipeline hallucinate quotes?
Not in this design. Quote extraction is handled by a deterministic algorithm rather than the model, matching concepts to source segments by exact code match, so every quote is traceable to a transcript position.
Primary source: Forster, E., Kartschmit, N., Klager, E., Mosor, E., Schuster, B., Mosor, E., Stamm, T., & Donsa, K. (2026). Multi-Stage LLM Pipeline to Support Qualitative Content Analysis — A Proof of Concept Experiment with Expert Validation. Studies in Health Technology and Informatics (dHealth 2026). Read the paper.
Last updated: 7 August 2026. This is an independent editorial summary of third-party research; it is not affiliated with or endorsed by the authors.