Do LLMs Agree With Human Coders? A 2025 Reliability Study
Qualitati Research Team · 2026-05-30 · 7 min read
Last updated: May 30, 2026
Short answer
In a 2025 study, GPT-4o and GPT-4.5 reached substantial agreement with human coders (Cohen's kappa > 0.6) on three of four themes in a real qualitative dataset, but only moderate agreement on a domain-general theme. The takeaway: large language models can match human inter-rater reliability on concrete, well-defined codes — but they still need human oversight on abstract constructs.
What the study tested
The question of whether LLMs can be trusted to code qualitative data the way a second human coder would is now an empirical one, not a hypothetical. According to Borse, Chatta Subramaniam, and Rebello (2025), in "Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis", the researchers compared two OpenAI models — ChatGPT-4o and ChatGPT-4.5-preview — against human raters on transcripts from 14 undergraduate student groups at a midwestern U.S. university discussing problem-solving strategies for a project.
Rather than asking the model to invent themes, the team applied a defined coding scheme and measured how often the LLM's code assignments matched the human raters'. They tuned model hyperparameters (such as temperature) and refined prompts to optimize agreement, then computed Cohen's kappa — the standard chance-corrected measure of inter-rater reliability — for each theme.
How reliable was the AI?
Reliability was strong on concrete themes and weaker on abstract ones. According to Borse et al. (2025), the LLMs achieved:
| Outcome | Agreement level | Cohen's kappa |
|---|
| Three coded themes | Substantial | κ > 0.6 |
| One additional theme | Moderate | 0.4 – 0.6 (below substantial) |
For context, Cohen's kappa runs from 0 (chance agreement) to 1 (perfect agreement). A kappa above 0.6 is conventionally read as "substantial," and many qualitative teams treat ~0.7 as a publishable threshold for two human coders. Hitting κ > 0.6 on three of four themes puts these models in roughly the same band a competent human second coder would occupy — at least for the domain-specific codes.
Where AI coding fell short
The weakest result was on what the authors describe as a domain-general construct — a theme requiring broad, context-dependent interpretation rather than recognition of a concrete physics problem-solving move. This mirrors a pattern seen across the LLM-coding literature: models are most reliable when a code has clear, surface-level markers, and least reliable when it demands inference about intent, tone, or meaning that humans resolve using tacit knowledge.
The practical implication is not "AI replaces a coder." It is that AI reliability is theme-dependent. A responsible workflow audits agreement code by code, leans on the model where kappa is high, and routes low-agreement codes to human adjudication.
What this means for researchers
If you are considering AI-assisted coding in 2026, this study supports a measured, human-in-the-loop approach:
- Use a defined codebook. The strong results came from applying an existing scheme, not free-form discovery. Give the model explicit code definitions and examples.
- Measure agreement, don't assume it. Compute kappa (or another chance-corrected statistic) between the AI and a human on a sample before trusting the model on the full corpus.
- Tune prompts and settings. The authors improved reliability by optimizing hyperparameters and prompts — reliability is partly an engineering problem, not a fixed property of the model.
- Treat abstract codes as human territory. Reserve domain-general, interpretive themes for human coders or dual review.
This is exactly the philosophy behind Qualitati's QDA Workspace and ThemeLens, our AI thematic analysis engine: the AI proposes and applies codes across transcripts, while researchers review, override, and validate every code — so you get the speed of automation without surrendering the rigor of human judgment. For teams collecting the data too, Qualitati's AI Interviewer feeds transcripts straight into that coding workflow.
How this fits the broader evidence
The 2025 reliability study is one data point in a fast-growing literature. Recent work converges on a consistent message: LLMs are best positioned as an additional coder — a second or third rater that accelerates the work, surfaces candidate codes, and improves consistency — rather than an autonomous analyst. The gains are real (speed, scalability, reduced repetitive labor), but they are conditional on transparent prompts, defined schemes, and human verification.
FAQ
Can LLMs replace human qualitative coders?
No. The 2025 study shows GPT-4o and GPT-4.5 can reach substantial agreement with humans on concrete codes, but they fall to moderate agreement on abstract, interpretive themes — so human review remains essential.
What is a good Cohen's kappa for AI-assisted coding?
A kappa above 0.6 indicates substantial agreement; many teams aim for ~0.7+ before treating codes as reliable. Below 0.4 signals the code likely needs human adjudication.
Which models were tested?
ChatGPT-4o and ChatGPT-4.5-preview, against human raters, on discussions from 14 undergraduate student groups (Borse et al., 2025).
Does prompt design affect reliability?
Yes. The authors improved agreement by tuning hyperparameters and refining prompts, indicating reliability is partly a function of how the task is set up, not just the model.
Sources: Borse, N. S., Chatta Subramaniam, R., & Rebello, N. S. (2025). Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis. arXiv:2508.14764.
This article is an independent editorial summary of third-party research. Statistics and findings are attributed to the original authors; Qualitati is not affiliated with the study.