Last updated: May 30, 2026
\n\nIn a 2025 study, GPT-4o and GPT-4.5 reached substantial agreement with human coders (Cohen's kappa > 0.6) on three of four themes in a real qualitative dataset, but only moderate agreement on a domain-general theme. The takeaway: large language models can match human inter-rater reliability on concrete, well-defined codes — but they still need human oversight on abstract constructs.
\n\nThe question of whether LLMs can be trusted to code qualitative data the way a second human coder would is now an empirical one, not a hypothetical. According to Borse, Chatta Subramaniam, and Rebello (2025), in \"Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis\", the researchers compared two OpenAI models — ChatGPT-4o and ChatGPT-4.5-preview — against human raters on transcripts from 14 undergraduate student groups at a midwestern U.S. university discussing problem-solving strategies for a project.
\nRather than asking the model to invent themes, the team applied a defined coding scheme and measured how often the LLM's code assignments matched the human raters'. They tuned model hyperparameters (such as temperature) and refined prompts to optimize agreement, then computed Cohen's kappa — the standard chance-corrected measure of inter-rater reliability — for each theme.
\n\nReliability was strong on concrete themes and weaker on abstract ones. According to Borse et al. (2025), the LLMs achieved:
\n| Outcome | Agreement level | Cohen's kappa |
|---|---|---|
| Three coded themes | Substantial | κ > 0.6 |
| One additional theme | Moderate | 0.4 – 0.6 (below substantial) |
For context, Cohen's kappa runs from 0 (chance agreement) to 1 (perfect agreement). A kappa above 0.6 is conventionally read as \"substantial,\" and many qualitative teams treat ~0.7 as a publishable threshold for two human coders. Hitting κ > 0.6 on three of four themes puts these models in roughly the same band a competent human second coder would occupy — at least for the domain-specific codes.
\n\nThe weakest result was on what the authors describe as a domain-general construct — a theme requiring broad, context-dependent interpretation rather than recognition of a concrete physics problem-solving move. This mirrors a pattern seen across the LLM-coding literature: models are most reliable when a code has clear, surface-level markers, and least reliable when it demands inference about intent, tone, or meaning that humans resolve using tacit knowledge.
\nThe practical implication is not \"AI replaces a coder.\" It is that AI reliability is theme-dependent. A responsible workflow audits agreement code by code, leans on the model where kappa is high, and routes low-agreement codes to human adjudication.
\n\nIf you are considering AI-assisted coding in 2026, this study supports a measured, human-in-the-loop approach:
\nThis is exactly the philosophy behind Qualitati's QDA Workspace and ThemeLens, our AI thematic analysis engine: the AI proposes and applies codes across transcripts, while researchers review, override, and validate every code — so you get the speed of automation without surrendering the rigor of human judgment. For teams collecting the data too, Qualitati's AI Interviewer feeds transcripts straight into that coding workflow.
\n\nThe 2025 reliability study is one data point in a fast-growing literature. Recent work converges on a consistent message: LLMs are best positioned as an additional coder — a second or third rater that accelerates the work, surfaces candidate codes, and improves consistency — rather than an autonomous analyst. The gains are real (speed, scalability, reduced repetitive labor), but they are conditional on transparent prompts, defined schemes, and human verification.
\n\nCan LLMs replace human qualitative coders?
No. The 2025 study shows GPT-4o and GPT-4.5 can reach substantial agreement with humans on concrete codes, but they fall to moderate agreement on abstract, interpretive themes — so human review remains essential.
What is a good Cohen's kappa for AI-assisted coding?
A kappa above 0.6 indicates substantial agreement; many teams aim for ~0.7+ before treating codes as reliable. Below 0.4 signals the code likely needs human adjudication.
Which models were tested?
ChatGPT-4o and ChatGPT-4.5-preview, against human raters, on discussions from 14 undergraduate student groups (Borse et al., 2025).
Does prompt design affect reliability?
Yes. The authors improved agreement by tuning hyperparameters and refining prompts, indicating reliability is partly a function of how the task is set up, not just the model.
Sources: Borse, N. S., Chatta Subramaniam, R., & Rebello, N. S. (2025). Investigation of the Inter-Rater Reliability between Large Language Models and Human Raters in Qualitative Analysis. arXiv:2508.14764.
\nThis article is an independent editorial summary of third-party research. Statistics and findings are attributed to the original authors; Qualitati is not affiliated with the study.
\n","related":[{"slug":"batching-synthetic-respondents-inflates-correlations-2026","title":"Batching Synthetic Respondents Inflates Correlations (2026)","description":"Generating many synthetic respondents in one LLM request inflates correlations, a 2026 LSE study finds. What it means for synthetic survey data validity.","category":"Methodology","date":"2026-10-13","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"ai-chatbot-qualitative-interviews-lessons-2026","title":"AI Chatbot Interviews at €0.04: Lessons From a 2026 Study","description":"Can AI chatbot interviews replace qualitative interviewers? A 2026 study of 74 chatbot-led interviews found cheap, engaging data collection but limited depth.","category":"Methodology","date":"2026-10-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-inductive-content-analysis-survey-responses-2026","title":"Can GPT-5 Do Inductive Content Analysis of Survey Text?","description":"Can GPT-5 do inductive content analysis? A 2026 study of 903 survey answers found moderate human-AI agreement (ARI 0.61 codes, 0.54 themes), varying by question.","category":"Methodology","date":"2026-10-05","readTime":"7 min read","author":"Qualitati Research Team"}]}