Short answer: Intercoder reliability measures how consistently two coders apply the same codes to the same data, usually with Cohen's κ. In AI qualitative coding, an LLM is treated as one coder and checked against a human gold standard. As of 2026, well-prompted frontier models reach substantial human–AI agreement (κ ≈ 0.60–0.70) on concrete, domain-specific codes, but drop on abstract constructs. Human oversight stays mandatory.
\n\nIntercoder reliability (ICR) — also called inter-rater reliability — is the degree to which independent coders assign the same codes to the same segments of qualitative data. It is the classic evidence that a codebook is clear and that coding is not just one researcher's private interpretation. When you bring AI into qualitative data analysis, intercoder reliability becomes the natural yardstick for a simple question: does the model code the way a trained human would? Treat the LLM as a second coder, run it against a human-coded gold standard, and compute an agreement statistic.
\nLast updated: July 13, 2026. All statistics below are attributed to named, dated 2025–2026 sources; where evidence is mixed, we say so.
\n\nThe workflow mirrors classic two-coder reliability, with the model standing in for the second human.
\n| Cohen's κ | Common interpretation | What to do in AI coding |
|---|---|---|
| < 0.20 | Slight | Codebook or unit problem; do not automate |
| 0.21–0.40 | Fair | Human-only; revise definitions |
| 0.41–0.60 | Moderate | AI as first-pass draft, full human review |
| 0.61–0.80 | Substantial | AI-assisted coding with human validation |
| 0.81–1.00 | Almost perfect | Spot-check; audit edge cases |
Thresholds are conventions (Landis & Koch, 1977), not laws. Many methodologists treat κ ≥ 0.60–0.70 as acceptable for exploratory work and demand higher for high-stakes coding.
\n\nThe research picture is encouraging but bounded. In an August 2025 study of engineering-design discussions (14 groups, 42 students, 204 text segments), ChatGPT-4o and 4.5-preview reached substantial human–AI agreement after tuning: κ = 0.70 for physics concepts and 0.60 for engineering-design codes, but only 0.55 for metacognitive thinking — the most abstract construct (arXiv 2508.14764). Earlier work found a sharp model-generation gap: GPT-4 reached human-equivalent interpretation where GPT-3.5 averaged κ ≈ 0.34 on the same prompts.
\nA second 2025 finding is a useful warning. A July 2025 paper showed that manipulating temperature and persona to make LLM agents agree with each other produced “minimal accuracy gains” against the human ground truth (arXiv 2507.11198). The lesson: consensus among models is not the same as correctness. Only agreement with a human gold standard counts as reliability evidence.
\n\nUse this original Qualitati checklist before you report any AI-assisted coding as reliable.
\nIntercoder reliability is most meaningful for deductive coding, where a fixed codebook defines what agreement even means. Here AI does well: the categories are pre-specified, so the model has a clear target. For inductive coding, where codes emerge from the data, ICR is a blunter instrument — two coders (human or AI) may generate different but equally valid code labels. For inductive work, reliability shifts from “identical labels” toward transparency, audit trails, and reflexivity. Report ICR where a codebook exists; report your process where it does not.
\n\nQualitati is an AI user research platform for product, UX, and insights teams, with qualitative data analysis built around human-in-the-loop coding. The QDA Workspace supports AI-assisted inductive and deductive coding, codebook generation, and theme visualization, with a human able to confirm, merge, rename, or reject every code — the exact override loop intercoder reliability assumes. For higher-level synthesis, ThemeLens runs a map-reduce thematic pipeline across up to 100 transcripts, mapping codes to research questions and anchoring themes in participant quotes. Qualitati documents which model and temperature each analysis tool uses, so you can record the reproducibility details this checklist requires. It works in 10 languages, and pricing is transparent: a free tier with 30 credits on signup, no credit card, and published per-credit rates — a stated alternative to NVivo, ATLAS.ti, and MAXQDA on AI-native analysis. See transparent pricing.
\n\nIntercoder reliability is necessary but not sufficient. A high κ proves consistency, not validity: two coders can reliably apply a flawed codebook and agree on the wrong thing. Kappa is also sensitive to prevalence — rare codes can produce low κ even with high raw agreement (the “kappa paradox”), so read it alongside percent agreement and base rates. The published AI figures come from specific domains (STEM education, small samples) and may not transfer to your interview data, sensitive topics, or non-English corpora. And forcing reliability onto genuinely interpretive, reflexive analysis can distort the method — not every qualitative tradition treats κ as the goal. Human-review note: verify any model-performance or reliability claim against your own data and deployment before publishing; do not infer it from category benchmarks.
\n\nCohen's κ of 0.61–0.80 (“substantial”) is a common acceptance bar for AI-assisted coding, with human validation. High-stakes coding warrants κ ≥ 0.80. Below 0.60, use AI only as a first-pass draft and revise the codebook.
\n\nOn concrete, well-defined codes, frontier models reached κ of 0.60–0.70 against human coders in a 2026 study — substantial agreement. On abstract or interpretive constructs, agreement drops, and human oversight remains essential.
\n\nUse Cohen's κ for two coders (e.g., one human vs one AI) and nominal codes. Use Krippendorff's α when you have more than two coders, missing data, or ordinal/interval codes. Both correct for chance agreement; raw percent agreement does not.
\n\nNo. A July 2025 study found that making LLM agents agree with each other yields minimal accuracy gains against ground truth. Reliability evidence requires agreement with a human gold standard, not model-to-model consensus.
\n\nLess directly. ICR assumes a fixed codebook, which fits deductive coding. For inductive coding, where codes emerge, favor transparency, audit trails, and reflexivity over a single κ number.
\n\nIntercoder reliability turns “the AI looks accurate” into a number you can defend. The 2025–2026 evidence is clear: well-prompted frontier models reach substantial human–AI agreement on concrete codes, weaker agreement on abstract ones, and model-to-model consensus proves nothing on its own. Treat the LLM as a second coder, validate against a human gold standard, and keep a human in the loop. Start free with 30 credits — run AI-assisted coding in the QDA Workspace with human validation built in, or compare transparent pricing against NVivo, ATLAS.ti, and MAXQDA.
\n","related":[{"slug":"ai-chatbot-qualitative-interviews-lessons-2026","title":"AI Chatbot Interviews at €0.04: Lessons From a 2026 Study","description":"Can AI chatbot interviews replace qualitative interviewers? A 2026 study of 74 chatbot-led interviews found cheap, engaging data collection but limited depth.","category":"Methodology","date":"2026-10-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-inductive-content-analysis-survey-responses-2026","title":"Can GPT-5 Do Inductive Content Analysis of Survey Text?","description":"Can GPT-5 do inductive content analysis? A 2026 study of 903 survey answers found moderate human-AI agreement (ARI 0.61 codes, 0.54 themes), varying by question.","category":"Methodology","date":"2026-10-05","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-human-surrogates-item-mean-personas-2026","title":"Can Richer Personas Make LLMs Human Surrogates? A 2026 Test","description":"A 2026 study of 400,000+ participants finds LLM human surrogates match item averages but explain only 3.05% of individual variation. Richer personas don't help.","category":"Methodology","date":"2026-10-01","readTime":"7 min read","author":"Qualitati Research Team"}]}