Which LLM Is Best at Thematic Analysis? A 2026 Study
Qualitati Research Team · 2026-07-27 · 8 min read
Can large language models produce reliable thematic analysis? A December 2025 study says yes — when you run the same model several times and measure agreement. Testing Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Sonnet on an interview transcript, all three exceeded the κ > 0.80 threshold for “excellent” inter-rater reliability, with Gemini scoring highest.
What did the study test?
The paper, “Multi-LLM Thematic Analysis with Dual Reliability Metrics” by Nilesh Jain, Hyungil Suh, Seyi Adeyinka, Leor Roseman, and Aza Allsop (arXiv, December 2025; revised February 2026), asks a practical question: if you ask an LLM to code a qualitative transcript, how do you know the output is trustworthy rather than a one-off hallucination?
Their answer is an ensemble approach. Instead of running a model once, they ran each model six independent times on the same psychedelic art-therapy interview transcript, varying random seeds (1–6) and temperature (0.0–2.0). They then measured how consistently each model reproduced the same themes across runs, using two complementary metrics.
How was reliability measured?
The study used two metrics that capture different kinds of agreement:
- Cohen’s Kappa (κ) — measures inter-rater agreement corrected for chance. It answers: do the runs assign the same codes to the same segments? Standard interpretation: below 0.40 is poor, 0.40–0.60 moderate, 0.60–0.80 substantial, and above 0.80 excellent.
- Cosine (semantic) similarity — measures whether theme descriptions mean the same thing even when worded differently. This catches agreement that exact-match kappa would miss.
Combining the two is the paper’s core methodological contribution: kappa alone penalizes harmless rewording, while semantic similarity alone can mask genuine coding disagreement. Together they give a fuller picture of reliability.
Which LLM was most reliable for thematic analysis?
According to Jain et al. (2025), all three models cleared the κ > 0.80 bar, but Gemini led on both metrics:
| Model | Cohen’s Kappa | Cosine Similarity | Consensus Themes |
|---|
| Gemini 2.5 Pro | 0.907 | 95.3% | 6 (50–83% consistency) |
| GPT-4o | 0.853 | 92.6% | 5 |
| Claude 3.5 Sonnet | 0.842 | 92.1% | 4 |
The gap between models is real but modest. More striking is that every model surpassed the excellence threshold once results were aggregated across runs — a reminder that the reliability gain comes as much from the ensemble method as from the choice of model.
Why does running the model multiple times matter?
Because LLMs are probabilistic, a single run can surface an idiosyncratic reading of a transcript. That is the same worry a qualitative researcher has about a single human coder. The classic fix in qualitative methods is a second coder and an inter-rater agreement statistic. The study reframes that fix for AI: treat each model run as a separate “rater,” then require themes to survive across runs before you trust them. Gemini’s six consensus themes, for instance, each recurred in 50–83% of runs — the themes that appear once and vanish are exactly the ones you should be skeptical of.
What this means for researchers
The takeaway is not “use Gemini.” It is that AI-assisted coding needs a reliability protocol, not a single prompt. Three practices transfer directly to everyday work:
- Run more than once. Multiple runs at varied temperature expose which themes are stable and which are artifacts.
- Report an agreement metric. A kappa (or an equivalent) makes AI coding auditable and defensible to reviewers, just as it does for human coders.
- Keep a human interpreter. High reliability means the model is consistent, not necessarily correct. Consistency across runs is a floor for trust, not a substitute for researcher judgment about meaning and context.
This is the workflow behind AI-assisted qualitative tools generally. If you want to try structured, auditable theme extraction on your own transcripts, ThemeLens applies AI thematic analysis with a transparent coding trail, and QualiTaTi’s AI Interviewer can generate the transcripts to analyze in the first place.
Limitations to keep in mind
This was a single interview transcript in one domain, so the exact kappa values should not be read as universal model rankings. The metrics validate internal consistency across runs — they do not verify that the themes match a human expert’s interpretation or the participant’s intended meaning. Reliability and validity remain distinct questions, and the paper measures the former.
FAQ
Is AI thematic analysis reliable?
In this study, yes — when aggregated over multiple runs, all three tested LLMs reached Cohen’s κ above 0.80, the conventional threshold for excellent agreement. Reliability from a single run is not guaranteed.
Which LLM did best?
Gemini 2.5 Pro (κ = 0.907, 95.3% semantic similarity), narrowly ahead of GPT-4o (0.853) and Claude 3.5 Sonnet (0.842).
Does high reliability mean the coding is correct?
No. The metrics measure consistency across runs, not agreement with human interpretation. A human researcher should still validate that the themes are meaningful and accurate.
How many runs should I use?
The study used six per model with varied seeds and temperature. The principle — run multiple times and keep only themes that recur — matters more than the exact count.
Last updated: July 27, 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the study’s authors. Read the original paper: Multi-LLM Thematic Analysis with Dual Reliability Metrics (arXiv:2512.20352).