AI-Assisted Thematic Analysis: What 18 Researchers Found
Qualitati Research Team · 2026-08-24 · 6 min read
AI-assisted thematic analysis keeps the researcher in the interpretive seat while the model handles the mechanical passes — background scoping, first-pass codes, cluster review. In a 2025 study of 18 qualitative researchers using the DeTAILS toolkit, model outputs matched what researchers ultimately kept at F1 scores of 0.86–1.00, with overall task workload rated 26.3/100.
That is a narrower and more useful claim than "AI can do thematic analysis." The question researchers actually face is not whether a model can emit codes — it obviously can — but whether a human-in-the-loop workflow produces codes the researcher would have kept anyway, without the analyst losing contact with the data. A recent preprint puts numbers on both halves of that question.
What did the study test?
Sharma, Cochrane and Wallace (2025) built DeTAILS, a toolkit that maps LLM assistance onto Braun and Clarke's six-phase thematic analysis rather than replacing it. Each phase produces model suggestions the researcher accepts, edits, or discards before the next phase runs — so the analysis is a sequence of reviewed checkpoints, not a single generation.
The evaluation was a controlled user study. According to Sharma et al. (2025):
- 18 qualitative researchers, stratified by experience: 6 novice (<1 year), 6 proficient (1–4 years), 6 expert (>4 years).
- 30 discussion post transcripts per participant, drawn from a subreddit of the participant's own choosing (each transcript = an original post plus its comments).
- Model: gemini-2.5-pro-preview-03-25 via Google Vertex AI.
- Session length: 2 to 2.25 hours, covering a full analysis from background scoping to theme generation.
Participants were instructed to treat the analysis as if it belonged to their own research — which matters, because it means the "ground truth" is not an external codebook but each researcher's own accepted output.
How accurate was the AI at each phase?
Accuracy varied sharply by phase, and the pattern is the interesting part. The authors report agreement between the model's suggestions and what participants kept after refinement:
| Analysis phase | Agreement (F1) | Variability (SD) |
|---|
| Background research — related concepts | 0.86 | 0.13 |
| Concept outline | 0.98 | 0.03 |
| Initial coding | 0.90 | 0.17 |
| Global coding | 0.97 | 0.04 |
| Reviewing codes | 0.90 (macro) | 0.20 |
| Generating themes | 1.00 | — |
Read the standard deviations, not just the means. Initial coding (SD = 0.17) and reviewing codes (SD = 0.20) are the phases where researchers diverged most from the model — exactly the phases that require judgement about what a fragment of talk means. Concept outline and global coding, which are closer to structured summarisation, were near-ceiling and near-uniform. The high theme-generation score is best read as a consequence of the pipeline: by that point the researcher has already curated the codes the themes are built from.
Did it actually reduce workload?
Yes, on self-report. Sharma et al. (2025) collected NASA-TLX ratings after each session:
- Overall workload: 26.3/100 (SD = 12.4)
- Mental demand: 39/100 · Effort: 37/100 · Temporal demand: 29/100
- Frustration: median 10/100
- Self-rated performance: 73/100
Perceived usefulness was high — mean 4.21/5, with 86% of responses at 4 or 5 — and on the AttrakDiff scale the tool scored 6.3/7 for "structured" and 5.7/7 for "straightforward."
One timing result deserves attention: initial coding took significantly longer than the related-concepts, concept-outline, initial-codebook and global-coding phases (Bonferroni-corrected Wilcoxon tests, p ≤ 0.004). The phase where the model agreed least with researchers is also the phase where researchers spent the most time. That is what a functioning human-in-the-loop system should look like — effort concentrating where the machine is weakest.
What this means for researchers
Three practical implications follow from the numbers rather than from the framing.
- Treat first-pass codes as drafts, not output. An F1 of 0.90 with SD 0.17 means roughly one participant in six had substantial disagreement with the model's initial codes. Any workflow that auto-accepts this phase inherits that variance silently.
- Budget your time to match the variance. The phases worth your attention are initial coding and code review. Background scoping and outline restructuring are safe to delegate and check quickly.
- Low workload is not automatically a benefit. The ISERN workshop of 25 software-engineering researchers (Ornelas et al., 2025) raised the counterpoint directly: skipping manual open coding removes the familiarisation with the data that makes later interpretation possible. A workload score of 26/100 could describe a well-supported analyst or a disengaged one; the metric can't tell you which.
The design lesson generalises beyond this one toolkit. Systems that expose each phase for review — rather than returning a finished theme set — let the researcher spend attention where model agreement is lowest. That is the same principle behind phase-by-phase review in ThemeLens, where codes and themes are inspected and edited as separate steps rather than accepted as a block.
Limitations worth stating
This is a preprint (arXiv, October 2025) and has not completed peer review. The corpus was Reddit discussion posts, not interview transcripts — public asynchronous text differs from elicited talk in structure, length, and the interviewer's presence — a gap that matters if your corpus comes from AI-moderated interviews rather than scraped forums. The agreement scores compare the model against each participant's own refinements, so they measure alignment with a single analyst's decisions, not correctness against an independent expert codebook. And with n = 18 and one model, results should not be generalised to other LLMs or to sensitive datasets.
FAQ
Is AI-assisted thematic analysis accurate enough for published research?
The evidence supports it as a drafting aid under review, not as an autonomous analyst. In this study, agreement with researchers' own final decisions ranged from 0.86 to 1.00 F1 across phases, but the variance in the interpretive phases means outputs need to be checked line by line before they enter a manuscript.
Which phases of thematic analysis should stay manual?
Initial coding and code review showed the highest disagreement (SD = 0.17 and 0.20). These are the phases where researcher judgement changes the result most, so keep them human-led even in an AI-supported workflow.
Does using an LLM mean losing familiarity with the data?
It can. Ornelas et al. (2025) report researchers warning that delegating open coding removes the immersion that later interpretation depends on. Reading a sample of transcripts in full before running any model pass is a cheap safeguard.
What model was used in the DeTAILS study?
gemini-2.5-pro-preview-03-25, accessed through the Google Vertex AI API. Results are specific to that model and should not be assumed to transfer to smaller or open-weight models.
Sources
- Sharma, A., Cochrane, K., & Wallace, J. R. (2025). DeTAILS: Deep Thematic Analysis with Iterative LLM Support. arXiv:2510.17575. https://arxiv.org/abs/2510.17575
- Ornelas, T., Araújo, A. A., Araújo, J., Araújo, M., Trinkenreich, B., & Kalinowski, M. (2025). LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations. arXiv:2511.14528. https://arxiv.org/abs/2511.14528
Last updated: 24 August 2026.
This is an independent editorial summary of third-party research. QualiTaTi is not affiliated with the authors or their institutions, and all figures are drawn from the cited preprints.