Last updated: June 2, 2026
\n\nCan large language models do thematic analysis on their own? Experienced researchers say no — not yet. In a 2025 reflective-workshop study of 25 software-engineering researchers, participants saw real gains in efficiency and scalability from LLM-assisted thematic analysis but flagged serious risks around bias, contextual loss, and reproducibility, concluding that LLMs can support interpretive analysis but not substitute for it (Ornelas et al., 2025).
\n\nThe study asked how experienced researchers conceptualize the opportunities, risks, and methodological implications of folding LLMs into thematic analysis. According to Ornelas et al. (2025), the authors ran a structured reflective workshop with 25 researchers from the International Software Engineering Research Network (ISERN). Rather than benchmarking a model against a gold-standard codebook, the design captured expert judgment: participants used color-coded canvases to document perceived opportunities, limitations, and recommendations across three stages of analysis — LLM-assisted coding, theme generation, and review.
\nThat framing matters. Most coverage of \"AI thematic analysis\" measures accuracy on a fixed dataset. This study instead asks the people who do thematic analysis for a living where they would and would not trust the tool — which is closer to the decision a working researcher actually faces.
\n\nParticipants did not dismiss LLMs. The headline benefits were efficiency and scalability: the ability to move through large volumes of qualitative data faster than manual coding allows, and to support early, exploratory passes over a corpus. In a field where hand-coding hundreds of interview transcripts can take weeks, that speed is a genuine draw.
\nThe optimism, though, was conditional. The same researchers who valued the speed were explicit that it only pays off when paired with disciplined human judgment — a theme that runs through every recommendation below.
\n\nAccording to Ornelas et al. (2025), participants converged on three recurring risks:
\n| Risk | What it means in practice | Why it matters for rigor |
|---|---|---|
| Bias | The model's training and prompting can tilt which codes and themes surface | Themes may reflect the model's priors, not the data |
| Contextual loss | Nuance, tone, and situated meaning get flattened when text is chunked and summarized | Interpretive depth is the whole point of qualitative work |
| Reproducibility | The same prompt and data can yield different themes across runs and model versions | Findings become hard to audit or replicate |
Reproducibility is the quietest but most corrosive of the three. Thematic analysis already lives with the charge that it is subjective; adding a non-deterministic model that shifts behavior between versions makes \"we got these themes\" harder to defend unless the process is documented carefully.
\n\nThe study's recommendations centered on two ideas. First, prompting literacy: researchers need to understand how prompt wording, examples, and framing shape what the model returns, rather than treating the LLM as a black box that \"finds the themes.\" Second, continuous human oversight: a human stays in the loop at every stage — checking codes, interrogating generated themes, and owning the interpretation.
\nThe overall conclusion, in the authors' framing, is that LLMs as tools can support but not substitute interpretive analysis. That lines up with a broader pattern in 2025–2026 methods research: AI is most defensible as an accelerant for the mechanical parts of coding, with meaning-making left to the researcher.
\n\nIf you run thematic analysis — in software engineering, UX, market research, or academic work — the takeaway is not \"avoid LLMs\" or \"automate everything.\" It is to design a workflow where the model accelerates coding while a human owns interpretation and auditability. Tools that keep the researcher in control of codes and themes, rather than hiding the reasoning, fit this brief best. Qualitati's ThemeLens is built around exactly this human-in-the-loop model for AI thematic analysis, and pairs naturally with an AI Interviewer when you are also collecting the data. Whatever tool you use, the study's bar is a fair one: speed is welcome, but the themes are still yours to defend.
\n\nCan LLMs replace researchers in thematic analysis?
No. According to Ornelas et al. (2025), experienced researchers concluded that LLMs can support but not substitute interpretive analysis, with human oversight needed at every stage.
What are the biggest risks of AI thematic analysis?
The study highlights three: bias in which themes surface, loss of context and nuance, and poor reproducibility across runs and model versions.
What is \"prompting literacy\"?
It is the researcher's understanding of how prompt wording, examples, and framing shape an LLM's output — treated in the study as a core skill for using these tools responsibly.
How was the study conducted?
It was a structured reflective workshop with 25 ISERN software-engineering researchers, who documented opportunities, limitations, and recommendations on color-coded canvases.
This article is an independent editorial summary of third-party research. Primary source: Ornelas, T., Araújo, A. A., Araújo, J., Araújo, M., Trinkenreich, B., & Kalinowski, M. (2025). LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations. arXiv preprint.
\n","related":[{"slug":"synthetic-participants-pilot-study-design-2026","title":"Can Synthetic Participants Pilot Your Study Design? (2026)","description":"Can synthetic participants pilot your study design? A 2026 study found LLM reasoning traces caught more survey flaws than a paid human pilot or a chatbot review.","category":"Methodology","date":"2026-10-20","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"ai-follow-up-interviews-vs-fixed-surveys-experiment-2026","title":"AI Follow-Up Interviews in Surveys: A 2,243-Person Test (2026)","description":"Do AI follow-up interviews improve survey answers? A 2026 POQ experiment with 2,243 people found richer reasons, stable satisfaction, and some answer shifts.","category":"Methodology","date":"2026-10-16","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"batching-synthetic-respondents-inflates-correlations-2026","title":"Batching Synthetic Respondents Inflates Correlations (2026)","description":"Generating many synthetic respondents in one LLM request inflates correlations, a 2026 LSE study finds. What it means for synthetic survey data validity.","category":"Methodology","date":"2026-10-13","readTime":"7 min read","author":"Qualitati Research Team"}]}