Can LLMs Scale Qualitative Analysis? A 167-Interview Study

Qualitati Research Team · 2026-07-24 · 7 min read

Large language models can make large-scale qualitative analysis feasible without abandoning rigor — but only when researchers design the division of labor deliberately. A January 2026 study by Ronaghi and colleagues applied a human-LLM framework to 167 interview transcripts across 12 health centers, work they estimated would otherwise take roughly 93 workweeks of manual coding.

What did the study test?

The study tested whether a structured human-LLM workflow could deliver rigorous qualitative analysis at a scale that manual coding could not reach in time. Ronaghi, Aveling, Levis, Ross, Alsentzer, and Singer (2026), in the arXiv preprint Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research, worked inside a live multi-site study of diabetes care at Federally Qualified Health Centers (FQHCs).

Two analytic tasks were run through the framework:

Qualitative synthesis — organizing roughly 31,200 words of researcher-generated summaries across 22 domains into comparative feedback reports for each site.
Deductive coding — coding 167 interview transcripts (about 1,327,000 words, roughly 8,600 minutes of interview time) against a framework of 19 coding areas expanded into 177 sub-questions, an average of nine per code.

The technical setup mattered for a health-data context: ChatGPT-4o for thematic organization, OpenAI o1 for cross-site synthesis, and ChatGPT-4o with retrieval-augmented generation for coding, all run on a HIPAA-compliant institutional deployment with a Qdrant vector store and OpenAI's text-embedding-3-large.

What is the four-step human-LLM framework?

The framework is model- and task-agnostic, and its central move is to decide which parts of the work carry tacit researcher judgment before any model is involved. According to Ronaghi et al. (2026), the four steps are:

Define the task. Complete the work manually on a small sample first — clarify the goal, fix the output format, document the workflow, and mark where researcher involvement is non-negotiable.
Design the human-LLM method. Break the task into discrete parts, state each part's purpose, identify the tacit contributions humans make, and choose how the model is configured for each part.
Evaluate at small scale. Compare outputs produced with and without LLM assistance using task-specific quality criteria rather than a single quantitative score.
Apply and evaluate at full scale. Document real-world use and assess the effect on research goals and efficiency, not just on speed.

Step 1 is the one most teams skip. Doing the task by hand first is what produces the output specification the model is later held to.

How did they validate the LLM's output?

They did not use a kappa. Notably, the team judged outputs against qualitative rigor standards rather than inter-rater reliability statistics — specifically grounding in the data, integration of theory and data, alignment with the research question, significance to the field, and usefulness to practitioners.

The small-scale evaluation was deliberately modest: two researchers independently produced outputs for two comparable domains in the synthesis task, and for six interviews across two sites covering four codes in the coding task. The threshold for scaling up was that variation between human-only and human-LLM outputs resembled what you would expect between two human researchers.

That benchmark is worth pausing on. It reframes the validation question from "does the model match a gold standard?" to "is the model's disagreement within the range of normal human disagreement?" — which is closer to how qualitative methodologists actually think about coding variation. It also sets a low bar if human variation is itself wide, which is why the authors pair it with the substantive rigor criteria above.

How much time did it actually save?

The reported gains differ sharply by task, and that difference is the study's most practical finding.

Task	Scale	Reported effect
Qualitative synthesis of summaries	~31,200 words, 22 domains	30–55% time reduction per researcher
Deductive coding of transcripts	167 transcripts, ~1.33M words	Manual estimate ~93 workweeks across 12 sites; LLM assistance made a nine-month post-collection deadline feasible

The synthesis figure is an efficiency gain. The coding figure is a feasibility gain — the analysis was not going to happen at all on that timeline without assistance. Those are different arguments, and teams evaluating AI-assisted analysis should be clear about which one they are making to their stakeholders.

Where did the LLM fail?

The authors report a specific and recognizable set of failure modes. Retrieval-augmented generation, which made the 1.33-million-word corpus tractable, also produced fragmented and decontextualized understanding — the model saw retrieved passages, not the arc of an interview. Related problems included overgeneralizing from a single data point, overlooking surrounding context, drifting toward generic descriptive summary, and showing example bias and positivity bias in responses.

One limitation is methodological rather than technical: when the model reads the transcripts, the researcher does not. Reduced familiarity with raw data is a real cost in interpretive work, and it does not show up in any accuracy metric. The authors are explicit that humans remained essential for final interpretation, contextual understanding, and methodological transparency.

What this means for researchers

Three practical takeaways follow from this study:

Pilot manually before you automate. The framework's first step exists because you cannot specify an output you have never produced yourself.
Match the validation to the claim. If your analysis is deductive and framework-driven, human-vs-human variation is a defensible benchmark. If it is inductive and interpretive, expect to defend more.
Budget for reading. Deliberately keep a subset of transcripts that a human reads end to end, so the team retains contextual grounding the retrieval pipeline strips away.

If you are running framework-based coding at this scale, tools like ThemeLens handle AI-assisted thematic analysis with the coding trail kept visible, and Qualitati's AI Interviewer covers the data-collection side when transcript volume is the bottleneck.

FAQ

Can LLMs do deductive coding reliably?

This study coded 167 transcripts against a 19-area framework and validated by comparing human-LLM output to human-only output on a six-interview subset, accepting it when the variation resembled ordinary between-researcher variation. That is evidence for feasibility under human oversight, not for unsupervised coding.

Does retrieval-augmented generation help or hurt qualitative analysis?

Both. RAG is what made a 1.33-million-word corpus tractable, but Ronaghi et al. (2026) report it also produced fragmented, decontextualized readings because the model works from retrieved passages rather than the whole interview.

Do I need inter-rater reliability statistics for AI-assisted coding?

Not necessarily. This team used qualitative rigor criteria — data grounding, theory integration, question alignment, relevance, practitioner usefulness — instead of a reliability coefficient. Which is appropriate depends on your method and your reviewers.

Is a 30–55% time saving typical?

That figure applies to the synthesis task in this study only, where researchers were reorganizing their own summaries. The coding task was framed as a feasibility gain rather than a percentage saving.

Primary source: Ronaghi, S., Aveling, E.-L., Levis, M., Ross, R. L., Alsentzer, E., & Singer, S. (2026). Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research. arXiv preprint. https://arxiv.org/abs/2601.14478

Last updated: July 24, 2026

This is an independent editorial summary of third-party research. Qualitati is not affiliated with the authors, and readers should consult the original preprint for full methodological detail.