How Often Do LLMs Fabricate Interview Quotes? 2026 Study
Qualitati Research Team · 2026-08-21 · 7 min read
When an LLM pulls a supporting quote out of an interview transcript, it is usually — but not always — quoting. In a 2026 JAMIA study of 122 heart failure transcripts, 68% of LLM-supplied quotations were verbatim, 20% were paraphrased, and 9% were hallucinated in whole or in part. Themes were reliable; the evidence attached to them was not.
What did the study test?
Bang and colleagues (2026) ran the same corpus through two complete qualitative workflows and compared them. The corpus was 122 transcripts — 61 clinical consultations and 61 reflexive interviews — collected from patients with heart failure. Both workflows used a hybrid deductive–inductive design: start from an a priori framework, then let new codes emerge from the data.
- Human-led arm. Trained qualitative researchers coded the transcripts in Dedoose using conventional thematic analysis.
- LLM-assisted arm. The same corpus was analyzed with ChatGPT Edu (GPT-5.2) inside an institutionally compliant environment, driven by a fixed eleven-prompt protocol rather than ad hoc chatting.
- Comparison arm. The team compared 63 human–LLM theme pairs, judged the validity of every LLM theme, and then manually verified each quotation the model offered as evidence.
That third step is what makes the paper unusual. Most evaluations of AI qualitative analysis stop at whether the themes look right. This one went back to the transcript and checked whether the sentences the model attributed to participants had actually been said.
How well did the AI themes match the human themes?
Well enough to be usable, and not so well as to be redundant. According to Bang et al. (2026), the human-led analysis produced 7 themes and the LLM-assisted analysis produced 9. Overlap between the two theme sets was moderate to substantial, with Jaccard coefficients between 0.44 and 0.51 and a hit rate of 1.00 — meaning every human theme had a corresponding LLM theme somewhere in the model's output.
Critically, the reviewing researchers judged all 9 LLM themes to be valid. Reconciling both sets produced a final structure of 14 themes across three domains. The model was not inventing themes that did not belong; it was cutting the same material at somewhat different joints, and a few of its cuts were ones the human team had not made.
How often did the model fabricate quotes?
Roughly one excerpt in eleven was not a real quotation. The quotation fidelity audit in Bang et al. (2026) broke down as follows.
| Excerpt category | Share of quotations | What it means in practice |
|---|
| Verbatim | 68% | Safe to quote as-is after a spot check |
| Paraphrased | 20% | Meaning preserved, wording changed — cannot be published in quotation marks |
| Partial hallucination | 6% | Real utterance with invented words spliced in |
| Full hallucination | 3% | No corresponding utterance in the transcript |
| Truncated | 3% | Cut in a way that can distort meaning |
The partial hallucinations are the dangerous category. A wholly invented quote often reads as slightly off-register and gets caught. A real participant sentence with three or four extra words grafted onto it reads perfectly natural, sits under a theme that is genuinely supported, and will survive any review that does not involve reopening the transcript.
Why do the themes hold up when the quotes do not?
Because the two tasks fail differently. Theme construction is a compression task: the model reads a large body of text and returns an abstraction, and abstractions are forgiving — a slightly differently worded theme is still recognizably the same theme. Quotation is a retrieval task with an exact-match success criterion, and a generative model is not a retrieval index. When it produces a quote, it is producing text conditioned on the transcript, not copying a span out of it. Fluent reconstruction is the default behavior, not a malfunction.
The practical consequence: the trustworthiness of an AI qualitative analysis is not a single number. The thematic layer and the evidentiary layer need to be validated separately, and only the second one requires going back to the source.
What this means for researchers
The authors' own conclusion is that LLMs are best positioned as analytic partners rather than autonomous coders, and that human oversight remains essential for contextual interpretation and verification. Four operational implications follow.
- Verify every quote you intend to publish. Not a sample — every one. At a 9% hallucination rate, a paper with 20 illustrative quotes is likely to carry one or two fabrications.
- Prefer tools that anchor quotes to transcript locations. A system that returns a span with a timestamp or line reference can be checked mechanically. A system that returns free text cannot. This is a tooling choice, not a prompting one — no prompt reliably eliminates the behavior. It is why ThemeLens ties each supporting excerpt back to its position in the source transcript.
- Use a fixed protocol, not a conversation. The study's eleven-prompt structure is what makes its results reproducible. Ad hoc prompting produces analyses no one else can repeat.
- Treat AI themes as a second coder, not a replacement. The hit rate of 1.00 says the model will find what your team found. The two extra themes say it may also surface something you missed — which is a reason to run both arms, not to drop one.
Transcript quality sets the ceiling on all of this. A model cannot return a verbatim quote from a transcript that garbled the utterance in the first place, which is why accurate capture at the interview stage compounds through every downstream step.
Limits of this evidence
This is one corpus, in one clinical domain, with one model. Heart failure consultations are structured, vocabulary-dense conversations; hallucination rates on rambling, emotionally complex, or multilingual interview data could differ in either direction. The study also used a compliant institutional deployment of GPT-5.2 — results say nothing about consumer chatbots handling identifiable health data, which raises separate governance problems entirely. And because both arms analyzed the same corpus, the comparison shows convergence, not accuracy against an external ground truth.
FAQ
Can LLMs be trusted to do thematic analysis?
For generating and organizing themes, the evidence here is encouraging: all 9 LLM themes were judged valid, and every human theme was recovered. For supplying verbatim evidence, no — 32% of quotations were not verbatim.
What is a hybrid deductive–inductive workflow?
An analysis that begins with a predefined coding framework drawn from theory or prior work, then allows additional codes to emerge inductively from the data. It is the design both arms of this study used.
How do I check AI-generated quotes efficiently?
Search the exact string in your transcript file. Exact-match failure is the signal — if the string is not present character-for-character, the excerpt is paraphrased, truncated, or fabricated, and cannot be published inside quotation marks.
Does this mean AI-assisted qualitative research is not publishable?
No. It means the verification step is not optional and should be described in your methods section, alongside the prompt protocol used.
Primary source: Bang, S.H., Han, S., Reading Turchioe, M., Ellison, M., Dai, S., Happ, M.B., Russell, D., & Masterson Creber, R. (2026). Developing and evaluating human-led and large language model–assisted hybrid deductive–inductive workflows for qualitative analysis. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag123
Last updated: 2026-08-21
This is an independent editorial summary of third-party research. Qualitati is not affiliated with the authors or their institutions.