Can Interview Transcripts Predict Survey Answers? A 2026 LLM Test

Qualitati Research Team · 2026-10-12 · 7 min read

Short answer: Partly. A 2026 study in the Journal of Educational Data Mining fed 19 participants' interview transcripts to GPT-4.1, Claude 3.7 Sonnet and Gemini 2.0 Flash and asked them to fill in a 15-item questionnaire as each person. The models matched the overall answer pattern but produced answers that were too uniform and too extreme, and they did not reproduce the scale's psychometric structure.

Many mixed-methods studies collect two kinds of data from the same people: an interview and a questionnaire. Researchers then have to reconcile them by hand. A team from the University of Arkansas, the University of Wyoming and the University of South Carolina tested a shortcut: give an LLM a participant's interview transcript and ask it to answer the survey as that participant would. If the synthetic answers line up with the real ones, the model has extracted something meaningful from the qualitative data. Where they diverge, the gap may point to a weak item or an inconsistent respondent. This post summarizes what the study tested, what it found, and what it means for anyone building interview-grounded personas or digital twins.

What did the study test?

The study tested whether interview-informed LLMs can reproduce individual survey answers, using a full factorial design across models, prompts and temperature. According to Zhang, Liang, Deng, Bonge, Tan, Zhang and Zarrett (2026), the data came from after-school program staff in the Southeastern United States who took part in a physical-activity trial between 2023 and 2024. Nineteen staff members completed both a semi-structured interview, lasting roughly 15 to 25 minutes, and the Behavioral Regulation in Exercise Questionnaire (BREQ), a 15-item, 6-point scale with four motivation subscales.

The design crossed three factors:

Model: OpenAI GPT-4.1, Anthropic Claude 3.7 Sonnet and Google Gemini 2.0 Flash.
Temperature: 0 (deterministic) and 0.5. In pilot runs, settings above 0.5, such as 0.7, produced outputs too loose to yield clean item-level answers.
Prompt content: four prompts built from research background, item wording, the participant's interview transcript and their demographics.

That gives 24 conditions. With one generated answer per participant per item, the authors analyzed 6,840 synthetic responses and compared them with the human answers at the item, person and whole-test level.

Which prompt worked best?

Prompts that included the interview transcript aligned best with human answers, and adding demographics on top contributed little. The table below summarizes the four conditions as the paper reports them.

Prompt	What the model saw	Average length	What the study found
Prompt 1 (baseline)	Research background + survey items	1,276 tokens	Answers nearly identical across "participants", so variance was low to zero
Prompt 2	Baseline + interview transcript	mid-length (Prompts 2 and 3 averaged 5,917 tokens)	Higher correlation with human answers; more varied output for Claude and GPT
Prompt 3	Baseline + demographics only	mid-length	Only slightly better than the baseline
Prompt 4	Baseline + transcript + demographics	longest (mean 5,952 tokens)	Comparable to Prompt 2; demographics added minimal improvement

The practical reading: what a person says about their own experience carries more signal than their age, gender or race. That matches a pattern seen elsewhere in persona research, where thin demographic personas tend to collapse toward a stereotype.

How accurate were the synthetic answers?

Accuracy was moderate. Per the study, average correlations between LLM-generated and human responses ranged from about 0.5 to 0.73. Claude showed the highest correlation with humans under every prompt, followed by GPT, with Gemini lowest. Claude also had the lowest item-level and person-level error.

The models agreed with each other far more than they agreed with people. GPT and Claude correlated between 0.92 and 0.95 across conditions, GPT and Gemini between 0.87 and 0.93, and Claude and Gemini between 0.81 and 0.88. High agreement between models is therefore not evidence that either model is right about the human, which is worth remembering whenever a vendor uses cross-model consistency as a validity argument.

Where did the LLMs go wrong?

The study identified four failure patterns, and each one matters for anyone using synthetic respondents.

Too little variance. Variances of LLM-generated answers were much lower than human variances in every condition. The models captured the average and missed the spread.
Too extreme. Where humans answered low, the models answered lower; where humans answered high, the models answered higher. The authors describe this as consistently extreme values.
Negative emotional wording. The three items with the largest errors used words like "ashamed", "failure" and "restless". Humans gave middling answers on these items; the models gave extreme ones.
Broken scale structure. At the whole-test level, measured with the relative autonomy index that weights the four BREQ subscales, only Claude with the richer prompts beat the baseline. The authors conclude that the models can mimic item patterns but struggle to reproduce the questionnaire's psychometric relationships.

Longer interviews did not reliably help either. The correlation between transcript length and person-level error was 0.404 with p = .086, which the authors treat as inconclusive; they suggest the relevance of the interview content matters more than its length.

What this means for researchers

The study's own framing is the most useful takeaway: interview-informed LLM answers are a diagnostic tool for mixed methods, not a replacement for respondents. Here is how to apply that.

Use the gap, not just the fit. When a synthetic answer built from someone's interview disagrees sharply with their survey answer, flag the item or the respondent for a closer look. The paper suggests such gaps can expose items that do not capture what people actually think.
Ground personas in what people said. If you build digital twins, interview transcripts beat demographic sketches. Tools like the Digital Twin Panel are most defensible when each twin is anchored to real qualitative data and validated against held-out answers.
Collect interviews that cover the constructs. Since relevance outweighed length, an interview guide that touches each construct you later measure will feed a twin better than a longer, wandering conversation. A structured AI Interviewer guide can help keep coverage consistent across participants.
Never treat synthetic variance as real. Any analysis that depends on spread, subgroup differences or scale reliability should come from human data.
Report the models and settings. Results differed noticeably by model, so name the model version, temperature and prompt components in your methods section.

How strong is the evidence?

The evidence is careful but small. Nineteen participants, one instrument, one population and one generated answer per condition limit how far the results generalize, and the authors say so. The models tested (GPT-4.1, Claude 3.7 Sonnet, Gemini 2.0 Flash) are also a generation behind the current frontier. The direction of the findings, though, lines up with larger synthetic-respondent studies: averages are reachable, variability and structure are not.

FAQ

Can an LLM answer a survey on behalf of someone it has only interviewed?

It can approximate the overall pattern. In this 2026 study, correlations with real answers ranged from about 0.5 to 0.73, but the synthetic answers were less varied and more extreme than the person's own.

Do demographics improve LLM persona accuracy?

Not much in this study. Adding age, gender and race on top of an interview transcript produced minimal improvement; the transcript did most of the work.

Which model performed best?

Claude 3.7 Sonnet had the highest correlations with humans and the lowest error, followed by GPT-4.1, with Gemini 2.0 Flash lowest. Newer models may behave differently.

Should I replace survey respondents with interview-based digital twins?

No. The authors position the method as a way to triangulate qualitative and quantitative data and spot measurement problems, not to substitute for human responses.

Source: Zhang, J., Liang, X., Deng, A., Bonge, N., Tan, L., Zhang, L., & Zarrett, N. (2026). Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI-Generated and Human Data. Journal of Educational Data Mining, 18(1), 1–24. Read the paper.

Last updated: October 12, 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the authors.