Short answer: Yes, with limits. In a 2026 study, Ivey and colleagues built InterviewPlayground, a simulation environment where AI interviewers question theory-grounded simulated participants. Scores from the simulation predicted how the same five AI interviewers performed with 450 real participants, averaging a Pearson correlation of 0.86 across 12 measures. It was weakest at predicting comfort and overall experience.
Last updated: November 9, 2026
\n\nAnyone building or choosing an AI interviewer faces the same problem. A good interview is a long, adaptive conversation, so you cannot judge an interviewer from a single prompt and response. Testing with real people works, but it is slow and expensive, so most published evaluations rest on one small pilot in one domain. Simulated participants are the obvious shortcut, but simulations that have never been checked against real data can mislead (the Sim2Real gap). The preprint InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers tests that shortcut directly by running matched simulated and human studies.
\n\nThe study asked whether an AI interviewer's scores in a simulation predict its scores with real people. According to Ivey, Liang, Wang, Mandell, Xiao and Field (Johns Hopkins University and Listen Labs, arXiv preprint, October 2026), the team ran five published AI interviewers through three simulated studies, then ran the same 15 interviewer-topic combinations as real qualitative studies with 450 human participants (30 per study, up to 30 minutes each).
\n\nThe three topics came from published interview studies: obesity and weight management, Asian American identity and politics, and generative AI in knowledge work. They differ in how sensitive and how concrete they are. All five interviewers ran on GPT-5.4 mini, and human participants were recruited through Listen Labs and paid $15 per hour.
\n\nEach simulated participant is a persona plus a memory bank, shaped by six behavioral traits taken from the interviewing literature. The authors reviewed qualitative-methods research, found 20 participant characteristics that affect how interviews go, and merged them into six traits:
\n\nEach participant gets 200 generated memories. Most are unrelated to the topic, and six hold study-relevant insights drawn from the original published research. When the interviewer asks a question, the five most relevant memories are retrieved, with noise added for participants with weaker memory. This design matters: the simulation does not try to guess what real people think. It plants known insights and measures whether the interviewer can draw them out.
\n\nFor most measures, very well. The authors scored every interview with InterviewReportCard, a set of LLM-judge measures covering participant responses, interviewer behavior and participant experience, plus two descriptive statistics. Ten of the 12 measures showed strong correlations between simulated and human studies (r = 0.74 to 1.00).
\n\n| Measure | What it captures | Pearson r (simulated vs human) |
|---|---|---|
| Support and rapport | Encouraging, rapport-building statements | 1.00 |
| Leading questions | Questions that suggest an answer or embed an assumption | 0.98 |
| Interview guide coverage | Share of planned subtopics actually addressed | 0.95 |
| Coherence | Logical flow and transitions between questions | 0.95 |
| Average turns | Conversation length | 0.94 |
| Adaptiveness | Questions that respond to what was just said | 0.92 |
| Novel responses | Relevant material outside the planned guide | 0.90 |
| Unclear questions | Confusing or ambiguous questions | 0.85 |
| Average response length | Words per participant answer | 0.85 |
| Relevant response volume | Total research-relevant material elicited | 0.74 |
| Overall experience | Participant rating of the interview | 0.67 |
| Comfort level | Feeling comfortable and fairly treated | 0.60 |
Each correlation rests on only 15 paired studies, so the confidence intervals are wide. For overall experience, the reported interval runs from 0.25 to 0.88. The ranking of interviewers was similar when the authors used Spearman correlations.
\n\nExpert raters could not tell them apart on realism. Three annotators with qualitative-research experience rated 100 excerpts, half human and half simulated. Simulated participants averaged 3.51 on a 4-point realism scale against 3.34 for humans, a difference that was not significant (p = 0.28). The experts could identify five of the six intended traits from transcripts but not memory, which is expected because they could not see the memory bank.
\n\nRoughly as much as a human coder. Across 1,800 expert judgments, agreement among the human annotators ranged from 0.62 to 0.70 (Krippendorff's alpha), and agreement between the median human rating and the LLM judge ranged from 0.60 to 0.80. For guide coverage, the judge's topic sets matched the humans' with an average Jaccard similarity of 0.69.
\n\nMostly yes. The authors reran the simulated studies with seven simulator models. Gemini 3.1 Pro and Gemini 3.7 Flash both averaged 0.86, and Qwen 3.5 4B averaged 0.83 across all measures. The smallest model tested, Qwen 3.5 0.8B, dropped to 0.62. The authors suggest using small models for early development and larger ones for final evaluations.
\n\nBeyond the numbers, the simulated transcripts showed the same behaviors seen with real people:
\n\nThat last gap is the clearest warning in the paper. Simulated people say out loud what real people keep to themselves, so a simulation can make an interviewer look more sensitive than it really is.
\n\nThe study supports a two-stage workflow: test in simulation, then confirm with people.
\n\nDisclosure: one of the five interviewers tested, the system described by Liu & Yu (2025), was developed by Qualitati's founders and underlies the Qualitati AI Interviewer. The paper's text does not single out that interviewer's results, and this summary makes no claim about its performance. Two of the paper's authors are affiliated with Listen Labs, which also provided participant recruitment.
\n\nIt is an open-source simulation environment, released as a Python package, for evaluating AI interviewers. Interviewers question simulated participants whose behavior follows six traits from interviewing research, and the transcripts are scored with the InterviewReportCard measures.
\n\nNo. In this study they predicted interviewer behavior and response quality well, but comfort (r = 0.60) and overall experience (r = 0.67) were predicted less reliably, and real participants expressed discomfort differently from simulated ones.
\n\n450 participants across 15 studies: five AI interviewers times three topics, with 30 participants each and interviews of up to 30 minutes.
\n\nAs of November 2026 it is an arXiv preprint (version 2, October 2026). Treat the findings as promising but not yet peer-reviewed.
\n\nSource: Ivey, J., Liang, A., Wang, A. Y. S., Mandell, M., Xiao, Z., & Field, A. (2026). InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers. arXiv:2610.12023.
\n\nLast updated: November 9, 2026. This article is an independent editorial summary of third-party research by the Qualitati Research Team. It is not affiliated with or endorsed by the paper's authors.
\n","related":[{"slug":"structuring-survey-histories-llm-digital-twins-2026","title":"How to Structure Survey Data for LLM Digital Twins (2026)","description":"How should you structure survey data for LLM digital twins? A 2026 Chicago Booth study found preference profiles beat raw transcripts, mostly on pricing.","category":"AI User Research","date":"2026-11-10","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-user-simulators-sim2real-gap-2026","title":"Can LLMs Simulate Real Users? The Sim2Real Gap (2026 Study)","description":"Can LLMs simulate real users? A 2026 study compared 451 people with 31 LLM user simulators and found synthetic users too cooperative and too positive.","category":"AI User Research","date":"2026-11-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"interviewbench-ai-interviewer-llm-benchmark-2026","title":"Do AI Interviewers Get Real Answers? InterviewBench (2026)","description":"InterviewBench tested 23 LLMs as AI interviewers: they ask every question but often fail to elicit clear answers. Key results and what researchers should do.","category":"AI User Research","date":"2026-10-28","readTime":"7 min read","author":"Qualitati Research Team"}]}