Short answer: Yes, with limits. In a 2026 study, Ivey and colleagues built InterviewPlayground, a simulation environment where AI interviewers question theory-grounded simulated participants. Scores from the simulation predicted how the same five AI interviewers performed with 450 real participants, averaging a Pearson correlation of 0.86 across 12 measures. It was weakest at predicting comfort and overall experience.
Last updated: November 9, 2026
Anyone building or choosing an AI interviewer faces the same problem. A good interview is a long, adaptive conversation, so you cannot judge an interviewer from a single prompt and response. Testing with real people works, but it is slow and expensive, so most published evaluations rest on one small pilot in one domain. Simulated participants are the obvious shortcut, but simulations that have never been checked against real data can mislead (the Sim2Real gap). The preprint InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers tests that shortcut directly by running matched simulated and human studies.
What did the InterviewPlayground study test?
The study asked whether an AI interviewer's scores in a simulation predict its scores with real people. According to Ivey, Liang, Wang, Mandell, Xiao and Field (Johns Hopkins University and Listen Labs, arXiv preprint, October 2026), the team ran five published AI interviewers through three simulated studies, then ran the same 15 interviewer-topic combinations as real qualitative studies with 450 human participants (30 per study, up to 30 minutes each).
The three topics came from published interview studies: obesity and weight management, Asian American identity and politics, and generative AI in knowledge work. They differ in how sensitive and how concrete they are. All five interviewers ran on GPT-5.4 mini, and human participants were recruited through Listen Labs and paid $15 per hour.
How are the simulated participants built?
Each simulated participant is a persona plus a memory bank, shaped by six behavioral traits taken from the interviewing literature. The authors reviewed qualitative-methods research, found 20 participant characteristics that affect how interviews go, and merged them into six traits:
- Knowledge: how much experience the person has with the topic.
- Understanding: how well they grasp the questions and the purpose of the interview.
- Reflexivity: how far they can reflect on and analyze themselves.
- Memory: how accurately they recall experiences.
- Verbosity: how long their answers are.
- Disclosure: how willing they are to share sensitive information.
Each participant gets 200 generated memories. Most are unrelated to the topic, and six hold study-relevant insights drawn from the original published research. When the interviewer asks a question, the five most relevant memories are retrieved, with noise added for participants with weaker memory. This design matters: the simulation does not try to guess what real people think. It plants known insights and measures whether the interviewer can draw them out.
How well did simulation scores predict real interviews?
For most measures, very well. The authors scored every interview with InterviewReportCard, a set of LLM-judge measures covering participant responses, interviewer behavior and participant experience, plus two descriptive statistics. Ten of the 12 measures showed strong correlations between simulated and human studies (r = 0.74 to 1.00).
| Measure | What it captures | Pearson r (simulated vs human) |
|---|---|---|
| Support and rapport | Encouraging, rapport-building statements | 1.00 |
| Leading questions | Questions that suggest an answer or embed an assumption | 0.98 |
| Interview guide coverage | Share of planned subtopics actually addressed | 0.95 |
| Coherence | Logical flow and transitions between questions | 0.95 |
| Average turns | Conversation length | 0.94 |
| Adaptiveness | Questions that respond to what was just said | 0.92 |
| Novel responses | Relevant material outside the planned guide | 0.90 |
| Unclear questions | Confusing or ambiguous questions | 0.85 |
| Average response length | Words per participant answer | 0.85 |
| Relevant response volume | Total research-relevant material elicited | 0.74 |
| Overall experience | Participant rating of the interview | 0.67 |
| Comfort level | Feeling comfortable and fairly treated | 0.60 |
Each correlation rests on only 15 paired studies, so the confidence intervals are wide. For overall experience, the reported interval runs from 0.25 to 0.88. The ranking of interviewers was similar when the authors used Spearman correlations.
Are the simulated participants realistic?
Expert raters could not tell them apart on realism. Three annotators with qualitative-research experience rated 100 excerpts, half human and half simulated. Simulated participants averaged 3.51 on a 4-point realism scale against 3.34 for humans, a difference that was not significant (p = 0.28). The experts could identify five of the six intended traits from transcripts but not memory, which is expected because they could not see the memory bank.
Can the LLM judges be trusted?
Roughly as much as a human coder. Across 1,800 expert judgments, agreement among the human annotators ranged from 0.62 to 0.70 (Krippendorff's alpha), and agreement between the median human rating and the LLM judge ranged from 0.60 to 0.80. For guide coverage, the judge's topic sets matched the humans' with an average Jaccard similarity of 0.69.
Does it work with small, cheap models?
Mostly yes. The authors reran the simulated studies with seven simulator models. Gemini 3.1 Pro and Gemini 3.7 Flash both averaged 0.86, and Qwen 3.5 4B averaged 0.83 across all measures. The smallest model tested, Qwen 3.5 0.8B, dropped to 0.62. The authors suggest using small models for early development and larger ones for final evaluations.
What failure modes did the simulation catch?
Beyond the numbers, the simulated transcripts showed the same behaviors seen with real people:
- Repetitive questioning was the most common cause of poor experience in both settings. Human and simulated participants complained in very similar ways when an interviewer kept asking the same thing after a weak answer.
- Specific, guide-anchored probing explained why the best interviewer on coverage (LLMRoleplay) succeeded. It asked concrete follow-ups for each subtopic, probed when detail was missing and moved on cleanly once a topic was covered.
- Discomfort on sensitive topics showed up with low-disclosure participants in both settings, but it was expressed differently. Simulated participants said outright that they did not want to go into it, which made interviewers move on. Real participants tended to stay quiet in the interview and flag the problem later in the post-survey.
That last gap is the clearest warning in the paper. Simulated people say out loud what real people keep to themselves, so a simulation can make an interviewer look more sensitive than it really is.
What this means for researchers
The study supports a two-stage workflow: test in simulation, then confirm with people.
- Use simulation to compare and debug interviewers. Coverage, coherence, adaptiveness, leading and unclear questions all transferred well, so simulation is a reasonable way to catch repetition, weak probing or poor transitions before you spend participant budget.
- Do not use simulation to certify participant experience. Comfort and satisfaction were the weakest predictions, and sensitive topics are exactly where simulated and real participants diverge.
- Keep a human pilot. A short pilot with real participants, as in our guide to pilot-testing an AI interview guide, is still the check that catches what simulations miss.
- Remember the scope. The validation covers three topics, five interviewers and one interviewer base model, and the authors present the three settings, not every possible study, as validated.
Disclosure: one of the five interviewers tested, the system described by Liu & Yu (2025), was developed by Qualitati's founders and underlies the Qualitati AI Interviewer. The paper's text does not single out that interviewer's results, and this summary makes no claim about its performance. Two of the paper's authors are affiliated with Listen Labs, which also provided participant recruitment.
FAQ
What is InterviewPlayground?
It is an open-source simulation environment, released as a Python package, for evaluating AI interviewers. Interviewers question simulated participants whose behavior follows six traits from interviewing research, and the transcripts are scored with the InterviewReportCard measures.
Can simulated participants replace human pilots for AI interviews?
No. In this study they predicted interviewer behavior and response quality well, but comfort (r = 0.60) and overall experience (r = 0.67) were predicted less reliably, and real participants expressed discomfort differently from simulated ones.
How many people took part in the human studies?
450 participants across 15 studies: five AI interviewers times three topics, with 30 participants each and interviews of up to 30 minutes.
Is the paper peer-reviewed?
As of November 2026 it is an arXiv preprint (version 2, October 2026). Treat the findings as promising but not yet peer-reviewed.
Source: Ivey, J., Liang, A., Wang, A. Y. S., Mandell, M., Xiao, Z., & Field, A. (2026). InterviewPlayground: A Validated Simulation Environment for Evaluating AI Interviewers. arXiv:2610.12023.
Last updated: November 9, 2026. This article is an independent editorial summary of third-party research by the Qualitati Research Team. It is not affiliated with or endorsed by the paper's authors.