Short answer: Mostly not yet. InterviewBench, a 2026 benchmark of 23 large language models as interviewers, found that models ask every question on a guide but often fail to get a clear, usable answer. In simulated interviews, the best models covered 100% of the questions yet elicited the target information only about half to two-thirds of the time.
Last updated: October 28, 2026
AI-moderated interviews are now routine in UX, market and academic research, and most evaluations ask one question: does the conversation feel natural? A new benchmark from Fudan University argues that is the wrong test. Fluency is easy for modern models; getting a respondent to say something specific and recordable is hard. This post summarizes what the study measured, what it found, and what it means if you run interviews with an AI moderator.
What is InterviewBench?
InterviewBench is a benchmark that scores large language models on the skills an interviewer needs, not just on how fluent their questions sound. It was published by Shangzhe Dai, Feiyu Duan and Zhongyu Wei in Findings of the Association for Computational Linguistics: EMNLP 2026 (October 2026), and the code is public.
The authors break interviewing into three capabilities:
- Information guidance — asking the right main question and following up on the informative part of an answer.
- Situated adaptation — handling answers that are vague, emotional, contradictory, off-topic or a refusal without derailing the conversation.
- Information extraction — turning the conversation into a structured record afterwards.
The benchmark is built from real broadcast interview transcripts (the INTERVIEW NPR media-dialog corpus) and contains 5,500 multiple-choice items, 3,000 open-generation items and 184 multi-turn interview scenarios.
How did the study test AI interviewers?
The study used two settings: a static test of single interviewer decisions and a dynamic test of full interviews with a simulated respondent.
| Setting | What the model does | What is scored |
|---|---|---|
| Static: multiple choice | Picks the best next move from four options | Accuracy against the real interviewer's move |
| Static: conditioned generation | Writes the next turn when told which move to make (e.g. "follow up") | LLM-judge score, 0–100 |
| Static: unconditioned generation | Writes the next turn and must decide the move itself | LLM-judge score, 0–100 |
| Dynamic interview | Runs a questionnaire-guided interview of up to 20 turns, then fills in the record | Completion, efficiency, "spoken truth", backfill accuracy, respondent satisfaction |
Each static task covers four interviewer behaviours: main questioning, follow-up, responding to the participant, and reacting to a disruptive event. In the dynamic setting, four workflows were compared with increasing autonomy: Structured (asks the guide in order), Probing (adds follow-ups to unclear answers), Adaptive (chooses between questions, probes and responses) and Agentic (also decides how to handle events and when to end).
Which LLM is the best interviewer?
Proprietary frontier models led the static tests. According to Dai, Duan and Wei (2026), Gemini-3.1-Pro-Preview averaged 85.1 and Claude Sonnet 4.6 averaged 83.0 across all static tasks, ahead of GPT-5.4 at 77.2. The strongest open-source models were GLM-5 (76.9), DeepSeek-V4-Pro (74.9) and Qwen3.5-122B (73.6). The smallest models tested, such as Qwen3-1.7B, scored around 47.
The more useful finding is where models fail. Almost every model wrote a better interviewer turn when it was told which move to make than when it had to choose the move itself. In other words, models can phrase a good follow-up; they are worse at knowing that a follow-up is what the moment calls for. The gap was stark on the multiple-choice "response" task, where Gemini-3.1-Pro scored 94.9 but GPT-4o scored 12.8 and Qwen3-max 12.0, even though both wrote fluent responses when the move was specified.
Do AI interviewers actually collect the information?
Only partly. In the dynamic interviews, nearly every model and workflow reached 100% of the guide's questions, but the share of target answers that were actually stated in the conversation — what the authors call "spoken truth" — was far lower. For Gemini-3.1-Pro it ranged from 54.4% (Agentic) to 65.4% (Structured); for Gemma-3-27B it fell as low as 39.1%.
The authors locate the failure precisely. Once a respondent did state the answer, the stronger models converted it into the structured record with over 97% accuracy. The bottleneck is elicitation: the interviewer asks the question, gets a hedge or a partial answer, and moves on.
Does giving the AI more autonomy help?
No — in this benchmark, more autonomy made interviews worse. The Structured and Probing workflows were the most stable, with efficiency around 85 in clean conditions. Agentic workflows dropped to roughly 61–72 on efficiency, and for every model they elicited less target information than the Structured workflow. DeepSeek-V4-Pro in the Adaptive workflow was the extreme case, with efficiency of 32.9 and spoken truth of 20.0; the authors attribute such failures to wasted turns, loops and stopping early.
What happens when a participant misunderstands or refuses?
The interview still finishes, but it gets worse for the respondent. When the authors injected a single disruptive event — a misunderstanding, refusal or contradiction — completion barely moved (99.94% to 99.90%), and backfill accuracy on stated answers changed by only 0.3 points. Simulated respondent satisfaction, however, fell by 8.9 points on average. The models kept going without repairing the interaction.
What are the study's limitations?
The results come from simulated respondents, and that matters. A sensitivity test showed that swapping the simulated respondent changed absolute scores (for one interviewer in the Structured workflow, direct backfill ranged from 77.0% to 86.7% depending on the respondent model), though the ranking of workflows held. The source data are English-language radio interviews, so clinical, hiring, fieldwork and cross-cultural interviews are not covered, and the benchmark is text-only, so it ignores pauses, tone and other voice cues. Judging was also done by LLMs, although the authors report high agreement between two judge models (Pearson 0.956 on conditioned generation) and a human audit.
What this means for researchers using AI interviewers
The practical message is that "the AI asked all my questions" is not evidence of good data. Four takeaways:
- Audit answers, not coverage. When piloting an AI-moderated study, check each transcript for whether the key questions received a clear, specific answer, not just whether they were asked.
- Prefer structure with probing over full autonomy. A guide-driven flow with explicit follow-up rules beat a free-roaming agent on every model tested.
- Write probes into the guide. Because models do better when told which move to make, specifying when to probe (for example, "if the answer is vague, ask for a concrete example") plays to their strength.
- Plan for difficult moments. Refusals and misunderstandings rarely stop an AI interview, but they lower the respondent's experience; review those segments in your pilot.
If you are setting up AI-moderated interviews, the QualiTaTi AI Interviewer lets you write follow-up instructions into the interview guide and review every transcript before analysis.
FAQ
What is InterviewBench?
InterviewBench is a 2026 benchmark from Fudan University that evaluates large language models as interviewers using 5,500 multiple-choice items, 3,000 generation items and 184 simulated interviews built from real interview transcripts.
Which AI model was the best interviewer in InterviewBench?
Gemini-3.1-Pro-Preview had the highest static average (85.1), followed closely by Claude Sonnet 4.6 (83.0). GLM-5 was the strongest open-source model (76.9).
Are AI interviewers good at follow-up questions?
They write good follow-ups when instructed to, but they are weaker at deciding on their own when a follow-up is needed. That is why explicit probing rules in the interview guide help.
Is a more autonomous AI interviewer better?
Not according to this study. Agentic workflows were less efficient and elicited less of the target information than structured, guide-led workflows.
Source: Dai, S., Duan, F., & Wei, Z. (2026). InterviewBench: Benchmarking Large Language Models as Interviewers. Findings of the Association for Computational Linguistics: EMNLP 2026, 45731–45787.
Last updated: October 28, 2026. This is an independent editorial summary of third-party research by the Qualitati Research Team; it is not affiliated with or endorsed by the paper's authors.