Which LLM Is the Best AI Interviewer? A 2026 Benchmark
Qualitati Research Team · 2026-09-01 · 7 min read
Different large language models make measurably different interviewers. In a controlled 2026 benchmark published in Scientific Reports, six frontier models ran the same 54-question semi-structured interview protocol. Expert psycholinguists scored over 2,900 items and found no single winner — only trade-offs between empathy, thoroughness, speed and reliability.
What did the study test?
The researchers built a modular LLM interviewing agent and ran it across six models under identical conditions. According to Panfilova and colleagues (2026), the agent worked through a fixed battery of 54 main questions covering biography, family, interests, challenges, values, work and health. After each answer it decided whether a follow-up was warranted, and if so generated one — capped at five follow-ups per main question.
The controlled part is what makes this benchmark unusual. Three things were held constant so that only the interviewer model varied:
- Interview context — answers to the 54 main questions were injected from transcripts of ten real human baseline interviews.
- Orchestration — identical prompts, guardrails, retry logic, memory window (2,000 characters) and temperature (0.7) across all models, accessed through one unified API.
- The respondent — every follow-up was answered by a single GPT-4o-based simulated interviewee holding a fixed Big Five profile, removing human response variability.
The models under test were Claude Sonnet 4, Gemini 2.5 Pro, GPT-5 Chat, Grok 4, Qwen3-235B A22B and DeepSeek Chat V3.1, evaluated in a single window in late September–early October 2025.
How was interviewer quality measured?
Quality was scored by humans, not by another model. Three expert psycholinguists trained in qualitative methods independently annotated 1,658 LLM-generated follow-up questions on four binary criteria, plus 1,275 turns where the agent chose not to follow up on a fifth:
- Benevolence — does the follow-up have an empathic, acknowledging tone?
- Necessity — was a follow-up genuinely warranted, given how complete the answer already was?
- Context-awareness — does it build on what the respondent disclosed earlier?
- Openness — does it avoid leading or closed phrasing?
- Justified skip — when no follow-up was asked, was that the right call?
Annotators were blind to model identity. Inter-rater reliability was strong: Fleiss' κ ranged from 0.67 on necessity to 0.93 on benevolence. That matters — a lot of AI-interviewer claims rest on user-satisfaction ratings or downstream task accuracy, neither of which tells you whether the questions themselves were any good.
Which model made the best AI interviewer?
Gemini 2.5 Pro took the best aggregate rank, but the more useful reading is per-profile. Averaging ranks across the five expert criteria (lower is better), the study reports:
| Model | Mean rank | Follow-ups per interview | Time per follow-up | Cost per interview | Profile |
|---|
| Gemini 2.5 Pro | 2.4 | 31.8 | — | $3.88 | Most empathic tone; highest openness |
| Grok 4 | 3.0 | 45.7 | 1 min 46 s | $5.03 | Exhaustive coverage; lowest necessity |
| Claude Sonnet 4 | 3.2 | 39.2 | 27 s | — | Balanced; dialogic, present-tense style |
| GPT-5 Chat | 3.6 | 19.4 | 9.8 s | $0.75 | Fastest and cheapest; most warranted follow-ups |
| Qwen3-235B A22B | 3.6 | 26.8 | 2 min 41 s | — | Strong context use, formal register, very slow |
| DeepSeek Chat V3.1 | 4.2* | 18.7* | — | — | Excluded: format instability |
*DeepSeek completed only three of ten interviews; its figures are reported for transparency but are not comparable.
Empathy and thoroughness pull in opposite directions
Gemini 2.5 Pro significantly outscored every other model on benevolence, with annotators flagging phrasings that acknowledged the effort of disclosure. Grok 4 sat at the other extreme: it asked the most follow-ups (45.7 per interview, 0.85 per main question) and scored lowest on necessity, because annotators judged many of those questions redundant against answers that were already detailed. Its questions averaged 43 words and a mean syntactic depth above eight, versus a corpus-wide mean of 26 words and depth 5.9.
Grok also over-used context. In one interview it referenced the respondent's high neuroticism score 24 times — often, annotators judged, awkwardly. More context is not automatically better rapport.
The skip decision is already reliable
The most reassuring finding for practitioners is the least dramatic. When a model decided not to probe, that decision was judged appropriate 85–93% of the time across models, and openness scores were above 0.80 everywhere. Contemporary LLMs, prompted properly, do not routinely ask leading questions and do not routinely probe when nothing is left to probe. The differences are in how aggressively they pursue what remains.
Schema compliance is a first-class research risk
DeepSeek Chat V3.1 accumulated 45 schema validation errors across just three interviews and had to be dropped. This is an under-discussed failure mode: an interviewer agent that cannot reliably emit a structured decision will silently drop follow-ups and produce interviews that are not comparable to each other. When you evaluate an AI interviewing platform, ask about output-schema failure rates, not just model names.
What this means for researchers
The practical takeaway is that "which model is best?" is the wrong question — "best for which interview?" is the right one. The paper's own guidance maps cleanly onto study design: efficiency-critical work such as large-scale screening favours a fast, selective model; sensitive autobiographical topics such as illness, trauma or identity favour a warmer, more acknowledging one; exhaustive discovery interviews can tolerate a high-intensity model if respondent fatigue is managed.
That framing has a design consequence. If model behaviour varies this much on empathy and probe frequency, then the model is a methodological choice, not an infrastructure detail — and it should be reported alongside your sampling and your interview guide. Teams running studies on an AI Interviewer should pilot the probing behaviour on a handful of real transcripts before fielding, and treat follow-up intensity as something to tune per protocol rather than accept as a default.
It also strengthens the case for the five-criterion rubric as a portable audit tool. You do not need three psycholinguists to apply it: reading 30 follow-up questions from your own pilot and marking each yes/no on necessity, context-awareness and openness will surface an over-probing agent within an hour.
Limitations worth knowing
Three caveats bound how far these results travel. The interviews were conducted in Russian, and the linguistic profiling used a Russian dictionary — stylistic findings may not transfer to other languages. The respondent was simulated, not human, so the study measures interviewer behaviour rather than the data quality real participants would produce. And the models were tested in a single window in late 2025; frontier model behaviour drifts, which is precisely why the authors frame their rubric as a re-runnable audit rather than a fixed leaderboard.
FAQ
Does the best-ranked model make the best AI interviewer for my study?
Not necessarily. Gemini 2.5 Pro had the best mean rank across criteria, driven largely by empathic tone. If your protocol is a high-volume screening interview rather than a sensitive life-story interview, the study's own recommendation points toward a faster, more selective model instead.
Do AI interviewers ask leading questions?
Rarely, in this benchmark. All models scored above 0.80 on openness — the criterion for avoiding leading or closed phrasing — when prompted with explicit non-leading instructions. Prompt design appears to do a lot of that work.
Is asking more follow-up questions better?
No. The model with the most follow-ups (45.7 per interview) scored lowest on whether those follow-ups were warranted. Probe volume and probe quality moved in opposite directions.
How much does an AI-moderated interview cost to run?
In this study, end-to-end model cost ranged from $0.75 to $5.03 per 54-question interview depending on the model — roughly $0.04 per follow-up at the cheap end. Those are inference costs only, not platform, recruitment or incentive costs.
Source
Panfilova, A., Bolshev, V., Mozikov, M., Latynov, V., Vanin, A., Nestik, T., Nourkova, V., Vlasova, A., Kozin, M., Serohvostov, A., Tarasova, E., & Nikolenko, S. (2026). The AI interviewer: multi-faceted evaluation of adaptive questioning by large language models. Scientific Reports, 16, 20257. Open access.
Last updated: 1 September 2026
This is an independent editorial summary of third-party research. Qualitati is not affiliated with the authors, and all figures are drawn from the published paper.