Short answer: In a 2026 randomized experiment, a locally hosted 7B-parameter AI interviewer matched trained non-expert human interviewers on how on-topic text-chat answers were, showed no detectable difference in specificity, and collected about 65% more respondent text per interview. The catch: only participants interviewed by the AI skipped questions.
AI interviewers are often pitched as a way to run qualitative interviews at survey scale. The open question has been whether the data they collect is actually any good compared with what a person would get. A new preprint from the University of Copenhagen, \"Can AI interview? Comparing AI-led to Human-led Chat Interviews\" by Semra Guven, Tobias Gårdhus, Andreas Bjerre-Nielsen, Anna Rogers and Hjalmar Bang Carlsen (arXiv, October 2026), describes itself as the first direct empirical comparison of data quality between the two. This post summarizes what the study tested and what it found, and what the results do and do not license.
\n\nThe study randomly assigned university students to a synchronous text-chat interview led either by a trained non-expert human interviewer or by an AI system. Both conditions used the same chat interface, the same interview guide and the same instructions. The topic was academic help-seeking: one section on a time the student asked for help and got it, and one on a time they needed help but did not ask.
\nAccording to Guven et al. (2026), 45 participants were randomized and 41 interviews were analyzed (20 AI-led, 21 human-led). On the first day the AI malfunctioned, repeating questions and in one case producing long off-topic passages. The authors discarded those four AI interviews and revised the system. Participants sat on site in rooms next to the interviewers and were told which condition they were in. The on-site setup was also meant to stop respondents from using generative AI tools to write their answers.
\n\nThe system splits the work across three LLM agents, a multi-agent design the authors adapt from earlier work:
\nThe AI was deliberately restricted. It could only probe around a predefined set of open-ended main questions and could not invent new questions under broad themes. All three agents ran on a 4-bit quantized OpenHermes 2.5 model served with Ollama on researcher-controlled hardware. The authors argue this addresses three problems with commercial APIs: confidentiality, GDPR compliance, and reproducibility when a provider silently updates or retires a model.
\n\nThe AI matched humans on relevance and was well ahead on volume per minute, while humans drew slightly longer individual answers. The table summarizes the headline numbers reported by Guven et al. (2026).
\n| Measure | AI interviewer | Human interviewer | Significant after correction? |
|---|---|---|---|
| Median interview duration | 28.9 min | 28.3 min | No |
| Questions per interview | 31 | 14 | Yes |
| Characters per answer | 112 | 150 | No |
| Total respondent text per interview | 3,244 chars | 1,968 chars | Yes |
| Answers per minute | 1.18 | 0.54 | Yes |
| Respondent characters per minute | 106 | 72 | Yes |
| Time to write the next question | 3 s | 39 s | Yes |
| Mean specificity (1–5) | 3.6 | 3.8 | No |
| Mean relevance (1–5) | 4.9 | 4.9 | No |
Two details matter when reading the question count. First, 7 of the 20 AI interviews reached the 44-question ceiling the guide allowed once every probe was used, so the number partly reflects the probe budget the researchers set rather than the model's own behavior. Second, mean duration diverged more than the median (30.3 vs 26.7 minutes) because five AI interviews ran past 40 minutes. That is why the authors report rates per minute instead of claiming equal exposure time.
\n\nThe study could not detect a difference, but it also could not rule out a meaningful one. Two coders rated 395 question-answer pairs, drawn from 909 in total. The estimated gap was 0.16 points in favor of humans, with an interval from −0.71 to 0.36. The authors are careful here: a half-point difference on a five-point scale \"remains possible\" with this sample. Coder agreement on specificity was also modest, with an ordinal Krippendorff's alpha of 0.65, just below the conventional 0.667 threshold for tentative conclusions.
\nRelevance is the stronger result, though it is a coarse one. Almost every answer in both conditions was scored 5, so the measure mostly separates on-topic answers from off-topic ones. The authors also note that both conditions struggled to get developed, experience-based accounts, which may have limited how sensitive the specificity measure could be.
\n\nThe design cannot say why, and the authors offer three candidate explanations. Declining a machine may feel less socially costly. Participants who knew they were talking to an AI may have treated the session more like a survey. And because the AI often probed up to the allowed limit, later probes may have felt repetitive. With only five participants who skipped, the study cannot tell these explanations apart. In practice, the third explanation is the one a research team can do something about: set the probe budget with fatigue in mind instead of maximizing question count.
\n\nThe authors draw a careful line around their result, and that line is the most useful part of the paper:
\nIf you are planning AI-moderated interviews, the practical lessons are to fix your main questions, cap probes deliberately, track skips as a data-quality metric, and pilot before you scale (the first-day malfunction in this study is a reminder why). Tools such as the QualiTaTi AI Interviewer follow the same guide-anchored pattern, and transcripts can then be coded in ThemeLens with a human checking the themes.
\n\nIn this 2026 text-chat experiment, a 7B AI interviewer produced answers as on-topic as those from trained non-expert humans, with no detectable difference in specificity, while collecting more text per minute. The study did not compare the AI with expert interviewers or with face-to-face interviews.
\nNot necessarily. This study used OpenHermes 2.5, a 7B open-weight Mistral model running locally. The authors say they believe it is the first AI-interviewer system built on a model that small. They also note that their results are not an upper bound, since the model was not fine-tuned for interviewing.
\n45 university students were randomized and 41 interviews were analyzed, which yielded 909 question-answer pairs. With about twenty participants per condition, the study can detect large differences, but it cannot establish equivalence on specificity.
\nQuestion skipping. Only AI-interview participants used the skip button (5 of 20 against 0 of 21, Fisher's exact p = 0.021, uncorrected), so non-response is worth monitoring in AI-led interviews.
\n\nLast updated: 2026-10-19. This article is an independent editorial summary of third-party research by Guven, Gårdhus, Bjerre-Nielsen, Rogers and Carlsen (2026). QualiTaTi was not involved in the study. Consult the original paper for full methods and results.
\n","related":[{"slug":"artificial-societies-benchmark-synthetic-population-validity-2026","title":"Is Your Synthetic Population Valid? An 11-Test Benchmark (2026)","description":"Is your synthetic population valid? A 2026 benchmark tested 9 LLMs on 11 validity checks: they match averages but compress scales and distort trait links.","category":"AI User Research","date":"2026-10-15","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"synthetic-consumers-visual-marketing-pretest-2026","title":"Can Synthetic Consumers Pretest Logos and Ads? A 2026 Study","description":"Can synthetic consumers pretest visual marketing? A 2026 study found GPT agents saw logos correctly but matched at most 2 of 6 human effects and reversed one.","category":"AI User Research","date":"2026-10-14","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"interview-informed-llm-survey-responses-2026","title":"Can Interview Transcripts Predict Survey Answers? A 2026 LLM Test","description":"Can interview-informed LLMs predict survey answers? A 2026 study tested GPT, Claude and Gemini against 19 real respondents. Here is what held up and what broke.","category":"AI User Research","date":"2026-10-12","readTime":"7 min read","author":"Qualitati Research Team"}]}