Short answer: Yes, adding an AI follow-up interview to a survey yields much richer answers. In a preregistered experiment with 2,243 US respondents, published in Public Opinion Quarterly in 2026, AI follow-ups more than doubled the length of open-ended explanations compared with a fixed \"why?\" prompt. Satisfaction barely dropped and dropout did not rise. However, the AI conversation also nudged some closed-ended answers.
Survey researchers have long added an open-ended \"Why do you hold this position?\" after a closed question, and they have long been disappointed by the short, vague answers it produces. AI-enabled interviews promise something closer to a semi-structured interview inside a survey, with follow-up probing at scale. A new randomized experiment by Austin, Hohe, Kennedy, Litman, Minozzi and Moses tests that promise directly, and it measures the side effects that vendors rarely mention.
\n\nThe study randomly assigned respondents to five versions of the same short survey on two US policy issues: whether undocumented immigrants should be able to stay legally if they meet certain requirements, and whether raising tariffs on imports is good or bad for the country. According to Austin et al. (2026), the sample of 2,243 US adults was recruited on CloudResearch Connect in December 2024, with roughly 450 people per arm and Census-based quotas for age, gender, ethnicity and race. The study was preregistered, and the AI interviews ran on a platform called Engage.
\n\n| Arm | What respondents saw |
|---|---|
| Closed Only (CO) | The two closed-ended policy items, with no follow-up |
| Closed then Fixed (CF) | Each closed item followed by a standard \"Why do you hold this position on this issue?\" prompt |
| Closed then AI (CAI) | Each closed item followed by an AI interview |
| AI then Closed (AIC) | An opening question and AI interview first, then the closed item |
| AI Only (AIO) | The opening question and AI interview, with no closed item |
This design lets the authors separate three questions that are often blurred together: does an AI follow-up produce better open-ended data than a fixed prompt, does it annoy respondents, and does it change what people say on the closed-ended measure?
\n\nYes, by a wide margin. The AI arms produced 70 more words on immigration and 57 more on tariffs than the fixed-follow-up arm, more than doubling that arm's word counts. AI interviewing also raised the number of distinct reasons respondents gave by 0.75 for immigration and 0.63 for tariffs. In pooled terms, the reason-provision rate reached 24% in both AI orderings against 14% with the fixed prompt.
\nTo code the open-ended text, the authors used GPT-4o to summarize and label the reasons. In a spot check of 300 sampled labels per group, 4 (1.3%) were clearly wrong and 16 (5.3%) were unclear. The authors judged the summaries good but did not treat the model's prevalence estimates as reliable, which is a useful caution for anyone who plans to let an LLM count themes unsupervised.
\n\nBarely. On a seven-point satisfaction scale, the estimated effect of the AI component was -0.06, with a 95% interval from -0.11 to 0.02, which is at most a very small decline. Rolloff did not increase: the authors report that the 10 people who failed to finish all dropped out before reaching the AI component. Of the 1,422 respondents who interacted with the AI, 1,057 (74%) left a comment about the survey. About 44% of those comments were positive and 12% were critical.
\nThe real cost is time. Median completion time ranged from under 7 minutes in the closed-only arm to 14.5 minutes when the AI interview came first. Because everyone received the same flat $3, the implied hourly pay fell from $25.90 to $12.41. Budget for that if you pay by the minute. The AI itself was cheap: the authors estimate about $0.008 per interview.
\n\nSometimes, and this is the finding researchers should take most seriously. When the AI conversation came before the closed item (AIC), immigration responses shifted toward support, with no comparable effect on tariffs. When the AI interview followed the closed item (CAI), respondents were more likely to change their answer between the interview and a later repeat of the item, and the responses became slightly more polarized on both issues. The authors describe these polarization effects as small, since most respondents did not move at all.
\nOrder also shaped the content of the reasons. For example, AI follow-up made tariff supporters more likely to cite domestic production, while placing the AI first made job protection more likely. In other words, the conversation is not a neutral recording device. It is part of the instrument, and it can prime or harden opinions.
\n\nThe sample was a nonprobability online panel, and quotas fell short for men, Hispanic and white respondents, and especially people over 60. Only one AI system was tested, and it did not always respect its three-follow-up limit. Both topics were salient, polarized US policy issues, so effects may differ on less charged or more sensitive subjects. The authors themselves call for work on other topics and populations and on interviewer traits such as tone and warmth.
\n\nFor depth, yes. In this experiment, AI follow-ups more than doubled open-ended word counts and raised the share of respondents giving a reason from 14% to 24%, compared with a fixed \"why?\" prompt.
\nNot much. Satisfaction fell by an estimated 0.06 points on a seven-point scale, dropout did not rise, and positive comments (44%) clearly outnumbered critical ones (12%).
\nIt can. Placing the AI conversation before a closed question shifted immigration answers, and an AI interview after it slightly increased polarization. Ask key closed items first and report question order.
\nThe paper is Austin, Hohe, Kennedy, Litman, Minozzi and Moses (2026), \"An Experimental Comparison of AI-Enabled Semi-Structured Interviews and Fixed Surveys,\" Public Opinion Quarterly: doi.org/10.1093/poq/nfag063.
\n\nLast updated: 2026-10-16.
\nThis article is an independent editorial summary of third-party research. Qualitati is not affiliated with the study's authors, and all figures are taken from the published paper.
\n","related":[{"slug":"batching-synthetic-respondents-inflates-correlations-2026","title":"Batching Synthetic Respondents Inflates Correlations (2026)","description":"Generating many synthetic respondents in one LLM request inflates correlations, a 2026 LSE study finds. What it means for synthetic survey data validity.","category":"Methodology","date":"2026-10-13","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"ai-chatbot-qualitative-interviews-lessons-2026","title":"AI Chatbot Interviews at €0.04: Lessons From a 2026 Study","description":"Can AI chatbot interviews replace qualitative interviewers? A 2026 study of 74 chatbot-led interviews found cheap, engaging data collection but limited depth.","category":"Methodology","date":"2026-10-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-inductive-content-analysis-survey-responses-2026","title":"Can GPT-5 Do Inductive Content Analysis of Survey Text?","description":"Can GPT-5 do inductive content analysis? A 2026 study of 903 survey answers found moderate human-AI agreement (ARI 0.61 codes, 0.54 themes), varying by question.","category":"Methodology","date":"2026-10-05","readTime":"7 min read","author":"Qualitati Research Team"}]}