Dynamic Surveys: Can LLMs Cluster Answers in Real Time?
Qualitati Research Team · 2026-09-02 · 6 min read
A dynamic survey is a survey that reads answers while they arrive: an LLM clusters open-ended responses into themes in real time, asks each respondent a personalized follow-up, and then shows them the emerging themes to rate and rank. A 2026 study found LLM clustering was about 89% accurate against human judgment.
What did the study test?
Lei and colleagues (2026) built a survey platform that does three things a normal survey cannot. First, it clusters incoming open-ended answers into themes as the survey runs. Second, it generates a follow-up question tailored to what each person just wrote. Third, it turns the clusters back into stimulus material: later respondents see the themes their peers produced and rate, rank, and react to them.
The team ran two field studies over roughly two months at a university. Study 1 collected panel questions for a career workshop (n = 52 student respondents, 18 of whom completed a post-study feedback survey, plus one recruiter interviewed). Study 2 asked students to identify gaps in an engineering curriculum (n = 41 respondents, 26 post-study surveys, plus three department administrators interviewed). Across both, 44 respondents gave structured feedback on the experience and four stakeholders were interviewed about whether the output was usable.
How accurate was the LLM clustering?
Accurate enough to be usable, with meaningful variation between topics. According to Lei et al. (2026), the career workshop survey produced 6 clusters with an average accuracy of 83.8 out of 100, while the curriculum survey produced 9 clusters at 92.7 out of 100 — an overall average of 88.9. Separately, 70.8% of respondents agreed the clusters they were shown were clear, coherent, and distinct.
The gap between the two studies is the interesting part. Curriculum complaints are concrete and vocabulary-stable ("we never learn version control"), so they cluster cleanly. Career-workshop questions are more diffuse and more likely to straddle two themes at once, which is exactly where automated clustering degrades. That pattern should be familiar to anyone who has run AI coding over both a tightly scoped product study and an open-ended exploratory one.
Reported results at a glance
| Measure | Career workshop | Curriculum gaps |
|---|
| Respondents | 52 | 41 |
| Post-study feedback | 18 | 26 |
| Clusters generated | 6 | 9 |
| Average cluster accuracy (/100) | 83.8 | 92.7 |
| Ranking score range | 0–68 | 18–63 |
Source: Lei et al. (2026). Overall cluster accuracy across both studies: 88.9/100.
Why does asking respondents to rank themes matter?
Because it fixes the oldest weakness of open-ended survey data: you get richness but no idea what is widely held. In a conventional study, the researcher reads 90 free-text answers, decides which themes matter, and that prioritization is a judgment call made by one person after the fact.
In the dynamic survey, prioritization becomes data. Later respondents rank the themes earlier respondents generated, so each theme carries both a qualitative description and a quantitative score. Lei et al. (2026) report that in the career workshop study, ranking scores correlated positively with the proportion of respondents who agreed with a theme — the two signals told a consistent story. In the curriculum study, that alignment held for the top-ranked clusters but weakened for lower-ranked ones, which the authors attribute to timing: themes that emerged late in the fielding period were seen by fewer people and had less opportunity to accumulate ranking votes.
That is a real design constraint, not a footnote. If you run this method, order-of-arrival advantages early themes. A theme that surfaces on day one gets ranked by everyone; one that surfaces on day ten gets ranked by the stragglers.
What did participants and stakeholders think?
Respondents were broadly positive about the personalized follow-ups and the ranking step, and the four stakeholders interviewed — one recruiter and three administrators — all saw value in the clustered, ranked output rather than a raw response dump. The consistent complaint was length: the follow-up and ranking stages add time, and some respondents reported fatigue. The authors report this as a qualitative pattern rather than a percentage, so treat it as a design warning rather than a measured effect size.
What this means for researchers
Three practical takeaways, in order of how much they should change your workflow.
- Clustering accuracy is topic-dependent, so pilot on your topic. The 9-point spread between two studies at the same university, on the same platform, is a warning against assuming a headline accuracy number transfers. Run 20 responses through your pipeline and read the clusters yourself before fielding at scale.
- Real-time follow-ups are the cheapest depth you can buy. A single tailored probe on a vague answer converts an unusable response into a codeable one. This is the same mechanism behind AI-moderated interviews — the difference is only how many probes you spend per participant.
- Budget respondent time explicitly. Dynamic surveys trade completion length for data quality. If your incentive and your recruitment pitch assume a five-minute survey, adding follow-ups and a ranking task will show up as drop-off.
Methodologically, the honest framing is that this is a promising prototype, not a validated instrument. Two field studies at one institution with student samples, and cluster accuracy assessed against the research team's own judgment, is early-stage evidence. The mechanism is convincing; the generalization is not yet established.
Frequently asked questions
What is a dynamic survey?
A survey in which an LLM processes responses as they arrive — clustering open-ended answers into themes, generating personalized follow-up questions, and presenting the emerging themes back to respondents for rating and ranking. It sits between a static questionnaire and a full interview.
How accurate is LLM clustering of open-ended responses?
In this study, 88.9 out of 100 on average, ranging from 83.8 to 92.7 depending on the topic (Lei et al., 2026), with 70.8% of respondents agreeing the clusters were clear and distinct. Accuracy varied enough between two studies to warrant piloting on your own data.
Does this replace qualitative interviews?
No. A dynamic survey gets one or two probes per respondent across a large sample; an interview gets dozens of probes with a handful of people. They answer different questions — breadth of themes versus depth of a single account.
Can I run this with standard survey tools?
Not the live clustering-and-ranking loop, which requires responses to be analyzed mid-fielding. Conversational survey tools such as AI Surveys and ThemeLens cover the adaptive-probing and theme-extraction halves; the peer-ranking step described here remains a research prototype.
Primary source: Lei, K., Ladenburg, A., Petiwala, Z., Wang, Z., Jhawar, D., Bisht, I., Kumar, A., & Lee, D. T. (2026). Dynamic Surveys: Using LLMs to Blend Qualitative Depth, Quantitative Structure, and Collaborative Interaction. Proceedings of the ACM on Human-Computer Interaction. arXiv:2608.00357
Last updated: September 2, 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the authors and all figures are drawn from the published paper.