Can LLMs Predict Survey Answers They've Never Seen? (2026)
Qualitati Research Team · 2026-09-11 · 7 min read
Short answer: Partly. In a 2026 study of Taiwan's national election survey, zero-shot LLMs predicted individual respondents' answers to questions they had never seen with about 52% exact accuracy, compared with 58% for a random forest trained on roughly 960 real respondents. Accuracy was high for partisan attitudes and poor for personal-judgment items.
Last updated: September 11, 2026
Silicon sampling, the practice of asking a large language model to answer a survey as if it were a particular person, has mostly been judged on aggregates: does the synthetic sample reproduce the real sample's distribution? A new preprint argues that is too easy a test, and proposes a harder one. This summary covers what the authors did, what they found, and what it means if you are considering synthetic respondents for your own work.
Key takeaways
- Individual-level accuracy is modest. The best zero-shot LLM reached 52.1% exact match on unseen items, against a 46.6% majority-vote baseline and 58.3% for a supervised random forest.
- Demographics alone carry almost no signal. A logistic regression on five demographic variables scored 47.0%, barely above the majority baseline. The predictive information sat in respondents' answers to other questions.
- Predictability follows partisanship. Party-linked constructs reached about 67%; sovereignty preferences only 23%.
- Variance collapse is not just an LLM problem. Every method narrowed the answer distribution, and the LLMs actually preserved more spread than the random forest.
- Safety alignment effects differ by model. Removing alignment cost one model 6.1 points and left another essentially unchanged.
What is cross-survey transfer?
Cross-survey transfer is an evaluation design in which an LLM sees a respondent's real answers to one set of survey questions and must predict that same person's answers to a different, never-shown set. According to Ku, Hsu, Huang, Liu, Cheng and Kang (2026), most earlier silicon sampling studies conditioned models on demographic profiles and compared aggregate distributions, which can reward pattern matching rather than coherent prediction of a single person.
The authors split the survey's items into two disjoint groups before running anything: a context set (Set A) shown to the model as a persona, and a prediction set (Set B) held out entirely. Roughly two-thirds of each construct's items went to Set A and one-third to Set B, so every predicted item had a related anchor question in the context.
What did the study test?
The study used the Taiwan Election and Democratization Study (TEDS) 2024, a nationally representative telephone survey of 1,206 people conducted after the January 2024 elections. The design worked as follows:
- Inputs: 5 demographic variables and 28 Set A items per respondent, written into a natural-language prompt, plus population-level response distributions for base-rate calibration.
- Targets: 14 Set B items spanning 13 constructs, from institutional trust to the independence–unification scale. Between 593 and 610 valid respondents were retained per item.
- Models: three open-weight LLMs run locally, Qwen3.5 (27B), gpt-oss (120B) and Gemma3 (27B), each also tested in an “abliterated” version with safety alignment removed.
- Baselines: a majority vote, a logistic regression on demographics, and a random forest on demographics plus Set A, both trained with 5-fold cross-validation.
The choice of a Mandarin-language survey from a non-Western democracy was deliberate: most silicon sampling evidence comes from English-language, WEIRD populations.
How accurate were the LLMs?
The zero-shot LLMs landed between the naive baselines and the trained model. The table below reports the paper's headline results.
| Method | Training data | Exact match | Within ±1 | Variance ratio |
|---|
| Majority vote | — | 46.6% | — | — |
| Logistic regression (demographics only) | Supervised | 47.0% | 83.1% | 0.38 |
| Random forest (demographics + Set A) | Supervised | 58.3% | 88.2% | 0.72 |
| Qwen3.5 27B | Zero-shot | 52.1% | 85.7% | 0.84 |
| gpt-oss 120B | Zero-shot | 51.3% | 85.2% | 0.85 |
| Gemma3 27B | Zero-shot | 47.6% | 86.2% | 0.67 |
Two details stand out. First, model size did not predict accuracy: the 27B Qwen3.5 edged out the 120B gpt-oss, which the authors attribute to Qwen's stronger Chinese-language training. Second, the authors frame the 6-point gap to the random forest as the value of about 960 labeled examples from the target population, which the LLMs never received.
Which attitudes can an LLM predict?
Attitudes anchored in party identification were the most predictable; attitudes built on personal judgment were the least. Averaged across LLMs, party competence reached 67.2%, government performance 62.7%, and institutional trust 60.4%. Economic evaluation fell to 44.4%, candidate evaluation to 24.7%, and sovereignty to 23.3%.
The authors read this hierarchy through Converse's theory of attitude constraint: when party identity structures a belief, answers to other questions reveal it. The ordering was stable across all three LLMs and the random forest, which suggests it reflects real attitudinal structure rather than a quirk of one model. The sovereignty item was the outlier, with a 33-point gap between the LLMs (23.3%) and the random forest (56.1%).
Is variance collapse really an LLM problem?
Not only. Variance collapse, the tendency of synthetic answers to cluster too tightly around the average, appeared in every method. The LLMs' variance ratios ranged from 0.67 to 0.85, compared with 0.72 for the random forest and 0.38 for the demographics-only regression. The authors argue that predicting individuals from limited context pulls every method toward the population mean.
That creates a trade-off researchers should notice: the LLMs were less accurate per person but kept more realistic diversity in the overall distribution. If your goal is population-level heterogeneity rather than individual prediction, that matters.
Does safety alignment distort synthetic respondents?
It depends on the model. After abliteration, gpt-oss lost 6.1 points of accuracy, Gemma3 lost 2.3, and Qwen3.5 was essentially unchanged at −0.1. Variance moved in opposite directions too: gpt-oss rose from 0.85 to 0.90, while Gemma3 fell from 0.67 to 0.45. The authors' advice is to check alignment effects per model rather than assume a universal pattern.
What this means for researchers
The authors conclude that current accuracy is insufficient to replace real respondents, but useful for piloting instruments and generating hypotheses. A few practical implications follow:
- Test individual-level prediction, not only distributions. Holding out questions is a cheap way to see whether a synthetic panel knows your respondents or just the population average.
- Give models real attitudinal context. Demographic personas added almost nothing here; prior answers did the work. This is the logic behind grounding digital twin panels in real response data.
- Expect uneven performance by topic. Partisan or identity-linked items may simulate well while sensitive or idiosyncratic ones fail.
- Consider hybrid designs. The paper points to combining LLM-generated priors with smaller human samples as the most promising near-term use.
Limitations are worth keeping in view: the study used a single survey and a fixed item partition, only open-weight models, and no human test-retest ceiling. The authors also note that the gain over the majority baseline, while statistically significant across more than 8,000 predictions, is modest in practical terms.
FAQ
What is silicon sampling?
Silicon sampling means prompting an LLM to answer survey questions as a specified person or persona, producing synthetic responses that are meant to approximate a real human sample.
Can LLMs replace survey respondents in 2026?
Based on this study, no. Zero-shot models reached 52% exact accuracy on unseen items, below a supervised model at 58%, and performance collapsed on sensitive and personal-judgment questions.
Do bigger models make better synthetic respondents?
Not in this study. A 27B model with strong Chinese-language training outperformed a 120B model on a Mandarin survey, suggesting language fit can matter more than size.
What is variance collapse?
It is the narrowing of predicted answer distributions relative to real ones. Here it affected all methods, including a random forest, not just LLMs.
Source: Ku, C.-T., Hsu, C., Huang, P.-C., Liu, F. C.-s., Cheng, I.-L., & Kang, Y. (2026). Silicon Sampling via Cross-Survey Transfer. arXiv preprint 2607.03091.
Last updated: September 11, 2026. This is an independent editorial summary of third-party research by the Qualitati Research Team; it is not affiliated with or endorsed by the paper's authors.