Short answer: Not yet. A 2026 study by Kumagae and Kaneko tested six LLMs as "silicon samples" on two US social surveys. The synthetic populations reproduced each question's answer distribution, but none covered the real respondents as well as a second human sample did. All six even trailed a simple baseline that draws every answer independently.
Last updated: October 26, 2026
Silicon sampling means asking a large language model to answer a whole questionnaire as a person with given demographics (say, a woman in her thirties), then repeating that across many personas to build a synthetic population. Most validation work checks the answers question by question. The preprint Silicon Samples Match the Answers but Not the Respondents (Yusuke Kumagae, Hakuhodo DY Holdings, and Masahiro Kaneko, MBZUAI; arXiv, October 2026) checks something different: whether the synthetic people look like the real ones.
What is the respondent coverage gap?
The respondent coverage gap (RCG) measures how far each real respondent sits from the nearest synthetic respondent, compared with how far they sit from the nearest other real person. Each respondent is treated as one point defined by their complete answer profile across all questions. The authors take the 95th percentile of those nearest-neighbour distances and divide it by the same figure computed among humans alone.
- RCG of about 1.0 means the synthetic population covers real respondents about as well as another human sample would.
- RCG above 1.0 means some real people have no close synthetic counterpart: they are left uncovered.
A companion measure, the off-support ratio (OSR), runs the other way: it asks whether synthetic respondents sit unusually far from any real person. Together the two ratios tell you not just that a synthetic panel fails, but how.
The point of the metric is simple. Two synthetic populations can match every question's answer distribution perfectly and still be made of people whose combinations of answers no real person holds. Per-question checks cannot see that.
What did the study test?
The study compared six LLMs and three statistical baselines on two surveys of US residents, using 100 questions from each.
- World Values Survey (wave 7, US sample): 2,594 respondents, 730 demographic cells.
- American National Election Studies (2024 wave): 5,453 respondents, 603 cells.
- Models: gpt-5.4-nano, gemini-3.1-flash-lite, Qwen3.5-9B, gemma-4-12B, gpt-oss-20b and Llama-3.3-70B, used off the shelf.
- Baselines: "Marginals" (draws each answer independently from the real per-question distribution), a Gaussian copula (also models how questions co-vary), and a two-class latent class model.
Each LLM answered one question at a time, with a persona and up to three of its own earlier answers in the prompt. Prompt wording, persona format, question order and answer-option order were randomised to reduce wording effects.
How well did LLM silicon samples cover real respondents?
None of them reached human-level coverage, and all of them lost to the simplest baseline. According to Kumagae and Kaneko (2026), uncalibrated LLM scores ranged from 1.199 to 1.275 on the World Values Survey and from 1.277 to 1.370 on ANES. A held-out human sample scored 1.001 and 1.000.
| Generator | RCG, World Values Survey | RCG, ANES 2024 |
|---|---|---|
| Another human sample (reference) | 1.001 | 1.000 |
| Gaussian copula baseline | 1.041 | 1.091 |
| Latent class baseline | 1.105 | 1.143 |
| Marginals baseline (answers drawn independently) | 1.158 | 1.250 |
| Best LLM (gemma-4-12B on WVS, Llama-3.3-70B on ANES) | 1.199 | 1.277 |
| Worst LLM (Llama-3.3-70B on WVS, gpt-5.4-nano on ANES) | 1.275 | 1.370 |
The Marginals result is the uncomfortable one. That baseline knows nothing about how a person's answers hang together; it just samples each question separately. Yet every LLM covered real respondents less well than it did, and the gap was statistically significant for every model on both surveys. The authors found no consistent ordering by model family, open versus proprietary weights, or model size.
Why don't standard fidelity metrics catch this?
Because they average along questions, not people. The authors show that the Marginals baseline scores 0.994 on a widely used correlation-structure metric for the World Values Survey, against a human split-half ceiling of 0.995, while its RCG is 1.158. They also prove that this correlation score is unchanged if you shuffle answers between respondents within a cell, which destroys every real person's answer profile.
Per-question distance metrics have the same blind spot. After calibration, the LLMs had the closest per-question distributions of any generator, yet still covered respondents worse than the copula and latent class baselines. In plain terms: a "matches the published numbers" chart is not evidence that a synthetic panel contains realistic respondents. We made a related point about marginal fidelity in synthetic survey panels; this paper supplies a direct respondent-level test.
How do synthetic respondents fail?
The off-support ratio splits the models into two failure patterns:
- Contraction. gemma-4-12B, gemini-3.1-flash-lite and Llama-3.3-70B produced respondents that stay close to typical real people while leaving others uncovered (OSR 0.722–0.840 on WVS). The synthetic population huddles around the centre and loses the tails.
- Off-support generation. gpt-oss-20b, gpt-5.4-nano and Qwen3.5-9B produced, on ANES, respondents with answer combinations unusually far from any real person (OSR 1.144–1.213).
Contraction echoes earlier findings on reduced variance and the flattening of group differences. Off-support generation is the opposite risk: plausible-looking individual answers that add up to people who do not exist.
Does calibration, reasoning or a bigger model fix it?
Each of these helped somewhat, but none closed the gap. The paper reports:
- Calibrating each question's answer distribution to the real one lowered RCG for all six models (to 1.126–1.150 on WVS and 1.178–1.232 on ANES), beating Marginals but not the copula or latent class baselines.
- More synthetic respondents helped slowly. With 10,000 generated respondents per cell (100 times the human sample), open-weight models still had RCG above 1 on the WVS subset, around 1.09–1.13. The authors' rough extrapolation puts human-level coverage at hundreds of thousands of synthetic respondents per cell.
- Full response history, higher reasoning effort and higher-tier models (gpt-5.4, gemini-3.1-pro-preview) brought some models close to or below Marginals, but in every setting tested, every LLM stayed above the Gaussian copula.
- Temperature barely moved RCG (at most 0.033 within a model).
What this means for researchers using synthetic respondents
Treat synthetic respondents as a tool for questionnaire piloting and hypothesis generation, not as a stand-in for a sample of real people. If you do use a synthetic panel, ask for respondent-level validation, not only per-question charts. Three practical checks follow from the paper:
- Ask how the panel was validated. Matching marginals or correlation matrices is necessary but not sufficient.
- Look at the tails. If your research question depends on unusual or minority answer patterns, contraction is the failure you should expect.
- Anchor with real people. Use synthetic runs to sharpen a guide, then collect real voices. Tools like the Digital Twin Panel are useful for rehearsal, while the AI Interviewer lets you reach real participants at scale.
Note the study's limits: it covers US respondents on two social surveys, uses off-the-shelf prompting, and does not test fine-tuned or backstory-based personas. It is also a preprint that has not yet been peer reviewed.
FAQ
What is silicon sampling?
Silicon sampling is using an LLM to answer a survey as a person with specified demographic attributes, then repeating this across many personas to build a synthetic population of respondents.
Can LLM synthetic respondents replace a human survey sample?
Not on current evidence. In this 2026 study, none of six LLMs covered real respondents as well as another human sample, and all trailed a baseline that samples answers independently.
Is matching the answer distribution enough to validate a synthetic panel?
No. The study shows synthetic populations can match per-question distributions and correlation scores while their individual respondents still differ from real people.
What is a good respondent coverage gap score?
Close to 1.0, which is what a second, independent human sample scores. Values above 1.0 mean some real respondents have no close synthetic counterpart.
Source: Kumagae, Y., & Kaneko, M. (2026). Silicon Samples Match the Answers but Not the Respondents. arXiv:2610.30900.
Last updated: October 26, 2026. This is an independent editorial summary of third-party research by the Qualitati Research Team; it is not affiliated with or endorsed by the paper's authors.