Does Silicon Sampling Work Across Countries? A 2026 ESS Test
Qualitati Research Team · 2026-09-22 · 7 min read
Short answer: Silicon sampling (using an LLM to simulate survey respondents) can roughly rank countries on some attitudes, but it does not reproduce individual people. A 2026 test against the European Social Survey found that the country name in the prompt did almost all the work, while individual-level accuracy stayed near zero in every condition.
Silicon sampling promises cheap, fast survey data: give a large language model a demographic profile and let it answer as that person would. Most validation studies so far have focused on the United States. A new working paper by Chuyao Wang at the London School of Economics tests the idea across 30 European countries and asks a sharper question: where does the signal actually come from?
What did the study test?
The study compared LLM-simulated answers with real answers from European Social Survey (ESS) Round 11, covering 30 countries and 42 items across 12 domains, including institutional trust, well-being, immigration attitudes, religion, safety and income. According to Wang (2026), two open-weight models were used, Qwen 2.5-7B-Instruct and Llama 3.1-8B-Instruct, each prompted in first person ("answer as this person") and third person ("predict how this person would answer").
Each simulated respondent received a backstory built from 20 ESS demographic variables. The study then ran a series of controlled experiments:
- Backstory conditioning: adding or removing blocks of the profile (country, region, political identity, socioeconomic status, household, migration and others) to see which part moves accuracy.
- Swapped-label placebo: giving every respondent the wrong country name while keeping the rest of their profile.
- Response-format 2×2: crossing country label (present/absent) with the answer scale (numbers only vs. numbers with the endpoints named in words).
- Non-LLM comparators: predicting each country from the average of its regional neighbors, and from region plus log GDP per capita.
In total the paper reports 34.6 million generation attempts across 43 experimental arms.
How accurate is silicon sampling across countries?
Aggregate accuracy was moderate and very uneven across items. Under the best-performing setup (Qwen, first person), the median per-item correlation between simulated and observed country means was 0.443. Only 20 of 42 items exceeded 0.5, and nine exceeded 0.7. Trust in the police, parliament and the legal system recovered best, at around 0.80.
A headline-friendly number hides this. The pooled correlation across all items was 0.822, but the paper shows this is inflated because pooling mixes items with different scale lengths. Researchers who report only a pooled figure can make a weak simulation look strong.
The model also ranked countries backwards on 11 items. The worst were whether gay men and lesbians should be free to live as they wish (−0.82) and whether same-sex couples should have the right to adopt (−0.73). Every country's simulated mean was biased upward, from 0.36 scale points in Finland to 1.40 in Bulgaria.
Where does the signal come from? The country name
The country label produced almost all of the aggregate accuracy. Adding the country name to a minimal three-variable profile (gender, age, birth year) raised the median per-item correlation from −0.03 to 0.52. The country name on its own, with no demographics at all, also reached 0.52. Richer profiles added no consistent gain, and political and socioeconomic information actually lowered aggregate accuracy.
The placebo confirmed that the model reads the label. Giving respondents the wrong country lowered recovery for their real country below the no-label baseline (0.34 vs. 0.40 on forward-coded items). Wang interprets this as a country-level assumption: the model attaches one expectation to a country name, and respondent detail does not revise it. The author notes the data cannot distinguish genuine survey knowledge from national stereotypes or online discourse about a country.
Why did the model rank countries backwards?
The inversions were mostly a response-format problem, not a knowledge problem. Ten of the 11 inverted items were reverse-coded, meaning the numeric scale runs against the direction of the question, and item direction alone explained 66% of the variance in item-level accuracy. When the scale endpoints were named in words instead of shown as a bare number range, the country label helped every forward-coded item and 8 of 10 reverse-coded items. The effect replicated in Llama.
The practical lesson: the prompt's answer format is part of the measurement instrument. A human ESS respondent sees a labeled show card; a model given "Scale: 0-10" has to guess which end is which.
Can silicon samples predict individual respondents?
No. Median within-country correlation never exceeded 0.028 in any model and prompt condition, which is less than 0.1% shared variance. By comparison, a simple regression on five demographic predictors reached 0.210 on the same human data. First- versus third-person framing made no detectable difference (0.027 vs. 0.029 for Qwen). In a replicate run, 95% of the within-country variation the model produced was random variation between runs. Simulated answers were also compressed, with a model-to-human standard-deviation ratio of 0.57 to 0.61.
Aggregate and individual accuracy did not even move together. Across the 30 countries they correlated at −0.21, and 232 of 435 country pairs were ranked in opposite orders by the two measures. A country that looks well simulated in aggregate is no safer for individual-level work.
Did the LLM beat a simple baseline?
The most sobering result is that a baseline with no LLM did better on level accuracy. Averaging each country's regional neighbors gave a median per-item correlation of 0.55 against the model's 0.44, and a much smaller normalized error (0.056 vs. 0.180), winning on level accuracy for 41 of 42 items.
| Approach | Median per-item country correlation | Median normalized error |
|---|
| LLM, full 20-variable profile (Qwen, 1P) | 0.44 | 0.180 |
| LLM, country name only | 0.52 | not reported |
| Regional-neighbor average (no LLM) | 0.55 | 0.056 |
| Region + log GDP per capita (no LLM) | 0.62 | 0.052 |
Source: Wang (2026), Sections 4.1–4.2. Normalized error is mean absolute error after rescaling each item to 0–1.
What this means for researchers
The paper's own conclusion is narrow: silicon sampling may support exploratory country-ranking comparisons on items validated against an external benchmark, with the response format reported. It does not support individual prediction, within-country heterogeneity or distributional inference. Four habits follow:
- Validate item by item. Pooled and country-profile correlations looked strong here while hiding inverted items.
- Anchor your scales in words and report the exact format you used.
- Always run a simple baseline. If a regional average beats your model, the model adds cost, not information.
- Do not treat richer personas as a fix. More demographic detail did not help and sometimes hurt.
For questions about why people in a given market think what they think, real respondents remain the evidence. Qualitati's AI Interviewer makes it cheaper to reach real people across countries and languages. Simulated panels such as Digital Twins are best used to pilot questions and stress-test a guide before fieldwork, not to replace it.
FAQ
What is silicon sampling?
Silicon sampling means conditioning an LLM on a respondent's demographic profile so it generates survey answers meant to approximate a comparable human. The term comes from Argyle et al. (2023).
Does adding more demographic detail make LLM respondents more accurate?
Not in this study. The country name alone reached the same aggregate accuracy as a fuller profile, and adding political or socioeconomic information lowered it.
Can I use LLM-simulated respondents to compare countries?
Only cautiously and exploratively, on items you have validated against real data, with verbally anchored scales. Check that the model beats a simple regional average before trusting it.
Were large commercial models tested?
No. The study used two open-weight 7–8B models run locally on a university cluster, so results for larger commercial models remain an open question.
Primary source: Wang, C. (2026). Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey. arXiv:2609.16395.
Last updated: September 22, 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the author.