Short answer: Not necessarily. A September 2026 preprint by Alexander Doudkin tested demographic synthetic survey panels against published multiselect survey results and found that matching the published averages mostly reflects how answers are elicited. A single no-persona question about population prevalence beat committed persona panels in 9 of 9 comparisons, so marginal agreement is not evidence that individual respondents are being simulated.
Synthetic survey panels are sold on a simple promise: build a few hundred LLM personas with real demographic profiles, ask them your questions, and get answers that look like a real population. The usual proof is a chart showing that the synthetic percentages line up with a published survey. Doudkin's paper, Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels (arXiv, 2026), asks what that chart actually proves. The answer is: less than most vendors and researchers assume.
\n\nThe study compared synthetic panels with real survey results on six multiselect (\"tick all that apply\") question batteries from four survey organisations in three countries. Three batteries, where the synthetic and human samples share the same population frame, carry the headline results:
\n\nThe other three batteries (Ofcom online activities, an ONS cost-of-living battery and a CES political-activity item) were used only as sensitivity checks because their population frames did not match the synthetic cohort. Each cohort had 500 personas described by four attributes such as gender, age band, region and education. The models included claude-sonnet-5, gpt-5.6-terra, gemini-3.7-flash, claude-haiku-4-5 and gpt-4o-mini.
\n\nThe core of the paper is that the same persona can be asked in different ways, and that choice moves the measured fidelity a lot. Doudkin calls it the response contract.
\n\n| Method | How it works | Average error vs. human data (aligned pairs) |
|---|---|---|
| Committed set | Each persona returns the list of options that apply to it, like a real respondent | 12.39 MAE (9 model-battery pairs) |
| Per-option probability | Each persona gives a 0–100 likelihood for every option; the panel averages them | 5.34 MAE when projected onto the question's selection limit |
| No-persona prevalence query | One neutral prompt asks the model what share of the population would pick each option | 6.27 MAE |
MAE here is mean absolute error in percentage points across the options of a question. Lower is better.
\n\nCommitted answers, the format that most resembles real respondents, performed worst. According to Doudkin (2026), across the three aligned batteries the committed-set panels left 66 of 128 model-battery option slots completely empty: no simulated respondent chose that option at all, even with panels of up to 500 personas. Probability elicitation left 0 of 128 empty. The paper calls this support collapse.
\n\nThe examples are concrete. On two of five models, no simulated American reported visiting an emergency room, against 22.1% of real CES respondents. On two of three models, no simulated German named immigration as a top issue, against 23.0% in the Eurobarometer data.
\n\nSwitching to per-option probabilities cut option-level error by 4.53 to 7.30 points in all eight uncapped model-instrument comparisons. That sounds like a fix, and for estimating population percentages it partly is. But it also changes what the panel is: an averaged set of forecasts, not a set of individual answers.
\n\nBecause a model can hit the averages without simulating anyone. The decisive comparison is the no-persona baseline. A single prompt asking for population prevalence, with no personas at all, averaged 6.27 MAE and beat the committed persona panels in 9 of 9 aligned comparisons. If the plain question does as well as or better than a 500-persona panel, then a good match to the published table tells you about the model's knowledge of population rates, not about its ability to stand in for individual people.
\n\nThe paper is careful here: constraint-aware probability panels (5.34 MAE) did beat the no-persona query on some pairs, so the query is not uniformly the best estimator. The point is narrower and more useful. Marginal agreement is \"not evidence of individual simulation,\" in the author's words.
\n\nTwo further results reinforce it. First, on a capped question (pick at most two), two models produced probability vectors summing to 620 and 760 points against a human total of 192, which reversed the advantage until the vectors were projected onto the cap. Second, on three demographic subgroups with no published table, neither persona panels nor the prevalence query beat simply reciting the national distribution (7.00 MAE). Subgroup estimates are where synthetic panels are most often sold, and that is where the evidence was weakest.
\n\nThe study does not say synthetic panels are useless. It says the standard validation, \"our percentages match a published survey,\" cannot carry the weight placed on it. A practical checklist drawn from the paper:
\n\nFor qualitative and UX teams, this fits a broader pattern: synthetic respondents are most defensible as a way to stress-test materials before fieldwork, not as a substitute for the people you need to hear from. A Digital Twin Panel is best used to pilot questions and spot gaps, followed by real conversations, for example with an AI Interviewer, to collect the evidence itself.
\n\nThis is a single-author preprint that has not been peer reviewed. The human targets differ in mode, date and organisation, so absolute error levels are not comparable across batteries. Personas used only four demographic attributes, so richer grounding (interview transcripts, past answers) was not tested. Training-data contamination cannot be fully ruled out. The author also discloses a commercial interest in synthetic-panel simulation, which makes the critical findings more notable, not less.
\n\nThey can approximate published population percentages, but according to Doudkin (2026) a single prompt without any personas does about as well, and committed persona answers left many real options entirely unselected. Accuracy on averages does not show that individuals are simulated.
\n\nIt is when no simulated respondent ever picks an option that real people do pick, so the panel's share for that option is exactly zero. In the study, 66 of 128 option slots collapsed this way under committed answers.
\n\nCompare it with a no-persona baseline, check coverage of every answer option, test on subgroups or questions without published benchmarks, and evaluate the individual or cross-question patterns you plan to rely on.
\n\nThey reduce average error by 4.53 to 7.30 points on uncapped questions, but they turn the panel into averaged forecasts. That is useful for population estimates, not proof of individual-level simulation.
\n\nLast updated: October 22, 2026. This article is an independent editorial summary of third-party research; Qualitati is not affiliated with the author. See the original paper for full methods and results.
\n","related":[{"slug":"llm-as-judge-qualitative-interpretation-2026","title":"Can LLM-as-Judge Rate Qualitative Interpretations? A 2026 Test","description":"Can LLM-as-judge replace human review of qualitative interpretations? A 2026 study of 3,560 AI interview readings finds it screens models but misses nuance.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"rehearse-interview-protocol-simulated-participants-2026","title":"Can You Rehearse Interviews with AI Participants? A 2026 Study","description":"Can you rehearse an interview guide with AI-simulated participants? A 2026 study of 21 researchers found real protocol revisions and three hidden risks.","category":"Methodology","date":"2026-10-21","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"synthetic-participants-pilot-study-design-2026","title":"Can Synthetic Participants Pilot Your Study Design? (2026)","description":"Can synthetic participants pilot your study design? A 2026 study found LLM reasoning traces caught more survey flaws than a paid human pilot or a chatbot review.","category":"Methodology","date":"2026-10-20","readTime":"7 min read","author":"Qualitati Research Team"}]}