Short answer: A synthetic population is valid only for the analyses it has been tested for. A September 2026 benchmark ran nine language models through eleven validity tests against twenty human data sources. It found that models often match average answers while answering too consistently, compressing response scales and distorting how traits relate. Validate against the specific analysis you plan to run, not against a single overall score.
Synthetic respondents are language-model agents asked to answer surveys as a specific kind of person would. Market researchers, UX teams and social scientists increasingly use them to pilot questionnaires, screen concepts or fill gaps in hard-to-reach samples. The open question is how to decide whether a given synthetic panel is good enough for a given job. A new preprint by Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis and James K. He, \"Artificial Societies Benchmark: A Validation Framework for Synthetic Research\" (arXiv, 2026), offers a structured answer: a battery of tests grouped by the kind of validity each one checks.
\n\nThe benchmark tests whether a synthetic population can support the specific analysis a researcher intends to run. The authors compared nine models, four proprietary (GPT-5.6 Sol, Claude Opus 5, Gemini 3.8 Flash, Grok 4.3) and five open-weight (Mistral Small 2603, DeepSeek Flash, Qwen3.8-27B, Gemma 4 31B, Llama 4 Scout), against twenty human data sources. Those sources include national surveys such as ANES, the General Social Survey and the European Social Survey, personality inventories, longitudinal panels and published survey experiments.
\nThe eleven tests fall into three overlapping domains:
\n\n| Validity domain | Tests | Question it answers |
|---|---|---|
| Internal validity | Repeated questions, linked questions, paraphrased questions, perturbed surveys | Does the synthetic respondent answer coherently, and react to wording and order changes the way humans do? |
| Construct validity | Item covariance, independent criteria, distinct constructs | Do scale items hang together and relate to outside measures the way they do in human data? |
| External validity | Response distributions, groups and individuals, experimental effects, latent structure | Do answer frequencies, subgroup results, treatment effects and measurement models match the human population? |
Rather than setting a universal pass mark, the authors recommend building a scorecard from only the tests a given research question needs, and fixing acceptable tolerances before looking at results.
\n\nMost models are more consistent than people, which is a validity problem rather than a strength. In one personality dataset, humans gave the same answer to a duplicated item about 73% of the time; Gemma did so 99% of the time, and seven of the nine models exceeded the human rate. On paraphrased value questions, humans agreed with themselves about 85% of the time, while eight models agreed 100% of the time.
\nConsistency and coherence are not the same thing, though. Llama 4 Scout repeated its answers about 93% of the time yet gave opposing endorsements to linked items about 21% of the time, compared with about 8% for humans. Real people are noisy and occasionally contradictory; synthetic respondents tend to be either rigidly uniform or contradictory in different ways.
\nSurvey experiments exposed the same gap. In a task where stated scientific consensus rose step by step, about 48% of human response sequences showed at least one drop in belief. No model sequence did.
\n\nNo. The benchmark found that every model left out response categories that humans used on European Social Survey items. On an 11-point immigration question, the OpenAI model used only positions 6 to 10. One model gave identical probabilities on 28 of 60 judgement items, leaving no variation to correlate with anything else.
\nThis matters because most downstream analysis depends on variance. If a synthetic panel compresses the scale, correlations, regressions and segment comparisons built on it will be distorted even when the mean looks right. The authors illustrate the point with two statistical control baselines that both matched the item distributions closely. The one that treated items as independent got trait correlations badly wrong (mean absolute error 0.170), while the one that preserved rank correlations stayed close (0.040).
\n\nPartly, and unevenly. In an argument-quality experiment with 24 messenger contrasts, where the average human effect was 0.52 rating points, Grok had the smallest mean absolute effect error (0.23 points) and Claude the largest (1.10 points). Some contrasts were shrunk, some exaggerated and some reversed. A synthetic panel that gets the direction of one treatment effect right is not evidence that it will get the next one right.
\n\nIt depends on the model and on how the profile is written. Using a twin dataset of 430 people, the authors gave models richer personal information, including answer histories and distilled text profiles. With answer histories plus distilled text, error fell for Gemini (by 0.047 and 0.041 on two measures) and for Claude (0.027 and 0.028); distilled profiles alone gave the OpenAI model and Gemini a smaller gain of about 0.02. Mistral's error rose slightly, although that increase was not firmly established. The practical lesson is that the profile format is itself part of the simulation design, and has to be validated with the same model you plan to use.
\n\nThe benchmark turns \"is synthetic data valid?\" into a more answerable question: valid for which analysis? A practical workflow based on the paper's recommendations:
\nIf you run a Digital Twin Panel, treat it the same way: use it to explore and pilot, and anchor any claim about how people differ in real responses, for example from an AI Survey fielded to a human sample.
\n\nThe authors list several caveats. Human sources are public, so training-data contamination cannot be ruled out. Models ran with different decoding settings and access routes, so results describe those configurations rather than model families in general. The distilled profiles were produced with the OpenAI model and audited with Claude Opus 5, which may favour those providers. The paper is a preprint and has not yet been peer reviewed.
\n\nIt is a validation framework of eleven tests, grouped into internal, construct and external validity, for judging whether a synthetic population can support a particular analysis. It reports results per test instead of as one overall score.
\nNone won across the board. Rankings changed by test: for example, Qwen had the lowest distributional error on one personality dataset but the highest contradiction rate on another.
\nNot on this evidence. They can approximate some averages, but they compress variance and distort relationships between traits, which undermines analyses of how people differ.
\nChoose the tests that match your planned analysis, compare against a human reference with uncertainty intervals, set tolerances in advance and report every failure or refusal alongside accuracy.
\n\nLast updated: October 15, 2026
\nThis is an independent editorial summary of third-party research by the Qualitati Research Team. Qualitati is not affiliated with the authors. Please consult the original paper for full methods and results.
\n","related":[{"slug":"synthetic-consumers-visual-marketing-pretest-2026","title":"Can Synthetic Consumers Pretest Logos and Ads? A 2026 Study","description":"Can synthetic consumers pretest visual marketing? A 2026 study found GPT agents saw logos correctly but matched at most 2 of 6 human effects and reversed one.","category":"AI User Research","date":"2026-10-14","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"interview-informed-llm-survey-responses-2026","title":"Can Interview Transcripts Predict Survey Answers? A 2026 LLM Test","description":"Can interview-informed LLMs predict survey answers? A 2026 study tested GPT, Claude and Gemini against 19 real respondents. Here is what held up and what broke.","category":"AI User Research","date":"2026-10-12","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"expertise-adaptive-ai-interviewer-local-llm-2026","title":"Expertise-Adaptive AI Interviewers: What a 2026 Study Found","description":"Can an AI interviewer adapt to each participant's expertise? A 2026 study of 246 interviews with a local LLM found 78.9% profiling agreement and kappa 0.80.","category":"AI User Research","date":"2026-10-09","readTime":"7 min read","author":"Qualitati Research Team"}]}