Total Simulated Survey Error: How to Audit LLM Survey Data

Qualitati Research Team · 2026-09-23 · 7 min read

Short answer: Total Simulated Survey Error (TS2E) is a 2026 framework that adapts the classic Total Survey Error model to survey answers generated by large language models. It sorts what can go wrong into measurement errors (what is being measured), representation errors (whose views are being simulated) and evaluation fallacies, so researchers can trace where a synthetic survey fails instead of reporting one overall accuracy score.

Synthetic survey respondents, also called silicon samples or digital twins, are now common in academic and market research. Most validation work asks one question: how closely do the LLM answers match a human benchmark? A new preprint from researchers at the University of Mannheim, GESIS, the University of Graz and the University of Duisburg-Essen argues that this is the wrong unit of analysis. A single match score cannot tell you whether the problem sits in your prompt, your personas, your model or your evaluation.

What is the Total Simulated Survey Error framework?

TS2E is a checklist of error sources mapped onto the life cycle of an LLM-generated survey. According to Sen, Ahnert, von der Heyde, Lasser, Weiß and Strohmaier (2026), it borrows the core split of the Total Survey Error tradition (Groves and Lyberg, 2010): measurement errors reduce construct validity, while representation errors reduce generalizability to the target population. The authors then add errors that only exist when the "respondents" are models.

The framework also separates two kinds of cause. Some errors come from a researcher's design choices, such as how a persona is written. Others come from inherent limits of current LLMs, such as stereotypes absorbed in training. That distinction matters because only the first kind can be fixed by redesigning the study; the second can only be documented or compensated for.

Which errors does TS2E identify at each stage?

The framework follows the survey from design to analysis. The table below summarizes the seven error types, as described in the paper.

Stage	Error	Type	What goes wrong
Before fielding	Specification error	Measurement	The question does not fully capture the construct (same as in human surveys).
Before fielding	Articulation error	Measurement	Rewording the question, answer options or output format for the model shifts its meaning.
Before fielding	Persona construction error	Representation	The chosen personas, their attributes or their wording do not match the target population; errors in the source survey are inherited.
Fielding	Response generation error	Measurement	The model answers badly: answer-order effects, social desirability, refusals, poor instruction-following, run-to-run instability.
Fielding	Persona simulation error	Representation	The model cannot portray some groups well, however carefully the persona is written.
After fielding	Response processing error	Measurement	Parsing free text or JSON, or using an LLM judge, misreads what the model said.
After fielding	Adjustment error	Representation	Reweighting with human survey weights assumes models and humans err in the same way.

Source: Sen et al. (2026), Section 3. Persona construction and persona simulation errors are new to LLM surveys and have no exact human-survey counterpart.

What are the three evaluation fallacies?

Even a well-designed simulation can be judged wrongly. The paper names three logical traps in comparing synthetic answers with human data:

Ground truth fallacy. Human survey data carries its own errors, so it is not a perfect gold standard. It may also be contaminated: the model may have seen the survey in training, or the "human" answers may themselves have been written with AI help.
Single best metric fallacy. No one metric settles validity. Individual-level scores, distribution distances and downstream effect sizes can disagree, and a good descriptive match does not guarantee the same substantive conclusions.
Context drift fallacy. A simulation validated in one setting (say, one election) is assumed to work in a "similar" one (the next election, another country). The authors note there is no agreed way to measure how similar two contexts are, and closed commercial models can change silently over time.

How did the authors test the framework?

They ran a multiverse analysis simulating vote choice in the 2024 U.S. presidential election, using the American National Election Studies (ANES) as the reference. Each design choice was varied systematically:

Task wording: two phrasings of the vote question.
Personas: demographics only vs. demographics plus attitudes (such as party and ideology), each in a direct "you are..." format or an interview format.
Models: six open-weight models from the Llama, OLMo and Qwen families, at two sizes each.
Response handling: JSON output, free text read by an LLM judge, or free text read by keyword matching.
Adjustment: with or without ANES survey weights.

That produced 288 configurations. The 144 configurations before weighting were each run with 5 random seeds over 4,779 ANES respondents, for 3,440,880 simulated responses in total. The code is public on GitHub.

Which design choices mattered most?

Persona content dominated; most other choices mattered far less. According to Sen et al. (2026), the overall weighted F1 score across all configurations was about 0.52. The key findings:

Attitudes in the persona were the biggest lever. Direct-format personas with demographics plus attitudes averaged an F1 of 0.624, against 0.434 for demographics alone. The authors caution that party identification is strongly tied to vote choice and may be missing in real-world use.
Question wording had no significant effect in this case (F1 0.52 vs. 0.518).
Configurations varied enormously. The best setup reached an F1 of 0.686; the worst reached 0.308. The authors read this sensitivity as evidence of how brittle current LLM simulations are.
The "best" model depended on the metric and the group. Qwen-30B scored highest on F1, OLMo-32B on distribution distance, and other models did better for conservative respondents.
Keyword parsing of free text lowered performance slightly but significantly.
Reweighting with ANES weights also slightly hurt overall results, contrary to what survey practice would suggest.

Results were stable across repeated runs of the same configuration, so for these open models reliability was less of a concern than sensitivity to design.

What this means for researchers using synthetic respondents

The practical message is to treat an LLM survey like any other instrument: document every design choice and test more than one. The paper recommends that researchers:

Preregister the simulation design, because the gap between the best and worst configurations leaves room for cherry-picking.
Report several metrics and break results down by subgroup, not only overall.
Test reliability and sensitivity separately: the same prompt repeated, and small harmless changes to the prompt.
Be careful with closed commercial models when simulating past surveys, since training data and version changes cannot be checked.
Validate out of context, for example on a later survey wave, before claiming the simulation generalizes.

The appendix includes a documentation checklist that extends existing reporting guidance, including the AAPOR standards. If you pilot questions on simulated panels such as Qualitati's Digital Twins, the TS2E stages are a useful list of what to record. When the goal is evidence about real people, a conversational instrument like AI Surveys fielded to human respondents avoids persona and simulation errors altogether.

FAQ

How is TS2E different from Total Survey Error?

It keeps the measurement vs. representation split but adds LLM-specific errors, namely persona construction, persona simulation and response generation error, plus a set of evaluation fallacies that arise when synthetic answers are benchmarked against human data.

Do richer personas fix synthetic survey bias?

Not automatically. In this study, adding attitudes helped a lot for vote choice, but the authors stress that those attitude variables may be unavailable in real use, and that some groups remain poorly simulated regardless of persona design.

Can I rely on one accuracy score to validate synthetic respondents?

No. The two metrics in the case study ranked different models as best, and subgroup results differed from the overall ranking. The authors recommend reporting multiple metrics chosen for your use case.

Does the case study generalize to all synthetic surveys?

The authors say it does not. It illustrates the framework on one U.S. vote-choice task with open-weight models, and design choices that were insignificant here may matter in other settings.

Primary source: Sen, I., Ahnert, G., von der Heyde, L., Lasser, J., Weiß, B., & Strohmaier, M. (2026). Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models. arXiv:2609.10280.

Last updated: September 23, 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the authors.