Batching Synthetic Respondents Inflates Correlations (2026)
Qualitati Research Team · 2026-10-13 · 7 min read
Short answer: Yes. If you ask an LLM to generate several synthetic survey respondents in one request, the earlier answers leak into the later ones. A 2026 LSE preprint found that generating ten respondents per request instead of one raised correlation error by 48–58% for Qwen and 114–127% for Llama, mostly by exaggerating how strongly variables move together.
Synthetic respondents are usually judged on whether each answer distribution looks human: the right share of people who trust parliament, the right spread of income. Much less attention goes to the relationships between variables, which is what most analyses actually use. A new preprint by Thomas S. Robinson of the London School of Economics isolates one overlooked design choice that changes those relationships: how many synthetic people you put into a single LLM completion.
What is shared-completion batching?
Shared-completion batching means asking a model to answer for several people inside one generated response. It is cheaper and faster than one call per person, and applied studies do it: Robinson cites papers that request 30 to 50 respondents per completion, and a qualitative interview simulation that requests four people's answers together. The catch is that a language model writes token by token. By the time it answers for person ten, the answers it already wrote for persons one to nine sit in its context window. If a generator is meant to sample each person independently given their profile, splitting the same profiles across requests should not change the results. Robinson calls this property request-partition invariance, and shows that current models do not have it.
What did the study test?
The main experiment held everything constant except request size. According to Robinson (2026), the setup was:
- Profiles: 2,000 real European Social Survey (ESS) respondents, 100 from each of 20 countries, each described by ten background characteristics.
- Outcomes: 19 numeric answers covering trust, political attitudes, well-being, income, age and interview year.
- Prompt: each target profile plus three examples pairing other people's profiles with their real answers.
- Models: Qwen3.8-27B and Llama-3.3-70B-Instruct, three random seeds each, temperature 1.
- Manipulation: the same targets generated one per request or ten per request.
- Metric: mean absolute error (MAE) between synthetic and human correlations within each country, alongside separate measures for single-variable distributions and regression estimates.
Two further datasets extended the test to US health data (NHANES 2017–2018) and income and work data (ACS 2022, employed Massachusetts adults), with 800 people each.
How much does batching distort synthetic data?
Substantially, and in every run. On matched profiles, correlation MAE rose from 0.183 to 0.278 for Qwen and from 0.187 to 0.415 for Llama when moving from one to ten respondents per request. Error increased for 155–159 of the 171 variable pairs in each Qwen run and 148–154 in each Llama run.
| Measure (ESS experiment) | Human reference | Qwen: 1 vs 10 per request | Llama: 1 vs 10 per request |
|---|
| Correlation error (MAE) | — | 0.183 → 0.278 | 0.187 → 0.415 |
| Relative increase across seeds | — | +48–58% | +114–127% |
| Average correlation strength (mean |r|) | 0.193 | 0.360 → 0.459 | 0.355 → 0.596 |
| Rank agreement with human ordering (batched) | — | 0.864 | 0.866 |
Source: Robinson (2026), arXiv:2609.32546, Section 3.1.
The table carries a second message that is easy to miss. Even one-at-a-time generation produced average correlations of about 0.36, nearly double the human 0.193. Batching took an existing tendency to overstate relationships and made it worse.
Is the pattern wrong, or just the strength?
Mostly the strength. Batched correlations still ranked variable pairs in roughly the human order, with Spearman agreement of 0.864 and 0.866. When Robinson rescaled each set of correlations by its single best multiplier, a diagnostic that uses the human answers and is not a usable fix, one-person and ten-person error both fell to about 0.10. The synthetic data knew which variables go together; batching made it too confident about how much.
Do lower temperature or better prompts fix it?
No. The penalty persisted at temperature 0.5, after removing an instruction to "preserve realistic individual variation", and under constrained structured output, where ESS error still rose 51% for Qwen and 122% for Llama and NHANES error rose 56% and 54%.
Why do earlier answers change later ones?
Because the model reads them. Robinson ran a controlled test: give the model nine completed respondents and ask only for the tenth, keeping the same answer values but re-pairing them so the nine showed a positive, negative or zero relationship. With Qwen on NHANES, the correlation between BMI and sedentary time among the newly generated tenth respondents swung from 0.963 to −0.670 depending only on how the earlier values were paired. Qwen on ESS trust items moved from 0.985 to −0.647, and Llama moved in the same direction in both domains. The supplied profiles and value distributions were identical across conditions, which makes answer history a causal channel.
Hiding earlier answers from the model reduced correlation error in every tested setting, for example from 0.367 to 0.197 when the target was tenth in line for Qwen on NHANES. That gain appeared even with a single preceding respondent, so the model does not need to infer a full correlation matrix to be influenced.
Is one respondent per request always better?
Not for every statistic, and this is the paper's most useful warning. Batched outputs had better single-variable distributions than one-person outputs in all six model and dataset settings. Hiding history improved correlations but raised marginal error by 8–11% in Qwen on NHANES. Exploratory post-hoc corrections that shrank correlations lowered correlation error in all six settings while worsening marginal accuracy in each, and a single shrinkage multiplier delivered 93–101% of the gain of a full pair-by-pair correction. In NHANES, the corrected data were still less accurate than simply predicting zero correlation for every pair. Regression coefficients improved in some settings and worsened in others. A synthetic sample can look better on one check and worse on the one your analysis depends on.
What this means for researchers using synthetic respondents
- Treat request construction as part of your method. Report how many synthetic people share a completion and whether they can see each other's answers, alongside the model, prompt and temperature.
- Validate the quantity you will analyse. If you will run correlations or regressions, matching percentages is not enough. Compare relationships and coefficients against a human benchmark.
- Do not trust a single fix. Isolated generation, hidden history and shrinkage each traded one kind of accuracy for another in this study.
- Keep humans in the loop for decisions. Synthetic panels are most defensible for drafting and pretesting instruments before a real survey, not as a replacement for one.
If you work with synthetic panels, Qualitati's Digital Twin Panel lets you compare simulated answers with a real sample collected through AI Surveys, rather than relying on synthetic output alone.
Limitations to keep in mind
This is a single-author preprint on arXiv and has not been peer reviewed. It covers two open-weight model families and one basic prompt design, mostly at temperature 1. Evidence from the ACS income data is weaker because the ten-person conditions there failed the author's validity threshold. The correction experiments are explicitly exploratory, and the study does not establish a cheaper generation method that works in general.
FAQ
Should I generate synthetic survey respondents one at a time?
If relationships between variables matter to your analysis, generating each respondent in a separate request avoided the batching penalty in this study. It did not make single-variable distributions better, so validate both.
Why do synthetic respondents show stronger correlations than real people?
In this study, even isolated synthetic respondents had average correlations near 0.36 against 0.193 for humans. Batching pushed them higher because the model conditions on answers it already wrote for earlier respondents.
Does this affect AI-simulated interviews and focus groups?
The experiments used numeric survey answers only. The mechanism, later outputs conditioning on earlier ones in the same context, also applies when several simulated participants share a prompt, but the paper does not measure qualitative outputs.
Which models were tested?
Qwen3.8-27B and Llama-3.3-70B-Instruct. The size of the effect differed between them, roughly twice as large for Llama in the main experiment.
Source: Robinson, T. S. (2026). Shared Autoregressive Context Can Distort Relationships in Synthetic Data. arXiv preprint arXiv:2609.32546. Read the paper.
Last updated: October 13, 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the author.