Why Synthetic Survey Data Breaks Confidence Intervals
Qualitati Research Team · 2026-08-25 · 7 min read
Synthetic survey data — responses generated by an LLM instead of collected from people — usually comes with confidence intervals that are far too narrow. A 2026 paper by Tan and Zrnic shows those naive intervals covered the true answer only 3% of the time on real election-survey data, and proposes a calibration method that restored valid coverage.
What problem does this paper solve?
It solves the error-bar problem in silicon sampling. Researchers already know synthetic respondents are biased; the harder question is what a synthetic estimate is actually evidence for. If you generate 2,000 LLM "respondents" and compute a 90% confidence interval, the interval reflects sampling noise inside the model's output distribution — not the distance between that distribution and reality. The interval is precise about the wrong quantity.
"Valid Inference with Synthetic Data via Task Exchangeability" (Lezhi Tan and Tijana Zrnic, arXiv preprint, June 2026, revised August 2026) attacks this directly. Its argument is that without some assumption linking the true distribution to the synthetic one, inference from synthetic samples alone is impossible — so the useful move is to name the weakest assumption that makes it possible again.
What is task exchangeability?
Task exchangeability is the assumption that your new research question is statistically interchangeable with a set of past questions where you do have real data. Concretely: you have run 30 similar surveys before, each with both real responses and synthetic ones. On each, you can measure how far the synthetic estimate landed from the truth. If your new task is drawn from the same population of tasks, the historical gaps tell you how wrong the new synthetic estimate is likely to be.
The method follows from that. According to Tan and Zrnic (2026), the procedure is:
- Compute the ordinary confidence interval from the synthetic sample.
- For every historical task, estimate the gap between the real-data answer and the synthetic-data answer.
- Take an empirical quantile of those historical gaps.
- Widen the synthetic interval by that quantile in both directions.
The result is an interval with a provable coverage guarantee. It is much wider than the naive one — deliberately so. The width is the finding: it is the honest price of not having collected the data.
How badly do naive synthetic intervals fail?
Badly, and consistently across three very different domains. The reported coverage figures (the share of intervals that actually contain the true value; the target was roughly 90–95%) are below.
| Benchmark | Synthetic source | Naive coverage | Calibrated coverage |
|---|
| ANES feeling thermometer (33 target tasks) | GPT-3.5 | 3% | 97% |
| Pew ATP presidential approval (8 forecast waves) | GPT-4o | 0% | 100% |
| Arena autorater, win rate (73 models) | LLM autorater | 19% | 100% |
| Arena autorater, Bradley-Terry score | LLM autorater | ~73% | 93.3% |
Two details are worth dwelling on. First, the ANES median calibrated interval width was 29.8 points on a 0–100 feeling thermometer — an enormous range, and yet that is what honest uncertainty looked like for that task. Second, the Pew case is the most uncomfortable: the naive intervals contained the truth in zero of the eight forecast waves, because the synthetic bias was systematic rather than noisy. More synthetic respondents would not have helped. That is the whole point.
What does this mean for researchers?
It means a synthetic study is only as trustworthy as the real-data history you can calibrate it against. Three practical implications:
- Keep your old real-data studies. Under this framework, a back-catalogue of human-collected studies stops being an archive and becomes the calibration set. Teams that have run the same tracker for years are in a far better position than teams starting synthetic-first.
- Synthetic-first is defensible for triage, not for point estimates. If a calibrated interval spans 30 points, the study can still separate a large effect from a null one — it just cannot support "approval is 46%."
- Ask which tasks your task resembles. The authors are explicit that exchangeability is a substantive requirement, not a formality. Reusing 2020 election calibration for a 2026 product-concept test is not covered by the theorem.
The same logic transfers to qualitative work. If you use a Digital Twin Panel or a Synthetic Focus Group for early-stage exploration, the disciplined version is to keep running real interviews on a subset of studies so you can see how far the simulated version drifts. Qualitati's AI Interviewer and synthetic tools are designed to sit alongside human data collection for exactly that reason, not to replace it.
What are the limits of the method?
The authors state them plainly. Coverage is marginal over the distribution of tasks, not conditional on any particular task's features — so a single study can still be badly wrong even when the procedure is correct on average, much as with conformal prediction. The method also requires real data for the historical tasks; it removes the need to collect real data for the current task, not for research generally. And identifying a genuinely exchangeable population of tasks is a judgment call that no algorithm makes for you.
Extensions in the paper soften the first constraint: a weighted variant degrades gracefully when exchangeability is only approximately true, and a hybrid version intersects real-data and synthetic-data intervals when you have some of both.
FAQ
Does this mean synthetic survey respondents are useless?
No. It means their uncertainty is routinely misreported. The paper's calibrated intervals were valid — they were simply wide. Wide-but-honest is usable for direction-finding and screening; narrow-but-wrong is not usable for anything.
How much historical data do you need?
The experiments used task counts in the tens — 33 historical ANES tasks, 40 calibration waves for the Pew analysis, and 73 leave-one-out models in the autorater study. The empirical-quantile step means the number of past tasks matters more than the number of respondents inside any one of them.
Does this apply outside surveys?
Yes. The third benchmark is AI model evaluation — 280,737 human-versus-autorater comparisons across 74 models — where "synthetic data" means an LLM judge standing in for human raters. Any setting where a model substitutes for human labels has the same structure.
Can I use it if I've never collected real data?
Not for a formal guarantee. Without real observations on comparable past tasks there is nothing to calibrate the synthetic gap against, and the paper is clear that inference is impossible in that case.
Primary source: Lezhi Tan and Tijana Zrnic, "Valid Inference with Synthetic Data via Task Exchangeability," arXiv preprint arXiv:2606.13629 (June 2026, revised August 2026).
Last updated: 25 August 2026. This is an independent editorial summary of third-party research; it is not affiliated with or endorsed by the authors, and the preprint has not completed peer review.