Can AI predictions make a survey more precise without replacing respondents? Yes, modestly. A 2026 study by Lai Wei and Yifei Zhu (University of Hong Kong) used prediction-powered inference to combine model predictions with real survey answers across 83 outcomes. Supervised models added about 7–12% effective sample size for means; LLMs added less, but still helped.
The paper takes a clear position in the synthetic-respondent debate: AI predictions cannot stand in for people, but they can stretch the value of every real interview. Below we explain the method, the numbers, and what it means for researchers who are tempted to swap respondents for language models.
What is prediction-powered inference (PPI)?
Prediction-powered inference is a way to use a model's guesses about survey answers while letting real responses correct the model's errors. It works in two steps:
- Predict at scale. A model predicts the outcome (for example, self-rated health) for every person in a very large file that has demographics but no answers. In this study that file was the American Community Survey (ACS), with roughly 3.2–3.4 million person records.
- Correct with humans. The same model predicts the outcome for the real survey respondents. The average gap between their actual answers and the predictions is added back as a correction.
Because the correction comes from real people, the estimate stays on target even when the model is biased. Precision improves because the survey only has to estimate the part of the outcome the model cannot predict. Sociologists have also called this approach a "mixed subjects design". For the same reason, our earlier piece on why synthetic survey data breaks confidence intervals argued that uncorrected synthetic answers should never feed population estimates directly.
What did the study test?
The study tested whether AI predictions improve the precision of two major U.S. probability surveys. According to Wei and Zhu (2026), the benchmark covered:
- 37 opinion outcomes from the 2022 General Social Survey (GSS), with 3,544 respondents in the target wave and 46,971 historical observations from 1988–2021.
- 46 health outcomes from the 2017 National Health Interview Survey (NHIS), with 78,132 observations in the target wave.
- Harmonized covariates available in both the surveys and ACS: sex, age, marital status, race, nativity, citizenship, education, employment, occupation, industry, income and poverty status.
Three kinds of predictor were compared: a supervised ensemble trained on past survey waves ("historical"), a supervised model cross-fitted within the current wave ("cross"), and LLM respondent simulation using GPT-5-mini, Gemini 2.5 Flash and Qwen3 (including a fine-tuned Qwen3 variant).
How much does AI improve survey precision?
AI predictions improved precision for population means, but the gains were moderate and supervised learners beat LLMs. The table summarizes the average results reported in the paper.
| Predictor | Survey | Confidence-interval width reduction (means) | Effective sample size gain (means) |
|---|---|---|---|
| Historical supervised learner | NHIS (health) | 5.4% | 12.2% |
| Cross-fitted supervised learner | NHIS (health) | 4.3% | 9.8% |
| Best LLM | NHIS (health) | 1.2% | 2.6% |
| Historical supervised learner | GSS (opinion) | 3.4% | 7.5% |
| Cross-fitted supervised learner | GSS (opinion) | 3.0% | 6.5% |
| Best LLM | GSS (opinion) | 1.4% | 2.8% |
Averages hide wide variation by outcome. In NHIS, the largest gains for means were roughly 34–36% for self-rated health, functional limitations and minutes of vigorous physical activity. In GSS, the best outcomes (computer use, marijuana legalization, home ownership and social class) reached about 21–23%. Many social and political attitudes, by contrast, were weakly predicted by demographics alone.
Why are regression coefficients harder?
Gains for regression coefficients were usually smaller and less consistent than gains for means. The authors explain that a predictor must reduce the variation relevant to a specific association, not just predict the outcome well. Their practical advice is to evaluate a predictor for the exact estimand you care about, not by overall accuracy.
Where do LLMs still help?
LLMs are most useful when no labeled training data exist yet, such as for a new survey question. Supervised models need past answers; an LLM only needs a demographic profile and the question wording. For self-rated health, Wei and Zhu (2026) report that a cross-fitted supervised model reached a prediction–response correlation of about 0.33 with 100 training cases, 0.46 with 1,000 and 0.49 with 5,000. GPT-5-mini scored about 0.443 with no outcome-specific training, roughly matching the supervised model at around 1,000 cases. Gemini 2.5 Flash was lower at about 0.422. As training data grow, the supervised model pulls ahead.
How small effects change test results
A modest prediction accuracy can still change whether a result reaches significance. The paper's worked illustration uses a survey of 1,000 people. With a predictor of R² = 0.30 and an auxiliary file of 10,000, the effective sample size rises to about 1,370, and a survey-only p-value of about 0.114 drops to about 0.064. With R² = 0.60, effective sample size reaches about 2,174 (10,000 auxiliary records) or 2,463 (100,000), and the p-value falls to roughly 0.020 or 0.013. The rule of thumb behind this: effective sample size is about n / (1 − R²), so an R² of 0.10 gives an 11% gain and 0.40 gives 67%.
What this means for researchers
The study supports a simple design principle: use AI to raise the return on each human response, not to remove the human. In practice:
- Keep a real probability sample. PPI only works because real answers correct the model. It does not fix nonresponse or coverage bias, and the survey and auxiliary file must describe the same population.
- Benchmark against simple baselines. The authors recommend auditing LLM outputs and comparing them with supervised models before trusting them.
- Expect more help for behaviors and health than for attitudes. Opinions tied loosely to demographics benefited least.
- Check subgroups separately. Strong overall accuracy can hide weak prediction in small groups.
The same logic applies beyond large public surveys. If you use digital twin panels to explore ideas, treat their answers as provisional predictions and validate the key estimates with real participants, for example through conversational AI surveys fielded to an actual sample.
Frequently asked questions
Can LLMs replace survey respondents?
No. In this study, LLM predictions were useful only after real survey answers corrected their errors. The authors argue that treating synthetic answers as respondents can make biased results look deceptively precise.
Is prediction-powered inference the same as using synthetic data?
No. Synthetic data replaces real answers with generated ones. PPI keeps every real answer and uses predictions only to reduce variance, with a built-in correction for the model's bias.
Which predictor worked best?
A supervised ensemble trained on past survey waves gave the largest average gains: 12.2% effective sample size for NHIS means and 7.5% for GSS means. LLMs gave smaller positive gains.
Where can I read the original paper?
The preprint is "Using AI Predictions to Augment Rather Than Replace Surveys" by Lai Wei and Yifei Zhu (arXiv, November 2026). The authors note that replication materials are hosted on OSF.
Last updated: November 12, 2026
This article is an independent editorial summary of third-party research by the Qualitati Research Team. Qualitati is not affiliated with the study's authors. All figures are taken from the paper; please consult the original for full methods and caveats.