Are LLM Survey Respondents Psychometrically Valid? A 2026 Audit
Qualitati Research Team · 2026-09-07 · 7 min read
Short answer: No — not psychometrically. A 2026 audit of 37 large language models by Lukauskas and Šarkauskaitė found that synthetic respondents reproduce the direction of human survey relationships but fail on structure: a plain Gaussian-copula statistical baseline beat every LLM, and models trained on synthetic answers lost predictive validity on real people.
What the study actually tested
The study asks a harder question than most synthetic-respondent evaluations. Instead of asking whether an individual answer looks plausible, it asks whether a synthetic sample preserves the joint properties that make survey data usable: the correlation structure, the latent factor structure, scale reliability, mediation pathways, and demographic effects.
According to Lukauskas and Šarkauskaitė (2026), the design was:
- A Lithuanian organisational-psychology dataset of 263 employees, answering 68 items across 12 subscales (Dunham Attitudes Toward Change, UWES-17, and the Koopmans Individual Work Performance Questionnaire).
- A lineup of 37 models spanning OpenAI, Anthropic, Google, and twelve open-weight families.
- Each model conditioned on real respondent profiles under a five-level persona-disclosure ladder, plus presentation and reasoning-effort ablations.
- Counterfactual demographic swaps (gender, role, education), a cross-language check, and a verbatim-recall probe to test whether results were driven by memorised training data.
- A Psychometric Similarity Score (PSS) anchored against five non-LLM statistical baselines and a held-out human-vs-human ceiling, with respondent-bootstrap confidence intervals and an item-permutation null for Tucker's phi.
That last point is worth borrowing regardless of what you think of LLMs. Most synthetic-data claims have no ceiling and no floor: without a human-vs-human benchmark you cannot tell whether a similarity score of 0.7 is good, and without a statistical baseline you cannot tell whether the model added anything a copula could not.
How close did the synthetic respondents get?
Close enough to look right, far enough to be misleading. The reported failures cluster into a few kinds.
| Failure mode | Reported finding |
|---|
| Beaten by a statistical baseline | A Gaussian-copula baseline outperformed every LLM on the sample-driven components of the PSS |
| Synthetic monoculture | Mean inter-LLM PSS of 0.73 — models resemble each other more than they resemble humans |
| Acquiescence bias | Every model showed an agreement shift of +0.84 SD |
| Lost predictive validity | Regressors trained on synthetic data scored mean R² of −0.18 on held-out humans, versus 0.28 otherwise |
| Fabricated mechanisms | Models produced indirect effects on 3 of 10 placebo mediation paths |
| Collapsed factor structure | Tucker's phi on the UWES fell inside the item-permutation null for 8 of 37 models |
Two of these deserve emphasis. A negative R² means the synthetic-trained model predicts real people worse than simply guessing the mean — not a weaker signal, an actively harmful one. And the placebo mediation result should worry anyone using synthetic panels for theory testing: the authors included mediation paths that should have shown nothing, and the models produced an effect on three of them.
Why the memorization result matters
A standing objection to synthetic-respondent benchmarks is that good scores come from models having memorised the instruments. This study probed for that with a verbatim-recall test and reports a recall-PSS rank correlation of 0.00 — recall ability did not predict leaderboard position.
That cuts both ways. It rules out memorization as the explanation for the successes, which strengthens the finding that models genuinely track the qualitative direction of human relationships. It also removes the comforting excuse for the failures: the models are not failing because they never saw these scales, they are failing because generating a coherent population is a different task from generating a coherent person.
The education effect nobody prompts for
The counterfactual swaps produced an ordering researchers rarely anticipate. Changing the described respondent's education moved answers with a mean absolute effect size of 0.56, dwarfing role (0.18) and gender (0.12).
If you build personas, you are probably specifying gender and job title carefully and treating education as a background detail. In this audit it was the dominant lever. That is a concrete, testable prompt-design implication: hold education constant, or vary it deliberately, or your synthetic sample's variance will be driven by a variable you did not intend to manipulate.
What this means for your research practice
The paper's conclusion is narrow and stated plainly: LLM samples are not a drop-in replacement for human survey data. It does not say synthetic respondents are useless. Read against the rest of the 2026 literature, a defensible position looks like this:
- Fine for pretesting instruments. Catching a confusing item, a ceiling effect, or a broken skip pattern does not require the joint distribution to be right.
- Not fine for effect sizes, mediation, or anything you will publish as a finding. The placebo-path result is disqualifying for that use.
- Never a training set for a model you will apply to humans. The negative R² is the clearest number in the paper.
- Always benchmark against a statistical baseline. If a copula fitted to a small real sample beats your LLM pipeline, the LLM is adding cost, not information.
The practical alternative is not "collect a full traditional sample or give up". It is to spend a small amount of real human data well. A short round of AI-moderated interviews yields the depth and the anchoring distribution that synthetic generation cannot produce on its own, and a Digital Twin Panel built on top of real respondent data is a different object from one conjured out of a persona string. This audit is an argument for grounding synthetic work in human data, not for abandoning it.
FAQ
Can I use LLMs as survey respondents at all?
For instrument pretesting and pilot mechanics, yes. For estimating effects, testing mediation, or training predictive models, this audit gives strong evidence against it.
Does using a better or larger model fix it?
Not in this study. The failures — acquiescence shift, monoculture, fabricated mediation — appeared across all 37 models, spanning four commercial families and twelve open-weight families.
What is a Gaussian-copula baseline and why does it matter here?
It is a standard statistical method for generating synthetic data that preserves the correlation structure of a real sample. It matters because it beat every LLM on the sample-driven PSS components — the simpler tool did the structural job better.
Is this specific to Lithuanian data?
The dataset is Lithuanian organisational psychology and the authors ran a cross-language check, but a single-country, single-domain sample of 263 is a real limit on generalisation. Treat the direction of the findings as robust and the exact figures as specific to this setting.
Last updated: 2026-09-07. This is an independent editorial summary of third-party research; it is not affiliated with or endorsed by the study's authors.