Short answer: Mostly, but not always. In an October 2026 preprint, three LLMs filled in held-out answers on two attitude surveys more accurately than a baseline that simply picked the answer most semantically similar to a respondent's earlier one. When only 20% of a respondent's answers were known, on a niche survey the models could not have seen in training, that advantage disappeared.
Most tests of synthetic survey respondents compare LLM output with human data and stop there. A new paper by Esko Lehtonen (VTT Technical Research Centre of Finland) and Tommi Buder-Gröndahl (University of Helsinki) adds a step that is still uncommon: it asks whether the LLMs did better than a deliberately simple method that knows nothing about the topic and only measures how similar the questions sound.
What did the study test?
The study tested survey data augmentation: whether an LLM, given part of a real person's answers, can predict that person's answers to the remaining questions. The authors argue this is a more promising use than asking LLMs to invent respondents from scratch.
The worry they address comes from the semantic theory of survey response. Items that are worded alike tend to get similar answers. According to work cited in the paper (Arnulf et al., 2014), semantic similarity alone explained 60%–86% of the variance in answers to one leadership questionnaire. If that is true, an LLM could look accurate just by matching wording, without any real understanding of the people or the topic.
The setup, step by step
- Two real surveys. AV measures acceptance of automated driving: 22 items, six factors, 500 respondents sampled at random from an open dataset on Zenodo. LEV measures views of light electric vehicles: 21 items in four factors plus eight further questions, 426 respondents (from 646 after removing missing and "don't know" answers). LEV is not publicly available.
- Three masking levels. For each respondent, roughly 20%, 50% or 80% of answers were shown and the rest were hidden.
- Three LLMs. Gemini 3.7 Flash, GPT 5.6 Luna and Mistral 4 Small were told to act as a consistent survey respondent and answer the hidden items on the same five-point agree–disagree scale.
- Two semantic baselines ("Semagents"). These picked each missing answer by finding the response sentence most similar, in embedding space, to one the person had already given. One used SBERT embeddings and one used Gemini embeddings.
- Scoring. Agreement with the real answers was measured with quadratic-weighted Cohen's kappa, with bootstrapped confidence intervals clustered by respondent. The authors also checked whether correlations between items and the known factor structure survived.
Do LLM survey respondents beat semantic baselines?
On the open, widely studied AV survey, yes, every time. According to Lehtonen and Buder-Gröndahl (2026), all three LLMs beat the best semantic baseline at every masking level on AV. At 50% known answers the models ranked in a consistent order: Gemini 3.7 Flash, then GPT 5.6 Luna, then Mistral 4 Small.
On the niche LEV survey, the advantage was much smaller. Only Gemini 3.7 Flash (at 50% and 80%) and GPT 5.6 Luna (at 50%) were significantly better than the baselines. At 20% known answers, all models performed poorly and the difference between LLMs and baselines disappeared.
For scale: in a 100-respondent prompt-robustness check at 50% known answers, the LLMs reached kappa values of 0.540 to 0.621. In the main analysis at the same masking level, the semantic baselines scored 0.336 to 0.413. When the LLMs were told to be an inconsistent respondent, kappa turned negative, which shows that the prompt really did drive their behaviour.
How much data does an LLM need to fill in answers?
About half. For the LLMs, going from 20% to 50% known answers produced a larger gain than going from 50% to 80%. The baselines were less sensitive to how much data they had.
Structure tells the same story. With 50%–80% of answers known, the LLM-completed datasets reproduced the surveys' validated factor structure. At 20%, only Mistral 4 Small's data could be fitted at all, and data from the semantic baselines never could. The table below shows how well each method preserved the covariance between items at the hardest setting.
| Method (20% of answers known) | AV: RV coefficient (higher is better) | LEV: RV coefficient | AV: Frobenius distance (lower is better) | LEV: Frobenius distance |
|---|---|---|---|---|
| Gemini 3.7 Flash | 0.94 | 0.91 | 5.50 | 5.68 |
| GPT 5.6 Luna | 0.91 | 0.87 | 5.86 | 6.63 |
| Mistral 4 Small | 0.93 | 0.89 | 5.31 | 5.39 |
| Semagent (Gemini embeddings) | 0.74 | 0.95 | 11.71 | 5.00 |
| Semagent (SBERT) | 0.71 | 0.91 | 14.42 | 11.26 |
Source: Lehtonen & Buder-Gröndahl (2026), Table 1.
The LEV columns are the interesting part. With sparse data on an unfamiliar topic, a simple embedding baseline kept the shape of the item correlations at least as well as the LLMs did.
Why did the LLMs do better on one survey?
The authors give two possible reasons and do not claim to have shown either. First, the AV data is openly available, so it may appear in the models' training data. Second, automated-vehicle acceptance is a heavily studied topic, so models have seen many similar questionnaires. LEV is unpublished and covers a narrower subject. When the models lacked both familiar material and enough of the person's own answers, they appear to have fallen back on the same wording-level cues a semantic baseline uses.
The paper also reports systematic item-level biases. Each LLM rated certain items consistently higher or lower than humans did, regardless of how many answers it was given.
What this means for researchers using synthetic respondents
- Add a cheap baseline to every validation. If an LLM cannot beat "pick the answer most similar to what this person already said", its apparent accuracy is not evidence that it understands your respondents.
- Test on data the model has not seen. Validating on a famous public dataset may overstate performance. Hold out unpublished data from your own study.
- Treat sparse inputs as a red flag. Completing a profile from half its answers behaved very differently from completing it from one fifth.
- Check structure and item bias, not only agreement. Kappa alone hid the factor-structure failures at 20% and the per-item biases.
The same logic applies to digital twins built from interview or survey data. If you use QualiTaTi's Digital Twin Panel, ground each twin in as much of the real person's data as you can, and compare its answers with held-out responses before you rely on it. For new questions, real answers collected through AI Surveys remain the reference point. For related evidence, see our summaries of LLMs predicting answers to unseen surveys and why silicon samples match answers but not respondents.
Limitations
This is a short, 16-page arXiv preprint. It covers three English-dominant models, two English-language surveys in one domain (vehicle attitudes), and Likert items only. The authors call for wider tests across domains, languages, models and population groups. The analysis code is to be released if the paper is accepted.
Frequently asked questions
What is a semantic baseline for LLM survey responses?
It is a simple method that predicts a missing answer by choosing the response whose wording is closest, in embedding space, to an answer the person already gave. It has no knowledge of the topic. It tests whether an LLM adds anything beyond similarity of wording.
Can LLMs fill in missing survey answers?
Partly. In this study they agreed with real answers better than semantic baselines in most conditions, and with at least half of the answers known they preserved the survey's factor structure. Performance fell sharply when only 20% of answers were known.
Which model performed best?
At 50% known answers, Gemini 3.7 Flash ranked first, GPT 5.6 Luna second and Mistral 4 Small third. The authors tested only these three models.
Does this mean synthetic respondents are useless?
No. It means their accuracy depends on the survey and on how much real data you condition them on, and that validations should include a simple baseline before anyone treats the results as evidence.
Source: Lehtonen, E., & Buder-Gröndahl, T. (2026). LLMs as survey respondents: Comparison to simple semantic baselines. arXiv:2610.29645 [cs.CL].
Last updated: November 2, 2026
This article is an independent editorial summary of third-party research. QualiTaTi is not affiliated with the authors, VTT or the University of Helsinki.