Methodology

Can LLM Digital Twins Answer a Personality Test as You? (2026)

Qualitati Research Team

2026-11-20

7 min read


Not reliably. In a November 2026 study, four LLMs read a person's Facebook posts and filled in a 60-item Big Five questionnaire as that person. Average errors looked acceptable, but similar scores came from the models' default answers with no text at all. The models also pulled people with very high or very low traits back toward the average.

The study is "Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based Evaluation" by Francesco Di Cursi, Chiara Boldrini, Andrea Passarella and Marco Conti. The authors are at the Institute of Informatics and Telematics of Italy's National Research Council (CNR) in Pisa, and Di Cursi is also at the University of Pisa. It was posted to arXiv on 12 November 2026. It asks a simple question with a large consequence for synthetic research: can a model answer a standard questionnaire as a specific individual, rather than as a plausible average person?

What did the study test?

The researchers treated personality prediction as a digital-twin task. The model does not output a trait label. It answers each questionnaire item as the person would, and the answers are scored the way a human questionnaire is scored.

  • Data: the MyPersonality Facebook dataset. The authors kept 125 users with at least 25 posts each and merged each user's posts into one document (median of about 867 words). Every user has self-reported Big Five scores.
  • Instrument: the BFI-2-60, with 12 items per trait (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) on a 1–5 scale.
  • Models: GPT-4 plus three small open models run locally: Gemma 7B, Llama 3.1 8B and Mistral 7B, all at temperature 0, with five runs per experiment.
  • Formats: "PerItem" (one call per question) and "OneBatch" (the whole questionnaire in a single call).

The baselines are what make the study useful. Each text-based prediction was compared with three population references and with the same model answering without any user text. The population references were a value sampled from the norms of a 130,000+ person internet sample, the population mean, and the scale midpoint of 3. That no-text condition shows what the model believes about "people in general" before it learns anything about the individual.

How accurate were the LLM digital twins on average?

On average they often beat the main population baseline, but that alone proved little. According to Di Cursi et al. (2026), Gemma and GPT-4 reached a mean absolute error (MAE) of 0.49 and 0.50 on Openness in the PerItem format, against 0.68 for the sampled population baseline. Yet Gemma and Llama scored 0.47 on Openness with no user text at all. Mistral reached 0.54 on Agreeableness with no text, against a baseline of 0.75.

Setting (MAE, lower is better)No user textWith user textPopulation baseline
Llama, Agreeableness, OneBatch1.330.68n/a
Mistral, Extraversion, OneBatch1.320.88n/a
GPT-4, Openness, PerItem0.580.500.68
GPT-4, Agreeableness, PerItem0.690.60n/a
Gemma / Llama, Openness, PerItem0.47n/a0.68

Some conditions improved a lot with text, but the pattern was not systematic. The authors report that several text and no-text comparisons were nearly identical and that no-text sometimes did better. Their first key finding is that a low aggregate error does not show that the model used anything person-specific.

Do LLM digital twins capture extreme personalities?

Mostly no. When the authors split error by where the person sits on each trait (lowest quarter, middle half, top quarter), the apparent successes shrank. Models were more convincing for typical people and drifted toward average answers for distinctive ones.

The clearest example is GPT-4 on Openness. Its aggregate MAE of 0.50 beat the 0.68 baseline. Broken down, the error was roughly 1.1 for the least open quarter of users, 0.4 for the middle and 0.2 for the most open quarter. The good average came almost entirely from one end of the scale. For low-Openness users, GPT-4 did worse than simply sampling from population norms.

A few cases did move away from the priors in a useful way, notably GPT-4 on Agreeableness and, less strongly, Llama and Mistral on Conscientiousness. The authors call these exceptions too limited to change the overall picture.

Does giving the model the target profile help?

Only for GPT-4, and only on some traits. In a stress test, each model received the person's true Big Five scores instead of their posts and was asked to answer the questionnaire to match. GPT-4 stayed below the baselines across all three regions of the distribution for Extraversion and Neuroticism. Gemma, Llama and Mistral stayed close to the baselines in at least one tail. So even when told exactly who to be, most models could not express a distinctive profile consistently.

Does more text make digital twins more accurate?

No. As the authors fed progressively longer segments of a person's writing, error did not fall steadily. It improved, stayed flat or got worse depending on the segment and model. GPT-4 was steadier than the open models, which oscillated more. Gemma began returning invalid output once inputs passed about 3,500 words.

What this means for researchers using digital twins

The study is about personality items, but the lesson carries over to any synthetic respondent or digital twin used in survey and market research:

  1. Always run a no-context baseline. Ask the same model the same questions without the person's data. If the twin barely beats this, it is reproducing the model's priors, not the person.
  2. Report error by segment, not only on average. Split results by low, middle and high scorers. Averages can hide a twin that fails exactly the respondents you most want to understand.
  3. Do not assume more context is better. Test how accuracy changes as you add background material, and watch for context-length failures in smaller models.
  4. Validate against real answers before using twins for decisions. Treat them as a fast way to explore hypotheses, then confirm with real people, for example through AI-moderated interviews with actual participants.

If you build panels with a tool such as QualiTaTi's Digital Twin Panel, these checks are worth keeping in your validation plan. For a related look at individual-level accuracy, see our summary of how accurate LLM digital twins of real respondents are.

What are the study's limitations?

The reported results come from one dataset (MyPersonality) and one questionnaire (BFI-2-60). The authors ran the Pandora dataset and other questionnaires as well but did not report them in the main paper. The model pool is small, GPT-4 is the only proprietary model, and the authors note that newer models may do better, especially with long context. The paper is a preprint and has not been peer reviewed.

FAQ

Can an LLM fill in a personality questionnaire as a specific person?

Not reliably yet. In this 2026 study, four models given a person's social-media posts produced plausible average answers, but their accuracy was often matched by their own no-text answers and by population norms.

Why do LLM digital twins struggle with extreme respondents?

The models tend to regress toward typical answers. People with very low or very high trait scores were often pulled back toward the middle, so average accuracy hid large errors in the tails.

How should I validate a digital twin panel?

Compare it with a no-context run of the same model and with a population baseline, report errors separately for low, middle and high scorers, and check the twin against real respondents before relying on it.

Last updated: 20 November 2026. This is an independent editorial summary of third-party research by Di Cursi, Boldrini, Passarella and Conti (2026); it is not affiliated with or endorsed by the authors. See the original paper for full methods and results.


Try Qualitati Free

Start conducting AI-powered qualitative research today. No credit card required.

Get Started Free
Related Articles