Short answer: Not faithfully yet. A 2026 Carnegie Mellon study ran the same customer-service tasks with 451 real people and 31 LLM user simulators. The best simulator scored 76.0 on the authors' User-Sim Index, against 92.7 for agreement between human groups. Simulated users were too cooperative, too polite and too uniform, and they rated the AI they talked to more kindly than humans did.
Synthetic users are now a standard shortcut. Product teams use LLM "customers" to stress-test chatbots, researchers use simulated participants to pilot interview guides, and agent developers score their systems against simulated users. All of this assumes the simulator behaves roughly like a person. The paper "Mind the Sim2Real Gap in User Simulation for Agentic Tasks" (Zhou, Sun, Ma, Xie, Liu, Du, Welleck, Yang, Neubig, Wu and Sap, 2026; accepted at COLM 2026) tests that assumption directly, and the answer matters well beyond agent benchmarks.
What did the study test?
The study compared real humans and LLM simulators playing the customer in the same conversations. The authors used τ-bench, a benchmark of airline and retail customer-service tasks, and ran the full protocol with people instead of models. According to Zhou et al. (2026), the human side involved 451 participants recruited on Prolific, covering 165 tasks in three independent batches. The AI agent on the other side was held fixed (GPT-5.2), so any difference came from who played the user.
On the simulator side they benchmarked 31 models: 18 proprietary, 9 open-source and 4 models fine-tuned specifically for user simulation or human-likeness. After each conversation, both humans and simulators rated the interaction on task success and quality dimensions such as efficiency, human-likeness and interaction flow. That second step is what makes the paper useful for researchers: it measures not only whether simulators act like people, but whether they judge like people.
How do the authors measure the Sim2Real gap?
They split the gap into a behavioural part and an evaluative part, then combine them into one score, the User-Sim Index (USI), on a 0–100 scale.
- Communication style (D1): politeness, verbosity and LLM-typical punctuation.
- Information pattern (D2): how much information the user shares per turn.
- Clarification (D3): asking questions and expressing uncertainty.
- Error reaction (D4): emotion, accusatory language and changes of strategy when things go wrong.
- Outcome calibration (ECE): whether the agent succeeds as often with the simulator as with humans, across easy and hard tasks.
- Evaluative alignment (Eval): how closely the simulator's post-task ratings match human ratings.
The four behaviour scores compare population-level distributions of simulator and human features, so a simulator does not need to copy any one person, only to look like the crowd. The authors stress that USI is a summary, and that the component scores carry the diagnostic information.
How close do LLM user simulators get to real people?
The best simulators land in the mid-70s, well short of the human benchmark. Agreement between the three human batches gives a ceiling of 92.7. The table below shows selected results reported by Zhou et al. (2026).
| User | Style (D1) | Clarification (D3) | Rating alignment (Eval) | USI |
|---|---|---|---|---|
| Humans (between batches) | 87.4 | 88.0 | 97.4 | 92.7 |
| DeepSeek-V3.1 | 45.1 | 74.5 | 74.3 | 76.0 |
| Gemini 2.0 Flash | 51.6 | 68.2 | 73.7 | 74.7 |
| GPT-5.1 | 47.3 | 73.3 | 72.1 | 73.5 |
| HumanLM-opinion (fine-tuned) | 30.1 | 38.5 | 61.6 | 46.9 |
Two patterns stand out. First, communication style is the weakest dimension for each of the general-purpose models shown: simulators simply do not talk like people. Second, a model fine-tuned to sound human did not win. HumanLM-opinion scored far below the general models here, which fits the authors' headline conclusion that higher general model capability does not necessarily produce more faithful simulation, and neither does a human-likeness label.
In what ways do simulated users behave differently?
Simulated users are nicer, wordier and more organised than real customers. The paper describes them as excessively cooperative and stylistically uniform, with little of the frustration or ambiguity real users bring. A few of the reported contrasts:
- Short replies: 29.0% of human turns were short, against 1.0% for GPT-4o as a simulator.
- Politeness: 49.0% of GPT-4o's turns were polite, against 15.3% for humans.
- Over-sharing: UserLM-8b packed 4.8 identifiers into each turn, against 2.6 for humans, meaning it volunteered booking numbers and details people usually give only when asked.
- Hedging: GPT-4o expressed uncertainty in 14.6% of turns, about twice the human rate of 7.3%.
- Changing tack: GPT-4o (19.1%) and CoSER (16.5%) pivoted strategy more than twice as often as humans (8.4%).
The authors call the result an "easy mode": cooperative users make the agent's job easier, so success rates measured with simulators run higher than the human baseline.
Can simulated users stand in for human feedback?
This is the finding researchers should notice most. When simulators rated the conversations, they were consistently more positive than humans across the quality dimensions. According to the paper, GPT-5.1 as a simulated user overestimated the assistant's human-likeness by 55% of the rating scale and its overall score by 18%. In other words, an AI asked to play a customer and then evaluate another AI gave that AI the benefit of the doubt.
For anyone using synthetic participants to evaluate a product, a prototype or an interview flow, that is the risk in plain terms: the synthetic panel tends to say things went better than real people would.
What this means for qualitative and UX researchers
The study is about customer-service agents, but its logic applies to most synthetic-participant work. Here is how we would read it for research practice:
- Use simulated users for rehearsal, not evidence. They are fast and repeatable for piloting a discussion guide or catching broken logic. They are poor stand-ins for how real people react, complain or rate an experience.
- Expect simulated data to be too clean. Real participants give short answers, withhold details until asked and get annoyed. A guide that works smoothly with synthetic users may still stall with people.
- Do not use synthetic ratings as satisfaction data. The evaluative gap was large and positive, so simulator-based scores will likely overstate quality.
- Validate against a small human sample. The paper's own method, running the same tasks with real people and comparing distributions, is a template any team can scale down.
If you want to see how a study plan holds up before fieldwork, a Synthetic Focus Group is a reasonable rehearsal step, and the findings above are a good reason to then run the real thing with an AI Interviewer that talks to actual participants.
What are the study's limitations?
The evidence comes from two customer-service domains, one fixed AI agent and mostly English-speaking participants recruited online, who were role-playing customers rather than facing their own problems. The authors also note known task and reward issues in the τ-bench version they used, although tasks were held constant across conditions. The gap may be larger or smaller in open-ended interviews, health or other cultures.
FAQ
What is the Sim2Real gap in user simulation?
It is the difference between how LLM-simulated users behave and evaluate and how real people do. Zhou et al. (2026) measure it on behaviour (style, information sharing, clarification, error reactions) and on evaluation (success calibration and post-task ratings).
Which LLM is the best user simulator?
In the paper's main table, DeepSeek-V3.1 had the highest User-Sim Index at 76.0, followed closely by Gemini 2.0 Flash and GPT-5.1. All of them remained well below the 92.7 agreement among human groups.
Do fine-tuned "human-like" models simulate users better?
Not in this study. The top User-Sim Index scores went to general-purpose models, while HumanLM-opinion, one of four specialised models tested, scored 46.9.
Can I use synthetic participants instead of real interviews?
For piloting and stress-testing, yes. For conclusions about real people's behaviour or satisfaction, this study suggests synthetic users are too cooperative and too positive to replace human participants.
Last updated: November 6, 2026
This article is an independent editorial summary of third-party research by the Qualitati Research Team. It is not affiliated with or endorsed by the paper's authors. Please consult the original paper for full methods and results.