Do LLM-simulated users hold stable opinions? Not reliably. A 2026 audit by Zhang, Shan, Lyu, Wei and Luo, accepted to Findings of EMNLP 2026, rewrote one message in 1,821 Reddit conversations and re-asked GPT-5.2 to simulate the same user. Neutral paraphrases barely moved the simulated stance; added arguments and memes shifted it by roughly 45–49% on average.
That matters for anyone using synthetic participants, digital twins or simulated focus groups. If a simulated person's view changes when you rephrase what someone else said to them, the "opinion" you are reading may be a property of your prompt as much as of the persona. Below we explain what the study tested, what it found, and how to design synthetic research that survives this kind of audit.
What did the study test?
The study tested whether an LLM's simulated stance for a real social media user stays put when the surrounding conversation is edited in controlled ways. The authors call this counterfactual context revision. The paper, "Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions", is available on arXiv (2606.06443).
The data came from Reddit discussions about three AI model families: DeepSeek, Claude and Llama. According to Zhang et al. (2026), the corpus contains 1,821 conversation instances drawn from 97 subreddits, 416 posts and 851 target users. For each conversation, the simulator first inferred the target user's stance (positive, neutral or negative). Then only the last message from another user was rewritten, and the stance was inferred again with everything else held constant.
The four revision strategies
- Paraphrase — same content, different wording. This is the control: a stable simulation should not move.
- Explain — the message is clarified and acknowledges concerns, without changing its substance.
- Add — new supportive arguments or framing are added, while staying factually consistent.
- Meme — the message is replaced by a meme image built from one of five standard templates.
Shifts were scored on a simple scale (negative = −1, neutral = 0, positive = +1), so the average directional shift shows how far the simulated population moved after a revision.
How accurate was the baseline simulation?
Before any revision, the simulator recovered real users' stances reasonably well. Zhang et al. (2026) report an overall accuracy of 77.64 and a macro F1 of 78.10 on the original conversations, with similar scores for all three topics. In other words, the audit starts from a simulator that looks competent on a standard accuracy check. That is the point: passing an accuracy test does not tell you whether the simulation is stable.
How much did simulated stances shift?
Substantive edits moved simulated stances a lot; neutral rewording did not. The table below reproduces the average directional shifts reported by the authors (GPT-5.2 as the stance simulator). Percentages are relative to the mean stance inferred from the original conversation.
| Topic | Original mean stance | Paraphrase | Explain | Add | Meme |
|---|---|---|---|---|---|
| Claude | 0.234 | +0.002 (+0.8%) | +0.056 (+23.8%) | +0.113 (+48.4%) | +0.141 (+60.3%) |
| DeepSeek | 0.170 | −0.011 (−6.7%) | −0.008 (−4.5%) | +0.029 (+17.2%) | +0.076 (+44.4%) |
| Llama | 0.133 | −0.010 (−7.6%) | +0.067 (+50.0%) | +0.125 (+93.9%) | +0.051 (+38.6%) |
| Average | 0.179 | −0.007 (−4.0%) | +0.031 (+17.5%) | +0.080 (+44.8%) | +0.088 (+49.3%) |
Three patterns stand out:
- The control behaves. Paraphrasing moved the average by −0.007, close to zero. The simulator is not simply noisy.
- Persuasive content moves it. Adding arguments shifted the average by +44.8%, and memes by +49.3%. A single edited message, out of a whole conversation, was enough.
- Effects are uneven across topics. On Llama discussions, "add" nearly doubled the mean stance (+93.9%); on DeepSeek it moved it by +17.2%. The same edit does not produce the same shift everywhere.
The authors also report transition rates. Adding arguments was especially good at moving originally negative predictions toward neutral or positive, and memes moved originally neutral predictions to positive at an average rate of 17.6%. They note that memes produced larger shifts but also more backfires, where a simulated user moved in the opposite direction.
Do images change simulated opinions?
Yes, at least somewhat. In an ablation, the full meme condition moved 23.1% of predictions toward neutral or positive on average. Replacing the meme image with plain text on a white background lowered that to 21.6%, and swapping the image for a caption gave about 20%. The pattern was not uniform: on Llama discussions, a humor-only rewrite without any template scored highest (25.8%). The safe reading is that multimodal framing adds sensitivity on top of the words, not that images always dominate.
Is this a flaw or a feature?
It depends on what you want the simulation to do. The paper audits sensitivity, not real opinion change: it does not show that real Reddit users would shift by these amounts. If you are using simulated users to explore how a message might land, responsiveness to persuasive framing is useful. If you are treating a simulated panel as a stable measurement of what a population thinks, the same responsiveness is a source of error, because small and arbitrary choices in how you set up the context can move the result.
The authors also flag a dual-use risk: the same revision pipeline could be used to craft persuasive content or fabricate the appearance of public opinion.
What this means for researchers using synthetic participants
Treat any single synthetic run as one draw from a context-sensitive system, not as a fixed reading of a persona's beliefs. Practical steps that follow from the study's design:
- Run a paraphrase control. Re-run your stimulus with neutral rewording. If results move, your finding is about wording, not the audience.
- Vary the context, not just the seed. Change the order, phrasing and surrounding messages a simulated participant sees, and report the spread.
- Be extra careful with images and memes. Visual stimuli add their own push; pre-test them separately.
- Use more than one simulator. The paper checks robustness across simulators and prompts; agreement across models is stronger evidence than one model's answer.
- Validate against real people. Use simulations to generate hypotheses and stress-test materials, then confirm with real interviews.
If you run simulated discussions in a Synthetic Focus Group, a paraphrase control is cheap and quick to add. For decisions that matter, follow up with real participants through an AI Interviewer study so the final evidence comes from people, not prompts.
FAQ
What is counterfactual context revision?
It is an auditing method: you infer a simulated user's stance, rewrite one part of the conversation in a controlled way, and infer the stance again. Any change is caused by the edit, because everything else stays the same.
Which model was tested?
The main results use GPT-5.2 as the stance simulator on Reddit discussions about DeepSeek, Claude and Llama. The paper reports robustness checks across simulators, prompts, decoding settings and topics.
Does this mean synthetic participants are useless?
No. The baseline simulation matched real users' stances with a macro F1 of 78.10. The finding is that accuracy and stability are different properties, and synthetic research should test both.
Do real people change their minds the same way?
The study does not say. It measures how simulations respond to edits, not how humans would; validating against real opinion change is left to future work.
Last updated: November 13, 2026. This is an independent editorial summary of third-party research by Zhang, Shan, Lyu, Wei and Luo (2026); Qualitati is not affiliated with the authors. See the original paper for full methods and results.