Can Synthetic Consumers Pretest Logos and Ads? A 2026 Study
Qualitati Research Team · 2026-10-14 · 7 min read
Short answer: Not reliably, according to a September 2026 study. When researchers at the University of Melbourne had GPT-based synthetic consumers rerun six classic visual marketing experiments, every setup identified the visual features correctly. None of them reproduced more than two of the six human effects, and one reversed a human finding. Use synthetic consumers to screen creative only after you calibrate them against human data, and never use them to judge how consumer segments will differ.
Synthetic consumers are LLM agents told to answer as a human respondent would. Vendors now sell them for pretesting logos, packaging and ads at a fraction of the cost of a human panel. The case for them comes mostly from text tasks such as opinion polls, economic games and willingness-to-pay questions. A new preprint by Yi-Lin Tsai and Yung-Hsiu (Arvin) Lai, "Seeing Is Not Perceiving: When Synthetic Consumers Can and Cannot Pretest Visual Marketing" (arXiv, 2026), tests whether that case holds once the stimulus is an image.
Key takeaways
- Seeing is not perceiving. All four model configurations passed every visual manipulation check, yet none reproduced more than two of six canonical human effects.
- One configuration got the answer backwards. GPT-4o-mini rated a product with an angular logo as more comfortable than the same product with a circular logo, the opposite of the human finding.
- Evidence in the prompt helps the average. Giving the model the original study's theory or statistics moved mean responses toward the human benchmark, much more for the newer model.
- Nothing fixed the spread. In all 12 combinations of configuration and intervention, synthetic responses were less varied than human ones. Every zero-shot SD ratio was below .50.
- The authors propose a gate: Calibrate, Intervene, Deploy. Use a human panel whenever a decision depends on how consumers differ.
What did the study test?
The study reran six well-known visual marketing experiments with synthetic respondents and compared the results with the published human findings. Tsai and Lai chose studies from the Journal of Consumer Research and the Journal of Marketing Research, each with publicly available stimuli and a stated psychological pathway. The original human samples ranged from 69 to 220 respondents. Each synthetic panel was matched to the original sample, which gave more than 800 synthetic respondents per configuration. The authors preregistered all six benchmark experiments.
The design crossed the two settings a marketer actually controls:
- Model generation: GPT-4o-mini (July 2024) vs. GPT-5.4-mini (March 2026).
- Input format: plain text vs. JSON.
Each synthetic respondent got its own API call and a demographic persona that matched the original sample. It then saw the image as a 512×512 PNG, rated it on the original dependent variable, explained its rating in text, and answered a manipulation check. The authors analysed the free-text rationales with Sentence-BERT and BERTopic. That let them separate seeing (getting the physical feature right) from perceiving (connecting it to the consumer meaning humans attach to it).
How accurate were synthetic consumers on visual tasks?
Accuracy was poor at zero-shot, meaning without any benchmark information in the prompt. According to Tsai and Lai (2026), no configuration reproduced more than two of the six effects, and most results were nonsignificant. The table summarises the plain-text runs. The authors report that JSON results were statistically indistinguishable at zero-shot.
| Benchmark effect (human finding) | GPT-4o-mini | GPT-5.4-mini |
|---|
| 1. More descriptive logos seem more authentic | Reproduced | Not reproduced |
| 2. Less saturated images fit temporally distant events | Not reproduced | Not reproduced |
| 3. Circular logos seem more comfortable | Reversed (p = .012) | Reproduced |
| 4. Asymmetric logos seem more exciting | Not reproduced | Not reproduced |
| 5. More saturated colour makes a product look bigger | Borderline (p = .052) | Reproduced |
| 6. Unstable logos raise utility of safety products | Not reproduced | Not reproduced |
The newer model did not do better across the board. It reproduced a different pair of effects, not more of them. The authors also address the memorisation objection. If the models were simply recalling famous papers, the oldest and most-cited effects should replicate best and none should reverse. The data showed the opposite.
Why do synthetic consumers miss what humans perceive?
The rationales point to two failure modes. In the first, the model reads the feature correctly but treats it as pure aesthetics. In Study 2, for example, saturation was judged as a visual property, with no link to time. In the second, the model forms a coherent association, but not the one people form. In Study 3, angular shapes were linked to "support" rather than to the softness people associate with round shapes. The authors frame this as a technical prior crowding out, or competing with, the psychological prior that carries the human effect.
This matters because a reversal is worse than a null result. A manager who trusted that GPT-4o-mini pretest would have chosen the wrong logo, and every manipulation check would have looked clean.
Can prompting fix synthetic consumer bias?
Partly. The authors tested two in-context interventions. Conceptual evidence placed the original paper's abstract in the system prompt. Empirical evidence added the original means, F-tests and p-values, which marks the upper limit of what steering could do. Both moved average responses toward the human effect, and the effect was far stronger for GPT-5.4-mini. Even so, adding empirical evidence brought every configuration closer to the human mean without the improvement reaching statistical significance within any single setup. Because the strongest prompt contains the answer, the authors describe the result as steerability, not spontaneous perception.
Do synthetic consumers capture variation between people?
No, and this is the most practical finding. Tsai and Lai measured spread with the SD ratio, the synthetic standard deviation divided by the human one, where 1.0 is a perfect match. All 12 configuration-by-intervention combinations fell below 1.0, and every zero-shot value was below .50. Under empirical evidence, GPT-5.4-mini improved slightly (plain text .335 → .400; JSON .354 → .493), while GPT-4o-mini got worse (plain text .430 → .283; JSON .498 → .326). Changing the temperature (0.5, 1.0, 1.5) did not fix the problem, and neither did varying persona demographics. Synthetic panels therefore understate how differently real consumers react. This is consistent with the variance shrinkage found in text-based digital twin research.
What this means for researchers
The authors turn their results into a three-step governance protocol:
- Calibrate. Before you trust a configuration in a stimulus domain, check that it reproduces a known human benchmark at the original sample size, matching both direction and significance.
- Intervene. If it fails, add conceptual or empirical evidence to the prompt and test again. If it still fails, go back to a human panel.
- Deploy. Screen new creative variants only with a configuration that passed, and redo calibration whenever the vendor changes the model or you move to a new stimulus type or judgment.
For qualitative and UX researchers, the rule extends beyond logos. A synthetic panel can help with early, average-effect screening, such as choosing which concepts go to real participants. It cannot tell you how segments differ or why a particular group reacts as it does. That evidence still has to come from real people, for example through an AI-moderated interview study with your target audience.
Keep the study's limits in view. It tested two low-cost models from one vendor, static images and numerical ratings. It benchmarked against published findings rather than ground truth. All six benchmarks predate the models' training cutoffs. The paper is a preprint and has not yet been peer reviewed.
FAQ
Can I use synthetic consumers to pretest ad creative?
Only with conditions. This study suggests calibrating the exact model and prompt format against a known human result in the same domain first. Use the output for average-effect screening, not for final decisions or segmentation.
Is a newer model more accurate for visual pretesting?
Not by default. GPT-5.4-mini reproduced a different two of the six effects than GPT-4o-mini rather than more of them. It did respond much better to evidence-based prompting.
Does JSON vs. plain-text prompting matter?
Format made no statistical difference at zero-shot. The authors still recommend treating the schema as a parameter to validate, because the older model performed worse when empirical evidence was tagged too finely.
Why do synthetic respondents give such uniform answers?
The paper finds the narrow spread in every configuration. Temperature, personas and prompts did not resolve it, which suggests it is a property of these agents rather than of any particular setup.
Last updated: October 14, 2026
This is an independent editorial summary of third-party research by the Qualitati Research Team. Qualitati is not affiliated with the authors. Please consult the original paper for full methods and results.