{"slug":"synthetic-stakeholders-interview-order-design-study-2026","title":"Should Synthetic Stakeholders Come Before Real Interviews? 2026","description":"Should you interview an AI-simulated stakeholder before or after a real one? A 2026 field study found human concerns shaped 90.7% of designs vs 68.8% for AI.","keywords":"synthetic stakeholders, AI-simulated stakeholders, synthetic participants, LLM personas user research, stakeholder interviews, AI in design research","date":"2026-11-19","author":"Qualitati Research Team","category":"AI User Research","readTime":"6 min read","content":"<p>Should you interview an AI-simulated stakeholder before or after you talk to a real one? A November 2026 field study from the University of Haifa and Wellesley College suggests that the order shapes how much weight the synthetic voice gets. In both orders, though, the concerns raised by real people were more likely than the AI's to end up in the final design.</p>\n\n<h2>What did the study test?</h2>\n<p>The study compared two interview sequences in a real design course. According to <a href=\"https://arxiv.org/abs/2611.22740\" target=\"_blank\" rel=\"noopener\">Mokryn, Ayalon, Dines and Shaer (2026)</a>, 21 students (19 undergraduate, 2 graduate) in a 12-week course on the ethical design of sociotechnical systems with generative AI worked in 8 teams of two or three. Each team built an <strong>AI-simulated stakeholder (AIS)</strong>, an LLM chatbot prompted to play a role such as a crisis-hotline volunteer, a special-education teacher or an elderly bus user. Each team also interviewed one real person in that role.</p>\n<p>Teams were randomly assigned to one of two orders:</p>\n<ul>\n<li><strong>AIS-first (4 teams):</strong> interview the synthetic stakeholder, then the human.</li>\n<li><strong>Human-first (4 teams):</strong> interview the human, then the synthetic stakeholder.</li>\n</ul>\n<p>Most students built their AIS on the free web version of ChatGPT (GPT-4o). The authors analysed interview transcripts, design artifacts, 13 written reflections, a class discussion and post-course interviews. They traced each pain point the students identified back to its source (AIS, human or both) and checked whether it showed up in the final design.</p>\n\n<h2>Did synthetic or human interviews shape the final designs more?</h2>\n<p>Concerns raised by human stakeholders carried through to the design more often. The paper counts pain points flagged as \"prominent\" (repeated or emphasised in an interview):</p>\n<table>\n<thead>\n<tr><th>Source of prominent pain point</th><th>Pain points</th><th>With a matching design element</th><th>Share</th></tr>\n</thead>\n<tbody>\n<tr><td>AI-simulated stakeholder</td><td>32</td><td>22</td><td>68.8%</td></tr>\n<tr><td>Human stakeholder</td><td>43</td><td>39</td><td>90.7%</td></tr>\n</tbody>\n</table>\n<p>The two groups overlap: 12 pain points came up in both kinds of interview. Team by team, the human-derived rate was higher in four teams, equal in three and lower in one. With 8 teams, the authors present these as qualitative patterns, not statistical effects.</p>\n<p>The coding held up reasonably well. A second trained coder recoded four of the eight teams, with Krippendorff's alpha of 0.746 for prominence, 0.850 for design-correspondence category and 0.849 for intervention level.</p>\n\n<h2>How did interview order change the way students used the AI?</h2>\n<p>Order changed the role the AI played. The students in both conditions treated the AIS as a reference point, the \"anchor\" in the paper's title, but they anchored on it in different ways:</p>\n<ol>\n<li><strong>AIS-first teams</strong> used the synthetic interview as a provisional frame and a rehearsal. All four described it as a useful preliminary check on their questions. The human interview that followed often revised that frame, steering them toward concerns they had not prioritised. One team said following the AI's suggested focus had delayed them.</li>\n<li><strong>Human-first teams</strong> mostly used the AIS afterwards to validate or extend what they had heard. It rarely became the main source of discovery. In one team the AIS surfaced a pain point the human had not raised.</li>\n</ol>\n<p>In both orders, students found that real interviewees gave narrower but more grounded and emotionally textured accounts. Half of the teams described the AIS responses as general and sometimes stereotypical. Students also said the AI tended to recast complaints as problem-and-solution pairs, and that its inconsistencies were harder to detect than a human's. Several teams did not trust what the AIS told them until a real interview corroborated it. One student summed up the AIS answers as ones that would \"tell us what we wanted to hear.\"</p>\n\n<h2>What this means for researchers using synthetic participants</h2>\n<p>The practical lesson is to treat synthetic interviews as preparation, not evidence. That echoes larger benchmark studies of <a href=\"https://qualitati.com/blog/synthetic-users-vs-real-participants-2026\">synthetic users versus real participants</a>, and this study adds an observation about process: whatever you hear first becomes the frame you test later. For UX, market and academic researchers, that suggests a few habits:</p>\n<ul>\n<li><strong>Use a synthetic stakeholder to rehearse the guide, not to answer the research question.</strong> The AIS-first teams valued the dry run, but the human interview did the real discovery work.</li>\n<li><strong>Write down what the AI told you before the human interview.</strong> Then compare explicitly. Otherwise the synthetic frame can quietly become the default you only adjust.</li>\n<li><strong>Watch for the problem-to-solution reflex.</strong> LLM personas tend to answer as if they were consultants. Real participants hesitate, contradict themselves and bring up issues no one asked about, and that friction is the data.</li>\n<li><strong>Keep humans as the source of record for design decisions.</strong> In this study, human concerns reached the final design at a higher rate.</li>\n</ul>\n<p>The authors' design suggestions for AIS tools point the same way: show uncertainty and scope limits, flag stereotypical patterns, and build in side-by-side comparison with stakeholder data. On QualiTaTi, the <a href=\"https://qualitati.com/synthetic-focus-group\">Synthetic Focus Group</a> is meant for exactly this rehearsal step, and the <a href=\"https://qualitati.com/ai-interviewer\">AI Interviewer</a> lets you run the real-participant interviews that should anchor the decisions.</p>\n\n<h2>What are the study's limitations?</h2>\n<p>The authors are explicit that this is a small, qualitative study. It covers one course and 21 novice designers, and there was no measure of prior experience. Interview order is confounded with project domain, with stakeholder type and with how teams refined their AIS. LLM choice and prompts varied and were not fully logged. Only 3 of the 21 students took part in the follow-up about four months later, all from AIS-first teams. The study did not assess design quality or stakeholder satisfaction. The paper is an arXiv preprint (cs.HC), and the authors note it was accepted to the <em>International Journal of Human-Computer Interaction</em>.</p>\n\n<h2>Frequently asked questions</h2>\n<h3>Can AI-simulated stakeholders replace real stakeholder interviews?</h3>\n<p>Not according to this study. Human-derived prominent concerns reached the final design 90.7% of the time, compared with 68.8% for AI-derived ones, and students described the AI's answers as more generic.</p>\n<h3>Is it better to interview the AI persona first or the real person first?</h3>\n<p>Each order produced a different pattern. Going AI-first gave teams a useful rehearsal, but the AI framing then had to be corrected. Going human-first kept the AI in a supporting, validating role. With only 4 teams per condition, the study does not establish that one order is better.</p>\n<h3>Which AI model did the students use?</h3>\n<p>Most used the free web version of ChatGPT (GPT-4o), though some tried other LLMs. The authors did not log the final model for each team.</p>\n\n<p><em>Source: Mokryn, O., Ayalon, O., Dines, N., &amp; Shaer, O. (2026). AI as Anchor: How Interview Sequencing Relates to the Appropriation of Synthetic Stakeholder Voices in Design Education. arXiv:2611.22740. <a href=\"https://arxiv.org/abs/2611.22740\" target=\"_blank\" rel=\"noopener\">https://arxiv.org/abs/2611.22740</a></em></p>\n<p><em>Last updated: November 19, 2026. This is an independent editorial summary of third-party research. It is not affiliated with or endorsed by the authors.</em></p>\n","related":[{"slug":"structuring-survey-histories-llm-digital-twins-2026","title":"How to Structure Survey Data for LLM Digital Twins (2026)","description":"How should you structure survey data for LLM digital twins? A 2026 Chicago Booth study found preference profiles beat raw transcripts, mostly on pricing.","category":"AI User Research","date":"2026-11-10","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"interviewplayground-simulated-participants-ai-interviewers-2026","title":"Can Simulated Participants Evaluate AI Interviewers? 2026 Study","description":"Can simulated participants evaluate AI interviewers? InterviewPlayground predicted real-study scores (r = 0.86, 450 people) but missed comfort. Key findings.","category":"AI User Research","date":"2026-11-09","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-user-simulators-sim2real-gap-2026","title":"Can LLMs Simulate Real Users? The Sim2Real Gap (2026 Study)","description":"Can LLMs simulate real users? A 2026 study compared 451 people with 31 LLM user simulators and found synthetic users too cooperative and too positive.","category":"AI User Research","date":"2026-11-06","readTime":"7 min read","author":"Qualitati Research Team"}]}