{"slug":"demographic-disclosure-llm-synthetic-respondents-2026","title":"Do Demographics Help LLM Synthetic Respondents? A 2026 Audit","description":"Do demographics help LLM synthetic respondents? A 2026 audit of 14 models found profiles barely improve answers and models follow the label, not the person.","keywords":"LLM synthetic respondents, demographic disclosure, synthetic survey panels, digital twins validity, data minimization synthetic surveys, silicon sampling, persona prompting","date":"2026-11-18","author":"Qualitati Research Team","category":"Methodology","readTime":"7 min read","content":"<p><strong>Adding a panelist's demographic record to an LLM prompt barely improves synthetic survey answers.</strong> A November 2026 audit of 14 language models found that once a model has seen eight earlier answers, age, education and ethnicity add little to individual predictions. They help group comparisons only when the record belongs to the right person, and even then the gain comes mostly from the label, not the person.</p>\n\n<p>The study is <a href=\"https://arxiv.org/abs/2611.20606\" target=\"_blank\" rel=\"noopener\">\"Necessary for What? Auditing Demographic Disclosure to LLM Synthetic Respondents\"</a> by Aspen Wang, Yen Yu Shan and Lun-Wei Ku of Academia Sinica, Taipei, posted to arXiv on 16 November 2026. It asks which uses of a synthetic survey actually need personal data.</p>\n\n<h2>What did the study test?</h2>\n\n<p>The authors tested whether a respondent's own demographic record makes an LLM a more faithful stand-in for that respondent. They used two U.S. surveys:</p>\n\n<ul>\n  <li><strong>Pew American Trends Panel, Wave 54</strong>: 1,000 respondents and 63 target questions on economic inequality, with a 15-field demographic record per person.</li>\n  <li><strong>ISSP 2023 National Identity and Citizenship</strong>: 1,513 U.S. respondents and 27 target questions on national pride, immigration, citizenship and institutional trust.</li>\n</ul>\n\n<p>For every respondent and target question, the model saw eight of that person's other answers, plus a demographic input that changed between conditions. The researchers ran 11 open-weight models from six families (7B to 72B parameters) on both surveys, and three hosted models (GPT-4.1, GPT-6.1 Terra and Claude Haiku 5.5) on the Pew panel only. Each model was compared with a statistical reference: a regularized logistic regression fit to human answers with the same information.</p>\n\n<h3>The conditions compared</h3>\n\n<ol>\n  <li><strong>History only</strong>: eight earlier answers, no profile.</li>\n  <li><strong>Own</strong>: the respondent's real record.</li>\n  <li><strong>Shuffled</strong>: another respondent's record, a control for whether the link to the right person matters.</li>\n  <li><strong>Substitutes</strong>: a census person record (ACS PUMS) or a synthetic persona (Nemotron-Personas-USA) instead of the real record.</li>\n  <li><strong>Minimized</strong>: the respondent's record cut to 10, 7 or 4 fields.</li>\n</ol>\n\n\n<h2>Do demographics improve individual synthetic answers?</h2>\n\n<p>Barely. According to Wang, Shan and Ku (2026), giving the model the respondent's own record lowered the median model's prediction error (Brier score) by 3% on the Pew panel and 1% on the ISSP. The more answers the model sees, the less demographics add.</p>\n\n<p>Population estimates moved even less. With answers alone, the models' population estimates were 22 to 51 percentage points (total variation distance) from the real sample, against about 2 points for the statistical reference. Adding the profile narrowed that distance by less than 1 point for the median model. For the open-weight models, the abstract reports that most of the population-level gain remained when the real record was replaced by census data or a synthetic persona, so it does not require the person's own data.</p>\n\n<h2>Where does the demographic record help?</h2>\n\n<p>Group comparisons are the one place the record pays off, and only with the right person's record. On the Pew panel, the median model reproduced 22.6% of the statistically supported human group differences with answers alone and 41.8% with the respondent's own profile. Contrast error fell for 7 of 11 open-weight models on Pew and 8 of 11 on the ISSP. The statistical reference still gained five to six times as much from the same profile.</p>\n\n<p>Using someone else's profile was worse than using none. The Shuffled condition raised contrast error for all 11 models on Pew and 9 of 11 on the ISSP, and cut the median share of reproduced differences from 22.6% to 11.6%.</p>\n\n<table>\n  <thead>\n    <tr><th>Condition (Pew panel, median open-weight model)</th><th>Human-supported group differences reproduced</th></tr>\n  </thead>\n  <tbody>\n    <tr><td>History only (no profile)</td><td>22.6%</td></tr>\n    <tr><td>Own record (15 fields)</td><td>41.8%</td></tr>\n    <tr><td>Own record cut to 10 fields</td><td>40.0%</td></tr>\n    <tr><td>Another respondent's record (Shuffled)</td><td>11.6%</td></tr>\n  </tbody>\n</table>\n<p><em>Source: Wang, Shan and Ku (2026), arXiv:2611.20606, Sections 4.2–4.3.</em></p>\n\n<h2>Are the models reading the person or the label?</h2>\n\n<p>Mostly the label. When the Shuffled predictions were grouped by the profile shown in the prompt rather than by respondents' real attributes, they reproduced 36% of human group differences on Pew and 25% on the ISSP, against 12% in both surveys when grouped by real attributes.</p>\n\n<p>The paper's household income example shows the effect. In the human data, respondents earning $100,000 or more were 45 points more likely than those earning $30,000–50,000 to hold stocks, bonds or mutual funds outside retirement accounts. In the median model, respondents <em>shown</em> those two income profiles differed by 51 points. Grouped by their <em>actual</em> incomes, the same predictions differed by only 2 points.</p>\n\n<p>That matters for validation: the usual check that a synthetic panel reproduces known group differences cannot tell a model of the person from a stereotype of the label.</p>\n\n<h2>Which group differences do synthetic respondents get wrong?</h2>\n\n<p>The models make groups look more alike than they are, especially on attitudes. With the own profile, the median model reproduced 76% of 474 differences in circumstances (such as savings and debts) but only 22% of 807 differences in attitudes (such as views on government responsibility and the causes of inequality). The statistical reference reproduced 92% and 77%.</p>\n\n<p>Errors were sharpest for religious affiliation and ethnic origin, both special categories of personal data under GDPR. A lower average error can also hide reversed conclusions. On the Pew panel, Llama 3.1 8B cut its contrast error by 1.9 points with the profile while reversing 104 more group differences.</p>\n\n<h2>Does disclosing less data cost accuracy?</h2>\n\n<p>Only a little, and the cost again falls on group comparisons. Cutting the record from 15 to 10 fields dropped the median model's reproduced differences from 41.8% to 40.0%, while aggregate measures showed no consistent trend. Fewer fields is not anonymity, however. On the Pew sample, four demographic fields alone identified 0.1% of respondents uniquely and eight prior answers identified 18%, but together they identified 90%.</p>\n\n<h2>What this means for researchers using synthetic respondents</h2>\n\n<p>The practical message is to justify each field by the estimate it serves, not by a general belief that richer profiles mean better twins. The authors outline an audit that a survey organization can run on questions it has already fielded:</p>\n\n<ol>\n  <li>List the estimates the synthetic survey will report (individual, population, group comparison) and how accurate each must be.</li>\n  <li>Compare answers alone with own and reassigned profiles to see what disclosure, and correct linkage, actually add.</li>\n  <li>Regroup reassigned predictions to see whether the model follows the person or the label.</li>\n  <li>Count group differences reproduced, missed and reversed for each attribute and question type, because average error alone hides these.</li>\n  <li>Among conditions that meet the study's requirements, prefer the one that discloses least.</li>\n</ol>\n\n<p>For qualitative and mixed-methods teams, the attitudes gap is the main warning. Synthetic panels are most trustworthy for factual circumstances and least trustworthy for how groups differ in opinion, which is usually what an interview or open-ended survey study sets out to learn. If you prototype with a <a href=\"https://qualitati.com/digital-twins\">digital twin panel</a>, use it to pilot questions and spot gaps, then confirm group-level attitude findings with real participants, for example through <a href=\"https://qualitati.com/surveys\">conversational AI surveys</a>.</p>\n\n<h3>Limitations to keep in mind</h3>\n\n<p>The authors note that the results cover two U.S. surveys, on economic inequality and national identity, with target questions fielded alongside the answers the models received. Small groups, intersections and alternative prompt formats were not tested, and hosted models ran on Pew only. Passing the audit on already-fielded questions does not show that a model will hold up on new ones. The paper is a preprint and has not yet been peer reviewed.</p>\n\n<h2>FAQ</h2>\n\n<h3>Should I put demographics in prompts for LLM synthetic respondents?</h3>\n<p>Only for the estimates that need them. In this 2026 audit, demographics added little to individual answers or population estimates. They helped group comparisons only when they were the respondent's own, and the models still leaned on the label rather than the person.</p>\n\n<h3>Can census data or synthetic personas replace real respondent records?</h3>\n<p>For population estimates from open-weight models, largely yes: most of the gain remained with census or synthetic personas. For group comparisons, no. Profiles that did not describe the actual respondent did not improve them.</p>\n\n<h3>Is removing demographic fields enough to protect panelists?</h3>\n<p>No. Four fields plus eight prior answers uniquely identified 90% of the Pew sample, so a minimized prompt can still single out most people.</p>\n\n<p><em>Last updated: 18 November 2026. This is an independent editorial summary of third-party research by Wang, Shan and Ku (2026); it is not affiliated with or endorsed by the authors. See the <a href=\"https://arxiv.org/abs/2611.20606\" target=\"_blank\" rel=\"noopener\">original paper</a> for full methods and results.</em></p>\n","related":[{"slug":"llm-generated-codebook-audit-eight-models-2026","title":"Do LLMs Write the Same Codebook? An 8-Model Audit (2026)","description":"Do LLMs write the same qualitative codebook? A 2026 audit of eight models across 668 interviews found different codes and only .69 cross-model agreement.","category":"Methodology","date":"2026-11-17","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-simulated-users-stance-stability-audit-2026","title":"Do LLM Simulated Users Hold Stable Opinions? A 2026 Stance Audit","description":"Do LLM simulated users hold stable opinions? A 2026 audit of 1,821 Reddit threads found one edited message shifted GPT-5.2's simulated stance by up to 49%.","category":"Methodology","date":"2026-11-13","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"prediction-powered-inference-ai-augment-surveys-2026","title":"Can AI Augment Surveys? Prediction-Powered Inference (2026)","description":"Can AI augment surveys without replacing respondents? A 2026 study of 83 GSS and NHIS outcomes found prediction-powered inference adds 7–12% effective sample size.","category":"Methodology","date":"2026-11-12","readTime":"6 min read","author":"Qualitati Research Team"}]}