Can LLMs Capture Expert Uncertainty? A 2026 Study
Qualitati Research Team · 2026-08-06 · 7 min read
Large language models can now pick out the values expressed in a long interview about as accurately as human experts can — but they rank those values differently, and they are uncertain about different things. A March 2026 study finds LLMs near the human ceiling on what they find and well below it on what matters most.
What did the study test?
The study asked whether LLMs can handle interpretive ambiguity the way expert qualitative analysts do. Kostina, Dikaiakos, Porcel and Stassopoulos (2026), in "Can LLMs Capture Expert Uncertainty? A Comparative Analysis of Value Alignment in Ethnographic Qualitative Research" (arXiv:2603.04897), took 12 open-ended, two-hour ethnographic interview sessions with local residents in China and asked both human experts and four LLMs to identify the top three human values each interview expressed, using the Schwartz Theory of Basic Values as the codebook.
Two design choices make this more useful than a typical accuracy benchmark. First, the task is genuinely ambiguous — there is no single right answer, which is true of most real interpretive coding. Second, the authors compared not just accuracy but uncertainty structure: where models hesitate versus where experts hesitate.
The four models tested
- DeepSeek-R1-Distill-Llama-8B
- Qwen3-30B
- Llama-3.3-70B
- Mistral-Small-3.2-24B
All are open-weight models, which matters for researchers working under data-governance constraints: this is evidence about models you can run without sending transcripts to a commercial API.
How close did the AI get to human experts?
Close on set overlap, not on ranking. According to Kostina et al. (2026), the best model — Qwen3 — scored within about 1.6 points of the expert baseline on F1 but roughly 15 points below it on rank-biased overlap, the metric that rewards getting the order of the top three values right.
| Metric (top-3 values) | Human experts | Qwen3 (best model) |
|---|
| F1@3 | 58.19 | 56.60 |
| Jaccard@3 | 44.54 | 43.96 |
| RBO@3 (rank overlap) | 51.97 | 37.09 |
Source: Kostina et al. (2026), arXiv:2603.04897. Higher is better; all figures are means.
Notice how low the human numbers are. Expert-to-expert agreement on this task reached a Krippendorff's α of 0.389 — a reminder that when two trained analysts read the same two-hour interview, they frequently disagree about which values dominate. Any claim that AI coding is "unreliable" needs that comparison in view, because the human ceiling here is not 90%; it is roughly 58 F1.
Why does ranking fail when detection succeeds?
Because identifying that a value is present is a recognition task, while ranking it first requires weighing an entire interview against itself. The models find broadly the right set of values — the sets overlap with expert sets almost as often as expert sets overlap with each other — but they order that set by different logic than a human analyst who has sat with the whole transcript.
This has a practical consequence. If your analysis output is a list of themes present in the data, LLM assistance is on reasonably solid ground. If your output is a claim about what mattered most to participants — the headline finding of most qualitative papers and most research readouts — the model's ordering is the weakest part of its output and the part a human should own.
What is the "Security" bias?
All four models assigned the Schwartz value Security substantially more often than the human experts did. The authors flag this as a possible model-induced value bias rather than a property of the interviews.
Systematic over-assignment of one code is a specific, checkable failure mode, and it is easy to miss because it does not look like an error at the level of any single transcript — each individual Security assignment is defensible. It only shows up when you compare the model's code distribution across the whole corpus against a human-coded subset. That comparison is worth building into any AI-assisted coding workflow.
Do ensembles help?
Yes, consistently. Combining the models' outputs with Majority Vote or Borda Count improved results by 8–10 points on F1 and RBO and 6–8 points on Jaccard, according to the paper — enough to close a meaningful part of the ranking gap.
The finding echoes what qualitative methodologists already do with human coders: independent passes reconciled by a documented procedure beat any single coder. The interesting version of this for practitioners is not four models voting, but one model and one human treated as two independent coders whose disagreements are surfaced rather than averaged away.
What this means for researchers
Treat LLM value or theme assignment as a candidate set to adjudicate, not a ranked finding to report. Three concrete moves follow from this study:
- Human-code a calibration subset. Without it you cannot detect a Security-style distribution bias, and you have no local estimate of the human ceiling to judge the AI against.
- Own the ranking. Let the model propose what is present; decide yourself what is central, and keep the reasoning in your audit trail.
- Use more than one pass. Multiple independent codings — different models, or model plus human — reconciled explicitly, outperform a single confident output.
This is the workflow ThemeLens is built around: the AI proposes codes and themes across the corpus and the researcher adjudicates and documents. The same division of labour applies upstream, where an AI Interviewer can run conversations at scale while interpretive authority over the findings stays with the researcher.
How much should you trust this study?
The corpus is small and culturally specific: 12 interviews with residents in China, coded against one value framework. The findings should not be read as a general accuracy rate for AI qualitative coding. What generalises better is the shape of the result — detection near the human ceiling, ranking well below it, systematic per-code bias, and reliable gains from ensembling. Those are patterns worth testing on your own data rather than numbers to cite as a benchmark.
FAQ
Can LLMs replace human coders in qualitative research?
Not on this evidence. They approach expert performance on identifying which codes are present but fall well short on ranking importance, and they carry systematic code-level biases that only a human-coded comparison will reveal.
Why is human agreement so low in this study?
The task — naming the top three human values in a two-hour open-ended interview — is inherently ambiguous. Experts reached a Krippendorff's α of 0.389, which the authors treat as a property of the task rather than a flaw in the annotation.
Which open-source model performed best?
Qwen3-30B came closest to expert-level agreement and aligned most closely with expert value distributions among the four models tested. The margins between models were modest, so cost, privacy, and stability are reasonable tiebreakers.
Does combining multiple models improve AI coding?
Yes. Majority Vote and Borda Count ensembling produced consistent gains — 8–10 points on F1 and RBO, 6–8 on Jaccard — making ensembling one of the cheapest available quality improvements.
Last updated: August 6, 2026. This is an independent editorial summary of third-party research; it is not affiliated with or endorsed by the study's authors. Primary source: Kostina, Dikaiakos, Porcel & Stassopoulos, "Can LLMs Capture Expert Uncertainty? A Comparative Analysis of Value Alignment in Ethnographic Qualitative Research," arXiv:2603.04897 (2026).