Can LLMs Annotate Ethnographic Texts? A 2026 Benchmark
Qualitati Research Team · 2026-09-15 · 7 min read
Short answer
Can LLMs annotate ethnographic texts reliably? Not yet. A January 2026 Oxford-led study tested 7 large language models on 567 ethnographic excerpts. The best setup reached a mean F1 of only 0.41 across 115 ritual features. Models did well on concrete, explicitly described features, but struggled with long texts, graded judgments and ambiguous concepts.
Many researchers now hope LLMs can turn "thick" qualitative description into structured variables at scale. This study is one of the most careful tests of that hope on hard, long-form material. Below we summarize what it tested, what it found, and what it means if you use AI to code interviews, fieldnotes or other long qualitative texts.
Key takeaways
- Across all models and prompting conditions, mean F1 ranged from 0.12 to 0.41 on the main dataset (Goodall et al., 2026).
- Open-weight DeepSeek V3.1 and GPT-OSS 120B tied for best (F1 = 0.41). Claude Sonnet 4.5 (0.39) and GPT-5 Nano (0.37) came close.
- Features with clear textual markers (funerals, dancing, singing) scored F1 > 0.60 for the best models. Interpretive features such as psychological discomfort scored F1 < 0.30.
- Features that humans disagreed on were also hard for LLMs. Even where humans agreed, models fell short.
- Different LLMs agreed with each other far less than two human coders did (κ = 0.23 vs. κ = 0.57).
What did the study test?
The study measured whether off-the-shelf LLMs can extract structured ritual features from real ethnographic prose. The paper is "Large language models struggle with ethnographic text annotation" by Leonardo S. Goodall, Dor Shilton, Daniel A. Mullins and Harvey Whitehouse. It was posted to arXiv on January 17, 2026, with authors from the University of Oxford, Tel Aviv University and Birkbeck, University of London.
The corpus came from the electronic Human Relations Area Files (eHRAF): 567 texts describing rituals in 73 cultures. The median text was 653 words, and texts ranged from 43 to 8,924 words. That is far longer than the tweets and short survey answers used in most earlier "LLMs match human coders" studies.
The authors used two datasets built from this corpus:
- Morphospace dataset: 115 ritual features (function, duration, actions, emotional states), mostly coded by one expert.
- Synchrony dataset: 6 features describing synchronous behavior (singing, chanting, praying, marching, dancing, generic movement), each coded independently by two humans. This allowed a direct comparison with human inter-coder reliability.
They compared three prompting strategies: one feature at a time (zero-shot), all features in a category at once (multi-task prompting), and multi-task prompting repeated 10 times with the most common answer kept (ensemble).
How accurate were the LLMs?
Accuracy was well below what unsupervised annotation needs. The table below shows the reported mean F1 scores on the 115-feature dataset.
| Model | Type | Mean F1 (best reported condition) |
|---|
| DeepSeek V3.1 (671B MoE) | Open-source | 0.41 |
| GPT-OSS 120B | Open-source | 0.41 |
| Claude Sonnet 4.5 | Proprietary | 0.39 |
| GPT-5 Nano | Proprietary | 0.37 |
| Perplexity Sonar (web-enabled) | Proprietary | 0.23 |
| Llama 3.2 Instruct (3B), Qwen 3 Instruct (4B) | Open-source | below 0.25 |
A majority-class baseline averaged F1 = 0.02, so the models picked up real signal. Still, according to Goodall et al. (2026), absolute performance stayed "far below" reliable levels. Web access did not help: Perplexity Sonar showed no advantage over offline models.
Prompting helped only a little. Multi-task prompting added 0.02 to 0.07 F1 points for most models. Ensembling added another 0.01 to 0.02. On the simpler synchrony dataset, the best result was F1 = 0.54 (GPT-5 Nano with multi-task prompting).
Why do LLMs struggle with ethnographic texts?
The authors point to three causes: text length, interpretive features and genuine ambiguity.
Longer texts produce more false positives
Longer excerpts predicted lower F1 scores and more false positives. The authors read this as spurious pattern matching: in long prose, models find surface cues that do not mean the feature is present. Small models were extreme. Llama 3.2 Instruct predicted "present" in nearly every case (false positive rate > 98%).
Graded and interpretive features are hardest
Multiclass features, which require choosing among ordinal or categorical options, had 90% lower odds of correct detection than binary features (OR = 0.10). Features such as "level of arousal" or "ritual form" cannot be read off explicit words. They need context and cultural knowledge.
Human disagreement sets a ceiling
Two human coders reached a mean Cohen's kappa of 0.57 on the synchrony features, with 89% raw agreement. Agreement ranged from κ = 0.92 for synchronous singing to κ = 0.25 for generic movement. LLM accuracy followed the same ranking. Where human agreement was poor (κ < 0.40), LLMs performed close to chance.
Do LLMs make the same mistakes as humans?
No, their errors run in opposite directions. Human coders were conservative: a mean false positive rate of 4% but a false negative rate of 51%. Most LLMs over-reported features instead. And models disagreed with each other much more than humans did, which limits how much simply pooling several models can fix.
One practical signal did emerge. Across 214,529 predictions, answers the ensemble repeated consistently were more often correct: 81% accuracy at 91–100% certainty versus 68% at 10–50%. The effect was strongest for GPT-OSS 120B (r = 0.36) and absent for Llama 3.2 (r = -0.01).
What this means for qualitative researchers
The lesson reaches beyond anthropology. Any long, interpretive qualitative data, such as in-depth interviews, fieldnotes or diaries, shares the features that tripped these models up.
- Check human agreement first. If two trained coders cannot agree on a code, do not expect an LLM to apply it reliably.
- Prefer concrete, binary codes for automation. Leave graded and latent judgments to humans, or treat AI output as a first draft.
- Chunk long documents. Text length drove false positives, so code shorter segments and keep the quote each code rests on.
- Use run-to-run consistency as a flag. Send inconsistent annotations to a human reviewer.
- Invest in the codebook. The authors did not test detailed codebooks with examples or fine-tuning, and name both as open questions.
This is the design logic behind human-in-the-loop tools such as ThemeLens, where AI proposes codes tied to source quotes and researchers review and revise them rather than accepting a single automated pass.
FAQ
Can ChatGPT or Claude code ethnographic fieldnotes?
They can help, but not unsupervised. In this 2026 study, Claude Sonnet 4.5 reached a mean F1 of 0.39 and no model exceeded 0.41 on 115 ritual features. Treat output as a draft for expert review.
Are open-source LLMs worse for qualitative annotation?
Not necessarily. Large open-weight models (DeepSeek V3.1, GPT-OSS 120B) matched or beat the proprietary models tested. Very small models (3B–4B parameters) performed worst.
Does better prompting fix the problem?
Only partly. Multi-task prompting and ensembling together added under 0.1 F1 points. The authors suggest codebooks with worked examples and fine-tuning as untested next steps.
Why did earlier studies find LLMs matched human coders?
According to the authors, those studies mostly used short texts, narrow coding schemes and less ambiguous categories. Long, culturally dense ethnographic prose is a harder test.
Last updated: September 15, 2026
This article is an independent editorial summary of third-party research by Goodall, Shilton, Mullins and Whitehouse (2026). Qualitati is not affiliated with the authors. See the original paper for full methods and results.