Can LLMs Code Open-Ended Survey Responses? A 2025 Study
Qualitati Research Team · 2026-07-16 · 8 min read
Can large language models reliably code open-ended survey responses? Not out of the box. A 2025 study in Survey Research Methods tested GPT, Llama, and Mistral on 5,072 German survey answers and found that only a fine-tuned model reached professional coding quality (macro F1 of 0.87). Zero- and few-shot prompting fell well short of trained human coders and even a classic machine-learning baseline.
What did the study test?
Researchers Leah von der Heyde, Anna-Carolina Haensch, Bernd Weiß, and Jessica Daikeler asked a focused question: can off-the-shelf LLMs classify open-ended survey responses into a predefined coding scheme as well as human experts? They used real data from the GESIS Panel — a random 20% sample of 5,072 responses to a question about why people participate in surveys. Two survey researchers had already coded this data independently with very high agreement (Cohen's kappa = 0.91), giving a trustworthy human benchmark across 22 categories.
The team compared three multilingual models — GPT-4o (proprietary), Llama 3.2, and Mistral NeMo (both open-source) — across several prompting strategies: zero-shot, few-shot, category descriptions, and fine-tuning. Because the categories were unevenly distributed, they used macro F1 (which weights rare categories equally) as the primary metric rather than raw accuracy.
How accurate was AI coding of survey responses?
Accuracy depended heavily on both the model and the prompting approach. According to von der Heyde et al. (2025):
- Fine-tuned GPT reached a macro F1 of 0.87 — the only setup the authors called "satisfactory," and a 16-point jump over few-shot prompting.
- Few-shot GPT was the best non-fine-tuned result at macro F1 0.71, with GPT hovering around 0.7 across prompting approaches.
- Open-source models (Llama 3.2, Mistral NeMo) trailed badly; even their best prompting setups did not come near GPT. For some categories, Llama produced no correct classifications at all.
- A classic support vector machine trained on the same data (Haensch et al., 2022) reached a median F1 of 0.83, beating every non-fine-tuned LLM setup (best 0.72).
The uneven-category problem
The models were not uniformly good or bad — they were lopsided. GPT coded common, concrete categories very well (macro F1 around 0.9 for incentive, interest, and fun) but collapsed on ambiguous or rare ones (macro F1 between 0.02 and 0.3 for no reason, non-identifiable/other, and other survey characteristics). Because performance varied so much by category, the AI-generated distribution of reasons diverged from the human-coded distribution — meaning that reported frequencies, not just individual labels, can be distorted when fine-tuning is skipped.
Prompting approach vs. model: which matters more?
| Setup | Macro F1 (GPT) | Verdict |
|---|
| Zero-shot prompting | ~0.70 (worst of the three) | Below supervised baseline |
| Few-shot prompting | 0.71 | Best without fine-tuning, still short of SVM |
| Fine-tuning | 0.87 | Only "satisfactory" setup |
| SVM baseline (2022) | 0.83 (median F1) | Beat all non-fine-tuned LLMs |
The headline pattern: model choice mattered more than prompt engineering. Few-shot beat zero-shot across the board, but the size of that gain depended on the model, and no amount of prompting closed the gap between open-source models and GPT. The single biggest lever was fine-tuning — retraining GPT on labeled examples — which is also the most resource-intensive to do.
What this means for researchers
The study is a useful corrective to the assumption that a capable general model plus a good prompt equals reliable coding. Three practical takeaways:
- Validate before you trust category frequencies. Uneven per-category performance can quietly skew your distributions, not just individual labels.
- Non-English, domain-specific coding is harder than English sentiment demos. Earlier optimistic results came from simpler English tasks; this German, 22-category scheme exposed the limits.
- Budget for fine-tuning or human-in-the-loop review if you need publication-grade reliability. Few-shot prompting is a fast first pass, not a final answer.
If you code open-ended survey data or interview transcripts, treating the model as an assistant that proposes codes for human confirmation — rather than an autonomous coder — matches the evidence. Tools like QualiTaTi's ThemeLens are built around that human-in-the-loop pattern for AI thematic analysis, and its AI Surveys feature keeps researchers in control of how open-ended answers are interpreted.
Frequently asked questions
Can ChatGPT code open-ended survey responses accurately?
Only partially. In this 2025 study, GPT-4o with few-shot prompting reached macro F1 of 0.71 — decent but below a trained SVM and short of expert-level reliability. Fine-tuning was needed to reach 0.87.
Are open-source LLMs good enough for survey coding?
Not in this test. Llama 3.2 and Mistral NeMo trailed GPT substantially and failed entirely on some categories, even with their best prompting setup.
Does prompt engineering fix weak AI coding?
It helps but has limits. Few-shot beat zero-shot, yet prompting alone never closed the gap to a supervised baseline. Fine-tuning drove the largest gains.
Should I still use AI for qualitative coding?
Yes, with guardrails: use it for a first-pass draft, validate per-category performance, quote verbatims, and keep a human coder in the loop for final decisions.
Last updated: July 16, 2026.
This is an independent editorial summary of third-party research. QualiTaTi is not affiliated with the study's authors. Primary source: von der Heyde, L., Haensch, A.-C., Weiß, B., & Daikeler, J. (2025). "AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation." Survey Research Methods (arXiv:2506.14634).