Can LLMs Code Open-Ended Survey Answers Better Than Humans?

Open-ended questions are cheap to add to a survey and expensive to analyse, so many datasets carry text columns nobody ever codes. A new ECONtribute discussion paper by sociologists Ansgar Hudde and Shannon Taflinger tests whether LLMs can close that gap. They also check whether LLM coding would lead a researcher to different substantive conclusions than human coding would.

What did the study test?

The study compared LLMs with trained student coders on one realistic, moderately complex coding task. The data came from a quota-representative survey of 1,419 US Americans aged 20–33, who were asked for their "reasons, thoughts, feelings" and other reflections about dating someone from the other political party.

According to Hudde and Taflinger (2026), the workflow had six steps:

1. Build the codebook by hand. The researchers ran a thematic qualitative text analysis and produced 15 codes in four categories: unwilling, situationally dependent, willing, and other.
2. Create a benchmark. The two researchers hand-coded every response, and that coding became the reference standard.
3. Code with humans and machines. Two Master's-level student assistants and eight LLMs received essentially the same written instructions.
4. Add training data. Each model was run again with 200 randomly selected hand-coded examples.
5. Split by ambiguity. Results were broken down by whether the human coders agreed on a response and by individual code.
6. Test substantive conclusions. Each set of codes was used to predict behaviour in a dating-profile experiment embedded in the same survey.

The models were GPT-5 nano, GPT-5 mini and GPT-5.2 from OpenAI; Claude 4.5 Haiku, Sonnet and Opus from Anthropic; and Mistral Small 4 and Large 3. All were called through their APIs from R.

How accurate were LLMs compared with human coders?

The best LLMs beat the students, and the smallest ones lost to them. The paper measures accuracy as Cohen's kappa against the researcher benchmark. The two student assistants scored 0.65 and 0.66, and their agreement with each other was only 0.60.

• Claude 4.5 Opus scored highest, with a kappa of 0.72, followed by Claude 4.5 Sonnet at 0.71.
• GPT-5.2 also significantly outperformed both students.
• Mistral Large 3 scored above the students, but the difference was not statistically significant.
• GPT-5 nano and Mistral Small 4 performed considerably worse than the students.

Within each provider, the pattern was the same: the more advanced the model, the more accurate the coding. The authors conclude that the choice of model tier matters more than the choice of provider.

Does giving the LLM examples help?

Yes. Adding 200 hand-coded examples improved almost every model, and some cheaper models with examples beat more advanced models without them. The only exception was GPT-5 nano, the weakest model, which did not improve. This suggests some models are simply too small for the task.

The gains were not spread evenly. Training data helped most on ambiguous responses and on the codes that models had initially handled worst. The authors read this as training data stabilising performance across the whole codebook rather than lifting every code equally. With examples included, Mistral Large 3 also clearly outperformed the students.

Where do LLMs make coding mistakes?

LLMs tended to go wrong on the same responses that people did. On the 55.6% of responses where both students matched the benchmark, the LLMs agreed at very high rates. On the 44.4% where at least one student diverged, the LLMs were also less accurate.

This matters for validation. LLM errors were concentrated on ambiguous text, which is where a human reviewer would look anyway, and were not scattered randomly. None of the three top models (GPT-5.2, Claude Sonnet 4.5 and Opus 4.5) performed substantially worse than both students on any single code. That rules out the worst case, where a model quietly misses one category entirely.

Would LLM coding change your conclusions?

In this study, no. This is arguably the paper's most useful test. Using the benchmark codes, respondents coded "unwilling" reacted strongly to a political mismatch in a dating profile, "situationally dependent" respondents reacted moderately, and "willing" respondents barely reacted. Those estimated reactions were almost identical whether the codes came from the researchers, the students or the LLMs. Hudde and Taflinger report that even mid-range models produced codes with predictive validity equivalent to human coding for this research question.

What does LLM coding cost?

Very little in money. Researcher time is the main cost. Coding all responses without training data cost about:

Source: Hudde & Taflinger (2026), batches of 10 responses, no training data. "Not singled out" means the paper's text does not compare that model directly with the students.

Batching is the main cost lever. For Claude Sonnet 4.5, sending one response per call cost about $7.00 and took around 60 minutes. Batches of 10 cost about $1 and took under 10 minutes. Adding training data raised Sonnet's cost from about $1 to about $3. The authors estimate that coding 10,000 similar responses would cost from under a dollar to a few dozen dollars.

How reproducible is LLM coding?

Reproducibility is good, though not perfect, and it depends on the provider. Anthropic and Mistral models accept temperature zero, and re-running them returned the same codes for more than 99% of responses. The GPT-5 family does not support temperature zero. Across three runs, GPT-5.2's pairwise consistency ranged from 92% to 94%. The authors therefore rate Anthropic and Mistral as the more reproducible options.

What this means for researchers

The study supports a division of labour rather than full automation. The researchers did the interpretive work themselves: reading the data, running the thematic analysis and building the codebook. The LLM then applied the finished scheme at scale. From the paper's recommendations:

• Develop the codebook by hand first. The authors expect this to take at least a few hundred responses in a case like theirs, until no new categories appear.
• Use each provider's stronger models for interpretive coding. The cheapest tier was not good enough here.
• Include about 200 hand-coded examples by default. The extra cost is small compared with the accuracy gain.
• Validate on your own data. Hand-code a random sample of about 200 responses, compute kappa against the LLM output, and read a random sample of coded responses to look for systematic errors.
• Upload as little as possible. The authors removed identifying information, sent only the answer text with a new random ID, and suggest Mistral or institution-hosted models when data protection matters most.
• Expect most of the benefit above about 1,000 responses. For smaller samples, hand coding may still be more efficient.

The authors note one important limitation themselves. This is one task, in English, with one US population and one team's codebook. Performance may differ with other text types, languages or coding schemes. The benchmark also reflects the judgements of a single research team.

If you collect open-ended answers inside conversational surveys, a tool like QualiTaTi AI Surveys keeps the text alongside the rest of each response. For the codebook-first, apply-at-scale workflow the paper describes, ThemeLens lets you review and edit AI-suggested codes against your own scheme. For background on the reliability metric, see our guide to intercoder reliability for AI coding.

FAQ

Can ChatGPT or Claude replace human coders for open-ended survey questions?

For applying an existing, well-defined codebook, the strongest models matched or beat trained student coders in this study. They did not replace the researchers who designed the codebook. Developing the codebook and validating the output stayed with humans.

What kappa should LLM coding reach?

The paper does not set a universal threshold. As a reference point, the trained student coders here scored 0.65–0.66 against the benchmark and the best model scored 0.72. Compare your LLM's kappa with what human coders achieve on your own data.

How many hand-coded examples should I give the model?

The study used 200 randomly selected hand-coded responses and recommends including training data as the default. The gains were largest on ambiguous answers and on the codes the models had found hardest.

Which LLM is best for qualitative coding?

In this study, Claude 4.5 Opus and Sonnet scored highest, followed closely by GPT-5.2. The authors stress that the gap between flagship models was small and that the model landscape changes quickly, so check results on your own data.

Source: Hudde, A., & Taflinger, S. (2026). A Golden Era for Open-Ended Questions? Using LLMs for Text Classification Tasks. ECONtribute Discussion Paper No. 416, University of Bonn and University of Cologne. (RePEc listing)

Last updated: October 30, 2026

This article is an independent editorial summary of third-party research. QualiTaTi is not affiliated with the authors or the ECONtribute series.