How to Analyze Open-Ended Survey Responses (2026)

Qualitati Research Team · 2026-07-24 · 11 min read

Last updated: July 24, 2026

Short answer

To analyze open-ended survey responses at scale, pick a method that matches your goal: LLM-assisted thematic coding when you need to know what people said, LLM pairwise comparison when you need to rank or score responses on a latent dimension, and few-shot classification when you have a fixed category list. In all three cases, validate against a human-coded benchmark sample before trusting the output.

Key takeaways

Open-text questions collect richer data than closed scales but historically cost too much to code — which is why most teams either drop them or skim a handful of quotes.
A March 2026 study in Public Opinion Quarterly shows LLM pairwise comparisons scale open text onto a continuous dimension more consistently than asking a model for a 0–10 rating, with Bradley–Terry estimates correlating above 0.87 across six models.
A 2025 study in International Health categorizing free-text answers from household vaccination surveys in the DRC reported accuracy from 61.5% to 96%, improving sharply with fine-tuning or 20–100 few-shot examples.
Accuracy is method-dependent, not just model-dependent. The single biggest lever is giving the model examples and a clear definition of the construct.
Use the Open-Text Analysis Method Selector and the Open-Ended Response Validation Checklist below to choose an approach and prove it worked.

Why open-ended survey responses stay unanalyzed

Almost every survey team knows the pattern. You add two or three open-text questions because closed scales cannot capture why someone answered the way they did. Fielding goes well. Then 4,000 free-text answers land in a spreadsheet column, and nobody has three weeks to code them. The open text becomes a quote mine for the deck — a few vivid verbatims, chosen by whoever scrolled furthest.

That is a measurement problem, not just a workload problem. Hand-picked quotes are a non-random sample of your own data. If you are going to collect open text, you need a defensible way to reduce all of it, not the top of the scroll.

Large language models changed the economics here. What they did not change is the requirement to demonstrate that the reduction is accurate. The useful question in 2026 is no longer "can AI read open text?" but "which method, and how do I show it worked?"

Three methods, three different jobs

Most confusion about how to analyze open-ended survey responses comes from treating it as one task. It is three.

1. Thematic coding: what did people say?

This is the classic qualitative job — inductively derive codes, group them into themes, report prevalence and anchor quotes. It is the right method when your research question is exploratory ("what is driving churn?") and you do not know the categories in advance.

LLM-assisted coding is now workable at survey scale, but the evidence says treat model output as a first draft. A 2026 comparison of open-source models in the International Journal of Qualitative Methods found codebook-guided deductive coding produced more usable output than free inductive coding, and that a substantial share of machine-generated codes were duplicative or thin (Misra et al., 2026). Human consolidation of the codebook is where the quality comes from.

2. Classification: does each response belong to a known category?

When you already have a category list — reasons for cancellation, barriers to adoption, product areas — the task is classification, and it is the most measurable of the three. You can build a benchmark, compute accuracy, and iterate.

The best public evidence comes from global health. Burstein, Mafuta, and Proctor tested zero-shot prompting, few-shot prompting, fine-tuning, and semantic embeddings on caregivers' free-text reasons for not vaccinating children, drawn from four years of household coverage surveys in the Democratic Republic of the Congo. Accuracy against a curated benchmark ranged from 61.5% to 96%, with the gains coming from fine-tuning or as few as 20–100 labeled examples (International Health, September 2025).

That range is the headline finding, and it is a warning as much as a promise. A zero-shot prompt with no examples can land near the bottom of it.

3. Scaling: how much of X does each response express?

Sometimes you do not want categories at all. You want a score — how knowledgeable, how negative, how specific a response is — so you can use open text as a variable alongside closed-ended items.

The March 2026 Public Opinion Quarterly paper by Matthew DiGiuseppe (Leiden University) and Michael Flynn (Kansas State University) is the most rigorous treatment of this so far. Instead of asking a model to rate each response 0–10, they have the model judge pairs of responses — which one better reflects the concept — then fit a Bayesian Bradley–Terry model to recover a continuous score with uncertainty intervals (DiGiuseppe & Flynn, POQ 90(3), 2026).

Across six models — GPT-4o and GPT-4o mini, Llama 3.1 405B and 8B, and Gemma 3 27B and 4B — Bradley–Terry estimates correlated above 0.87, and the largest models at 0.95. Direct 0–10 ratings were less consistent. Against expert human coders on 300 pairs, frontier models exceeded an F1 of 0.8. Using knowledge of interest rates as the test concept, the resulting scores behaved as they should: high scorers were more familiar with the Federal Reserve and answered factual questions correctly.

The methodological point generalizes beyond political science. Relative judgments ("is A more X than B?") are easier for a model to make consistently than absolute ones ("rate A from 0 to 10"). If you need open text as a measure, comparison is the more defensible route.

The Open-Text Analysis Method Selector

This is a Qualitati-owned decision matrix. Pick the row that matches what you actually need out of the open text.

Your goal	Method	What you must supply	How to validate	Effort
Discover unknown reasons or themes	LLM-assisted inductive coding, then human codebook consolidation	Research questions; a review pass on the draft codebook	Second coder on a 10–20% subsample; check theme–quote fit	Medium
Count responses against a known list	Few-shot or fine-tuned classification	20–100 labeled examples per the DRC study	Held-out benchmark set; report accuracy and per-class errors	Low–Medium
Score responses on a latent dimension	LLM pairwise comparison + Bradley–Terry	A precise definition of the dimension	Expert-coded pairs (F1); correlate with closed-ended items	High
Summarize for a stakeholder readout	Thematic synthesis with participant-anchored quotes	Coded data first — never summarize raw text directly	Trace every claim in the summary back to coded segments	Low
Detect low-effort or off-topic answers	Rule-based screens + model flagging, reviewed by a human	Length, duplication, and relevance criteria set in advance	Manual review of all flagged cases before exclusion	Low

Alt text suggestion: five-row decision matrix mapping open-ended survey analysis goals to methods, required inputs, validation approach, and effort level.

The Open-Ended Response Validation Checklist

Whatever method you choose, this is the minimum evidence you should be able to produce when someone asks whether the numbers are real.

Before you analyze

Write down the construct or research question the open text is supposed to answer.
Hand-code a random sample of 100–300 responses as your benchmark, before seeing any model output.
Decide the accuracy threshold you would accept — and what you will do if the model misses it.
Screen for AI-generated respondent answers and low-effort text as a separate step.

During analysis

Record the model name and version, the prompt, and any examples supplied.
Give the model examples. The DRC evidence is that this is the largest single accuracy lever.
Keep raw model output separate from human-edited output.
For scaling tasks, prefer pairwise comparisons over direct ratings and carry the uncertainty forward.

Before you report

Report agreement with the human benchmark, not just the final percentages.
Check minority categories specifically — aggregate accuracy hides rare-class failure.
Anchor every theme to verbatim quotes from identifiable respondents.
State plainly in the method section which steps were AI-assisted.

Where Qualitati fits

Qualitati is an AI user research platform for product, UX, and customer insights teams. Two parts of it apply directly to open-ended survey data.

First, conversational surveys change the input. Instead of one static open-text box, the AI asks a follow-up when an answer is thin — so you are analyzing elaborated responses rather than three-word fragments. Much of the difficulty in open-text analysis is that the raw material is too sparse to code reliably; probing at collection time is cheaper than rescuing sparse text at analysis time.

Second, ThemeLens handles the reduction. It runs a map-reduce pipeline across up to 100 transcripts at once, maps codes to your research questions, and synthesizes themes anchored to participant quotes. The QDA Workspace supports both inductive and deductive coding with codebook generation, so you can consolidate a machine-drafted codebook by hand rather than accepting it wholesale. Studies run in any of 10 languages, which matters when open text arrives multilingual.

What Qualitati does not do is remove the validation step described above. No platform can. The benchmark sample is yours to code.

Limitations and trade-offs

Several caveats deserve to be stated before anyone builds a workflow on this.

Accuracy varies enormously by construct. The 61.5%–96% range in the DRC study is not noise around a mean — it reflects real differences between prompting strategies and category difficulty. Assume you are at the low end until you measure.

Scaling methods need a clean latent dimension. DiGiuseppe and Flynn are explicit that their approach requires a clearly identifiable dimension, that domain-knowledge boundaries for LLMs remain unmapped, and that some concepts show less consistent agreement across models. "Sentiment" or "topic knowledge" may be tractable; "brand meaning" probably is not.

Pairwise comparison is computationally expensive. The number of possible pairs grows quadratically with sample size. That is a real cost consideration, and deciding which pairs to compare becomes part of the method.

Respondents now use AI too. The POQ authors flag this directly: survey respondents may increasingly use LLMs to write open-ended answers, which contaminates the very data being analyzed. Screening for this is becoming a standard data-quality step rather than an edge case.

Model drift is unresolved. Hosted models change. An analysis run in January and repeated in July may not reproduce. Record versions and, where reproducibility matters most, consider open-weight models you can pin.

A note for sensitive research: for regulated, clinical, or policy-consequential work, treat AI-assisted open-text analysis as requiring human review by a qualified methodologist, not as a validated instrument.

Who this is for — and when not to use it

Use these methods when: you have more than a few hundred open-text responses, a clear analytic goal, and the ability to hand-code a benchmark sample.

Do not use them when: you have fewer than roughly 100 responses (read them yourself — it is faster and better), when the construct is contested or highly interpretive, when the data is sensitive enough that a coding error carries real-world harm, or when you cannot produce a validation sample. In that last case the honest move is to report the open text descriptively rather than quantify it.

FAQ

What is the best way to analyze open-ended survey responses?

There is no single best way — it depends on the goal. For discovering themes, use LLM-assisted coding with human codebook consolidation. For counting against known categories, use few-shot classification with 20–100 examples. For scoring responses on a dimension, use pairwise comparison with a Bradley–Terry model, which the 2026 POQ study found more consistent than direct 0–10 ratings.

How accurate is AI at coding open-ended survey responses?

Published accuracy varies widely. The 2025 International Health study reported 61.5% to 96% against a human benchmark, depending on the prompting strategy, with few-shot examples and fine-tuning driving the gains. Because the range is that wide, you should measure accuracy on your own data rather than assume a published figure transfers.

How many examples does an LLM need to classify survey text well?

The DRC vaccination study found that as few as 20–100 labeled examples produced high accuracy in categorizing free-text responses. That is a practical threshold: hand-labeling 100 responses is a morning's work, and it is the highest-return hour in the whole workflow.

Why are pairwise comparisons better than asking a model for a rating?

Relative judgments are more stable than absolute ones. In the POQ study, Bradley–Terry estimates from pairwise comparisons correlated above 0.87 across six models of different sizes, while zero-shot 0–10 ratings were less consistent between even the largest models. The Bayesian setup also produces uncertainty intervals, which point-estimate ratings do not.

Should I still write open-ended survey questions if I cannot analyze them all?

Yes, but fewer and better. One well-targeted open question with an AI follow-up probe yields more codable material than four generic boxes. Conversational surveys are designed around exactly this trade-off.

Do I need to disclose that AI helped analyze the responses?

For academic publication and most client work, yes. State which steps were AI-assisted, which model and version, and what human review was applied. This is now standard expectation in qualitative reporting, and it is also the basis of a defensible audit trail.

Bottom line

The bottleneck in analyzing open-ended survey responses has moved. It is no longer coding capacity — it is validation discipline. The 2026 evidence is encouraging: pairwise LLM comparisons scale open text consistently onto latent dimensions, and few-shot classification reaches high accuracy with a hundred labeled examples. But both results come with measured, published error rates, and neither transfers to your data without a benchmark of your own.

Choose the method that matches your goal, hand-code 100 responses before you start, and report the agreement alongside the findings. That is the whole discipline.

Start free with 30 credits — no credit card required — and run a conversational survey, then analyze the open text with ThemeLens. Or view transparent pricing to see per-credit usage rates before you commit.