Does Prompt Engineering Improve AI Qualitative Coding?
Qualitati Research Team · 2026-07-20 · 8 min read
Prompt engineering helps AI qualitative coding far less than most researchers assume. In a 2026 controlled study of three large language models coding 116 real forum quotes, switching from zero-shot to multi-shot prompting moved agreement with human coders by only about 0.03 Cohen's kappa — and every model stayed in the "fair to moderate" range.
What did the study test?
The study compared two prompting strategies against a human-coded gold standard. Alshaikh and colleagues (2026), writing for the PROMPT-SE workshop at EASE 2026, asked three inexpensive LLMs — Claude Haiku, DeepSeek-Chat, and Gemini 2.5 Flash — to perform closed coding on 116 quotes drawn from the Stack Exchange Project Management and Software Engineering communities.
The codebook contained seven behavioural categories derived from Amy Edmondson's psychological safety framework: Admitting Mistakes, Disagreeing with Suggestions/Ideas, Drawing Attention to Errors, Expressing Concerns, Recommending Changes, Seeking Help, and Sharing Negative Feedback. Human coders built the gold standard through multi-researcher coding with consensus resolution, reporting a 4.67% disagreement rate between researchers.
Crucially, the authors ran each model ten independent times per prompt configuration — 60 classification runs in total — rather than trusting a single pass. That design choice turns out to be the paper's most useful contribution.
How accurate was AI qualitative coding?
Agreement with human coders was moderate at best, and multi-shot prompting produced small gains. According to Alshaikh et al. (2026), the mean Cohen's kappa values were:
| Model | Zero-shot kappa | Multi-shot kappa | Change |
|---|
| Gemini 2.5 Flash | 0.403 | 0.437 | +0.034 |
| Claude Haiku | 0.392 | 0.426 | +0.034 |
| DeepSeek-Chat | 0.332 | 0.331 | −0.001 |
Only Claude Haiku's improvement reached statistical significance (Wilcoxon p = 0.004). Gemini showed a positive trend, and DeepSeek-Chat was effectively unchanged — the examples in the prompt did nothing for it.
For context, kappa values in the 0.33–0.44 band are conventionally read as fair to moderate agreement. That is well below what most qualitative teams would accept between two human coders, and it means none of these configurations could be trusted to code unsupervised.
Why does the same model give different answers each run?
Run-to-run instability differed sharply by model, which is why single-run evaluations mislead. Across ten repetitions of an identical prompt on identical data, DeepSeek-Chat and Claude Haiku each showed a kappa standard deviation of roughly 0.017, while Gemini 2.5 Flash showed 0.038 — more than double.
The practical consequence: if you evaluate Gemini once and get a lucky draw, you could overstate its agreement by an amount comparable to the entire benefit of multi-shot prompting. The authors argue that multi-run evaluation should be treated as a minimum methodological standard for anyone reporting LLM coding performance. That is a reasonable bar, and it is one very few published applications currently clear.
Which codes did the AI get wrong?
Errors were systematic, not random — the models had consistent category preferences. The clearest pattern was over-prediction of "Sharing Negative Feedback," a rare code appearing in only 4 of 116 quotes. The models assigned it at 4.75× its true frequency (Claude Haiku and DeepSeek-Chat) and 5.25× (Gemini 2.5 Flash).
In the opposite direction, all three models under-predicted "Expressing Concerns" — the single most common code, at 50 of 116 quotes — with ratios between 0.64 and 0.74 under zero-shot prompting.
This shape of error is familiar to anyone who has audited AI coding output. The model gravitates toward vivid, lexically distinctive categories and away from the broad, contextually defined ones. A quote expressing worry reads as unremarkable text; a quote with sharp wording trips the "negative feedback" pattern even when the underlying construct is different. If your codebook mixes rare-but-salient codes with common-but-diffuse ones, expect the distribution of AI codes to be skewed in exactly this way.
What this means for researchers
Three practical takeaways follow from the findings:
- Calibrate prompt strategy per model, not globally. The authors state this directly, and their data support it: the same multi-shot examples helped one model significantly, one marginally, and one not at all. A prompt tuned on one model does not transfer.
- Report multi-run results. A single kappa from a single run is not a stable estimate of anything. Run the coding several times and report the spread alongside the mean.
- Audit the code distribution, not just the headline agreement. An acceptable overall kappa can hide a rare code being applied five times too often. Compare the frequency of each code in the AI output against your human-coded subsample.
None of this argues against using LLMs for qualitative coding. It argues for treating them as a first pass that a human verifies, which is how tools like ThemeLens are designed to be used — AI proposes codes and themes, the researcher adjudicates and keeps an audit trail. The same logic applies upstream at data collection: an AI Interviewer can run the conversation at scale, but the interpretive claims still belong to the researcher.
How much should you trust this study?
The design is careful but the scope is narrow, and the authors say so. The corpus is 116 quotes from two Stack Exchange communities, so the findings may not generalise to interview transcripts, open-ended survey text, or other domains. Several psychological safety categories overlap semantically, which inflates the difficulty of the task and depresses kappa for reasons that are partly about the codebook rather than the models. And the human gold standard, while consensus-based, does not report a formal inter-rater reliability statistic — so we cannot say how the models' 0.33–0.44 compares to the human–human ceiling on this same data. A human baseline of, say, 0.60 would tell a very different story than one of 0.85.
Read the study as evidence about the shape of the problem — per-model prompt sensitivity, run variance, systematic code bias — rather than as a benchmark of how well AI codes in general.
FAQ
Does multi-shot prompting improve AI qualitative coding?
Sometimes, and modestly. In this study it improved two of three models by about 0.03 kappa and had no effect on the third. It is worth testing on your own model and codebook rather than assuming it will help.
What is a good Cohen's kappa for AI qualitative coding?
There is no single accepted threshold, but the 0.33–0.44 range observed here is generally read as fair to moderate — not high enough to code without human review. The more informative comparison is against the agreement your own human coders achieve on the same data.
Why should I run the same AI coding prompt multiple times?
Because output varies between runs. This study found kappa standard deviations of 0.017 to 0.038 across ten identical runs, meaning a single run can over- or under-state performance by roughly as much as a prompting improvement would gain you.
Which model coded best?
Gemini 2.5 Flash had the highest mean kappa (0.437 multi-shot), but also the least stable output. Claude Haiku was close behind (0.426) and more consistent. The differences are small enough that model choice should probably turn on cost, privacy, and stability rather than on these accuracy numbers alone.
Last updated: July 20, 2026. This is an independent editorial summary of third-party research; it is not affiliated with or endorsed by the study's authors. Primary source: Alshaikh et al., "Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study," arXiv:2605.07422 (2026).