Last updated: May 31, 2026
\n\nYes. How you word a prompt measurably changes the codes a large language model assigns to qualitative data. A 2025 study introducing HALC (Reich, Thoms & Schrimpf, 2025) tested 1,512 prompts across roughly two million model requests and found that systematically optimized prompts lifted agreement with expert human coders into the \"substantial\" range (Krippendorff's alpha up to 0.78). Prompt design is not a cosmetic detail — it is a core validity decision in AI-assisted coding.
\n\nThe paper, \"Introducing HALC: A general pipeline for finding optimal prompting strategies for automated coding with LLMs in the computational social sciences,\" was posted to arXiv on 29 July 2025 by Andreas Reich, Claudia Thoms, and Tobias Schrimpf. HALC is a pipeline that systematically builds and tests prompts for a given codebook and model, rather than rewriting the codebook to suit the model. The authors ran 1,512 distinct prompts — totalling about two million requests — against local open-weight models, then compared the resulting codes against expert-coded ground truth on two social-science corpora (a \"climate\" dataset and a \"movement\" dataset).
\n\nUsing Mistral NeMo with HALC-optimized prompts, the authors reported the following agreement with human experts, measured in Krippendorff's alpha (where 0.667 is a common minimum threshold and 0.80 is considered strong):
\n\n| Coding task | Dataset | Krippendorff's alpha |
|---|---|---|
| Single-variable coding | Climate | 0.76 |
| Single-variable coding | Movement | 0.78 |
| Multi-variable coding | Climate | 0.71 |
| Multi-variable coding | Movement | 0.74 |
According to Reich, Thoms & Schrimpf (2025), single-variable tasks (deciding one code at a time) reached higher reliability than multi-variable tasks (juggling several codes at once) — a pattern that mirrors how human coders also struggle more as a codebook grows more complex. Critically, these numbers were achieved with an open, locally hosted model, not a frontier commercial API, suggesting that careful prompting can partly substitute for raw model size.
\n\nThe pipeline's logic can be broken into four steps:
\nThe headline lesson is that a single fixed prompt is a hidden methodological assumption. Two researchers coding the same transcripts with the same model but different prompt wording can reach different conclusions — a reproducibility risk that traditional intercoder-reliability checks were never designed to catch. The study reframes prompt design as something to be documented, tested, and reported, just like a codebook or an interview guide.
\nFor applied UX and market researchers, the practical takeaway is to (1) keep a human-authored codebook as the source of truth, (2) test more than one prompt before trusting the output, and (3) report the reliability of AI codes against a human-coded sample. Tools that keep every AI code anchored to the underlying data make this far easier. In Qualitati, ThemeLens proposes themes that stay linked to participant quotes, and the QDA Workspace lets researchers review, edit, and validate every AI-assigned code before it counts as a finding. Pair that with AI-moderated interviews to collect the raw data in the first place.
\n\nThis is one study on two social-science corpora using specific open models, so the exact alpha values will not transfer to every domain, codebook, or language. The two-million-request scale also implies real compute cost, which smaller teams may not be able to replicate. And high agreement with expert codes measures reliability, not interpretive depth — an LLM can be consistent and still miss the nuance a skilled qualitative researcher would surface. The result argues for prompt optimization as a discipline, not for removing humans from the loop.
\n\nDoes prompt wording really change qualitative coding results?
Yes. The 2025 HALC study tested 1,512 prompts and found prompt design materially affected agreement with human coders, with optimized prompts reaching Krippendorff's alpha up to 0.78.
What is a \"good\" reliability score for AI coding?
Krippendorff's alpha of 0.667 is a common minimum and 0.80 indicates strong agreement. HALC's best results (0.71–0.78) fall in the substantial range, below the 0.80 strong threshold.
Do I still need human coders if I optimize prompts?
Yes. Optimized prompts improve reliability but do not guarantee interpretive validity. Best practice is to validate AI codes against a human-coded sample and keep researchers in the loop.
Where can I read the original paper?
The preprint is on arXiv: arxiv.org/abs/2507.21831.
Primary source: Reich, A., Thoms, C., & Schrimpf, T. (2025). Introducing HALC: A general pipeline for finding optimal prompting strategies for automated coding with LLMs in the computational social sciences. arXiv:2507.21831. https://arxiv.org/abs/2507.21831
\n\nLast updated: May 31, 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the authors.
\n","related":[{"slug":"batching-synthetic-respondents-inflates-correlations-2026","title":"Batching Synthetic Respondents Inflates Correlations (2026)","description":"Generating many synthetic respondents in one LLM request inflates correlations, a 2026 LSE study finds. What it means for synthetic survey data validity.","category":"Methodology","date":"2026-10-13","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"ai-chatbot-qualitative-interviews-lessons-2026","title":"AI Chatbot Interviews at €0.04: Lessons From a 2026 Study","description":"Can AI chatbot interviews replace qualitative interviewers? A 2026 study of 74 chatbot-led interviews found cheap, engaging data collection but limited depth.","category":"Methodology","date":"2026-10-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-inductive-content-analysis-survey-responses-2026","title":"Can GPT-5 Do Inductive Content Analysis of Survey Text?","description":"Can GPT-5 do inductive content analysis? A 2026 study of 903 survey answers found moderate human-AI agreement (ARI 0.61 codes, 0.54 themes), varying by question.","category":"Methodology","date":"2026-10-05","readTime":"7 min read","author":"Qualitati Research Team"}]}