Can AI Build a Qualitative Codebook? What a 2026 Study Found
Qualitati Research Team · 2026-08-14 · 9 min read
Short answer. Yes — with a hard limit. In a 2026 study, an LLM pipeline built a 67-item qualitative codebook from 45,000 messages, but three human coders applying it to fresh data found five whole categories the model had never surfaced, despite the model processing roughly eight times more data than the humans read.
What did the study actually test?
The study asked a narrower question than "can AI code qualitative data?" It asked which phases of codebook development an LLM can take over, and what has to stay human.
Liu, Sun, Esbenshade, Xiao, Tian, Zhang and He (2026), in a preprint posted to arXiv on 30 July 2026, documented a three-phase human-LLM pipeline that adapted grounded theory's open, axial and selective coding to a corpus of 45,000 messages exchanged between K-12 educators and a generative AI teaching platform. The output was a hierarchical codebook. The test of that codebook was not model self-consistency — it was three trained human coders applying it, unaided, to an independent sample the model had never touched.
The three LLM-assisted phases
- Open coding. Claude 3 Sonnet labelled a pilot of 256 "trios" — an educator prompt, the AI's reply, and the educator's follow-up — returning content domain, pedagogical strategy and requested support as structured JSON. Two researchers read the raw outputs and refined the label vocabulary by hand.
- Axial coding. The corpus expanded to 9,352 trios with the model free-generating labels. Researchers reviewed a random 1,000 outputs, then clustered labels by semantic similarity and pedagogical intent — merging "differentiated strategies" with "differentiated support", but deliberately keeping "inquiry-based learning" and "project-based learning" apart.
- Selective coding. Claude 3.5 Haiku applied the now-closed vocabulary to 11,539 new trios, choosing only from existing codebook categories. This is where the instrument was stress-tested for coverage at scale.
Note the deliberate design choice: the authors treat the LLM as a labelling instrument, not an interpretive agent. That is why switching models mid-study did not, in their argument, threaten validity — the model proposed candidate labels, humans decided which survived.
What did the human validation phase find?
It found gaps that scale had not. Three coders with education domain expertise applied the 67-item draft codebook to 2,560 fresh messages drawn from 500 conversations collected over the following year. According to Liu et al. (2026), they added five new codes, taking the instrument from 67 to 72 items and from 18 to 19 categories:
| Code added by humans | Why the draft codebook missed it |
|---|
| Culturally Responsive Teaching | Frequent enough to warrant its own item rather than sitting inside "Differentiated Instructional Strategies" |
| Foreign Language Skill Development | World-language teachers' requests had a distinct character the existing ELA code did not capture |
| Classroom Management | Proactive routines and procedures, as opposed to the reactive "Student Behavioral Intervention" already present |
| Class, School, or Community Event | Assemblies, family nights and field trips fit neither existing communication nor documentation codes |
| School District Finance | A small administrator population — budgeting and resource allocation — absent from an instruction-centric draft |
Read the pattern rather than the list. Each addition corresponds to a user population, a professional context, or a change in usage over time. Those are exactly the things a model sampling text at volume smooths over: they are rare in aggregate but coherent to a reader who knows the domain.
How reliable were the human coders?
Moderately, and the authors are candid about it. Because a single educator message often carries several codes at once — planning plus differentiation, or assessment design plus standards alignment — conventional chance-corrected statistics like kappa are a poor fit: they score partial overlap between two coders' code sets as total disagreement.
The team instead reported Jaccard similarity over code sets across a 289-message hold-out (11.3% of the validation sample) after three calibration rounds. Mean pairwise Jaccard was 0.52 at the item level, and ranged from 0.46 to 0.55 depending on whether agreement was assessed at item, category or domain level. Disagreement clustered at semantic boundaries between adjacent codes, not on whether a message belonged in the codebook at all.
That granularity gradient is the useful methodological takeaway: a single coefficient at the finest level of a 72-item hierarchy tells you almost nothing. Report agreement at each level.
What the instrument shows about how people actually use AI
The 2,560 validated messages produced 5,237 code applications — roughly two codes per message. At domain level, Liu et al. (2026) report Instructional Practices as the largest share (1,913 applications, 36.5%), then Curriculum and Content Focus (909, 17.4%), Assessment and Feedback (603, 11.5%), Professional Responsibilities (450, 8.6%) and Student Needs and Context (396, 7.6%).
The striking number is the residual: an "Other" domain combining discourse continuity and non-educational use accounted for 924 applications, or 17.6%. Nearly a fifth of all coded content is conversational maintenance — reformat this, extend that, keep going — plus occasional off-task chat. Anyone analysing chat-based research data should budget a code for it rather than discarding it as noise.
What this means for researchers using AI in qualitative work
Three practical implications follow from the design, not just the findings.
Use the model for breadth, not for conceptual authority. The paper's own summary is that the model was dependable for consistency of structured annotation and for surfacing recurring patterns across tens of thousands of units — tasks where its lack of fatigue is a real advantage. It was not a source of category definitions at any phase. If your workflow lets an AI both generate and ratify the codebook, you have removed the only step that caught five missing categories. Tools like ThemeLens are best used this way: to propose a candidate structure fast, which you then argue with.
Pick the unit of analysis on purpose. Some conversations in this corpus exceeded 500 messages, so the authors coded overlapping three-message windows to keep dialogic context visible. More than 40% of validation-sample conversations ran longer than three messages. Chunking that severs a participant's turn from what prompted it will cost you meaning — a live consideration whether you are analysing chat logs or transcripts from an AI interviewer.
Budget for the human validation phase; it is not a formality. In this study it produced the reliability evidence, extended the instrument's coverage, and forced the inclusion and exclusion criteria to be written clearly enough to teach. A pipeline that stopped after the selective phase would have shipped a plausible codebook with none of those warrants.
Limitations worth stating
All the data come from one AI platform, so the codebook's domains reflect that platform's user base; the authors expect the structure to transfer more readily than the items. The specific commercial models used are not reproducible in a strict sense — the prompts, procedures and human decision points are what generalise. Human agreement, while workable for multi-label coding at this size, leaves room for improvement. And the corpus captures what educators asked the AI for, not what they subsequently did in a classroom.
FAQ
Can an LLM build a qualitative codebook on its own?
It can generate a usable draft. In this study the LLM-assisted phases produced 67 of the final 72 items. But human coders applying that draft to unseen data found five categories the model had missed, so the draft was not the finished instrument.
Why did the researchers use Jaccard similarity instead of Cohen's kappa?
Because coders assigned multiple codes to a single message. Chance-corrected coefficients assume one label per unit and treat partial overlap between two coders' code sets as full disagreement, which understates real convergence in multi-label coding.
Is a Jaccard score of 0.52 good?
There is no established threshold equivalent to kappa's conventions for set-valued agreement, and the authors themselves call it adequate but improvable. Its interpretive value here comes from the gradient — agreement rising from item to category to domain level, which localises disagreement at code boundaries rather than at the level of what the message is about.
Does switching AI models mid-analysis invalidate the results?
The authors argue not, on the specific condition that the model is used as a labelling instrument whose output is inspected and consolidated by humans before it enters the codebook. If model output were reported as a finding in itself, the change would matter.
Primary source: Liu, A., Sun, M., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., & He, K. (2026). Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use. arXiv:2607.28889.
Last updated: 14 August 2026. This is an independent editorial summary of third-party research; Qualitati is not affiliated with the authors, and readers should consult the original preprint for full methodological detail.