LLM Codebook Revision: Where Expert Time Pays Off (2026)
Qualitati Research Team · 2026-09-24 · 7 min read
Short answer: A September 2026 study from Penn State and Cornell found that the fastest way to improve an LLM codebook is to let several LLMs from different providers label the data, find the cases where they disagree most, and have experts label and explain only those cases. That approach reached 64.9% accuracy against expert labels, beating a codebook experts refined over six months (57.8%).
Anyone who has coded qualitative data with an LLM knows the bottleneck is not the model. It is the codebook. Definitions that looked clear in a team meeting fall apart at the boundaries, and fixing them usually means months of weekly calibration sessions. A new preprint, "Experts Rise Where LLMs Disagree" by Zeyu He, Zhuqian Zhou, Kirk Vanacore, Rene F. Kizilcec and Ting-Hao 'Kenneth' Huang (arXiv, 2026), tests a shortcut: use disagreement between LLMs to decide where scarce expert time should go, and then ask for that time in the form LLMs can use best.
Key takeaways
- Cross-model disagreement is a practical difficulty signal. When three LLMs agreed on a label, majority-vote accuracy was 66.2%; when all three disagreed, it fell to 12.8%.
- How experts give feedback matters more than how much. Labeling hard cases with a short rationale beat both answering clarification questions and editing AI-drafted codebook revisions.
- About 2.8 hours per expert with Rationale Labeling produced a better LLM codebook than the expert-refined final version, whose development each expert estimated at roughly 300 hours.
- Letting an LLM revise the codebook alone did not help: the AI-only revision scored 53.1% versus 53.4% for the untouched initial codebook.
- Diversity has to come from different model providers. Persona prompts that imitated individual experts produced little variation in labels.
What did the study test?
The study compared three ways of collecting expert feedback for revising a codebook that LLMs apply at scale. The setting was a real, long-running annotation project: two education experts built a taxonomy of 29 tutoring moves (30 labels including "None") for math tutoring transcripts from three providers, meeting weekly for six months to refine it.
The researchers took the earliest version of that codebook and had six LLMs from six providers (GPT 5.1, Gemini 2.5 Pro, Claude Sonnet 4.5, Qwen3 Coder, DeepSeek V3.2 and Mistral Large 3) label 6,595 tutor utterances the experts had never seen. They scored each utterance by normalized vote entropy, a measure of how widely the six models' labels were spread, and ranked labels by average disagreement. Each workflow then targeted 12 labels spread across high, medium and low disagreement.
Every resulting codebook was evaluated on a held-out set of 4,032 expert-consensus labels from 110 sessions, with the same six models labeling and their votes aggregated eight different ways.
Which expert feedback method worked best?
Rationale Labeling worked best, followed by structured question answering, while verifying AI-drafted edits did little. The three workflows were:
- Codebook Verifying (CV): an LLM drafted codebook revisions from disagreement cases and experts reviewed and edited them.
- Question Answering (QA): an LLM grouped disagreements into ambiguity patterns, generated clarification questions (six per label), and rewrote the codebook from the experts' answers.
- Rationale Labeling (RL): experts labeled 10 selected disagreement utterances per target label and explained their choice, without seeing any model predictions; an LLM then rewrote the relevant codebook entries.
| Codebook | Accuracy | Weighted F1 | Expert time (per expert) |
|---|
| Initial codebook (no revision) | 53.4% | 0.536 | None |
| AI-only revision | 53.1% | 0.547 | None |
| Codebook Verifying (best variant) | 53.7% | 0.557 | 0.9 hours |
| Question Answering (best variant) | 60.5% | 0.613 | 3.0 hours |
| Rationale Labeling | 64.9% | 0.658 | 2.8 hours |
| Final expert codebook (six months) | 57.8% | 0.583 | ~300 hours (retrospective estimate) |
According to He et al. (2026), both Rationale Labeling and the best QA setting beat the final expert codebook under all eight aggregation methods in paired tests. The authors note that the 300-hour figure covers the whole annotation project, not just codebook edits, so it is context rather than a like-for-like cost comparison.
Why does disagreement between LLMs point to hard cases?
Disagreement between models tends to show up exactly where the codebook is ambiguous, missing a rule, or drawing a fine boundary. In a preliminary test on 825 expert-labeled utterances, the authors found accuracy of 66.2% when GPT 5.1, Claude Sonnet 4.5 and Gemini 2.5 Pro all agreed (408 utterances), 36.0% when two agreed (308), and 12.8% when all three disagreed (109).
The gains were concentrated where they were most needed. On the ten highest-disagreement labels, weighted F1 rose from 0.295 with the initial codebook to 0.568 with Rationale Labeling, compared with 0.429 for the six-month expert codebook. Across every condition, low-disagreement labels scored highest and high-disagreement labels lowest.
One design detail is worth copying. The authors first tried prompting models to imitate each expert, using their writing and dissertations. Those personas barely changed the labels. Predictions varied far more across providers than within one provider, so the final panel used six different vendors.
Why didn't editing AI-drafted revisions help?
Asking experts to verify an LLM's proposed codebook edits was the cheapest workflow in time but produced almost no gain, and one variant scored below the initial codebook (52.0%). The authors suggest that on boundary cases an LLM can only guess without expert guidance, so its drafted revisions give experts the wrong thing to react to. Automatic prompt optimizers fared no better: without gold labels, both APO and SPO fell below the unrevised codebook.
The experts' own ratings add nuance. QA was rated the most mentally demanding (7 out of 7) but also best at surfacing distinctions they had not considered before, while Rationale Labeling was the least demanding (3.5 out of 7). QA may therefore be useful early in a project, when teams still need to discover their ambiguities.
What this means for qualitative researchers
The practical lesson is to spend expert time on labeled examples with reasons, not on proofreading AI text. If you use LLMs for deductive coding at scale, a workable routine based on this study looks like this:
- Draft a codebook with explanations, examples and near misses for each code.
- Run it through several models from different providers on data you have not coded yet.
- Rank segments and codes by how much the models disagree.
- Have the team code a small set of the most contested segments blind, with a one-line rationale each.
- Let an LLM fold those labels and rationales into the codebook, then re-test on a held-out, human-coded sample.
Tools that let you inspect and correct AI-applied codes make this loop easier to run. In ThemeLens, for example, you can review AI-applied codes before they feed your themes. For a broader walkthrough of codebook structure, see our guide on how to build a codebook for AI qualitative coding.
Keep the limits in view. This is one project, with two experts and a label set that stayed largely fixed, so the results may not carry over to inductive coding where codes are still emerging. The authors also flag a real trade-off: slow codebook work is how experts learn their data, and compressing it into hours may cost some of that understanding.
FAQ
What is cross-model disagreement in LLM coding?
It is the spread of labels that several different LLMs assign to the same text segment under the same codebook. The study measured it with normalized vote entropy, where 0 means all models agree and 1 means every model chose a different label.
How much expert time does Rationale Labeling need?
In this study each expert labeled 60 contested utterances with rationales, which took about 2.8 hours on average.
Can an LLM improve a codebook without human input?
Not in this study. The AI-only revision rewrote all 30 labels yet scored 53.1% accuracy, slightly below the 53.4% of the original codebook.
Does this replace intercoder reliability checks?
No. The workflow decides where to focus expert effort, but you still need a human-coded held-out sample to confirm that the revised codebook actually improves agreement with expert judgment.
Last updated: September 24, 2026.
This article is an independent editorial summary of third-party research by Qualitati. It is not affiliated with or endorsed by the paper's authors. Read the original preprint for full methods and results: arXiv:2609.26926.