Methodology

Do LLMs Write the Same Codebook? An 8-Model Audit (2026)

Qualitati Research Team

2026-11-17

7 min read


Do different LLMs write the same qualitative codebook? No. A November 2026 audit by Tamme, Hantel and colleagues gave eight models from five vendors the same interview samples and the same instructions. Every model built a different category system. Each model coded its own runs consistently (Krippendorff's alpha .76 to .92), but agreement between models on one shared codebook fell to about .69.

That gap matters to anyone who lets an LLM draft the codebook before coding interviews or open-ended survey answers. Below is what the study tested, what it found, and how to run a similar check on your own project.

What did the study test?

The study tested whether the model that creates a codebook changes the coded evidence a researcher ends up with. Most LLM coding benchmarks hand the model a fixed codebook and score how well it applies it. This paper looks one step earlier, at who writes the categories in the first place.

According to the preprint (Tamme, Hantel et al., 2026, arXiv:2611.19872), researchers from the Technical University of Munich and MIT worked with:

  • 668 interviews from six deployments of an AI-led interview platform inside one organisation, split into 13,166 fixed text units.
  • Eight models from five vendors: four Anthropic models (Opus 4.8, Sonnet 5, Haiku 4.5, Fable 5), GPT-4.1, Gemini 2.5 Pro, Grok 4.5 and DeepSeek v4-pro.
  • 48 codebooks (one per model per theme), each built from the same 15 randomly selected interviews per theme.
  • 270 coding runs: every model coded every unit three times with its own codebook and three times with a shared reference codebook written by Opus.

The authors call the procedure the Qualitative Variance Audit Framework (QVAF).

How different are LLM-generated codebooks?

They differ a lot in granularity, coverage and measurement type. On average, GPT-4.1 produced 15.0 substantive subcodes per theme, while Fable 5 produced 24.8 and DeepSeek v4-pro 25.0.

More codes did not mean more of the data got coded. GPT-4.1 coded the smallest share of units (.606), but the two most granular models, DeepSeek (.663) and Fable (.700), still trailed Opus (.741) and Sonnet (.745).

The models also disagreed on which codes are ordinal (codes with ordered levels, such as low to high workload). On the German-language Employee Engagement theme, Haiku declared 15 of its 16 subcodes ordinal, while DeepSeek declared none of its 27 ordinal. The authors warn that accepting these declarations unchecked brings model-specific decisions about measurement into the analysis.

One example from the paper: for workload in team check-ins, Opus split the topic into several subcodes (sustainability, working hours, workload drivers, protected time and recovery, stress or energy), while GPT-4.1 used three broader ones. The same interviews would support different findings depending on which instrument you used.

How reliable is AI coding within and across models?

Each model is consistent with itself but much less consistent with other models. The table below summarises the paper's Table 4. Reliability is three-run Krippendorff's alpha with MASI distance, which handles units that carry several codes.

ModelMean subcodesCoverageAlpha, own codebookAlpha, shared codebook
Opus 4.8 (reference)22.2.741.861.861
Sonnet 522.0.745.861.843
Haiku 4.517.5.670.804.819
Fable 524.8.700.917.909
GPT-4.115.0.606.837.838
Gemini 2.5 Pro17.3.657.798.821
Grok 4.523.5.681.831.856
DeepSeek v4-pro25.0.663.764.782

When all eight models applied the same Opus-written codebook, agreement between them ranged from .661 to .724 across the six themes. That is well below every model's agreement with itself. In plain terms, a single model can give you a very stable answer that another model would not reproduce.

Two more findings stand out:

  • Models do not code better with their own codebook. The own-minus-shared difference ranged from -.024 to +.018, and five of the seven non-reference models were slightly more consistent on the shared codebook.
  • Vendor explains only part of it. The four Anthropic models agreed with each other at about .71, against about .68 for the four models from other vendors. Both figures sit below within-model reliability, and the authors caution that the design cannot isolate a vendor effect, partly because the shared codebook came from an Anthropic model.

How much does auditing several models cost?

Coding is cheap compared with the cost of drawing conclusions from a single model's categories. At logged token counts and list prices during the study, coding 1,000 units once cost about US$0.54 with GPT-4.1 and up to US$8.96 with Fable 5. The authors note that some figures are lower bounds and that cost says nothing about validity.

How to audit an AI-generated codebook: the eight steps

The QVAF procedure can be reused on any project that relies on an explicit codebook. In short:

  1. Fix the audit corpus. Define themes and exclusions, segment the text once and freeze the unit IDs.
  2. Match elicitation inputs. Give every model the same sample of interviews, prompt and output schema.
  3. Generate codebooks. Ask for labels, definitions, inclusion and exclusion rules, examples and a nominal or ordinal declaration for each code.
  4. Set comparison conditions. Pair each model's own codebook with one shared reference codebook chosen in advance.
  5. Repeat corpus coding. Run each condition several times and log settings, retries and failed outputs.
  6. Assess representation and agreement. Compare codebook size, ordinal share, coverage, and within- and between-model reliability side by side.
  7. Measure dispersion. Within the shared codebook, report how code use spreads (entropy, effective number of codes, largest code share).
  8. Interpret and document. Link discrepancies to your substantive claims and flag them for human review.

The authors stress that no single number in steps 6 and 7 should be read as overall quality. More codes can mean fragmentation, high coverage can come from vague categories, and high entropy can sit alongside low agreement.

What this means for researchers

If an LLM drafts your codebook, the choice of model is a methodological decision, and you should report it like one. Practical takeaways from the paper:

  • Compare at least two clearly different model configurations before you treat coded frequencies as findings.
  • Report the codebook's provenance: the model, its settings, the sample it was built from and the date.
  • Read the definitions and the coded excerpts, not just the reliability score. A label can be applied reliably and still draw a poor conceptual line.
  • Treat high within-model agreement as a sign of consistency, not correctness.

The study also has limits, which the authors spell out. There was no human reference analysis, so it cannot say which codebook is best. Each model built only one codebook per theme. Sampling and reasoning settings differed between providers. The data came from one organisation's AI-led interviews and is mostly in English. The paper is a preprint and has not yet been peer reviewed.

Tools like ThemeLens keep the researcher in charge of the codebook, so you can review and edit generated codes before applying them. If your data comes from an AI Interviewer, as in this study, the same audit logic applies to the analysis stage.

Frequently asked questions

Is it safe to let an LLM create my qualitative codebook?

It can be a useful first draft, but this study shows that a different model would probably give you a different set of categories. Review the definitions yourself, and check the results against a second model or a human coder before you report findings.

Does high intercoder reliability mean AI coding is accurate?

No. Reliability measures consistency, not correctness. In this study, repeated runs of one model agreed at up to .917, while different models on the same codebook agreed at only about .69.

Does this apply to reflexive thematic analysis?

Not directly. The framework sits in the coding-reliability tradition, where explicit codebooks and intercoder agreement are quality criteria. Reflexive thematic analysis does not use agreement as its quality standard.

Which LLM is best for qualitative coding?

The study does not say. It does not rank codebook quality, and the authors say the results describe the recorded model configurations, not a buying recommendation.

Last updated: 2026-11-18. This article is an independent editorial summary of third-party research by Tamme, Hantel and colleagues (2026). It is not affiliated with or endorsed by the authors. Please consult the original paper for full methods and results.


Try Qualitati Free

Start conducting AI-powered qualitative research today. No credit card required.

Get Started Free