Can a Local 7B Model Do Auditable Thematic Analysis?
Qualitati Research Team · 2026-09-03 · 6 min read
An auditable thematic analysis workflow keeps every AI-generated code traceable back to the transcript segment that produced it, plus a written justification. A 2026 study by Jeldtoft and Yousef shows a locally-run 7B model can do this on 99 Danish interviews — matching human code coverage, but collapsing 19 human themes into 10 broader ones.
What did the study test?
The researchers built a two-phase workflow and compared it against human coders on real interview data. According to Jeldtoft and Yousef (2026), the corpus was 99 semi-structured Danish interview transcripts (94 for development, 5 held out for evaluation), yielding 291 coded segments. Three independent human annotators coded the same material.
Crucially, the model was Mistral 7B v0.3 Instruct — small enough to run on local hardware. That is a deliberate methodological choice, not a cost-saving one: interview data often cannot leave the institution, so a workflow that depends on a frontier API is unusable for a large slice of qualitative research.
The two phases
- Text to codes. The LLM reads a segment in context and produces codes plus an explicit justification for each. Deterministic code — not the model — handles bookkeeping, deduplication, and the link back to the source segment.
- Codes to themes. The accumulated codes are clustered into themes with written theme descriptions, again with traceable links down to the original quotes.
The separation matters. The model is confined to interpretation; everything procedural is handled by ordinary software. That is what makes the output auditable rather than a black box that emits a theme list.
How did the AI codes compare to human codes?
Comparable in coverage, noticeably less varied in vocabulary. Across all 291 segments the workflow produced 2.33 codes per segment against 1.85 for the human annotator who coded the full set. But on the five closely-inspected segments the picture inverts: the LLM averaged 1.2 codes per segment against 1.6–2.0 for the three humans, and generated only 4 unique codes where the humans produced 8–9 each.
Read together, those numbers describe a familiar failure mode: the model applies a small set of labels liberally, while humans coin new language when a segment demands it.
What did the expert raters say?
Independent experts scored code coverage and justification quality on 5-point scales. The LLM did not merely hold its own — on justifications it beat every human annotator.
| Dimension (5-point scale) | LLM workflow | Human annotators (mean) |
|---|
| Code coverage | 3.40 | 3.13 |
| Justification quality | 3.80 | 2.73 |
The justification gap is the most practically interesting result in the paper. Human coders rarely write out why a code applies; the workflow forces it every time. Whatever the model lacks in interpretive range, it is relentlessly consistent about showing its reasoning — and that reasoning was rated higher than the humans'.
Where the AI and humans diverged most
At the theme level. On the same 291 segments the workflow produced 10 themes (mean 46.7 codes per theme, median 29, largest theme 148 codes), while the human analysis produced 19 themes (mean 20.79 codes per theme, median 18, largest 54 codes).
The AI thematic structure is roughly half as granular, with one theme absorbing 148 codes. In practice that means the sharp, small themes — the ones carrying an unexpected finding — get folded into a large, safe one. Anyone who has watched a junior analyst produce four themes called "Communication," "Trust," "Barriers," and "Support" will recognise the shape of the problem.
What this means for researchers
Three takeaways worth acting on:
- Treat AI coding as a first pass, not a final one. Coverage is comparable; granularity is not. Budget human time for the theme stage specifically, where the compression is worst.
- Demand justifications, not just codes. The single largest measured advantage came from making the model explain itself. If your tool returns bare labels, you are leaving the best-validated benefit on the table. This is the logic behind traceable, quote-linked output in ThemeLens.
- A small local model is viable. These results came from a 7B model, not a frontier system. For sensitive data under GDPR or an IRB restriction, that is the finding that unblocks projects.
The caveats are real: a single language, a single domain, five segments in the close expert comparison, and a proof-of-concept implementation. The authors are explicit that domain transfer mainly requires prompt adjustment — which is another way of saying the prompting strategy is doing heavy, under-specified work.
FAQ
What makes an LLM thematic analysis "auditable"?
Every code and theme keeps an explicit link back to the source segment, alongside a written justification, and all procedural steps are handled deterministically rather than by the model. A reader can follow any theme down to the quotes that produced it.
Can a local 7B model really replace human coders?
No. It matched human code coverage and beat humans on justification quality in this study, but produced roughly half as many themes and a much smaller code vocabulary. It is a scaling aid for a human analyst, not a substitute.
Does this work outside Danish interviews?
Untested. The evaluation used Danish semi-structured interviews from one domain. The authors argue transfer mainly requires prompting changes, but that claim is not yet empirically demonstrated across domains or languages.
Why does the AI produce fewer themes?
The study does not isolate a cause, but the pattern — a small reused code vocabulary feeding into broad clusters — suggests the compression begins at the coding stage, not only at clustering.
Primary source: Jeldtoft, N. J., & Yousef, T. (2026). Designing an Auditable LLM-Supported Workflow for Qualitative Thematic Analysis. arXiv:2608.30543. https://arxiv.org/abs/2608.30543
Last updated: 3 September 2026. This is an independent editorial summary of third-party research; QualiTaTi is not affiliated with the authors.