Short answer: An AI council for thematic analysis uses three language models from different families that code data independently, with deterministic checks on every quote and a researcher who keeps final judgment. In a November 2026 test, the TACT procedure matched a published human coding at F1 0.71–0.72 and beat a single-model baseline (F1 0.65).
Last updated: November 5, 2026
Most teams that hand interview or survey responses to a chatbot get back a tidy list of themes and no way to check it. A new preprint from Oregon State University and Microsoft Research argues there is a third option between "leave the data unanalyzed" and "trust one model's reading." The authors, Rudrajit Choudhuri, Christian Bird, Carmen Badea and Anita Sarma, call it TACT (Traceable AI Council for Thematic analysis), posted to arXiv on November 3, 2026. This summary covers what the procedure does, how well it held up against an independent human coding, what it cost, and what researchers can take from it even if they never run it.
What is an AI council for qualitative analysis?
An AI council is a group of models from different vendors that each do the same analytic task without seeing one another's output, so that no single model's blind spots dominate the result. In TACT the council has three members, one each from Anthropic's Claude, Google's Gemini and OpenAI's GPT families. In the validation run these were Claude Opus 5, Gemini 3.6 Flash and GPT-5.6.
The authors frame this as bounded delegation. Models handle the explicit, checkable work: proposing codes and applying a codebook. Every one of those steps has to cite evidence. Researchers keep every interpretive decision, and the procedure records which model or person made each call.
How does the TACT procedure work?
TACT runs a data-preparation step and then five stages. Two of them belong to researchers, and two are plain scripts rather than models.
- Independent code discovery. Three AI analysts read the whole corpus and propose codes at three layers: semantic (what people say), latent (the assumptions behind it) and contrastive (where an answer departs from the respondent's overall stance). A script then checks that every quoted piece of evidence appears verbatim in the data and that no unit was skipped.
- Reconciliation. One reconciler model merges the three code lists into a candidate codebook. It works from an overlap table of shared units, and codes supported by fewer than three respondents are dropped (researchers can restore them).
- Human review. Before seeing any AI output, researchers open-code a blind sample of 10% of the units (at least 50, at most 150). Only then do they review, edit and approve the codebook. The blind pass is there to stop the AI's suggestions from anchoring them.
- Systematic coding. Three AI coders, again one per model family, apply the approved codebook to every unit. Each writes a rationale before assigning a code, and none can invent new codes.
- Consensus and reliability. A script keeps a code when at least two of the three coders assign it and computes Krippendorff's alpha per code (default threshold 0.7). Researchers then compare the result with their own coding of a separate held-out sample.
A heavier variant, TACT-Full, adds adversaries: a model from another family critiques each analyst, the reconciler and each coder, and the critiqued model decides whether to accept the critique. TACT-Full also adds a calibrated decision model (TypeSafe's Jev) that flags low-probability code assignments for human review.
How accurate was it against human coders?
TACT came close to an independent human team's coding and did better than one model working alone. The authors reran a published 2023 study by Zhang et al. that had manually coded Stack Overflow threads about GitHub Copilot. Using the 256 threads still online, they scored agreement against Zhang et al.'s labels on 192 evaluation threads. No model saw those labels.
| Measure | TACT-Lite | TACT-Full | Single model (GPT-5.6 alone) |
|---|---|---|---|
| Candidate codebook F1 vs. human labels | 0.75 | 0.77 | — |
| Final coding F1 vs. human labels | 0.71 | 0.72 | 0.65 |
| Median Krippendorff's alpha among AI coders | 0.77 | 0.87 | — |
| Cohen's kappa vs. authors' own held-out coding (50 threads) | 0.79 | 0.85 | — |
| Model-call cost per 100 threads | $13.77 | $33.59 | — |
According to Choudhuri et al. (2026), both variants beat the single-model baseline's F1 in 97% (Lite) and 99% (Full) of 1,000 paired bootstrap resamples. Precision ran below recall partly because the council's codes were finer than the original categories. For example, it split one "difficulty of integration" category into separate problems such as corporate proxies blocking access.
The process itself produced numbers worth noting. The provenance script traced all 1,405 quotes the analysts cited. In the first coding round, 34 of the 80 codes fell below the alpha threshold. The researchers consolidated them into a final codebook of 72 codes and recoded. That loop is the procedure catching weak codes, not a failure.
What does AI-assisted thematic analysis cost?
The whole validation run cost $35.26 in model calls under TACT-Lite and $86.00 under TACT-Full for 256 threads, at September 2026 list prices. The authors recommend starting with TACT-Lite. Its final coding agreed with the reference nearly as well at less than half the cost. If a team adds anything, it should add the discovery adversaries first, because a code missed early is harder to recover than a coding error that the other two coders would expose.
They also say plainly when not to bother. If two researchers can code the corpus within a week, TACT may add little. Teams that cannot send data to three commercial providers could use three open-weight model families run locally, checked against a held-out sample in the same way.
What can go wrong with AI qualitative coding?
The paper's pitfalls table may be its most reusable part. Three risks stand out because high model agreement can hide all of them:
- Evidence that only looks right. A quote quietly paraphrased, or a code citing units that don't support it, passes every agreement check. The fix is a script that matches each quote to the source, followed by a human judgment of whether the quote supports the code.
- Anchoring. Researchers who read AI codes before the data tend to see the data through those codes. Code a blind sample first.
- Shared model error. Coders that see each other's work, or come from one model family, agree more than independent coders would. As the authors put it, consensus cannot expose an error every coder shares, so human-to-council agreement must be measured separately.
What this means for researchers
You don't need TACT itself to borrow its safeguards. In any AI-assisted analysis, use more than one model family, verify quotes against the transcripts, code a slice yourself before looking at AI themes, and report human-to-AI agreement on held-out data, not just agreement between models. In ThemeLens, Qualitati's AI thematic analysis tool, researchers can review codes against their source quotes. The study suggests checks like that should be routine, not optional.
The limits are clear. The validation used one English-language forum dataset that has been public since 2023, so the models may have seen it in training. The authors note that the single-model baseline had the same exposure and still scored lower. The two people who coded the held-out sample also designed TACT. The authors release their prompts and scripts and present TACT as a starting point, not a universal result.
FAQ
Is an AI council better than one LLM for thematic analysis?
In this study, yes, modestly. Final-coding F1 was 0.71–0.72 for the council versus 0.65 for GPT-5.6 alone on the same task, and the council's advantage held in most bootstrap resamples.
Does TACT replace human qualitative researchers?
No. Researchers code a blind sample, approve the codebook, decide on low-agreement codes and check a held-out sample. The models handle volume; people keep interpretation.
What reliability threshold does TACT use?
A default per-code Krippendorff's alpha of 0.7 for exploratory work, which the authors suggest raising to 0.8 for conservative conclusions.
Can I use TACT with grounded theory or deductive coding?
Partly. The authors suggest using discovery and reconciliation for open coding in grounded theory, or only the coding stages when a codebook already exists.
Source: Choudhuri, R., Bird, C., Badea, C., & Sarma, A. (2026). TACT: A Pragmatic Approach to Qualitative Analysis with an AI Council. arXiv:2611.05702. https://arxiv.org/abs/2611.05702
Last updated: November 5, 2026. This is an independent editorial summary of third-party research by the Qualitati Research Team; it is not affiliated with or endorsed by the paper's authors.