Can On-Device LLMs Do Private Qualitative Coding?
Qualitati Research Team · 2026-07-22 · 8 min read
Can on-device large language models do private qualitative coding? Partially. A 2026 CHI study of ChatQDA, an on-device tool built on an open-source 20B-parameter model, found researchers trust local LLMs for surface-level tasks like excerpt extraction but not for interpretive coding — one participant estimated extraction accuracy at just "50–60%." The paper calls this pattern "conditional trust."
What the study tested
The study introduced ChatQDA, a chat-based, on-device framework that runs an open-source LLM entirely on the researcher's own machine — no data leaves the device. The goal was privacy-preserving qualitative data analysis (QDA): upload interview transcripts or documents, then query and code them without sending sensitive text to a cloud API.
According to Ngo, Nguyen Van, Nguyen, Do, and Nguyen-Quoc (2026), presented at the CHI Conference on Human Factors in Computing Systems, the tool ran on gpt-oss-20b — a 20.9-billion-parameter open-weights model with roughly 3.6B parameters active per pass and a ~12.8 GiB checkpoint, small enough to run locally. The researchers conducted a mixed-methods user study, combining Likert-scale ratings with qualitative interviews and deductive thematic analysis of participants' feedback.
How accurate is on-device LLM qualitative coding?
Accuracy was good enough for extraction but not for interpretation. Participants judged ChatQDA "acceptable" for pulling relevant excerpts from transcripts, but "less dependable" for labeling and first-order coding, citing hallucination and inconsistency. One participant estimated excerpt-extraction accuracy at "about 50–60%" and stressed that "the researcher still needs to review the data."
This mirrors a recurring finding across the LLM-QDA literature: models are useful for widening the funnel — surfacing candidate quotes and organizing large corpora — but the interpretive judgment that defines qualitative rigor still belongs to the human analyst.
Do researchers trust local LLMs with sensitive data?
Less than you might expect, even when the data never leaves their machine. The study's central concept is conditional trust: participants valued the tool for efficient, surface-level work while remaining skeptical of its interpretive nuance and consistency. Strikingly, despite the technical security of local deployment, participants still reported epistemic uncertainty about data protection — they weren't fully convinced their data was private, even when it provably was.
In other words, running a model on-device solves the technical privacy problem but not the perceived privacy problem. Trust in AI-assisted research is social and psychological, not just architectural.
What the ratings showed
With only four participants (two novices with under two years of QDA experience, two experienced researchers with prior AI-QDA exposure), the numbers are directional, not definitive. Still, the pattern is consistent — high marks for usability, lukewarm marks for trust and security.
| Dimension (5-point Likert) | Result |
|---|
| Perceived ease of use (learnability) | 4 / 5 (all participants) |
| Perceived usefulness (faster task completion) | M = 3.75; 50% rated 5 |
| Trustworthiness | M = 3.00 (neutral) |
| Perceived security | Median = 3.0 |
| Excerpt-extraction accuracy (participant estimate) | ~50–60% |
Source: Ngo et al. (2026), ChatQDA user study, n = 4.
Why on-device LLMs matter for qualitative research
Qualitative data is often the most sensitive data a researcher handles: health interviews, employee grievances, trauma narratives, minors' voices. Ethics boards and regulations like GDPR increasingly discourage uploading raw transcripts to third-party AI services. On-device, open-source models offer a genuine alternative — the analytical convenience of an LLM without the data-transfer risk.
The catch, as this study shows, is capability. A locally-runnable 20B model is far smaller than a frontier cloud model, and that gap shows up as weaker interpretive coding. The practical takeaway: use local LLMs for the retrieval-heavy, low-stakes parts of your workflow, and reserve human effort (or carefully governed cloud tools) for the interpretive core.
What this means for researchers
- Treat AI output as a first draft, not a finding. Even the study's participants — who liked the tool — insisted on reviewing everything. Build a human verification step into every AI-assisted coding pass.
- Match the task to the model. Extraction and organization: fine for a local LLM. First-order coding and theme interpretation: keep a human in charge. Platforms like QualiTaTi's ThemeLens are designed around this human-in-the-loop principle, pairing AI thematic analysis with reviewable, auditable outputs rather than one-click "answers."
- Don't assume privacy equals trust. If your team or IRB needs confidence in a workflow, document exactly where data goes and why — technical guarantees alone won't settle the "conditional trust" question.
For teams collecting the data in the first place, the same logic applies upstream: an AI Interviewer can scale data collection, but the resulting transcripts still deserve careful, human-led interpretation.
Limitations to keep in mind
This is a small, exploratory study — four participants and a single tool. It measures perceptions and self-reported accuracy, not a rigorous benchmark against a human gold standard. The findings are best read as a set of design signals and open questions for privacy-preserving QDA, not as a verdict on whether local LLMs "work." As always, replication with larger, more diverse samples is needed.
Frequently asked questions
What is on-device (local) qualitative coding?
It means running the LLM that assists with coding directly on your own computer, so interview transcripts and other sensitive text are never uploaded to an external server. It trades some model capability for stronger data privacy.
Are local open-source LLMs accurate enough for qualitative analysis?
For excerpt extraction and organizing data, they can be helpful. For interpretive coding, the 2026 ChatQDA study found them inconsistent — participants estimated extraction accuracy around 50–60% and insisted on human review.
What is "conditional trust" in AI-assisted research?
It describes researchers trusting an AI tool for narrow, low-stakes tasks (like finding quotes) while withholding trust for higher-stakes interpretive work — and remaining uncertain about data protection even when the tool is technically secure.
Does running an LLM locally guarantee my data is private?
Technically, local processing means data stays on your device. But the study found researchers still felt uncertain about protection, so privacy also depends on clear documentation and governance, not just architecture.
Last updated: July 22, 2026. This article is an independent editorial summary of third-party research (Ngo et al., 2026, CHI Conference on Human Factors in Computing Systems) and is not affiliated with or endorsed by the study's authors.