Can open-source LLMs running on your own device code qualitative data privately? A 2026 CHI study testing ChatQDA — an open-source, on-device coding tool — found researchers will use local LLMs for basic extraction, but \"conditional trust\" and a privacy-perception gap mean technical security alone does not convince them their interview data is safe.
\n\nThe study evaluated whether a privacy-preserving, on-device large language model can support qualitative open coding without sending sensitive transcripts to commercial cloud services. According to Ngo, Nguyen Van, Nguyen, Do, and Nguyen-Quoc (2026), the team built ChatQDA, a framework powered by the open-source gpt-oss-20b model (a 20.9-billion-parameter mixture-of-experts model chosen for a compact, on-device hardware footprint), and ran a mixed-methods user study comparing manual open coding against ChatQDA-assisted coding on a five-page semi-structured interview transcript.
\nThe sample was small and deliberately mixed: four participants, split between two novice qualitative researchers (fewer than two years of experience) and two experienced researchers (two or more years). This is an early-stage, formative study — its value is in the patterns it surfaces, not in generalizable effect sizes.
\n\nMost AI coding tools send transcripts to external servers. For interview data that may contain health details, identifiable quotes, or IRB-restricted material, that transfer raises real concerns: data sovereignty, re-identification risk, and institutional-review-board compliance. An on-device model keeps the data on the researcher's own machine — in principle removing the transfer entirely. The study's contribution is testing whether researchers actually believe that, and whether the coding output is good enough to rely on.
\n\nParticipants found the tool easy to learn but were cautious about trusting its interpretations. On a 5-point Likert scale reported by the authors, the system scored well on usability and speed but only moderately on trust and depth of interpretation.
\n\n| Dimension (5-point scale) | Mean rating | Read |
|---|---|---|
| Ease of learning & clarity | 4.0 | Strong — quick to pick up |
| Task completion speed | 3.75 (50% rated 5) | Fast for extraction |
| Overall usefulness | 3.00 | Moderate |
| Overall trustworthiness | 3.00 | Conditional |
| Reliability | 3.00 | Doubted for interpretation |
| General security measures (perceived) | 2.75 | Weakest score |
According to the authors, participants showed \"conditional trust\": they valued ChatQDA for basic extraction tasks — pulling out candidate codes and segments — but doubted its interpretive accuracy and reliability for the parts of coding that require human judgment.
\n\nThe most striking finding is what the authors call a privacy-perception gap. Even though the model ran locally — technically keeping data off external servers — perceived security was the lowest-rated dimension (M=2.75 for general security measures, M=3.0 for protection against third-party interception). Participants reported \"epistemic uncertainty regarding data protection\": because the security is invisible, it did not translate into confidence.
\nThe authors' core recommendation follows directly: privacy-preserving analysis tools must prioritize verifiable privacy and methodological rigor, not just technical infrastructure. Telling researchers \"it runs locally\" is not enough; they need to be able to see and confirm where their data goes.
\n\nThree practical takeaways:
\nFor teams weighing where AI fits in their coding pipeline, the lesson is to scope it to mechanical, reviewable steps and route interpretation through a transparent QDA workspace where every AI-suggested code can be checked against the source.
\n\nYes. The ChatQDA study used gpt-oss-20b on-device, with no cloud transfer of transcripts, to perform open coding on interview data.
\n\nTechnically yes — local models keep data off external servers. But the 2026 study found researchers still felt uncertain about data protection, so verifiable privacy matters as much as the architecture itself.
\n\nNot unsupervised. Participants in the study trusted AI for basic extraction but doubted its interpretive reliability. Keep theme interpretation and final coding decisions with a human researcher.
\n\nFour participants — two novice and two experienced qualitative researchers — coding a five-page transcript. It is a formative user study, so treat findings as design signals rather than generalizable benchmarks.
\n\nOn-device, open-source LLMs make private qualitative coding technically feasible today. But the 2026 ChatQDA study shows the harder problem is trust: researchers will let AI extract, not interpret, and they need privacy they can verify, not just privacy they are promised.
\n\nPrimary source: Ngo, T. T., Nguyen Van, D., Nguyen, A.-M., Do, P.-A., & Nguyen-Quoc, A. (2026). Qualitative Coding Analysis through Open-Source Large Language Models: A User Study and Design Recommendations. Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. arxiv.org/abs/2602.18352
\n\nLast updated: June 1, 2026.
\nThis is an independent editorial summary of third-party research, written to help researchers understand the study. It is not affiliated with or endorsed by the original authors.
\n","related":[{"slug":"batching-synthetic-respondents-inflates-correlations-2026","title":"Batching Synthetic Respondents Inflates Correlations (2026)","description":"Generating many synthetic respondents in one LLM request inflates correlations, a 2026 LSE study finds. What it means for synthetic survey data validity.","category":"Methodology","date":"2026-10-13","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"ai-chatbot-qualitative-interviews-lessons-2026","title":"AI Chatbot Interviews at €0.04: Lessons From a 2026 Study","description":"Can AI chatbot interviews replace qualitative interviewers? A 2026 study of 74 chatbot-led interviews found cheap, engaging data collection but limited depth.","category":"Methodology","date":"2026-10-06","readTime":"7 min read","author":"Qualitati Research Team"},{"slug":"llm-inductive-content-analysis-survey-responses-2026","title":"Can GPT-5 Do Inductive Content Analysis of Survey Text?","description":"Can GPT-5 do inductive content analysis? A 2026 study of 903 survey answers found moderate human-AI agreement (ARI 0.61 codes, 0.54 themes), varying by question.","category":"Methodology","date":"2026-10-05","readTime":"7 min read","author":"Qualitati Research Team"}]}