Can LLMs Analyze Online Forum Discussions? A 2026 Study
Qualitati Research Team · 2026-09-25 · 7 min read
Short answer: LLMs can help researchers analyze online forum discussions, but mostly as a navigation layer, not a summarizer. In a 2026 study of 21 researchers, participants valued AI that sorted comments by topic, discourse role, and reasoning type, but distrusted AI summaries that dropped fine detail and emotion. They wanted every AI label traceable back to the raw comments.
Reddit threads, community forums, and public comment sections are some of the richest unsolicited qualitative data available. People explain problems, propose fixes, and argue with each other without an interviewer in the room. The difficulty is the structure: hundreds of nested replies, repeated points, and side arguments. A new paper asks what researchers actually want from LLM tools that promise to make sense of that mess.
The study is "Disentangling Threads: Exploring the Potential of LLM-Supported Discussion Forum Analysis for Community Insight" by Tony W. Li, Zhiqing Wang, Thanh-Nha Tran, Yu-Chun Grace Yen, and Steven P. Dow (UC San Diego and National Yang-Ming Chiao-Tung University), published at the ACM Collective Intelligence Conference (CI '26).
Key takeaways
- Researchers want LLMs to structure forum data (topics, who is proposing vs. disagreeing, how people argue), not to replace reading it.
- According to Li et al. (2026), 19 of 21 participants used topic views to discover specific subtopics, and 16 of 21 used them to find common pain points.
- The main barriers were doubts about who the commenters are, distrust of AI summaries, and fixed categories that did not match each researcher's question.
- A preliminary check found that LLM topic generation covered 89% of the authors' manually coded topics.
What did the study test?
The study tested how researchers would use an LLM-style dashboard to explore one public forum thread. The authors first hand-coded a Reddit thread on "How can the US improve its K-12 education system?" with roughly 320 comments posted over several years. From that analysis and prior literature they built a three-part coding scheme, the Topic-Discourse-Reasoning (TDR) framework:
- Topic: what a comment is about (for example, teacher compensation or school district integration).
- Discourse role: what the comment does in the conversation: states a problem, proposes a design, agrees, or disagrees.
- Reasoning type: how it argues, using the classical rhetorical triad of logos (logic), ethos (expertise or experience), and pathos (emotion).
They then built a static prototype with a topic selection checklist, an aggregate topic overview, a subtopic view that breaks down reply trees, and a comment view color-coded by TDR labels with an optional AI-summary toggle. Finally, they ran one-hour semi-structured interviews with 21 researchers who had qualitative analysis experience. Each session combined free exploration of the raw Reddit thread, think-aloud tasks with the prototype, and a closing interview.
One design choice matters for interpreting the results. The prototype was populated with the authors' human labels, and participants were asked to imagine the output came from an LLM. The study therefore measures what researchers want from LLM forum analysis, not how accurately a specific model performs it.
Can LLMs accurately label forum discussions?
Early signs are positive, but this paper is not an accuracy benchmark. In a preliminary feasibility check, the authors report that bottom-up LLM topic generation covered 89% of their manual topics, and that LLM classification of TDR labels outperformed BERT, RoBERTa, and logistic-regression baselines. These checks were run on a single thread, so treat them as a proof of concept rather than a validated error rate. If you use an LLM to code forum data, you still need your own agreement check against a human-coded sample.
How did researchers want to use AI forum analysis?
Participants used the structure mainly to find things faster and to form hypotheses they could test elsewhere. The table below summarizes the use cases the authors counted across the 21 participants (Li et al., 2026, Table 1).
| Framework dimension | Use case | Participants (of 21) |
|---|
| Topic | Discover specific subtopics | 19 |
| Topic | Identify common pain points | 16 |
| Topic | Gauge breadth of the discussion | 9 |
| Discourse role | Target follow-up research | 15 |
| Discourse role | Find popular problems | 14 |
| Discourse role | Locate disagreement | 12 |
| Reasoning type | Filter by combinations of reasoning | 15 |
| Reasoning type | Find ethos (expertise/experience) comments | 14 |
| Reasoning type | Find logos (logical) comments | 14 |
| Reasoning type | Find pathos (emotional) comments | 3 |
Two patterns stand out. First, discourse role was treated as a research instrument: problem statements and design proposals told participants where a community's needs and ideas were, and where a follow-up study should dig. Second, only 3 participants filtered for emotional comments, even though emotion was exactly what participants said AI summaries tended to lose.
Why don't researchers trust AI summaries of forum threads?
Researchers distrusted summaries because summaries strip out the detail and voice that make forum data worth studying. Participants said AI summaries missed fine details and the human voice, worried about hallucinated content, and preferred reading full comments, especially where emotion was involved. They wanted any AI output to be checkable against the original text.
The authors identify three barriers:
- User representativeness. Forum commenters are anonymous and self-selected. Participants wondered whether commenters were legitimate or trolling and wanted demographic or professional context, while recognizing that exposing identity would undermine the open expression that makes forums useful.
- LLM semantic limitations. Summaries flattened nuance and emotion, and the risk of hallucination meant participants wanted to validate claims against raw text.
- Analytical inflexibility. Preset categories did not always fit the question. One participant wanted a topic such as early childhood education that the fixed scheme did not offer; others wanted to filter ethos comments by specific professions.
What this means for researchers using AI on community data
Use LLMs to organize forum data and point you to the right comments, then read those comments yourself. The paper's design recommendations translate into a practical workflow:
- Structure first, summarize last. Ask the model to tag comments by topic, discourse role, and reasoning type, and to surface representative quotes. Treat summaries as an index, not a finding.
- Keep every label traceable. Each code should link to the exact comment it came from, so you can audit it. This is the same principle behind a sound AI thematic analysis workflow: codes and themes should always be grounded in verbatim excerpts.
- Let your question define the categories. Add, split, or rename topics to match your research question instead of accepting a fixed taxonomy.
- Report the representativeness limit. Forum voices are not a sample of a population. Say who is likely missing.
- Triangulate with primary research. Participants saw forums as a starting point that shows where to look next. A practical follow-up is to turn the problem statements and proposals you found into interview prompts and test them with recruited participants, for example through an AI Interviewer study.
Limitations of the study
The findings are exploratory. The study used one Reddit thread and 21 participants recruited from a single institution, most of them aged 18 to 24. The prototype was static and filled with human labels, so it does not show how live LLM output, with its inconsistencies, would change researcher trust. The authors also note that the TDR framework was designed to elicit insights, not validated as a standalone analytic method.
FAQ
Can I use ChatGPT or another LLM to analyze Reddit threads for research?
Yes, as long as you use it to structure and search the data rather than to write your findings. Check a sample of its labels against your own coding, keep links back to the original comments, and follow the platform's terms and your ethics board's guidance on public online data.
What is the Topic-Discourse-Reasoning framework?
It is the three-dimension scheme Li et al. (2026) used to organize forum comments: the topic a comment covers, the role it plays in the discussion (problem, proposal, agreement, disagreement), and whether it argues through logic, expertise, or emotion.
Are forum comments representative of a community?
No. Commenters are self-selected and often anonymous, and loud or frequent posters can dominate a thread. Treat forum analysis as exploratory evidence and confirm important patterns with sampled participants.
Did the study measure LLM accuracy?
Only in a preliminary check: LLM topic generation covered 89% of manually coded topics, and TDR classification beat BERT, RoBERTa, and logistic-regression baselines. The main study focused on how researchers would use such output, not on model accuracy.
Last updated: September 25, 2026
This article is an independent editorial summary of third-party research by the Qualitati Research Team. It is not affiliated with or endorsed by the paper's authors. Read the original paper for full methods and results.