Are LLM Analyses Reproducible? What Qualitative Researchers Need to Know

Qualitati Research Team · 2026-07-12 · 11 min read

Short answer: Not by default. Large language models can return different outputs for identical inputs — even at temperature 0 — and hosted models change under your feet through silent version updates and deprecations. For qualitative researchers this threatens dependability and confirmability. Reproducibility is achievable in a documented, statistical form: log your full configuration, run the analysis at least twice, report test-retest agreement (Cohen's kappa between runs), and validate against a human-coded sample.

Are LLM analyses reproducible?

Reproducibility in qualitative analysis has always been a nuanced claim — two human coders never agree perfectly either. But when an LLM does the coding, two distinct instabilities enter the pipeline that most methods sections currently ignore. The first is within-version non-determinism: the same model, same prompt, and same settings can produce different codes on Tuesday than on Monday. The second is across-version drift: the model you cite in your paper may behave differently — or no longer exist — by the time a reviewer or replicator tries to re-run your analysis.

Last updated: July 12, 2026. This guide covers why both instabilities happen, what the empirical evidence says, why it matters for qualitative rigor, and a concrete logging-and-agreement protocol you can adopt today.

Key takeaways

Temperature 0 does not mean deterministic. A large empirical study of ChatGPT code generation found that setting temperature to 0 reduces but does not eliminate output variation across identical requests (Ouyang et al., ACM TOSEM, 2025).
The variation is structural, not a bug you can toggle off. Batch-dependent GPU reduction kernels and Mixture-of-Experts routing mean identical prompts can take different numerical paths through the model (Thinking Machines Lab, 2025).
Models drift between versions. GPT-4's accuracy on a prime-number identification task fell from 84% to 51% between its March and June 2023 releases, with instruction-following degrading in parallel (Chen, Zaharia & Zou, 2023).
Annotation reliability can fall below scientific thresholds. Repeating identical classification inputs to ChatGPT produced output consistency that fell short of standard reliability criteria; pooling multiple runs improved it (Reiss, 2023).
The fix is procedural: pin a model snapshot, log everything, run the analysis twice, report between-run kappa, and keep a human validation loop.

Why do LLMs give different answers even at temperature 0?

Sampling temperature is the obvious source of randomness: at typical settings the model draws tokens probabilistically, so two runs diverge by design. Most researchers know to set temperature to 0 (greedy decoding) for analytical tasks. Fewer know that this does not produce identical outputs either.

Ouyang and colleagues systematically re-ran identical code-generation requests and found substantial instability even under “deterministic” settings; under default settings, the majority of tasks produced zero test-equivalent outputs across repeated requests (ACM TOSEM, 2025). A separate study of supposedly deterministic configurations across multiple models reached the same conclusion: greedy decoding narrows but does not close the gap (Atil et al., 2024).

The mechanism is now well understood. In one demonstration, sampling 1,000 completions of the same prompt at temperature 0 produced 80 unique outputs. The dominant cause is not floating-point noise alone but batch-size dependence: the numerical result of a GPU reduction depends on how many other requests are batched with yours on the server, which you cannot see or control as an API user (Thinking Machines Lab, 2025). Mixture-of-Experts architectures add another path: expert routing can shift with server load. When two candidate tokens are nearly tied, a one-bit difference in an intermediate activation flips the output — and every token after it.

The practical upshot for a researcher: you cannot promise bit-identical re-runs of a hosted LLM analysis. What you can promise is a documented configuration and a measured level of run-to-run agreement.

What is model version drift, and why do deprecations matter?

Hosted models are moving targets. Providers retrain, fine-tune, and silently update the models behind stable-sounding names. The canonical demonstration is Chen, Zaharia and Zou's longitudinal evaluation: between March and June 2023, GPT-4's accuracy at identifying prime numbers dropped from 84% to 51%, code formatting errors increased, and willingness to follow user instructions declined — while the same period improved other tasks (How Is ChatGPT's Behavior Changing over Time?, 2023). The trends for GPT-3.5 and GPT-4 often pointed in opposite directions, which means you cannot even assume drift is monotonic improvement.

Deprecation is the harder deadline. Providers retire model snapshots on schedules of months to a few years. Once the snapshot you used is gone, exact replication of your analysis becomes impossible — no amount of logging brings the instrument back. Two mitigations exist: pin dated snapshots rather than floating aliases (for example, a versioned model ID rather than a “latest” alias), and archive the model's raw outputs alongside your data so the analysis itself remains auditable after the model is retired.

Why does non-reproducibility threaten qualitative rigor?

In Lincoln and Guba's trustworthiness framework, dependability asks whether findings are consistent and could be repeated, and confirmability asks whether findings are shaped by the data rather than by analyst (or instrument) idiosyncrasy. Both are conventionally demonstrated through an audit trail: a reader should be able to trace every theme back through coding decisions to raw data.

An unstable AI coder undermines both at once. If the “coder” produces different codes on re-run, the coding decisions in your audit trail describe one realization of a random process, not a stable procedure. And the evidence says the instability is large enough to matter for research: Reiss found that repeating identical annotation inputs to ChatGPT produced consistency below accepted reliability thresholds, and that even minor prompt rewording shifted outputs (Reiss, 2023). Task type matters too: in a medical-education coding study, GPT-4 applying a fixed human codebook reached a mean Cohen's kappa of 0.71 against humans, but in inductive mode only 31% of AI-generated codes matched human ones (Journal of Learning Analytics, 2024). Deductive coding with a tight codebook is far more stable ground than open inductive generation.

What should you log to make an LLM analysis reproducible?

Treat the model as an instrument and log its calibration. The following table is the minimum a methods section (or supplementary file) should contain.

What to log	Why it matters
Provider and exact model ID (pinned, dated snapshot)	Floating aliases silently change; a dated snapshot is the closest thing to a fixed instrument
Access date(s) of every analysis run	Anchors the analysis against version drift; providers update models without notice
Sampling parameters (temperature, top_p, max tokens, seed if offered)	These change output distributions dramatically; “defaults” differ across providers and over time
Full prompt text: system prompt, task instructions, codebook, few-shot examples	Minor wording changes measurably shift annotations (Reiss, 2023)
Number of runs and the aggregation rule (single run, majority vote, pooling)	Pooled or majority-voted outputs are more reliable than single runs and must be described to be replicated
Post-processing rules (code merging, normalization, exclusions)	Silent cleanup steps are a classic source of irreproducibility
Raw model outputs, archived	The only guaranteed replication artifact once a snapshot is deprecated
Between-run agreement statistic	Quantifies the instability instead of hiding it (see next section)

How do you measure whether your LLM coding is stable?

Borrow test-retest logic from psychometrics: run the identical analysis twice (or more), treat the runs as two coders, and compute an agreement coefficient — Cohen's kappa for two runs on categorical codes, Krippendorff's alpha for more runs or more complex data. This turns “the model might vary” into a number a reviewer can evaluate. By the common Landis-and-Koch reading, between-run kappa above 0.80 is strong; 0.61–0.80 is substantial but worth tightening; below that, your codebook or prompt is underspecified and the single-run analysis should not be trusted.

Three practical refinements:

If between-run agreement is low, fix the prompt before scaling. Ambiguous code definitions produce both human and machine unreliability; the model is amplifying a codebook problem. Our guide on how to code qualitative data covers writing code definitions that survive this test.
Pool runs for production coding. Majority-voting across three or five runs measurably improves reliability over any single run (Reiss, 2023).
Report machine-human agreement separately from machine-machine agreement. A model can agree perfectly with itself and still be wrong. See our deep dive on LLM–human inter-rater reliability for benchmarks and study design.

Where does human-in-the-loop validation fit?

Test-retest agreement guards against instability; it does not guard against a model that is stably wrong. The complementary control is human review of a stratified sample: draw a random sample of AI-coded segments (stratified by code, so rare codes are checked too), have a researcher code them blind, compute human–AI kappa, and adjudicate disagreements. Keep the disagreement log — it is the confirmability evidence reviewers actually want. Run a deliberate disconfirmation pass as well: search for segments that contradict the emerging themes before accepting the AI's synthesis. Our checklist for auditing AI qualitative coding walks through this end to end.

The LLM reproducibility checklist

Pin a dated model snapshot; never analyze on a floating “latest” alias.
Set temperature to 0 (or the minimum) — and still do not assume determinism.
Freeze prompts and codebook under version control before the main analysis.
Run the full analysis at least twice; report between-run kappa or alpha.
Use majority voting or pooling across runs for the final code assignments.
Archive raw model outputs with timestamps and run metadata.
Validate a stratified human-coded sample; report human–AI agreement.
Record every post-processing step applied after model output.
State the model's access dates and acknowledge deprecation risk in limitations.
Prefer deductive codebook application over unconstrained inductive generation when reproducibility is the priority.

Where Qualitati fits

Qualitati's analysis tools are built around the assumption that AI output must be auditable rather than trusted. ThemeLens anchors every synthesized theme to verbatim participant quotes that are verified against the source transcripts, so a human can always trace a claim back to data — the confirmability requirement, operationalized. The QDA Workspace keeps AI-assisted coding under human override, letting you re-run, compare, and correct code assignments instead of accepting a single stochastic pass. And because runs are cheap, the run-twice-and-compare protocol described above is practical rather than aspirational. Pricing is transparent and pay-as-you-go, with a free tier to test the workflow.

Limitations and trade-offs

Perfect replication of hosted models is not attainable. Logging and agreement statistics bound the uncertainty; they do not eliminate it. Only locally hosted open-weight models with fixed inference stacks approach bit-level reproducibility, at a large capability and infrastructure cost.
Test-retest agreement measures stability, not validity. High between-run kappa with low human–AI kappa means a consistently miscalibrated instrument.
The evidence base is young and shifting. Most published reliability studies tested 2023–2025 models; newer models may be more stable on some tasks and drift-prone on others. Re-verify on your own data.

Frequently asked questions

Does setting temperature to 0 make an LLM deterministic?

No. It removes sampling randomness but not the batch-dependent numerical variation in server-side inference or Mixture-of-Experts routing effects. Empirical studies find repeated identical requests still diverge at temperature 0 (Ouyang et al., 2025).

Is naming the model in my methods section enough?

No. “We used GPT-4” describes a family of changing systems, not an instrument. Report the dated snapshot ID, access dates, sampling parameters, full prompts, number of runs, and the aggregation rule — and archive raw outputs.

What agreement statistic should I report for repeated LLM runs?

Cohen's kappa for two runs over categorical codes; Krippendorff's alpha for three or more runs, missing data, or non-nominal codes. Report it alongside human–AI agreement on a validation sample.

Will journals accept LLM-assisted qualitative coding?

Increasingly yes, when the reliability case is made explicitly: logged configuration, between-run agreement, human validation sample, and an audit trail from themes to quotes. Undocumented single-run AI coding is what reviewers reject.

Conclusion

LLM analyses are not reproducible in the bit-identical sense, and pretending otherwise is a rigor problem waiting for a reviewer to find it. They are reproducible in the sense that matters for qualitative research: as a documented procedure with measured stability. Pin the snapshot, log the configuration, run it twice, report the kappa, and keep a human in the loop. If you want a workflow where the audit trail comes built in, try the Qualitati QDA Workspace — start free, no credit card required.