A study decomposes the non-determinism in large language model (LLM) brand recommendations into four separable sources: within-prompt resampling, prompt paraphrase, model identity, and query language. Using a crossed random-effects analysis on 12,933 responses across three models and eight languages, the authors identify query language as the largest source of variance at 26.5%, while brand identity accounts for only 1.5%.
- Query language explains 26.5% of response variance, whereas brand identity contributes just 1.5% (ICC 0.0146).
- Pure resampling accounts for 34.8% of variance, and the brand-in-context interaction explains 29.6%.
- Brand-by-language interactions account for 8.6% of variance, indicating a bilingual penalty.
- Brand-by-model and brand-by-prompt interactions are near zero.
- Adding languages and models reduces relative-error variance more effectively than adding repeats; the fifth repeat reduces error by only 0.0003.
The authors conclude that brand-ranking reliability remains low (near 0.01 for a single answer) and must be improved by spreading evaluations across languages and models rather than repeating prompts.