A controlled study of 162 participants demonstrates that user evaluations of large language models are driven primarily by pre-interaction framing rather than the model's true capability. Participants were told their model was either cutting-edge, older, or weaker than it actually was, which significantly shifted their opinions and interaction behaviors despite identical underlying performance.
- Oversold users rated the model more favorably and used more directive prompting.
- Undersold users wrote longer, more collaborative prompts.
- The quality of co-created output depended only on the model's true capability, not on what users were told.
- Changes in user impressions were predicted by whether the model met expectations and user confidence, not task performance.
The findings suggest that user-elicited LLM evaluations, including preference data used for public leaderboards, measure expectation management at least as much as the model itself.