Study finds LLM prompt robustness depends on whether questions are objective or subjective

A study challenges the assumption that prompted responses reliably reflect a model's values by comparing how large language models handle objective versus subjective queries. The researchers evaluated four instruction-tuned model families across three objective datasets (MMLU, ARC, CulturalBench) and three subjective datasets (Political Compass Test, ValueBench, World Values Survey). By applying various prompt changes to each question, they measured whether models maintained consistent answers across variants.

• Significant effects were found for model, dataset, prompt category, and their interactions using binomial generalized estimating equations.
• The interaction between dataset type and prompt category was particularly large, indicating that robustness varies significantly based on the nature of the question.
• Results demonstrate that prompt robustness is not uniform but depends heavily on the specific question type, the kind of prompt change applied, and the model architecture.

These findings suggest that survey-style evaluations treating responses as evidence of political values or social attitudes are fragile, as consistency cannot be assumed across different question types.