Researchers introduce Pluralis v0.1, a novel multimodal, multi-regional, and multilingual dataset designed to evaluate AI risk and reliability through a culture-first perspective. Spanning 6,448 prompts across six Asia-Pacific countries and eight languages, the benchmark natively sources localized safety hazards rather than adapting Western datasets.
- The framework introduces a multimodal evaluation paradigm where text and image inputs synergistically trigger specific legal or cultural violations.
- Pluralis disentangles universal safety violations from localized cultural appropriateness, establishing the latter as a first-class evaluation axis.
- The authors present Judge-Pluralis, an agreement-gated LLM-as-a-Judge ensemble trained on examples classified in an empirically derived cultural taxonomy.
- Observations reveal recurring, locale-specific failure modes such as image misidentifications and inadequate refusals that globally averaged metrics conceal.
The dataset serves as a catalyst for future innovation, aiming to advance the science of multilingual, multicultural evaluation to better support global AI cultural alignment.