SynthAVE introduces LLM-Arena framework for scalable synthetic labeling in e-commerce

The authors present SynthAVE, a large-scale benchmark for attribute value extraction designed to address the high cost of human labeling in e-commerce. The dataset spans 12,726 products across 229 product types, 792 attributes, and four languages.

• Validation uses a multi-LLM arena framework where samples are evaluated by 21 judge configurations (7 model families × 3 prompts).
• Final labels are determined via majority voting, which agrees with human experts at Cohen's κ=0.92.
• Individual judges show substantial inter-model agreement with Fleiss' κ=0.76.

This approach demonstrates that diverse models can aggregate into highly reliable predictions, enabling cost-effective validation at scale while maintaining quality parity with human review.