This study presents the first systematic meta-evaluation of LLM-generated rubrics designed for assessing open-ended outputs in paper reproduction tasks. The authors reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models.
- Rubrics are assessed intrinsically via semantic similarity and extrinsically through score alignment with ground-truth rubrics.
- Augmented generation settings substantially improve downstream evaluation alignment, with the strongest setting approaching human baselines.
- Intrinsic gains from augmentation are noted to be more modest compared to extrinsic improvements.
- Further analysis reveals that LLM-generated rubrics tend to be overly fine-grained, biased toward high scores, and less adaptive to specific paper domains.
The findings highlight both the affordances and limitations of using LLMs for scalable benchmark construction, indicating that while augmented settings enhance alignment, significant biases remain.