The authors present FinResearchBench II, a scalable pipeline for generating high-quality evaluation rubrics for financial deep research agents without requiring human experts in the final loop. The benchmark is built from 104 real-world user queries and automatically synthesizes 14,450 candidate rubrics from model-generated reports.
- LLM-based evaluation demonstrates 98.67% label-level agreement with human experts on jointly unanimous items, validating its use for large-scale screening.
- Consensus-derived gold rubrics are derived using a strict consistency filter and a distinguishability filter, resulting in 2,600 final rubrics.
- The benchmark produces clearly differentiated rankings across 10 deep research systems, with item-level pass rates ranging from 58.58% to 22.23%.
This approach removes human-expert execution from rubric generation and evaluation, making it naturally scalable for benchmark evaluation, automatic system comparison, and future studies of evaluation-driven system improvement.