Researchers introduce FormalRx, a comprehensive diagnostic evaluation framework designed to transform autoformalization assessment from opaque binary verdicts into actionable feedback. The system utilizes the SCI Error Taxonomy to decompose errors into 28 distinct categories and provides capabilities for alignment verdicts, error categorization, localization, and correction.
- FormalRx includes a diagnostic model, FormalRx-8B, trained on 56,287 NL-FL pairs with fine-grained annotations.
- The framework releases FormalRx-Test as the first fine-grained diagnostic benchmark for autoformalization.
- FormalRx-8B achieves F1-scores of 0.88 for verdicts and 0.71 for categorization, alongside accuracies of 0.75 for localization and 0.73 for correction.
- The model substantially outperforms both general-purpose LLMs and specialized baselines in these diagnostic tasks.
By connecting evaluation with actionable insights, FormalRx enables the systematic diagnosis and improvement of autoformalization systems.