LLM-as-a-judge signals fail to optimize closed-loop table recognition

A study using FinTabNet and OmniDocBench demonstrates that LLM-as-a-judge signals are insufficient for optimizing closed-loop regeneration in table recognition. The research reveals that judge scores frequently tied, rankings were not reproducible, and the system failed to recover better candidates produced by iteration.

• Judge signals showed weak performance, with selection policies beating random only due to tie rules rather than actual score quality.
• Severe losses occurred even without specific judge feedback, linked to target-preservation failure under unconstrained regeneration.
• Structure-preserving instructions reduced severe-loss rates but did not improve overall results when judge feedback was retained.

The findings indicate that evaluation ability does not imply optimization utility, suggesting iterative refinement requires deterministic verification signals rather than relying solely on LLM judge scores.