CDR-Bench reveals LLMs fail at faithful execution of compositional data refinement recipes

Researchers introduce CDR-Bench, a benchmark designed to evaluate whether large language models can directly and faithfully execute multi-step data refinement recipes where composition and order matter. The benchmark features 3,462 tasks across four domains and 29 operators, using deterministic reference outputs for exact evaluation.

• Experiments on over 10 state-of-the-art LLMs show consistent failure patterns in compositional settings.
• Performance degrades sharply when tasks require composition, and success rates for order-sensitive recipes collapse.
• The benchmark isolates text editing from code execution to specifically test procedural faithfulness.

These findings indicate that current models lack the necessary procedural faithfulness for reliable compositional data refinement.