The study investigates whether correcting position bias enables single-pass attention sorting to match the performance of iterative methods in long-context language models. Experiments on LLaMA-2 and YaRN-Llama-2 models refute the hypothesis that debiasing alone is sufficient to bridge the performance gap.
- On LLaMA-2-7B-32K-Instruct, debiasing produced identical containment accuracy (94.83%) to uncalibrated single-pass sorting.
- On YaRN-Llama-2-7b-64k, debiasing improved accuracy by 8.67 percentage points but remained 14.84pp behind iterative sorting.
- The corrected method closed only 37% of the performance gap between single-pass and iterative sorting approaches.
The results indicate that position-bias correction is insufficient to match iterative sorting, suggesting that repeated reordering provides additional benefits beyond bias correction.