The article diagnoses how attention-based KV cache eviction methods like H2O disproportionately retain noise on schema-dense inputs, such as nested JSON, where structural tokens carry significantly more energy than content tokens. This bias causes exact-match accuracy to collapse from 88% to 0% at a 5% budget due to the over-retention of KEY tokens relative to VALUE tokens.
- Structural KEY tokens are retained at roughly 1.8x the rate of answer-carrying VALUE tokens, acting as a non-stationary filter that degrades signal-to-noise ratio.
- Suppressing KEY tokens is identified as the most effective deployable filter for this specific bias.
- A retraining-free, role-conditional allocation method over SnapKV's windowed score closes 63-98% of the H2O gap at sub-20% budgets.
- At higher budgets, the proposed method modestly matches or exceeds full-cache accuracy with a small denoising effect.
- A 15 MB linear role probe supplies these labels at negligible inference cost, though matching parser-level downstream accuracy remains an open challenge.