RaBitQCache is a novel sparse attention framework designed to address the memory and computational bottlenecks of Key-Value (KV) cache in long-context Large Language Model inference. It utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights.
- The method employs an unbiased proxy score with a proven error bound to enable adaptive Top-p retrieval, dynamically adjusting the token budget based on actual attention sparsity.
- A hardware-aware system implementation incorporates asynchronous pipelining and lazy updates to mask computational overhead.
- Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines.
The approach offers a more efficient alternative to static fixed-budget retrieval or computationally expensive proxy scores, improving performance without sacrificing output quality.