arxiv arXiv cs.CL · 1h ago · research

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

from English

Researchers propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along contraction dimensions without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off across language models ranging from 1B to 70B parameters.

Reduction tolerance depends on the model family, task, component, and retention ratio, often improving with model scale.
RMM remains robust across discriminative, autoregressive generation, and long-context settings under moderate reduction.
The principle extends to multimodal vision-language inference.
Mechanistic ablations reveal attention-side computations are substantially more reducible than MLP components.
Wall-clock benchmarks on an NVIDIA A100 show practical runtime gains, especially at longer sequence lengths.

These results position RMM as a scalable direction for input-adaptive inference-time optimization.

Importance 2/3 arXiv cs.CL Inference efficiency Research paper

Related articles

lab NVIDIA Research · 3d ago · 10 views

DSTP mitigates visual token pruning failures in complex MLLM reasoning

Researchers identify Relevant Visual Information Shift (RVIS) during decoding as the primary cause for the failure of existing visual token pruning methods in Multimodal Large Language Models (MLLMs). To address this, they propose Decoding-stage Shift-aware Token Pruning (DSTP), a training-free framework that aligns visual tokens with shifting reasoning requirements.

media Hugging Face Forums · 10d ago · 1 view

Open-source Decay-Gated O(N) Causal Linear Attention with fused Triton kernel

A developer has open-sourced a Decay-Gated O(N) Causal Linear Attention architecture featuring fused Triton/CUDA state accumulation kernels designed to bypass quadratic MHA bottlenecks.

media Hugging Face Forums · 16d ago · 5 views

Jason Van Pham documents Gemma 4 multi-turn greedy degeneration under letter-level constraints

Jason Van Pham and co-engineer Grok (xAI) published a diagnostic report detailing a near-vanilla multi-turn failure mode in the Hydrodynamic Swarm harness when using a custom Gemma 4 instruct path. The study identifies that combining greedy decoding with long token budgets and letter-level constraints triggers classical text degeneration, including repetition and orthographic thrash.

arxiv arXiv cs.CL · 17d ago · 2 views

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention

PIVOT (Proxy Indexing Via One full-prefix Traversal) is a training-free, drop-in replacement for the DeepSeek Sparse Attention (DSA) indexer that reduces computational redundancy by sharing one prefix scan across a group of nearby queries. Instead of scoring every preceding token for each query individually, PIVOT aggregates the group into a single proxy query to obtain a candidate set, from which top-k tokens are selected for each query.

arxiv arXiv cs.CL · 29d ago · 3 views

ShortOPD recovers pruned LLMs with short-to-long on-policy distillation

The authors propose ShortOPD, a short-to-long on-policy distillation schedule designed to recover the generative capabilities of structured-pruned large language models that collapse during free-form generation. By detecting teacher-confirmed repetitive suffixes and treating the surviving prefix as the effective rollout length, the method allocates training budget more efficiently than standard approaches.