media Hugging Face Forums · 1h ago · open_models

Open-source Decay-Gated O(N) Causal Linear Attention with fused Triton kernel

from English

A developer has open-sourced a Decay-Gated O(N) Causal Linear Attention architecture featuring fused Triton/CUDA state accumulation kernels designed to bypass quadratic MHA bottlenecks.

Context Scaling: 128K context sequence executed in 108.78ms with <185MB VRAM on NVIDIA T4.
Kernel Acceleration: Fused Triton kernel delivers a 3.60x speedup over standard PyTorch implementation.
Precision Guard: Uses FP32 Master Accumulator + RMS Normalization to prevent FP16/BF16 gradient overflow during long-context training.
Retrieval Accuracy: Passed 16K Needle-In-A-Haystack (NIAH) benchmark with 1.89x SNR.

The author is seeking feedback on integrating the model into Hugging Face Transformers or conducting benchmark comparisons with existing linear models.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Research paper

Read original

media Hugging Face Forums · 6d ago

Jason Van Pham documents Gemma 4 multi-turn greedy degeneration under letter-level constraints

Jason Van Pham and co-engineer Grok (xAI) published a diagnostic report detailing a near-vanilla multi-turn failure mode in the Hydrodynamic Swarm harness when using a custom Gemma 4 instruct path. The study identifies that combining greedy decoding with long token budgets and letter-level constraints triggers classical text degeneration, including repetition and orthographic thrash.

arxiv arXiv cs.CL · 8d ago · 2 views

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention

PIVOT (Proxy Indexing Via One full-prefix Traversal) is a training-free, drop-in replacement for the DeepSeek Sparse Attention (DSA) indexer that reduces computational redundancy by sharing one prefix scan across a group of nearby queries. Instead of scoring every preceding token for each query individually, PIVOT aggregates the group into a single proxy query to obtain a candidate set, from which top-k tokens are selected for each query.

arxiv arXiv cs.CL · 19d ago

ShortOPD recovers pruned LLMs with short-to-long on-policy distillation

The authors propose ShortOPD, a short-to-long on-policy distillation schedule designed to recover the generative capabilities of structured-pruned large language models that collapse during free-form generation. By detecting teacher-confirmed repetitive suffixes and treating the surviving prefix as the effective rollout length, the method allocates training budget more efficiently than standard approaches.

arxiv arXiv cs.CL · 19d ago

Adaptive filtering corrects structural-role bias in LLM KV cache eviction

The article diagnoses how attention-based KV cache eviction methods like H2O disproportionately retain noise on schema-dense inputs, such as nested JSON, where structural tokens carry significantly more energy than content tokens. This bias causes exact-match accuracy to collapse from 88% to 0% at a 5% budget due to the over-retention of KEY tokens relative to VALUE tokens.

media r/LocalLLaMA · 20d ago

Solo researcher releases Wave Field LLM, replacing dot-product attention with FFT wave convolution

A solo researcher has introduced Wave Field LLM, a new base completion model architecture that replaces standard O(N²) dot-product attention with FFT wave convolution. This approach allows for constant speed and memory usage during inference regardless of context length, enabling 128K context windows where standard attention typically runs out of memory.