media Hugging Face Forums · 1h ago · open_models

Noah DeAngelis builds experimental routing-based attention for GPT models

from English

Noah DeAngelis has developed an experimental routing-based attention mechanism for GPT models, implemented in a PyTorch project named RoutingGPT. The approach uses a learned linear layer to select a subset of tokens as keys and values while keeping all tokens as queries.

With a default 25% routing ratio, the complexity reduces from T × T for dense attention to T × 0.25T.
The goal is to reduce attention computation and enable longer context lengths on low-end hardware.
The project is released under the Apache 2.0 license and is currently experimental without rigorous benchmarking.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Training methods

Read original

media Hugging Face Forums · 3d ago · 2 views

Measurements suggest revisiting q_proj/v_proj default for LoRA target modules

The author measures gradient subspace overlap across four models to evaluate the standard LoRA default of targeting query and value projections. The analysis indicates that this default may be suboptimal when the goal is retaining previous capabilities rather than maximizing accuracy on a new task.

media Hugging Face Forums · 6d ago · 13 views

SpiralCoreAttention prototype selects token blocks for long-sequence QLoRA fine-tuning

A developer is seeking feedback on SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning. Instead of processing complete long sequences during every forward and backward pass, the system selects specific token blocks from the sequence to train on, while evaluating the resulting adapter on held-out full-context sequences.

lab Hugging Face Blog · 13d ago · 37 views

Multiverse Computing cuts knowledge distillation VRAM with fused chunked KL loss

Multiverse Computing introduces a memory-efficient approach to large language model knowledge distillation by combining offline top-K logits caching with a fused chunked Kullback-Leibler (KL) divergence loss. This method eliminates the need to keep both teacher and student models in memory simultaneously, drastically reducing VRAM requirements.

media r/LocalLLaMA · 19d ago · 31 views

Apple's AFM3 20B uses instruction-following pruning to activate ~20% of layers

Apple has introduced a new architecture for its third-generation foundation models (AFM3) that utilizes "Instruction-Following Pruning" to significantly reduce active parameter count. The model is designed to activate approximately 20% of its MLP layers by making routing decisions once per prompt rather than per token.

media Hugging Face Forums · 23d ago · 35 views

PIN v2 decouples model width from storage cost via weight tying

PIN Architecture v2 introduces a method to pre-select a neural network's size and inference speed before training by using weight tying. This technique forces groups of cells to hold identical values, allowing the network to maintain its width while drastically reducing the number of stored weights.