media Hugging Face Forums · 1h ago · open_models

Measurements suggest revisiting q_proj/v_proj default for LoRA target modules

from English

The author measures gradient subspace overlap across four models to evaluate the standard LoRA default of targeting query and value projections. The analysis indicates that this default may be suboptimal when the goal is retaining previous capabilities rather than maximizing accuracy on a new task.

On Llama-3-8B fine-tuned for code-to-prose, HumanEval retention improved from 54.1% with the q,v default to 75.6% using a measured placement that excluded v_proj entirely.
The type ordering of module importance reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B, and TinyLlama-1.1B, with o_proj first and v_proj last or absent.
A cheap rule based on module type and layer depth ranks well (AUROC 0.83 to 0.92) but selects a different set of modules, differing by about a third from the gradient-based selection.

The author provides a Colab notebook and GitHub repository for reproducing these measurements, noting that the implementation is AGPL-licensed.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Training methods

Benchmarks

Benchmark	Model	Score
HumanEval	Llama-3-8B	54.1%

Read original

media Hugging Face Forums · 3d ago · 13 views

SpiralCoreAttention prototype selects token blocks for long-sequence QLoRA fine-tuning

A developer is seeking feedback on SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning. Instead of processing complete long sequences during every forward and backward pass, the system selects specific token blocks from the sequence to train on, while evaluating the resulting adapter on held-out full-context sequences.

lab Hugging Face Blog · 10d ago · 30 views

Multiverse Computing cuts knowledge distillation VRAM with fused chunked KL loss

Multiverse Computing introduces a memory-efficient approach to large language model knowledge distillation by combining offline top-K logits caching with a fused chunked Kullback-Leibler (KL) divergence loss. This method eliminates the need to keep both teacher and student models in memory simultaneously, drastically reducing VRAM requirements.

media r/LocalLLaMA · 16d ago · 29 views

Apple's AFM3 20B uses instruction-following pruning to activate ~20% of layers

Apple has introduced a new architecture for its third-generation foundation models (AFM3) that utilizes "Instruction-Following Pruning" to significantly reduce active parameter count. The model is designed to activate approximately 20% of its MLP layers by making routing decisions once per prompt rather than per token.

media Hugging Face Forums · 20d ago · 35 views

PIN v2 decouples model width from storage cost via weight tying

PIN Architecture v2 introduces a method to pre-select a neural network's size and inference speed before training by using weight tying. This technique forces groups of cells to hold identical values, allowing the network to maintain its width while drastically reducing the number of stored weights.

arxiv arXiv cs.LG · 29d ago · 26 views

SkewAdam uses tiered optimizer state to reduce MoE training memory by 97%

Researchers introduce SkewAdam, an optimizer designed for mixture-of-experts (MoE) models that allocates different state types to distinct parameter populations to drastically reduce memory usage. By assigning float32 momentum and factored second moments to the backbone, factored second moments to experts, and exact second moments to the router, SkewAdam reduces optimizer state from 50.6 GB to 1.29 GB for a 6.78B-parameter model.