Super and Supra use activation-weighted pruning scores for sparse fine-tuning

Researchers propose Super, a sparse parameter-efficient fine-tuning method that fixes a small trainable support using Wanda-style activation-weighted magnitude scores computed from a calibration pass. They also introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule.

• Super uses a fixed support selected by Wanda-style activation-weighted magnitude scores.
• Supra combines the sparse update with LoRA using a budget-splitting rule.
• In Math17K experiments on Llama-3.2-1B and Meta-Llama-3-8B, Super/Supra variants achieved the highest average accuracy among tested schedule-selected adapter configurations.
• A PaFi-style magnitude-only support served as a training-free sparse baseline.

The results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.