Recent work identified Super Weights as individual parameters whose removal degrades model performance by orders of magnitude, but this study shows that training these specific parameters is ineffective. Training Super Weights in isolation on OLMo-1B and OLMo-7B drops accuracy to random-guessing levels, even when expanding to local neighborhoods of up to 36K parameters.
- Training isolated Super Weights causes performance collapse, whereas training an equal number of randomly chosen positions improves over the baseline.
- Vanilla LoRA succeeds with only 0.16% of parameters by updating every position in attention weight matrices through low-rank structure.
- Applying the same low-rank update to down_proj layers also succeeds, while constraining LoRA updates at Super Weight coordinates yields statistically indistinguishable results from random guessing.
These findings establish that parameter importance does not imply parameter trainability in isolation and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.