Attention Head Reweighting adapts LLMs with 200-1000x fewer parameters

Researchers propose Attention Head Reweighting (AHR), a data-efficient method that adapts large language models to new text-classification tasks by learning only a single scalar per attention head. This approach drastically reduces the number of trainable parameters, modifying approximately 0.0001% of the model's total.

• AHR leverages the functional specialization of individual attention heads to adapt LLMs with minimal data.
• Experiments on diverse open-source text classification datasets show that AHR outperforms standard baselines like LoRA when learning from limited samples.
• The method requires 200-1000x fewer trainable parameters than comparable adaptation techniques.
• Learned weights are interpretable, allowing analysis of the mechanisms and attention heads responsible for in-context learning abilities.

The authors consider this important because it addresses the challenge of effective learning from limited data in domains like security, where labeled examples are scarce.