Structured Pruning of LLMs via Power Transformation and Sign-Preserving Score Aggregation

This paper proposes an improved structured pruning method for large language models that adapts Adaptive Feature Retention (AFR) to overcome challenges in heterogeneous pruning scores, loss of sign information, and outlier influence.

The approach combines power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that the method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup.

The authors consider this important because it enables structured pruning to retain the accuracy benefits of unstructured techniques while providing the inference efficiency gains inherent to structured sparsity.