Researchers propose using spectral shape-based metrics derived from Heavy-Tailed Self-Regularization theory to address the challenges of managing and quantifying large language models at scale. This approach utilizes the shape information of the weight empirical spectral density as a compact signature that captures intrinsic properties of pretrained models while remaining robust during post-training.
- The metric is data-free, computationally efficient, and scale-invariant, enabling practical large-scale analysis.
- A curated corpus of major open-source LLM families was used to benchmark spectral and non-spectral metrics across models and downstream tasks.
- The spectral signature supports tracking model lineage, unsupervised clustering of similar models, and quantifying model performance.
The proposed method provides a meaningful proxy for broad performance trends, enabling efficient organization, comparison, and analysis of large model collections.