Multiverse has released a paper detailing a method that reframes large language model pruning as a constrained binary optimization problem mapped to an Ising glass. By treating transformer blocks as spins with pairwise couplings derived from the loss Hessian, the approach identifies optimal block removal configurations without requiring iterative benchmarking.
- The method computes a Hessian matrix once on a calibration dataset to capture interactions between blocks, allowing energy calculations to serve as proxies for model quality.
- It utilizes classical and quantum-inspired solvers to find low-energy states in the configuration space, enabling the search of billions of combinations efficiently.
- On Llama-3.3-70B-Instruct at 50% compression, the method gains nearly 23 percentage points on MMLU compared to the best competing block-removal baseline.
- The approach generalizes to heterogeneous architectures like NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, where it outperforms baselines by identifying non-consecutive removal patterns.
The authors argue that exploring low-energy excited states rather than just the ground state often yields superior pruned models, particularly in deep compression regimes where block interactions are critical.