MAESTRO prunes MoE experts via Markov-chain routing to improve retention

Researchers introduce MAESTRO, a structured pruning framework for sparsely-activated Mixture-of-Experts (MoE) language models that addresses the memory bottleneck of full expert banks. The method models autoregressive expert activation trajectories as Ergodic Markov chains to capture cross-layer dependencies, providing a globally aware importance heuristic.

• MAESTRO evaluates expert importance using transition-based routing rather than local heuristics.
• It achieves up to 10.61% higher average performance retention compared to state-of-the-art baselines under a strict 50% compression regime.
• The approach demonstrates substantially lower cross-task variance, indicating more consistent generalization across heterogeneous tasks including Safety, Bias, and Ethics.