Researchers propose UMoE, a pipeline that realigns the expert pool of Mixture-of-Experts models to a target domain before supervised fine-tuning. The method prunes low-saliency experts and regrows the pool via perturbation-based expansion while preserving original parameter counts and inference costs.

  • Tested on Qwen3-30B-A3B and Qwen3.5-35B-A3B architectures across math, code, science, tool-use, and agentic coding domains.
  • Achieved 3.4-point improvement in average math accuracy and 6.0 points on SWE-bench Verified compared to direct supervised fine-tuning.
  • Raised average score on a strong in-house math corpus to 84.17, outperforming Qwen3-30B-A3B-Thinking's 81.06.
  • Demonstrates robustness as training data scales and lowers training loss by converting redundant capacity into useful domain capacity.

UMoE consistently improves performance across multiple benchmarks without per-domain hyperparameter tuning, effectively turning unused expert capacity into targeted domain expertise.