SplitMoE is an alternative Mixture of Experts architecture that breaks down wide feed-forward layers into smaller, specialized sub-components to improve token representation flexibility and computational modularity.
- It adopts finer granularity than standard MoEs like Mixtral, allowing tokens to compose nuanced representations using specialized parameter paths.
- The approach slices expert hidden dimensions so that tokens activate only the specific sub-networks they need, resulting in lower FLOPs per activated path.
- Unlike DeepSeekMoE, SplitMoE explores splitting and grouping mechanisms across intermediate dimensions without strictly forcing standard fixed shared pathways.
- It focuses on a streamlined partitioning strategy to keep routing lightweight and avoid heavy dispatch overhead.
The author provides a modular PyTorch implementation for benchmarking and invites feedback on routing stability and scaling behavior.