Apple has introduced a new architecture for its third-generation foundation models (AFM3) that utilizes "Instruction-Following Pruning" to significantly reduce active parameter count. The model is designed to activate approximately 20% of its MLP layers by making routing decisions once per prompt rather than per token.
- The system keeps the entire model in flash storage and loads only a small set of "expert" parameters (between 1 and 4 billion) into working memory at any time.
- A core of shared experts remains always active, while the rest of the parameters are loaded on demand based on the input prompt.
- This approach allows a 30B active MoE to achieve read bandwidth performance comparable to a 14B active model, and effectively turns a 9B dense model into a 3B active one.
This method aims to improve efficiency by minimizing memory usage while maintaining performance through sparse activation patterns.