A.L.F.R.E.D. distills large model knowledge into 2B models for faster execution

The article introduces A.L.F.R.E.D., an approach called Adaptive Local-First Routing and Execution Distillation, which aims to reduce compute costs by routing simpler tasks to smaller models.

• The system captures patterns from successful queries handled by larger models and stores them in a database.
• When similar requests arise, the system routes them to a 2B model using the stored pattern instead of invoking the larger model.
• This method reduces token usage from approximately 7k to ~1k for simple tasks, resulting in roughly 4x less speed cost while maintaining validation.

This approach allows systems to delegate reasoning-heavy tasks to large models while handling repetitive or patterned requests with smaller, faster models.