MTPLX V2 introduces Turbo Mode for faster MLX model inference

The developer behind MTPLX has released version 2 of the tool, which brings significant performance improvements to running MLX models on Apple Silicon hardware. The update focuses on optimizing inference speed through new kernel implementations and cache management features.

• Turbo Mode utilizes custom verify-specialized quantized-matmul kernels and a compiled verify step to achieve 82 tokens per second on a MacBook Pro M5 Max.
• The release includes significant changes to the SSD KV cache system.
• Improvements have been made to long context tool calling capabilities.
• Preliminary benchmarks comparing MTPLX against oMLX and DGX spark were provided by Ivan Fioravanti.

These updates aim to establish MTPLX as a high-performance runtime for local large language models, particularly for users seeking faster generation speeds on Mac devices.