github MLX · 1h ago · inference

MLX v0.32.1 adds head dimension 72 support in Metal full attention

from English

The MLX library released version 0.32.1, introducing support for a head dimension of 72 within its Metal full attention implementation.

The update enables the use of head dimension 72 in Metal full attention via pull request #4330.

Importance 1/3 Trust 2/3 MLX Apple ML Inference efficiency

Read original

media r/LocalLLaMA · 14d ago · 24 views

Apple's AFM3 20B uses instruction-following pruning to activate ~20% of layers

Apple has introduced a new architecture for its third-generation foundation models (AFM3) that utilizes "Instruction-Following Pruning" to significantly reduce active parameter count. The model is designed to activate approximately 20% of its MLP layers by making routing decisions once per prompt rather than per token.

media r/LocalLLaMA · 34d ago · 9 views

Apple in talks with startup PrismML that shrinks AI models to run on an iPhone

Apple is currently in discussions with the startup PrismML regarding technology that compresses artificial intelligence models for deployment on iPhones.

github MLX · 41d ago · 11 views

MLX v0.32.0 fixes conv2 gradients in grouped strided case on Metal

The MLX library released version 0.32.0, which addresses a specific bug in the Metal backend.

arxiv arXiv cs.CL · 44d ago · 1 view

BaseRT delivers best-in-class LLM inference on Apple Silicon via native Metal

Researchers present BaseRT, a native Metal inference runtime for large language models on Apple Silicon that achieves the highest reported inference throughput to date. By utilizing chip-specific kernel fusion and unified memory-aware optimization, it overcomes the overhead found in existing frameworks like llama.cpp and MLX.

media r/LocalLLaMA · 58d ago · 3 views

You can now convert EXL3 quants on Apple Silicon Mac

Users can now convert and run EXL3 quantized models on Apple Silicon Macs with 64GB+ RAM. Tests show that models like MiniCPM5 and Qwen3.6-27B achieve performance on par with or slightly behind RTX-card-based conversions, with EXL3 offering superior quantization quality compared to MLX.