The llama.cpp b10994 release includes a fix for the Metal backend that resolves NaN outputs in the `mul_mm_id` operation when activation values exceed the f16 range. This defect previously caused models like Mistral Small 4 to produce entirely NaN vocabularies during prefill steps of 32 tokens or more on Apple Silicon devices.

  • The fix rescales src1 by a power of two to fit within f16 limits and undoes the scale on the f32 accumulator, ensuring exact results without precision loss.
  • Performance benchmarks on an M2 Max show a median overhead of +1.14% for the affected paths, with larger batch sizes incurring slightly higher costs.
  • The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, CUDA, ROCm, OpenVINO, SYCL), Android, and Windows across various hardware configurations.