A patch to llama.cpp's CUDA backend fixes a crash in multi-token prediction (MTP) on pre-Ampere GPUs by adding robust capability checks. The update ensures BF16 is used only when supported, falling back to FP16 or FP32 for older architectures like Kepler.
- Fixes cuBLAS crashes caused by incorrect BF16 compute selection on unsupported hardware.
- Implements a fallback chain: BF16 if supported, FP16 if fast, otherwise FP32.
- Tested with Qwen3.6 35B A3B q4xl on four Tesla K40c (Kepler) GPUs.
- MTP enabled performance reached ~25 tok/s compared to ~22 tok/s stock and ~17.5 tok/s with MTP disabled.
This change allows MTP decoding to function correctly on older cards without affecting newer GPUs.