A pull request submitted by z-sachin introduces Q8_0 quantization support to the ggml-zendnn backend within llama.cpp, enabling optimized inference on AMD EPYC processors. The submission includes benchmark results comparing the new ZenDNN implementation against standard GGML CPU performance across several large language models.
- Llama-3.1-8B-Instruct showed gains ranging from 54.75% to 93.49% for prompt sizes between 256 and 2048 tokens.
- Mixtral-8x7B demonstrated significant improvements, with gains reaching up to 213.38% at a 2048-token prompt size.
- gemma4 31B achieved throughput increases between 68.05% and 114.58% for larger prompts.
- gemma-4-26B-A4B-it exhibited more modest gains, ranging from 4.72% to 19.02%.
- Decoding performance (tg128) remained comparable to standard ggml-cpu across all tested models.
The changes provide substantial speedups for prompt processing on AMD hardware while maintaining decoding parity with the baseline CPU implementation.