The ggml project has added Q2_0 quantization support for CPUs, specifically to enable the use of Ternary Bonsai models. This update completes the Q1_0, Q2_0, Q4_0, and Q8_0 quantization family within the library.
- The implementation supports ARM NEON and includes a generic scalar fallback.
- It enables running 1.7B, 4B, and 8B Ternary Bonsai models on CPU hardware.
- x86, Metal, CUDA, and Vulkan backends are planned for future submission.
This addition allows users to run Ternary Bonsai models using the newly supported Q2_0 format on compatible CPUs.