A pull request has been merged into llama.cpp to add support for the newly released Hy3 model, enabling users to run it using GGUF formats.
- The implementation is available in PR #25395 on the ggml-org/llama.cpp repository.
- Early testing indicates coherent output from Q2_K quantized models running at approximately 10-11 tokens per second on an RTX 5090 with Zen 4 architecture and 96GB DDR5.
- GGUF checkpoints for Hy3 are available via the Hugging Face repository maintained by satgeze.
This update allows users to locally run the Hy3 model using llama.cpp's inference engine.