Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.

  • Compatibility focuses initially on local inference for Apple Silicon, starting with the Qwen3.5 architecture.
  • The implementation reuses ggml Metal kernels through the new kernels library and reduces overhead in the generate function.
  • Users can load GGUF models by passing the Hub model ID and filename to from_pretrained without extra configuration.
  • The transformers serve command exposes an OpenAI-compatible API for serving these models.
  • Benchmarks on a MacBook Pro M2 Max show Transformers performing close to llama.cpp across dense and mixture-of-experts checkpoints.

This integration brings the model definition foundation of Transformers closer to the local inference foundation of llama.cpp, enabling developers to experiment with GGUF models using familiar PyTorch tools for inspection, evaluation, and fine-tuning.