How to run Prism Bonsai 27B

This guide explains how to run the Prism ML Ternary-Bonsai-27B model locally by using a specific quantization format and a custom build of llama.cpp. The process involves downloading the Q2_0 GGUF file from Hugging Face and compiling the PrismML-Eng fork of llama.cpp with hardware-specific flags.

• Download the Ternary-Bonsai-27B-Q2_0.gguf file using huggingface-cli or directly from the Hugging Face repository.
• Clone the PrismML-Eng/llama.cpp repository and compile it, enabling GGML_METAL for Mac or GGML_CUDA for CUDA-compatible GPUs.
• Launch the llama-server binary pointing to the model file, using recommended defaults of --temp 0.7, --top-p 0.95, and --top-k 20.
• Access the interactive interface via localhost:8080 in a web browser.

The article provides the necessary technical steps to execute this specific model variant, which requires the custom llama.cpp build due to its unique quantization format.