A user is planning to build an uncensored, Greek-native large language model for local deployment using the ilsp/Llama-Krikri-8B-Instruct checkpoint. The proposed plan involves abliterating the model with Heretic (heretic-llm), converting it to GGUF Q4_K_M format, and running inference on a GMKtec EVO-X3 equipped with an AMD Ryzen AI MAX+ 395 processor via Vulkan.

The user is seeking community advice on several technical aspects of this workflow:

  • Whether Heretic supports abliterating Llama-Krikri-8B or other Greek fine-tunes.
  • The optimal sequence for processing: abliterating safetensors before conversion versus using abliterate.cpp directly on GGUF files.
  • The best quantization level (Q4_K_M, Q5_K_M, or Q6_K) to balance quality and speed on the Strix Halo hardware.
  • Recommended llama.cpp Vulkan flags, such as -ngl 99, -c 32768, flash attention, q8_0 KV cache, and speculative decoding.
  • Identification of any other existing uncensored Greek models beyond Krikri, Meltemi, or Sophea.