media r/LocalLLaMA · 50d ago · open_models

How to Set Optimal llama.cpp Parameters for AMD GPU

from English

Users seeking optimal llama.cpp settings for gemma 4 models on an AMD GPU with 16GB VRAM ask whether trial and error is necessary. They reference Google's default settings for temperature, top-p, and top-k but note inconsistent results, indicating a need for more targeted guidance beyond official documentation.

Importance 1/3 r/LocalLLaMA Code generation Inference efficiency Open weights

Read original

media r/LocalLLaMA · 50d ago · 2 views

You can now convert EXL3 quants on Apple Silicon Mac

Users can now convert and run EXL3 quantized models on Apple Silicon Macs with 64GB+ RAM. Tests show that models like MiniCPM5 and Qwen3.6-27B achieve performance on par with or slightly behind RTX-card-based conversions, with EXL3 offering superior quantization quality compared to MLX.

media r/LocalLLaMA · 50d ago · 7 views

Best Settings for 48GB VRAM with Qwen 3.6 27B

A user shares optimized settings for running Qwen 3.6 27B with Q8_0 quantization on an RTX 4090 and RTX 3090 setup using llama.cpp. The configuration includes tensor split, 999 layers on GPU, 250k context, speculative decoding, and unified KV cache, achieving 75-100t/s throughput with vision and MTP support.

media r/LocalLLaMA · 50d ago · 3 views

7900XTX 24GB VRAM Runs Qwen 3.6 27B with 131k Context

A user reports successfully running a Qwen 3.6 27B model with Q6K+MTP quantization and 131k context length on a 7900XTX with 24GB VRAM. This is achieved using kvcache quantization (Q5_0/Q4_0), which reduces VRAM usage by 12% compared to Q8, enabling the model to run at 55-60 tokens per second with specific compile flags and llama.cpp arguments.

media r/LocalLLaMA · 52d ago · 2 views

Repurposing an Old Multi-GPU Node for Local Inference

The node features 8 NVIDIA Quadro RTX 6000 GPUs with 192 GB VRAM and 512 GB RAM, enabling large-scale local AI model inference. Models like LLaMA-3 or Mistral with 8-13 billion parameters could run efficiently here, offering faster, private, and low-latency performance compared to single-GPU setups, making it worthwhile for internal use.

media r/LocalLLaMA · 53d ago · 2 views

Cheapest way to run GLM 5.x locally without unified memory

A user explores cost-effective methods to run GLM 5.x locally using 4-bit quantization, such as IQ4_XS, without relying on unified memory. Options include CPU-only setups like Sapphire Rapids ES with DDR5, multi-GPU offloading, or similar-sized models. The user runs a 5900X + 128GB DDR4 + 7900XT 20GB system, successfully handling Minimax 2.7 at Q4_K_S and Qwen 3.6 27B at IQ4_XS.