A user explores cost-effective methods to run GLM 5.x locally using 4-bit quantization, such as IQ4_XS, without relying on unified memory. Options include CPU-only setups like Sapphire Rapids ES with DDR5, multi-GPU offloading, or similar-sized models. The user runs a 5900X + 128GB DDR4 + 7900XT 20GB system, successfully handling Minimax 2.7 at Q4_K_S and Qwen 3.6 27B at IQ4_XS.
media
r/LocalLLaMA
·
53d ago
·
open_models
Cheapest way to run GLM 5.x locally without unified memory
from English
Related articles
media
r/LocalLLaMA
·
42d ago
·
3 views
Running GLM5.2 on budget hardware < $2500
A Reddit user demonstrates how to assemble a local AI inference rig for under $2500 using affordable second-hand components, specifically targeting the ability to run large language models like GLM-5.2 without expensive enterprise hardware.
media
r/LocalLLaMA
·
45d ago
·
8 views
GLM-5.2 on 4x DGX Spark: Reconstructing Missing Build Steps for MTP Speculative Decode
The author successfully deployed GLM-5.2 with MTP speculative decode on a cluster of four NVIDIA GB10 (DGX Spark) nodes, achieving approximately 9.4 tokens per second. This setup utilizes vLLM with tensor parallelism, ported sparse-MLA Triton kernels, and a deterministic 15% expert pruning to fit AWQ-INT4 weights. A critical finding is that the original Docker image build instructions are incomplete, requiring reconstruction of missing patches for deep_gemm.py and sparse_attn_indexer.py. The author also identified that using any vLLM version other than the specific pinned commit causes real AWQ weights to crash during loading due to CUDA errors. To replicate the environment, users must apply a custom script that bakes in kernels and routes functions to sm12x fallbacks. Performance benefits include roughly double the speed of previous llama.cpp implementations, though inter-node bandwidth remains a bottleneck for dual-rail scaling.
media
r/LocalLLaMA
·
49d ago
·
4 views
GLM-5.2 Beats Gemini and GPT-5.4 in Coding but Is Inefficient
GLM-5.2 surpasses GPT-5.4 and the entire Gemini lineup in coding performance on the DeepSWE benchmark. However, it requires significantly more output tokens, making it substantially less efficient in terms of cost-per-task compared to models like GPT-5.5 and Claude Opus 4.8.
media
r/LocalLLaMA
·
50d ago
·
2 views
You can now convert EXL3 quants on Apple Silicon Mac
Users can now convert and run EXL3 quantized models on Apple Silicon Macs with 64GB+ RAM. Tests show that models like MiniCPM5 and Qwen3.6-27B achieve performance on par with or slightly behind RTX-card-based conversions, with EXL3 offering superior quantization quality compared to MLX.
media
r/LocalLLaMA
·
50d ago
·
7 views
Best Settings for 48GB VRAM with Qwen 3.6 27B
A user shares optimized settings for running Qwen 3.6 27B with Q8_0 quantization on an RTX 4090 and RTX 3090 setup using llama.cpp. The configuration includes tensor split, 999 layers on GPU, 250k context, speculative decoding, and unified KV cache, achieving 75-100t/s throughput with vision and MTP support.