DiffusionGemma 26B runs at up to 475t/s on a 4090 via vLLM with INT4 AWQ quantization, achieving speeds between 290t/s and 700t/s based on output length. However, it suffers from single-user operation, lower response accuracy, rapid context loss, and slower time-to-first-token compared to standard 26B models.
media
r/LocalLLaMA
·
51d ago
·
open_models
DiffusionGemma 26B on 4090 reaches 475t/s with limitations
from English
Related articles
media
r/LocalLLaMA
·
50d ago
·
1 view
GLM 5.2 Achieves 98% Max Intelligence with Less Than Half Tokens
GLM 5.2 demonstrates 98% of maximum intelligence in coding tasks using less than half of its total token budget, according to a technical report by z_ai. The model's reasoning efficiency has improved significantly, with token usage increasing from 16.7k to 36.7k between GLM 5.1 and GLM 5.2, though high-level settings may strain local hardware performance.
media
r/LocalLLaMA
·
50d ago
·
2 views
$1800 GPU cost runs Qwen3.6-27B with 262K context and 55 tok/s
A setup using four 5060 Ti GPUs (totaling $1800) achieves 55 tokens per second with Qwen3.6-27B-FP8, supporting 262K context length and bfloat16 KV cache. The configuration uses P2P and FlashInfer, with benchmark results showing 55.67 output token throughput and 65.25% speculative decoding acceptance rate.
media
r/LocalLLaMA
·
52d ago
·
1 view
Llama Bench vs Real-World Performance Discrepancy
The user reports a significant gap between Llama benchmark results and actual model performance. Benchmarks show 754 tk/s prefill and 36 tk/s generation, but real usage reveals only 7.98 tokens per second, with high latency and poor throughput. The discrepancy is attributed to real-world usage conditions, not benchmark settings, suggesting the model's actual performance is far below the benchmarked speed.
arxiv
arXiv cs.CL
·
53d ago
·
2 views
SwiftTrans Improves LLM Code Translation Efficiency
SwiftTrans addresses runtime efficiency gaps in LLM-based code translation by introducing Multi-Perspective Exploration and Difference-Aware Selection. The framework extends CodeNet, F2SBench, and introduces SwiftBench to evaluate runtime performance, showing consistent improvements in both correctness and efficiency across benchmarks.
media
Hugging Face Forums
·
6d ago
·
2 views
Qxern-v6 uses latent tokens and AST sidecar for faster, accurate code transfer
The Qxern-v6 system enables two LLMs to communicate via 32 compressed latent tokens while preserving exact symbol accuracy through an adaptive deterministic Abstract Syntax Tree (AST) sidecar. Built by a 15-year-old developer, the architecture uses Qwen2.5-Coder-1.5B to compress code and a frozen Qwen3.5-0.8B decoder to interpret it without seeing raw text.