llama.cpp version b9741 introduces new binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures. The release includes support for Vulkan, CUDA 12.4 and 13.3, OpenVINO, SYCL, and ROCm, with updated versions for iOS and Ubuntu.
A patch addresses random failures in the test-args-parser on Windows by modifying argv override to only apply when argc matches, preventing clobbering of programmatic arguments. This fixes a fastfail assertion in the OpenVINO Windows workflow while preserving UTF-8 handling for real binaries.
Users can now convert and run EXL3 quantized models on Apple Silicon Macs with 64GB+ RAM. Tests show that models like MiniCPM5 and Qwen3.6-27B achieve performance on par with or slightly behind RTX-card-based conversions, with EXL3 offering superior quantization quality compared to MLX.
A cost analysis estimates that hosting diffusiongemma at different user token levels results in monthly costs per user ranging from 1.7€ to 122.8€. The study finds agentic AI usage is economically unsustainable for collective hosting, though costs could decrease with new GPUs or ASICs and a shorter GPU depreciation period.
z.AI, ranking as the number 2, has publicly praised the number 1 open source model. The post highlights admiration for the model's capabilities, emphasizing its performance and contributions to the community.
A user asks which local LLM currently performs best at summarizing long English stories. The query highlights the need for accurate, local LLMs capable of handling multi-page narratives in English.
The ggml project has optimized AMX performance by flattening the partition over n_batch * M, ensuring all threads participate in quantization. This change improves speed by up to 1.47x across various models and hardware configurations on CPU and GPU platforms, with results showing consistent gains in inference time.
SupraLabs has released a curated chat title dataset with 115K samples, surpassing the previous record of 10K samples. The filtered dataset is available as `SupraLabs/chat-titles-filtered-115K`, while an unfiltered version with 150K samples is also provided, along with a legacy 12K dataset.
A user shares optimized settings for running Qwen 3.6 27B with Q8_0 quantization on an RTX 4090 and RTX 3090 setup using llama.cpp. The configuration includes tensor split, 999 layers on GPU, 250k context, speculative decoding, and unified KV cache, achieving 75-100t/s throughput with vision and MTP support.
A user is designing a local, offline document retrieval and LLM pipeline with storage, ingestion, query, and highlighting features. They seek advice on vector databases (e.g., pgvector in Postgres vs Qdrant), GraphRAG feasibility offline, and open-source tools for document highlighting with citations.
A user reports successfully running a Qwen 3.6 27B model with Q6K+MTP quantization and 131k context length on a 7900XTX with 24GB VRAM. This is achieved using kvcache quantization (Q5_0/Q4_0), which reduces VRAM usage by 12% compared to Q8, enabling the model to run at 55-60 tokens per second with specific compile flags and llama.cpp arguments.
A user asks for book recommendations to build a strong mathematical foundation for understanding and contributing to machine learning and deep learning, especially given their interest in AI architectures and large language models. They acknowledge that intuitive understanding is limited without proper mathematical background and seek structured resources to complement their current learning through channels like 3b1b.
A local agent can access the web without paid APIs by using self-hosted SearXNG for search and Scrapling with Trafilatura for page extraction. The setup avoids vendor dependencies, uses open-source tools, and delivers search results and page content in Markdown format, with fallbacks for CAPTCHAs and security challenges.
SupraLabs has launched supra-title-FFT-preview, a chat title generation model trained on 115K samples from a filtered dataset, expanding coverage beyond its previous 12K-sample model. The model uses full fine-tuning on LiquidAI/LFM2.5-350M-Base with BF16 precision and is designed for single-purpose chat title generation, available via Hugging Face and supporting direct loading or vLLM deployment.
Attention Algebra is a prototype that translates natural language into algebraic expressions, maps them to mathematical dynamics, and visualizes the result as a spectrogram. It treats language as a lossy projection of high-dimensional states, proposing that raw attention patterns grouped into functions serve as the 'DNA' of text, enabling efficient reasoning chains by reducing token usage from 20k to 4k.
The ggml-webgpu project has added adapter toggles for half-precision (F16) support on Vulkan and NVIDIA GPUs. This update enables improved performance on compatible hardware across multiple platforms, including macOS, Linux, Android, Windows, and openEuler, with specific builds available for ARM and x64 architectures.
A discussion thread identifies the best local AI agents available today, emphasizing open-weight models and local hardware execution. The post defines 'agents' as autonomous software that self-determines actions without pre-programming, distinguishing them from tools like IFTTT or Apple Shortcuts, and sets rules requiring local deployment and open-source agent software as a primary focus.
LLaMA.cpp releases version b9729 with binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures. The release includes CPU, Vulkan, OpenVINO, SYCL, and ROCm support, along with a new UI package. Internal references to 'webui' have been removed.
Users seeking optimal llama.cpp settings for gemma 4 models on an AMD GPU with 16GB VRAM ask whether trial and error is necessary. They reference Google's default settings for temperature, top-p, and top-k but note inconsistent results, indicating a need for more targeted guidance beyond official documentation.
GLM-5.2-REAP50-GGUF models are available on Hugging Face, offering two quantized versions: Q3_K_M (182 GB) and Q2_K (139 GB). The models are compared in a Reddit post to Qwen 3.6 27b, though no direct performance evaluation is provided.