User benchmarks local models with RAG, finding accuracy improves significantly

A Reddit user tested the accuracy of local large language models on technical questions by benchmarking them against documentation for projects like Node, Langchain.js, and TypeScript. The experiment compared unsloth Gemma QAT models, Apple Intelligence AFM 2 3B, and Qwen models with and without Retrieval-Augmented Generation (RAG).

• Without RAG, local models performed poorly on technical accuracy.
• Adding a RAG system that injected relevant documents significantly improved scores, making the models "very good."
• Enabling "thinking" capabilities provided only a marginal 1% improvement while increasing latency.
• Apple Intelligence AFM 2 3B achieved an 86% score despite a limited 4k context window.

The results indicate that local LLMs are highly effective for technical queries when connected to a knowledge base via RAG, whereas standalone performance is insufficient.