User runs 100B+ LLMs on low-end hardware using MoE, NVMe offloading, and quantization

A Reddit user demonstrates running large language models exceeding 100 billion parameters on a laptop with an Intel i7-8750H, 20GB RAM, and a GTX 1050 Mobile GPU. The setup relies heavily on offloading model parameters to a Samsung NVMe drive using memory-mapped file (mmap) access.

• The user strictly employs Mixture of Experts (MoE) models to avoid overwhelming the CPU, while dense models are deemed unsuitable.
• Quantization settings include Q3_K_M for standard large models and Q2 for those exceeding 700B parameters with over 20B active parameters.
• Specific tested models include Deepseek-V4-Flash (UD-IQ3_XXS) achieving 1.0-1.8 tokens per second and Nemotron-3-Super-120B-A12B (UD-Q3_K_M) achieving 1.5-2.5 tokens per second.
• The workflow utilizes LM Studio with batch-style processing for tasks like reverse engineering and code auditing, accommodating a context window of 16K to 90K tokens.

This configuration allows users in regions with restricted international payment options to utilize powerful local models as an alternative to cloud services.