User distills DeepSeek v4 Pro into Gemma 4 26B-A4B and 12B models

A user fine-tuned Gemma 4 26B-A4B and Gemma 4 12B using QLoRA to compare dense versus Mixture-of-Experts architectures during training. The dataset was generated by DeepSeek v4 Pro from Natural Questions, costing $0.36 for 1200 requests.

• The 26B model consumed ~2x the VRAM of the 12B (28.6 vs 14.3 GB) but achieved a significantly lower train loss (0.18 vs 0.71).
• The dense 12B was faster in wall-clock time (54 vs 72 min) and had higher per-GPU throughput (345 vs 261 tok/s).
• The 12B's gradient norm was ~5.4x noisier than the 26B's, and the user noted likely overfitting in the smaller model.

The author provides GGUF checkpoints for both models and the dataset to allow for reproducibility of the fine-tuning process.