Google DeepMind has released the Gemma 3 technical report, introducing a multimodal update to the Gemma family of lightweight open models ranging from 1 to 27 billion parameters. This version adds vision understanding, supports at least 128K tokens of context, and modifies the architecture to reduce KV-cache memory usage.
- The model architecture increases the ratio of local to global attention layers and keeps local attention spans short to manage long-context memory.
- Training utilizes distillation, with a novel post-training recipe that significantly enhances math, chat, instruction-following, and multilingual capabilities.
- Gemma3-4B-IT achieves performance comparable to Gemma2-27B-IT, while Gemma3-27B-IT is competitive with Gemini-1.5-Pro across benchmarks.
The authors release all models to the community, aiming to provide superior performance over previous versions for both pre-trained and instruction-finetuned applications.