The Falcon team has released the Falcon3 family of open models, a new iteration emphasizing expanded science, math, and code capabilities. The release includes five base models: Falcon3-1B, 3B, 7B, and 10B (transformer-based), plus Falcon3-Mamba-7B (SSM).
Key innovations include single large-scale pre-training for the 7B model using 14 trillion tokens, depth up-scaling to create a state-of-the-art 10B model, and knowledge distillation for efficient tiny models. The family supports context lengths up to 32k tokens and is available in various formats including Instruct, GGUF, and quantized variants.
Benchmark results show Falcon3-10B-Base achieving state-of-the-art performance under 13B parameters, with strong scores on MATH-Lvl5 (22.9) and GSM8K (83.0). The models are released under the Falcon LLM license, with multi-modal variants planned for January 2025.