Cactus has post-trained the Gemma 4 E2B model to generate a confidence score between 0 and 1 for every response, enabling developers to route uncertain queries to larger cloud models. The team achieved this by adding a lightweight 68k parameter probe layer that reads intermediate hidden states during decoding to predict the probability of an error.
- The probe averages 0.814 AUROC across 12 hold-out benchmarks, significantly outperforming token entropy heuristics which scored 0.549.
- By routing only 15-35% of queries to Gemini 3.1 Flash-Lite, the hybrid system matches that model's performance on most benchmarks while reducing costs.
- The technique extracts a modality-independent correctness signal, allowing the probe to perform well on audio benchmarks despite being trained on zero audio data.
- Weights are available on HuggingFace with code provided for Transformers, MLX, Llama.cpp, and Cactus, under an MIT license.
This approach allows developers to balance privacy and speed with accuracy by accepting on-device answers when confidence is high and handoffing to the cloud when it is low.