A user tested Anthropic's Global Workspace / Jacobian Lens paper on several open-source models, including Gemma 4 and Qwen variants, to evaluate its utility for detecting confident hallucinations in local inference.
- The study analyzed workspace trajectories across Gemma E4B, 12B, 12B abliterated, 26B MoE, and Qwen 3.6 27B models using 500 TriviaQA questions per model.
- On Gemma models, clean workspaces correlated with 77% correctness while noisy ones dropped to 42%, with workspace features outperforming output confidence alone.
- A logistic regression router trained on workspace trajectory features achieved high AUC scores for predicting wrong answers, particularly on Gemma E4B (0.787 combined) and 12B (0.843 combined).
- The approach failed on Qwen 27B, where output confidence was already well-calibrated and workspace features provided no additional benefit.
- Abliterated models showed increased fabrication rates for fake entities compared to base models, suggesting a loss of "I don't know" signals.
The findings suggest that Jacobian-lens workspace trajectory features can serve as an effective one-pass risk signal for local-to-cloud routing when output confidence is miscalibrated.