An evaluation of Anthropic's J-Space hallucination signal on Qwen3-4B reveals that workspace noise effectively routes high-confidence errors in fact-retrieval tasks but is blind to internalized myths and incompatible with mathematical reasoning.
- On PopQA, routing by workspace noise caught errors with 100% precision, whereas output logprobs yielded worse-than-chance precision (87.5%).
- The metric collapsed on TruthfulQA, where the model was wrong 84.9% of the time even when workspace noise was low.
- Static thresholds calibrated on factual datasets broke on GSM8K because step-by-step math is a structurally high-entropy activity.
The author notes that workspace noise detects epistemic guessing but cannot detect ontological falsehoods, and plans to test parameter scaling to find where surface logprobs match internal clarity.