Disentangling latent directions for mathematical solvability in LLMs

Researchers have successfully disentangled the internal representations of mathematical solvability knowledge and verbalization within large language models, showing they are encoded as distinct, linearly decodable vectors. The study reveals that model fabrication is primarily driven by changes in verbalization rather than underlying knowledge.

• Knowledge and verbalization are identified as separate latent directions in model hidden states across multiple LLMs.
• Fabrication correlates with shifts in verbalization representations rather than the core solvability knowledge.
• Prompting with unsolvability cues reduces fabrication by shifting the verbalization direction.
• Activation steering can mechanistically manipulate these representations to improve model abstention.

This work allows for the independent analysis and manipulation of solvability beliefs, enabling more effective control over when models should abstain from answering.