Research across five instruction-tuned models from three families (2B to 14B) identifies two distinct axes for LLM abstention: answer correctness and question answerability. The study finds that ordinary answer-confidence tracks whether an answer is right but remains nearly blind to whether a question is answerable, while a linear probe on hidden states detects unanswerability.
- On false-premise questions (CREPE), standard metrics stay near chance, whereas the hidden-state probe reaches 0.69 to 0.77 AUROC.
- Instructing models to check premises backfires with 57% false challenges, but routing instructions via the probe triples challenge precision.
- A calibrated policy certifying both answerability and correctness scores at 0.75 coverage achieves 0.31 correct answer coverage, compared to a single threshold's failure.
This approach allows the unanswerable-answer rate to be controllable at every scale while capping the wrong-answer rate by model accuracy.