Research across five instruction-tuned models from three families (2B to 14B) demonstrates that model refusal involves two distinct axes: answer correctness and question answerability. The study finds that standard confidence scores track correctness but are nearly blind to whether a question is actually answerable, particularly on false-premise questions.
- A linear probe on hidden states can detect unanswerable questions with 0.69 to 0.77 AUROC, whereas standard metrics like P(IK) and P(True) remain near chance.
- Instructing models to check premises backfires, causing them to dispute sound premises (57% false challenges), while routing instructions via the probe triples challenge precision.
- A calibrated policy using separate answerability and correctness scores certifies 0.75 coverage of correct answers at threshold 0.75, compared to only 0.31 for a single threshold.
This approach allows the unanswerable-answer rate to be controllable at every scale while capping the wrong-answer rate by model accuracy, providing a tighter guarantee than single-threshold methods.