A Multiverse Computing paper introduces a method to train language models to refuse only specific harmful subsets of a topic, rather than the entire category, addressing the limitations of blunt topic-level guardrails. The approach uses boundary-aware self-distillation with coverage repair and in-distribution benign data to maintain safety while reducing false refusals on legitimate prompts.
- Escalating retry strategy reduces prompt generation failures from 19.88% to 0.20%, preserving 40,293 harmful training examples.
- Inclusion of 11,955 verified surface-dangerous benign prompts mitigates false refusals on safe content.
- Held-out harmful-benign pairs measure boundary precision directly, showing that adding benign boundary data reduces over-refusal from 32.94% to 4.16% with only a minor drop in harmful refusal (91.88% to 87.72%).
The authors argue that safety tuning must evaluate both harmful and benign sides of the boundary, as optimizing for harmful refusal alone can render models useless on legitimate queries.