Researchers propose a real-time monitoring system for large language models that uses an external model's verifier signal to detect unsafe outputs at deployment time. The system raises an alarm by thresholding this signal, with the threshold calibrated through risk control.
- The approach relies on a simple design of thresholding a verifier signal from an external model.
- Experiments were conducted on mathematical reasoning and red teaming datasets.
- The method is shown to be competitive with more advanced monitors based on sequential hypothesis testing.