Researchers propose a method for automatically detecting inconsistencies in end-to-end generated Task-Oriented Dialogues (TODs) by modeling them as a Constraint Satisfaction Problem (CSP). This approach addresses the critical issue of hallucinations in Large Language Models, where system responses may fail to adhere strictly to domain knowledge bases.
- The pipeline conceptualizes TODs as a CSP where variables represent dialogue segments referencing the conversational domain and constraints capture properties like turn coherence.
- It identifies variables in a target dialogue and applies a CSP solver to find valid solutions for comparison.
- Inconsistencies are detected by comparing the target dialogue with valid variable assignments, allowing for suggestions of minimal changes to ensure consistency.
The study demonstrates the high accuracy of this CSP-based approach in detecting inconsistencies and provides a detailed analysis of the findings.