Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector

This paper analyzes non-sequential multimodal sentence-level embeddings, focusing on the SONAR model, to demonstrate that specific embedding dimensions are sensitive to perturbations and can indicate decoding anomalies. By leveraging consistency between successive encoding and decoding, the authors successfully build an accurate anomaly detector.

• The study focuses on non-sequential multimodal sentence-level embeddings with a particular emphasis on the SONAR model.
• Certain embedding dimensions are identified as sensitive to perturbations, serving as indicators of decoding anomalies.
• An accurate detector is built by leveraging consistency between successive encoding and decoding processes.
• The authors explore modifying specific dimensions of interest in an attempt to correct detected anomalies.

This work underscores the importance of understanding and analyzing embeddings themselves to enhance the reliability of multimodal representations.