Multimodal framework combines visual and trajectory representations for autonomous driving scenario search

Researchers present a unified retrieval pipeline for autonomous-driving scenarios that integrates visual embeddings with trajectory-based representations to address the limitations of single-modality approaches.

• The study investigates two trajectory methods: Exo-Trajectory, an explicit matching method based on surrounding-agent motion, and ScenarioFormer, a transformer-based representation learned via contrastive learning.
• Trajectory representations demonstrate strong performance for motion-centric events such as cut-ins, turning maneuvers, and traffic queueing.
• Visual embeddings excel when appearance cues are informative.
• Combining visual and trajectory information consistently improves retrieval quality, yielding the best overall performance.

These findings indicate that appearance and motion capture are complementary notions of scenario similarity, motivating multimodal systems for data mining, dataset curation, and scenario-based validation.