The authors propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text for video captions. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus.
- VEGAS uses egocentric activities and instructional slides paired with synchronized gaze and reference annotations for evaluation.
- Captions are selected via rejection sampling without model retraining.
- Experiments show VEGAS-selected captions align significantly better with human focus.
- The method improves downstream caption-to-video retrieval.
The work demonstrates the practical utility of incorporating viewer attention during inference.