VTaMo introduces explicit multi-granularity alignment for sign language translation

Researchers present VTaMo, a framework that improves sign language translation by introducing explicit multi-granularity alignment rather than relying on implicit cross-modal alignment. The model aligns video and text at three levels: local frame-to-token correspondence via entropy-regularized optimal transport, global embedding calibration using Earth Mover's Distance, and position-aligned contrastive learning.

• Local alignment uses a learnable null token for fine-grained correspondences.
• Global alignment employs an orthogonal transformation to calibrate embedding space geometry.
• Position-aligned contrastive learning provides discriminative token-level representations.

Experiments on Phoenix-2014T, CSL-Daily, How2Sign, and OpenASL demonstrate consistent state-of-the-art performance, with ablations confirming the complementary contributions of each component.