Researchers propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface for multi-vector vision-language retrieval.
- SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing and requires no ground-truth bounding boxes or detectors at inference time.
- The method adapts only the shared projection layer with frozen vision and language backbones.
- With K=64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09x while improving R@1 on Flickr30K and MSCOCO.
- The approach outperforms compression baselines and shows stronger phrase-level grounding compared to pruning or feature-only pooling methods.
These gains arise because object-aware merging preserves query-selectable object evidence that other compression methods can remove or collapse, suggesting efficient multi-vector retrieval depends on preserving the evidence future query tokens need to select.