arxiv arXiv cs.CL · 2h ago · research

Douyin releases DME multimodal embedding model with state-of-the-art performance

from English

Douyin has released the technical report for its Multimodal Embedding (DME) model, which combines large-scale contrastive pre-training with latent reasoning to achieve strong discrimination and efficiency.

Trained in two stages: Stage 1 establishes a unified multimodal embedding space via contrastive pre-training, while Stage 2 uses Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction to enhance semantic sufficiency.
Both mechanisms act only during training, allowing DME to serve with marginal query-side overhead comparable to standard contrastive encoders.
On MMEB-v2, the 2B and 9B variants reach state-of-the-art results with scores of 74.8 and 78.4, showing particular strength in video and visual-document tasks.
In production, DME delivers a 2.92% relative gain on Douyin's offline evaluation set and a 0.1% Lifetime (LT) gain in online A/B testing for search.

The model is deployed across Douyin scenarios including generative, image, and AI search, addressing the need for both efficiency under billion-scale indexing and fine-grained discrimination.

Importance 2/3 arXiv cs.CL ByteDance Multimodal Research paper

Related articles

media Hugging Face Forums · 2d ago

High school student seeks arXiv endorser for AttnRoute-MoE Vision Transformer paper

Hanyu (Olivia) Yu, a high school sophomore, is seeking an arXiv endorser in cs.LG or cs.CV to upload her independent preprint titled "AttnRoute-MoE: Attention-Prior Routing for Mixture-of-Experts Vision Transformers."

arxiv arXiv cs.AI · 7d ago

ClinFusion introduces vision-centric MLLM for holistic medical understanding

Researchers introduce ClinFusion, a vision-centric multimodal large language model designed to address the challenges of deploying AI in clinical practice by unifying 2D and 3D medical image understanding. The system features a compositional cascaded vision encoder with a Cascade Spatial-Aware Locality Fusion operator and includes a new evaluation framework comprising MedIF-Bench and region-of-interest-grounded metrics.

arxiv arXiv cs.CL · 7d ago

ClinFusion: vision-centric MLLM sets new state of the art in medical benchmarks

Researchers introduce ClinFusion, a vision-centric multimodal large language model designed for holistic medical understanding that unifies 2D and native 3D image analysis. The system utilizes a compositional cascaded vision encoder with a Cascade Spatial-Aware Locality Fusion operator to address the challenges of heterogeneous medical imaging.

lab NVIDIA Research · 7d ago · 3 views

Multimodal AI analysis of non-verbal communication in Delta Force

Researchers analyze gameplay videos from the competitive shooter game Delta Force to extract and understand emergent non-verbal communication, such as gestures and movement patterns. This work addresses a gap in prior studies that have largely focused on MOBA games or verbal coordination in other shooters.

arxiv arXiv cs.CL · 19d ago

CSSEL-P2P enables simultaneous speech translation without architectural changes

The article presents CSSEL-P2P, a data-driven approach for simultaneous speech translation that avoids the brittle architectural changes and explicit read/write policies often required by decoder-only LLM systems. The method utilizes fixed-length chunks for cumulative streaming decoding with a rewind-based committed prefix, combined with teacher-labeled prefix-to-prefix targets and bounded waiting for fine-tuning.