Frozen-backbone study finds domain adaptation efficacy depends on pre-existing target-domain knowledge

A case study evaluating frozen embedding backbones like Qwen3-Embedding, RoBERTa-base, and FinBERT reveals that the benefit of explicit domain adaptation varies significantly based on the model's prior knowledge. The research trained lightweight MLP adapters using Domain-Adversarial Neural Networks (DANN), Maximum Mean Discrepancy (MMD), and Supervised Contrastive Learning (SCL) to transfer sentiment analysis from consumer reviews to movie reviews (SST-2) and financial news.

• On the SST-2 dataset, domain adaptation provided negligible performance gains regardless of the backbone's scale.
• For a restricted subset of financial news, explicit domain adaptation recovered substantial performance for small general-purpose backbones.
• Adversarial alignment via DANN degraded performance for domain-specialized backbones like FinBERT by eroding pre-existing domain-specific structure.
• Supervised contrastive loss was found to preserve the domain-specific structure in specialized models better than adversarial methods.

The findings suggest that explicit domain adaptation is only effective when the frozen backbone lacks sufficient target-domain coverage, and inappropriate alignment methods can harm performance on already specialized models.