User seeks advice on CPT for DiffusionGemma or alternative text diffusion models

A user is seeking advice on adapting the DiffusionGemma model to a marine-domain dataset, specifically looking for methods to perform Continued Pretraining (CPT) for domain adaptation.

• The user attempted Supervised Fine-Tuning (SFT) but observed increased hallucination and degraded instruction-following quality.
• They note that DiffusionGemma only provides an instruction-tuned checkpoint without a base model, preventing the standard CPT followed by SFT workflow.
• The user is asking for successful CPT examples with DiffusionGemma or recommendations for other text diffusion models that support CPT and offer good inference speed.
• Hardware constraints include NVIDIA DGX Spark for training and an RTX A6000 (48 GB) for inference.