Researchers present Text2Sign, a text-conditioned diffusion model designed to generate short sign-language clips on a single NVIDIA L4 GPU, addressing the high costs associated with training and evaluating video diffusion models. The system combines a frozen vision-language text encoder with a 3D encoder-decoder and factorized spatiotemporal attention to reduce computational requirements while preserving motion coherence.
- On a signer-disjoint How2Sign split, the best short-run ablation reached a validation loss of 0.0648, while a longer-run checkpoint achieved 0.00999.
- The longer checkpoint attained an SSIM of $0.2403 \pm 0.0238$, a PSNR of $15.11 \pm 0.42$ dB, and temporal consistency of $1.0000 \pm 0.0000$ using 8-step DDIM sampling.
- The model generates a 32-frame, $64 \times 64$ clip in 12.60 seconds (2.54 frames per second) with a peak inference memory usage of 3.12 GB.
- A held-out denoising audit revealed weak prompt sensitivity, with removing text increasing late-timestep loss from 0.9875 to 0.9891 and shuffled prompts performing similarly to correct ones.
The authors characterize Text2Sign as a single-GPU research baseline rather than a complete production system due to its restriction to low-resolution, short clips and the lack of expert linguistic evaluation.