Researchers introduce Smooth Maximum Mean Discrepancy (SMMD) to address the unreliability of large language models in numerically precise tasks caused by standard cross-entropy training objectives. SMMD incorporates value-distance kernels over numeric tokens and graph-based smoothness to align predicted distributions with targets while encouraging local consistency.
- SMMD builds on classic MMD by using a kernel defined over a numeric sub-vocabulary that accounts for metric structure.
- The method aligns the predicted numeric distribution to the target via kernel matching and smooths the prediction-target residual over an induced kernel graph.
- Evaluations cover four numeric-target tasks: mathematical reasoning, arithmetic calculation, clock-time recognition, and chart question answering.
- SMMD consistently improves accuracy over cross-entropy and recent numeric-target losses across multiple open-weight LLM and VLM backbones.
The approach addresses the mismatch between standard training objectives and the need for metric structure in numeric outputs, offering a method to improve numerical precision in language models.