A new theoretical framework establishes that the negative Evidence Lower Bound (ELBO) in discrete diffusion models is exactly equal to data entropy plus the path KL divergence from the oracle reverse process, rather than serving merely as a bound. This "Oracle Distance" theorem identifies the unique optimizer as the conditional expectation of the true reverse jump rate given the current noisy state.
- The irreducible cost of training is defined as the rate at which the forward process destroys information about clean data.
- For token-factorizing noise, the optimizer has three exact coordinates: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them.
- The framework recovers MDM, UDM, SEDD, and GIDD as special cases and explains why denoiser and cavity parameters coincide for masked diffusion but not uniform diffusion.
- It proves that a denoiser parameterization causes the uniform ELBO to diverge at initialization, whereas the bridge plug-in remains finite.
This work provides exact calibration of ELBO implementations at initialization and clarifies which underlying law each existing loss function optimizes.