Tri-modal Masked Diffusion Models pretrained from scratch with SDE reparameterization and 3B demo
This paper introduces the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image–text, and audio–text data, presents a large-scale systematic study of scaling laws, modality mixing, noise schedules and batch-size effects, and proposes an SDE-based reparameterization that decouples physical and logical batch sizes. The authors also release a preliminary unified model at ~3B parameters (trained on ~6.4T tokens) showing competitive results in text generation, text-to-image and text-to-speech tasks.