RESEARCH · RESEARCH · #1973
Tri-modal Masked Diffusion Models pretrained from scratch with SDE reparameterization and 3B demo
This paper introduces the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image–text, and audio–text data, presents a large-scale systematic study of scaling laws, modality mixing, noise schedules and batch-size effects, and proposes an SDE-based reparameterization that decouples physical and logical batch sizes. The authors also release a preliminary unified model at ~3B parameters (trained on ~6.4T tokens) showing competitive results in text generation, text-to-image and text-to-speech tasks.
KEY POINTS
- This paper introduces the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image–text, and audio–text data, presents a large-scale systematic study of scaling laws, modality mixing, noise schedules and batch-size effects, and proposes an SDE-based reparameterization that decouples physical and logical batch sizes.
- The authors also release a preliminary unified model at ~3B parameters (trained on ~6.4T tokens) showing competitive results in text generation, text-to-image and text-to-speech tasks.
- The work advances multimodal discrete diffusion modeling by providing the first from-scratch tri-modal MDM, a practical SDE reparameterization for batch-size robustness, and empirical scaling insights useful for building unified text/image/audio generators.
WHY IT MATTERS
The work advances multimodal discrete diffusion modeling by providing the first from-scratch tri-modal MDM, a practical SDE reparameterization for batch-size robustness, and empirical scaling insights useful for building unified text/image/audio generators.