Contrastive Mixture Diffusion Models.

Abstract

Continuous diffusion model is feasible to perform smoothed transitions for flexible text generation in a continuous latent space. A challenging issue in continuous diffusion is to implement the denoising process and manipulate the generated tokens to reflect physical phenomenon in diffusion steps towards a contextually and semantically meaningful sentence. Basically, a high-quality text generation can be fulfilled by an easy-first strategy, which prioritizes the generation of high-frequency simple or general tokens in the beginning and then low-frequency complex or specific tokens in the end via mask language model. This paper introduces the mask noise as an absorbing state and develops a new continuous-discrete denoising process in a diffusion model. Each diffusion step is performed by generating either a continuous word embedding or a discrete mask token where the latent transition is modeled by a Gaussian-Dirac mixture distribution. The easy-first text generation is then implemented and strengthened via a contrastive loss to disentangle the generation between simple and complex tokens. A contrastive mixture diffusion model is accordingly exploited by minimizing a variational bound of negative log likelihood, which is regularized by aligning the denoising network posterior with the Gaussian-Dirac posterior in each diffusion transition based on an approximate Kullback-Leibler divergence. The experiments on text paraphrasing and other tasks demonstrate the effectiveness and efficiency of sentence generation by using the proposed method where the easy-first strategy in generation behavior is illustrated.