Diffusion

Add Gaussian noise and then reverse

Forward diffusion process:

• Gradually adds noise to input
  Reverse denoising process:
• Learns to generate data by denoising
  

Training Objective

Network structure

Diffusion models often use U-Net architectures with ResNet blocks and Self-Attention Layer to represent ${ϵ}_{\theta }(x_t,t)$
Time representation: Sinusoidal positional embeddings or random Fourier features

• Time features are fed to the residual blocks using either simple spatial addition or using adaptive group normalization layers
  

Latent Diffusion Models

Map data into compressed latent space
Train diffusion model efficiently in latent space

Advantages

1. Compressed latent space: Train diffusion model in lower resolution
   1. Computationally more efficient
2. Regularized smooth / compressed latent space
   1. Easier task for diffusion model and faster sampling
3. Flexibility
   1. Autoencoders can be tailored to data

Condition information is fed into the latent diffusion model by cross-attention
Query:

• Visual features from U-Net
• Key and value: text features

Compression and Encoding

Diffusion Model

Latent Diffusion Model

Diffusion Transformers

• Replace U-Net with Transformers