How Diffusion Models Work provides a technical and practical introduction to diffusion models, one of the fundamental generative AI architectures used for image generation. The course moves beyond using pre-trained models and APIs by developing a diffusion model from its underlying principles, including the forward diffusion process, noise prediction, iterative denoising, conditioning, and accelerated sampling.

The central idea is to learn how data can be progressively transformed into noise and how a neural network can learn to reverse this process. During training, noise is added to images at different timesteps and a neural network is trained to predict that noise. During generation, the process starts from random noise and repeatedly applies the trained model to progressively construct a new image.

A U-Net architecture provides the neural network used for noise prediction. The course demonstrates how timestep information and contextual information can be incorporated into the network, allowing the diffusion process to become conditional and therefore control what type of image is generated.

The course also examines the computational cost of iterative diffusion. Standard diffusion sampling may require many denoising steps, while Denoising Diffusion Implicit Models (DDIM) provide a more efficient sampling strategy that can substantially reduce the number of steps required to generate an image.

Overall, the course provides an understanding of diffusion models from the perspective of both their mathematical process and their implementation. By constructing and training the model rather than simply using an existing image-generation service, the course provides a technical foundation for understanding how contemporary diffusion-based generative image systems operate. (?deeplearning.ai)

Course topics

Intuition

The introductory section develops the intuition behind forward and reverse diffusion. Training images are progressively corrupted by adding Gaussian noise over a sequence of timesteps. The generative process reverses this concept: starting from random noise, a trained neural network progressively predicts and removes noise until a structured image emerges.

Sampling

Sampling demonstrates how a trained diffusion model generates new images. Starting from random noise, the model repeatedly predicts the noise present at each timestep and progressively denoises the sample. This iterative process transforms an initially random distribution into images following the distribution learned from the training data.

Neural Network

The course introduces a U-Net as the neural network architecture responsible for predicting noise. The network processes noisy images together with information about the current diffusion timestep, learning representations at multiple spatial resolutions and using them to estimate the noise that should be removed.

Training

The training section demonstrates how a diffusion model learns the reverse diffusion process. Clean training images are sampled at different timesteps, appropriate levels of noise are added, and the neural network learns to predict the introduced noise. By repeating this process across the dataset and diffusion timesteps, the network learns how to progressively recover image structure from noisy inputs.

Controlling

The diffusion model is extended with context or conditioning, allowing the generation process to be controlled. Contextual information is provided alongside the noisy image and timestep, enabling the network to generate images belonging to selected categories rather than producing entirely uncontrolled samples.

Speeding Up

The final technical section introduces DDIM sampling as a more efficient alternative to the standard diffusion sampling process. By reducing the number of denoising steps required during inference, DDIM can substantially accelerate image generation while maintaining useful output quality.

Topics and techniques

• Diffusion Models – generative modelling through progressive noise addition and removal.
• Forward and Reverse Diffusion – transforming images into noise and reconstructing data from noise.
• Noise Prediction – training neural networks to estimate the noise contained in an image.
• U-Net Architecture – multi-resolution neural network for diffusion-based image generation.
• Sampling – iteratively generating images from random noise.
• Conditioning – controlling generated output through contextual information.
• DDIM – accelerating the iterative sampling process.

Practical exercises

The practical exercises use Python and Jupyter notebooks to build and experiment with a diffusion model rather than relying on a pre-trained image-generation API. The labs progressively cover sampling, training, neural-network-based noise prediction, contextual conditioning, and accelerated generation. 

The implementation generates small 16×16 image sprites, providing a computationally manageable environment in which the complete diffusion process can be examined. The exercises train a U-Net to predict noise, use the resulting model to iteratively generate new images, introduce contextual information to control the generated content, and finally implement DDIM sampling to significantly reduce generation time. 

Together, the exercises provide an end-to-end implementation of the diffusion workflow—from adding noise to training data and training the noise-prediction network to generating controlled images from random noise and optimizing the sampling process.