CS492D: Diffusion Models and Their
Applications
Denoising Diffusion Implicit Models 2 /
Classifier-Free Guidance
LECTURE 6
MINHYUK SUNG
Fall 2024
KAIST
CS492(D): Diffusion Models and Their Applications (Fall 2024)
RECAP: DDPM vs. DDIM
𝛼! 1 − 𝛼*!"# 𝛼*!"#𝛽! 1 − 𝛼*!"#
𝑞 𝒙!"#|𝒙! , 𝒙$ = 𝒩 𝒙! + 𝒙$ , 𝛽! 𝐈
1 − 𝛼*! 1 − 𝛼*! 1 − 𝛼*!
For each time step 𝑡 = 𝑇, … , 1, repeat:
#
1. Compute 𝒙$|! = 𝒙! − 1 − 𝛼*! 𝛆1 ( (𝒙! , 𝑡)
&!
'
'! #"&
'!"# & !"# )!
'
2. Compute 𝜇5 = 𝒙! + 𝒙$|! .
#"&'! #"&
'!
3. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
#"&
'!"#
4. Compute 𝒙!"# = 𝜇5 + 𝛽! 𝒛! .
#"&
'!
CS492(D): Diffusion Models and Their Applications (Fall 2024) 2
RECAP: DDPM vs. DDIM
𝒙! − 𝛼*! 𝒙$
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝛼*!"#𝒙$ + 1 − 𝛼*!"# − 𝜎!+ ; , 𝜎!+𝐈
1 − 𝛼*!
For each time step 𝑡 = 𝑇, … , 1, repeat:
#
1. Compute 𝒙$|! = 𝒙! − 1 − 𝛼*! 𝛆1 ( (𝒙! , 𝑡)
&!
'
& ! 𝒙$|!
𝒙! " '
2. Compute 𝜇5 = 𝛼*!"#𝒙$|! + 1 − 𝛼*!"# − 𝜎!+ ; .
#"&
'!
3. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
4. Compute 𝒙!"# = 𝜇5 + 𝜎! 𝒛! .
CS492(D): Diffusion Models and Their Applications (Fall 2024) 3
RECAP: DDPM vs. DDIM
Q. What is 𝜇! with respect to 𝒙! and 𝛆! for both DDPM and
DDIM cases?
CS492(D): Diffusion Models and Their Applications (Fall 2024) 4
RECAP: DDPM vs. DDIM
1 1 − 𝛼! 1 − 𝛼*!"#
𝑞 𝒙!"#|𝒙! , 𝒙$ = 𝒩 𝐱! − 𝛆! , 𝛽! 𝐈
𝛼! 1 − 𝛼*! 1 − 𝛼*!
For each time step 𝑡 = 𝑇, … , 1, repeat:
# #"'!
1. Compute 𝜇5 = 𝐱! − 𝛆1 (𝒙 , 𝑡) .
'! '! ( !
#"&
2. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
#"&
'!"#
3. Compute 𝒙!"# = 𝜇5 + #"&
'!
𝛽! 𝒛! .
CS492(D): Diffusion Models and Their Applications (Fall 2024) 5
RECAP: DDPM vs. DDIM
𝒙! − 1 − 𝛼*! 𝛆!
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆! , 𝜎!+𝐈
𝛼*!
For each time step 𝑡 = 𝑇, … , 1, repeat:
'! 𝛆- & (𝒙! ,!)
𝒙! " #"&
1. Compute 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆1 ( (𝒙! , 𝑡).
&!
'
2. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
3. Compute 𝒙!"# = 𝜇5 + 𝜎! 𝒛! .
CS492(D): Diffusion Models and Their Applications (Fall 2024) 6
RECAP: DDIM
Q. Check out that DDPM is a special case of DDIM when
" "
1 − 𝛼)!#$
𝜎! = 𝜎!! = 𝛽! .
1 − 𝛼)!
CS492(D): Diffusion Models and Their Applications (Fall 2024) 7
RECAP: DDIM
Controlling Stochasticity
% !"#
$#&
Let 𝜎! = 𝜂 𝛽! .
$#%
&!
• 𝜂 = 0: Deterministic process.
For the same 𝒙 ' ~𝒩 𝟎, 𝐈 , we always obtain the same 𝒙( .
• 𝜂 = 1: Same as DDPM.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 8
RECAP: DDIM
Q. What is the maximum 𝜎! ?
CS492(D): Diffusion Models and Their Applications (Fall 2024) 9
RECAP: DDIM
𝒙! − 1 − 𝛼*! 𝛆!
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆! , 𝜎!+𝐈
𝛼*!
A. 1 − 𝛼#!"# − 𝜎!$ must be non-negative.
The maximum 𝜎! is 1 − 𝛼#!"# .
CS492(D): Diffusion Models and Their Applications (Fall 2024) 10
RECAP: Accelerating Sampling Process
The reverse process for a sub-sequence time steps 𝜏 =
[𝜏$ , 𝜏" , … , 𝜏) ]:
𝑝* 𝒙+ = 𝑝 𝐱 ' : 𝑝* 𝐱 +$"# |𝐱 +$
!,$
CS492(D): Diffusion Models and Their Applications (Fall 2024) 11
RECAP: Accelerating Sampling Process
• As smaller time steps are used, the quality of the generated
data can worsen.
• However, quality degradation is mitigated when the DDIM
reverse process becomes more deterministic.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 12
RECAP: DDIM Summary
• DDPM is a special case of DDIM.
• No need to retrain the noise predictor.
• The reverse process becomes deterministic when 𝜎! = 0.
• Quality degradation with fewer time steps is mitigated
when 𝜎! is small.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 13
Song et al., Denoising Diffusion Implicit Models, ICLR 2021.
RECAP: Accelerating Sampling Process
• When 𝜂 = 0 (deterministic),
• the quality does not get bad too much even when S = 10.
• The quality when S = 1000 is even better than DDPM!
CS492(D): Diffusion Models and Their Applications (Fall 2024) 14
Assignment 2
• Assignment 2 is about changing DDPM to DDIM.
• It builds on your solution from Assignment 1. You can start
Assignment 2 when you finish Assignment 1!
CS492(D): Diffusion Models and Their Applications (Fall 2024) 15
Classifier-Free Guidance (CFG)
Ho and Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 16
Ho and Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop.
Classifier-Free Guidance (CFG)
How to make a conditional diffusion model?
Dog
CS492(D): Diffusion Models and Their Applications (Fall 2024) 17
Classifier-Free Guidance (CFG)
Modify the noise prediction network to take the condition
(label) 𝑦 as an additional input:
𝛆> * 𝒙! , 𝑡 → 𝛆> * 𝒙! , 𝑦, 𝑡
CS492(D): Diffusion Models and Their Applications (Fall 2024) 18
Classifier-Free Guidance (CFG)
We can jointly train the conditional and unconditional
diffusion models by introducing a null label 𝜙.
𝛆> * 𝒙! , 𝜙, 𝑡 : Noise prediction without condition.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 19
Classifier-Free Guidance (CFG)
How can we enhance the conditioning?
We can extrapolate the conditional noise 𝛆> * 𝒙! , 𝑦, 𝑡 from
the null-condition noise 𝛆> * 𝒙! , 𝜙, 𝑡 :
𝛆! * 𝒙! , 𝑦, 𝑡 = 1 + 𝜔 𝛆> * 𝒙! , 𝑦, 𝑡 − 𝜔>𝛆* 𝒙! , 𝜙, 𝑡
where 𝜔 ≥ 0.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 20
Classifier-Free Guidance (CFG)
How can we enhance the conditioning?
We can extrapolate the conditional noise 𝛆> * 𝒙! , 𝑦, 𝑡 from
the null-condition noise 𝛆> * 𝒙! , 𝜙, 𝑡 :
𝛆! * 𝒙! , 𝑦, 𝑡 = 1 + 𝜔 𝛆> * 𝒙! , 𝑦, 𝑡 − 𝜔>𝛆* 𝒙! , 𝜙, 𝑡
where 𝜔 ≥ 0.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 21
Meng et al., On Distillation of Guided Diffusion Models, CVPR 2023.
Classifier-Free Guidance (CFG)
CS492(D): Diffusion Models and Their Applications (Fall 2024) 22
Classifier-Free Guidance (CFG)
• (+) Very easy to implement.
• (+) Versatile; can be used not only for labels but also for
any additional information (e.g., text descriptions).
• (−) The noise predictor needs to be evaluated twice in the
generation process.
CS492(D): Diffusion Models and Their Applications (Fall 2024) 23
Assignment 2
Implement CFG based on the DDIM implementation.
Wildlife
Cat
Dog
CS492(D): Diffusion Models and Their Applications (Fall 2024) 24
