reference: Very intuitive example shown in this blog
https://wiseodd.github.io/techblog/2016/12/21/forward-reverse-kl/
Let our true distribution defined as
, and the approximate distribution as
.
Forward KL:
Discussion in different cases:
- If P(x)=0, then log term can be igored so that the Q(x) can be any shape when P(x)=0. Q(x) is able to assign any probabilities when P(x)=0)
- If P(x)>0, then the log term have effect during optimization so that Q(x) assign probabilities as close as possible to P(X) when P(x)>0.
- The following graph is a specific optimal optimization for Forward KL.
Reverse KL:
- If Q(x)=0, log term can be ignored so that Q(x) able to assign 0 probabilities to P(x)>=0.
- If Q(x) > 0, log term is taken into account in optimization step so that Q(x) assign probabilites as close as possible to P(X) when P(x)>0
- The following graph is a specific optimal optimization for Reverse KL.