Kullback-Leibler (KL) Divergence

The Kullback-Leibler (KL) divergence or relative entropy is a measure of mismatch between two PDFs $P(X)$ and $Q (X)$:

$$\mathop{\mbox{KL }}(P \parallel Q) = E_{P(X)} \Bigg[ \log \frac {P (X)} {Q (X)} \Bigg].$$ (67)

The KL divergence is always nonnegative. It is zero if and only if $P(X)$ and $Q (X)$ are exactly the same. It is not symmetric and does not follow the triangle inequality. Hence, it is not a true distance measure.