Mutual Information

The mutual information between two RVs $X$ and $Y$ is a measure of the information contained in one RV about another [154]:

$$I (X, Y) = \int_{\mathcal{S}_X} \int_{\mathcal{S}_Y} P (x) P (y) \log \frac {P (x,y)} {P (x) P (y)} dx dy.$$ (68)

Rewriting $I (X, Y) = h (X) - h (X \vert Y) = h (Y) - h (Y \vert X)$ allows us to interpret mutual information as the amount of uncertainty reduction in $h (X)$ when $Y$ is known, or vice versa. Statistically-independent RVs have zero mutual information. We can see mutual information as the KL divergence between the joint PDF $P (X,Y)$ and the individual PDFs $P(X)$ and $P (Y)$. For independent RVs, i.e., when $P (X, Y) = P (X) P (Y)$, the mutual information is zero. The notion of mutual information extends to $N$ RVs and is termed multi information [162]:

$$I (X_1, \ldots, X_N) = \int_{\mathcal{S}_{X_1}} \ldots \int_{\mathcal{S}_{X_N}} P (x_1, ... ..._N) \log \frac {P (x_1, \ldots, x_N)} {P (x_1) \ldots P (x_N)} dx_1 \ldots dx_N$$

$$= \sum_{i=1}^{N} h (X_i) - h (X_1, \ldots, X_n).$$ (69)