Conditional Entropy

The conditional entropy of an RV $X$ given RV $Y$ is a measure of the uncertainty remaining in $X$ after $Y$ is observed [154]. It is defined as the weighted average of the entropies of the conditional PDFs of $X$ given the value of $Y$, i.e.,

$$h (X \vert Y) = \int_{\mathcal{S}_Y} P (y) h (X \vert y) dy.$$ (65)

Thus, functionally-dependent RVs will have minimal conditional entropy, i.e., $- \infty$. This is because, for a given $y$, the value $x$ is exactly known thereby causing $h (X \vert y)= 0, \forall y$. For independent RVs, however,

$$h (X \vert Y) = \int_{\mathcal{S}_Y} P (y) h (X \vert y) dy$$

$$= \int_{\mathcal{S}_Y} P (y) h (X) dy$$

$$= h (X).$$ (66)