Inverse Problem: KL-Divergence Optimality

We want the corrupted-signal PDF $\hat P_C ({\bf\tilde z})$, derived from the uncorrupted-signal model $P_\mathcal{U} ({\bf Z})$, to match the Markov PDF $P_C ({\bf\tilde z})$ estimated from the observed corrupted data. We propose the Kullback-Leibler (KL) divergence as a measure of the discrepancy between the two PDFs. If we define $\Theta = \{ {\bf z}_u \}_{u \in \mathcal{U}}$, then we want to find

$$\{ \Theta^*, \sigma^* \} = \mathop{\mbox{argmin }}_{\Theta, \sigma} \mathop{\mbox{KL }}(P_C \parallel \hat P_C)$$

$$= \mathop{\mbox{argmin }}_{\Theta, \sigma} E_{P_C} \Bigg[ \log \frac {P_C} {\hat P_C} \Bigg]$$

$$= \mathop{\mbox{argmin }}_{\Theta, \sigma} E_{P_C} \Big[ \log P_C - \log \hat P_C \Big]$$

$$= \mathop{\mbox{argmax }}_{\Theta, \sigma} E_{P_C} \Big[ \log \hat P_C \Big]$$

$$\approx \mathop{\mbox{argmax }}_{\Theta, \sigma} \sum_{t \in \mathcal{T}} \log \hat P_C ({\bf\tilde z}_t)$$

$$= \mathop{\mbox{argmax }}_{\Theta , \sigma} \sum_{t \in \mathcal{T}... ...igg( \sum_{u \in \mathcal{U}} G ({\bf\tilde z}_t - {\bf z'}_u, \Psi'_u) \Bigg).$$ (122)

What we have here is a ML optimization problem. ML estimation procedures, however, are well known to need regularization to reduces the chances of the optimization getting stuck in local maxima and to produce effective estimates, e.g., the classic method-of-sieves regularization by Grenander [68]. We propose to regularize the ML estimation by fixing the value of $\sigma$ beforehand. The enforcement of this regularization is similar in spirit to that used by Geman and Hwang [63] for nonparametric density estimation.

We can produce an effective optimal estimate for $\sigma$ as follows. We first find a ML-based estimate $\tilde \sigma$ for the nonparametric Markov PDF of the corrupted observed sample $\{ {\bf \tilde z}_t \}_{t \in \mathcal{T}}$ (details in [9,5]). We know that a significant fraction of intensities in the image are much larger than the noise level $\sigma _R$ where the Rician noise model is close to an additive independent Gaussian noise model. Therefore, we approximate

$$\sigma^* \approx \sqrt {\tilde \sigma^2 - \sigma_R^2}.$$ (123)

Fixing this $\sigma$ value, we subsequently obtain an optimal ML estimate for the set $\Theta$ relying on the EM algorithm. We have found that this approximation for $\sigma$ works effectively in practice.