Parzen-Window Density Estimation

Emanuel Parzen [125] invented this approach in the early 1960s, providing a rigorous mathematical analysis. Since then, it has found utility in a wide spectrum of areas and applications such as pattern recognition [48], classification [48], image registration [170], tracking, image segmentation [32], and image restoration [9].

Parzen-window density estimation is essentially a data-interpolation technique [48,171,156]. Given an instance of the random sample, ${\bf x}$, Parzen-windowing estimates the PDF $P(X)$ from which the sample was derived. It essentially superposes kernel functions placed at each observation or datum. In this way, each observation $x_i$ contributes to the PDF estimate. There is another way to look at the estimation process, and this is where it derives its name from. Suppose that we want to estimate the value of the PDF $P(X)$ at point $x$. Then, we can place a window function at $x$ and determine how many observations $x_i$ fall within our window or, rather, what is the contribution of each observation $x_i$ to this window. The PDF value $P (x)$ is then the sum total of the contributions from the observations to this window. The Parzen-window estimate is defined as

$$P (x) = \frac {1} {n} \sum_{i=1}^{n} \frac {1} {h_n^d} K \Bigg( \frac {x - x_i} {h_n} \Bigg),$$ (53)

where $K (x)$ is the window function or kernel in the $d$-dimensional space such that

$$\int_{\Re^d} K (x) dx = 1,$$ (54)

and $h_n > 0$ is the window width or bandwidth parameter that corresponds to the width of the kernel. The bandwidth $h_n$ is typically chosen based on the number of available observations $n$. Typically, the kernel function $K (\cdot)$ is unimodal. It is also itself a PDF, making it simple to guarantee that the estimated function $P (\cdot)$ satisfies the properties of a PDF. The Gaussian PDF is a popular kernel for Parzen-window density estimation, being infinitely differentiable and thereby lending the same property to the Parzen-window PDF estimate $P(X)$. Using (2.53), the Parzen-window estimate with the Gaussian kernel becomes

$$P (x) = \frac {1} {n} \sum_{i=1}^{n} \frac {1} {(h \sqrt {2 \pi})^d} \exp \Bigg( - \frac {1} {2} \bigg( \frac {x - x_i} {h} \bigg)^2 \Bigg),$$ (55)

where $h$ is the standard deviation of the Gaussian PDF along each dimension. Figure 2.5 shows the Parzen-window PDF estimate, for a zero-mean unit-variance Gaussian PDF, with a Gaussian kernel of $\sigma = 0.25$ and increasing sample sizes. Observe that with a large sample size, the Parzen-window estimate comes quite close to the Gaussian PDF.

Figure 2.5: The Parzen-window PDF estimate (dotted curve), for a Gaussian PDF (solid curve) with zero mean and unit variance, with a Gaussian kernel of $\sigma = 0.25$ and a sample size of (a) 1, (b) 10, (c) 100, and (d) 1000. The circles indicate the observations in the sample.