Information Theory

Several algorithms in this dissertation enforce optimality criteria based on fundamental information-theoretic concepts that help us analyze the functional dependence, information content, and uncertainty in the data. In this way, information theory forms an important statistical tool in the design of unsupervised adaptive algorithms. This section presents a brief review of the relevant key information-theoretic concepts.

In the 1920s, Bell Labs researchers Harry Nyquist [116] and Ralph Hartley [72] pioneered the mathematical analysis of the transmission of messages, or information, over telegraph. Hartley was the first to define a quantitative measure of information associated with the transmission of a set of messages over a communication channel. Building on some of their ideas, another Bell Labs researcher Claude E. Shannon first presented [154], in the year 1948, a concrete mathematical model of communication from a statistical viewpoint. This heralded the birth of the field of information theory. The principles underpinning the statistical theory have a universal appeal--virtually all practical systems process information in one way or the other--with information theory finding applications in a wide spectrum of areas such as statistical mechanics, business and finance, pattern recognition, data compression, and queuing theory [34,85].

Information theory deals with the problem of quantifying the information content associated with events. If an event has a probability of occurrence $p$, then the uncertainty or self-information associated with the occurrence of that event is $\log \Big( \frac {1} {p} \Big)$ [154]. Thus, the occurrence of a less-certain event ($p \ll 1$) conveys more information. The occurrence of events that are absolutely certain ($p = 1$), on the other hand, conveys no information.