Probability Theory

Probability theory is concerned with the analysis of random, or chance, phenomena. Such random phenomena, or processes, occur all the time in nature in one form or the other. Pierre Simon de Laplace established the theory of probability in the year 1812, after publishing the Theorie Analytique des Probabilites. The theory now pervades a wide spectrum of scientific domains including thermodynamics, statistical mechanics, quantum physics, economics, information theory, machine learning, and signal processing.

Probability theory deals with random experiments, i.e., experiments whose outcomes are not certain. The set of all possible outcomes of an experiment is referred to as the sample space, denoted by $\Omega$, for that experiment. For instance, let us consider the experiment of picking up a random pixel from an $N$ $\times$ $N$ pixels digital image. The sample space is all possible coordinates of the grid image domain, i.e., $\Omega = \{ \{ 0,1,2,\ldots,N-1 \} \times \{ 0,1,2,\ldots,N-1 \} \}$.

An event is a collection of the outcomes in the sample space, or a subset of the sample space. Consider an event $A$ in the sample space $\Omega$. The probability $P (A)$ of the event $A$ is the chance that the event will occur when we perform the random experiment. The probability is actually a function $P (\cdot)$ that satisfies the following properties:

$$\forall A, P (A) \geq 0,$$ (1)

$$P (\Omega) = 1,$$ (2)

$$P (A \cup B) = P (A) + P (B), \forall A \mathrm { and } B \mathrm { such that } A \cap B = \phi,$$ (3)

where $\phi$ is the empty set.