PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

Lecture Notes-1
Introduction to Probability
Engineering systems are designed to operate well in the face of uncertainty of
characteristics of components and operating conditions. In some case, uncertainty is introduced
in the operations of the system, on purpose. Understanding how to model uncertainty and how to
analyze its effects is an essential part of an engineer's education. Randomness is a key element of
all systems we design. Communication systems are designed to compensate for noise. Internet
routers are built to absorb traffic fluctuations. Building must resist the unpredictable vibrations of
an earthquake. The power distribution grid carries an unpredictable load. Integrated circuit
manufacturing steps are subject to unpredictable variations. Searching for genes is looking for
patterns among unknown strings. What should you understand about probability? It is a complex
subject that has been constructed over decades by pure and applied mathematicians. Thousands
of books explore various aspects of the theory. How much do you really need to know and where
do you start?
To start with, the first thing that should be done is to define what is mean by random signal. A
Random signal is a time waveform that can be characterized only in some probabilistic manner.
In general it can be either desired or undesired waveform. Undesirable waveforms will appear in
almost all communication systems. In communication systems undesired signal is termed as
Noise. Some examples of effect of random signals include background hiss in radio receiver,
snow in Television system, randomly generated sea sounds in SONAR systems. Due to the
random nature there is a need to study the phenomena of these signals in probabilistic manner.
All the random signals exist in nature vary with time, there comes the concept of random
processes. Here in this course we first introduce the mathematical tool probability and describe
how this tool will help to characterize the random signals.

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

Contributors to Probability:
Adrien Marie LEGENDRE, 1752-1833
Best use of inaccurate measurements: Method of Least Squares.
Jacob BERNOULLI, 1654-1705
Making sense of uncertainty and chance: Law of Large Numbers.
Abraham DE MOIVRE, 1667 1754
Bounding the probability of deviation: Normal distribution.
Thomas SIMPSON, 1710-1761
A First attempt at posterior probability.
Thomas BAYES, 1701-1761
The importance of the prior distribution: Bayes' rule.
Pierre Simon LAPLACE, 1749-1827
Posterior distribution: Analytical methods.
Carl Friedrich GAUSS, 1777 1855
Least Squares Estimation with Gaussian errors.
Andrei Andreyevich MARKOV, 1856 1922
Markov Chains
Andrei Nikolaevich KOLMOGOROV, 1903-1987
Kolmogorov was one of the most prolific mathematicians of the 20th century. He made
fundamental contributions to dynamic systems, ergodic theory, the theory of functions and
functional analysis, the theory of probability and mathematical statistics, the analysis of
turbulence and hydrodynamics, to mathematical logic, to the theory of complexity, to geometry,
and topology. In probability theory, he formulated probability as part of measure theory and
established some essential properties such as the extension theorem and many other fundamental
results.

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Andrei Marei Legendre

Thomas Bayes

Jacob Bernoulli

Pierre Simon Laplace

Abraham De Moivre

Markov

Unit-I

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

Set Theory
Prior to the introduction of the probability, it is necessary to develop certain concepts on set
theory.
Set definitions:
A set is a collection of objects, which are the elements of the set. If A is a set and x is an
x A
x A
element of B, we write
. If x is not an element of A, we write
. A set can have no

elements, in which case it is called the empty set or null set, denoted by . A set can contain a
finite or infinite number of elements. If a set A contains an infinite number of elements which
can be enumerated in a list (there exists a bijection between the elements of A and the set of
natural numbers), we say that A is countably infinite. If the elements of A cannot be enumerated
in a list we say that A is uncountable. If every element of a set A is also an element of a set B, we
A B
BA
A B
B A
say that A is a subset of B, and we write
or
. If
and
, the two sets are
equal, and we write A = B. We denote by S the universal set, which contains all objects of
interest in a particular context. All sets are subsets of universal set.
Set Operations:
Venn diagram
It is a geometric representation of elements of sets by closed-plane figures. The universal set S is
represented by rectangle, B is subset of A, C is disjoint from A and B as shown in Fig 1.1.

Fig 1.1: Illustration of subsets and mutually exclusive sets

Equality and difference

A B

Two sets A and B are equal if all elements in A are present in B and Vice versa. i.e if
and
B A
,for equal sets we write A=B. The difference of two sets A and B, denoted by A-B, is the
set containing all elements of A that are not present in B.
Union and Intersection

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

C A B

The union of two sets A and B is written as

.It is the set of all elements of A or B or
both. The union is sometimes called the sum of two sets.
D AB
The intersection of two sets A and B is written as
.It is the set of all elements common
to both A and B. The intersection is sometimes called the product of two sets. Fig 1.2 illustrates
the Venn diagram associated with union and intersection of sets.

Fig 1.2: Illustration of Union and intersection of sets

Complement

The complement of a set

ASA
Thus,

A
, denoted by , is the set of all elements not in A.

Algebra of sets

Commutative law

Associative law

Distributive law

A B B A
A B B A
A ( B C ) ( A B) ( A C )
A ( B C ) ( A B) ( A C )
( A B) C A ( B C ) A B C
( A B) C A ( B C ) A B C

De Morgans Laws
It states that the complement of a Union (Intersection) of two sets A and B equals the intersection
A
B
(Union) of the complements
and .Thus

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

( A B) A B
( A B) A B

Duality principle

It states that if in an identity we replace unions by intersections, intersections by unions,

and

by

by

, then the identity is preserved.

Probability Definitions and Axioms

We shall introduce some of the basic concepts of probability theory by defining some
terminology relating to random experiments (i.e., experiments whose outcomes are not
predictable).
Basic Terminology
Def: Outcome
The end result of an experiment. For example, if the experiment consists of throwing a die, the
outcome would be anyone of the six faces, F1,........,F6.
Def: Random experiment
An experiment whose outcomes are not known in advance. (e.g. tossing a coin, throwing a die,
measuring the noise voltage at the terminals of a resistor etc.)
Def: Random event
A random event is an outcome or set of outcomes of a random experiment that share a common
attribute. For example, considering the experiment of throwing a die, an event could be the 'face
F1 ' or 'even indexed faces' (F2,F4,F6). We denote the events by upper case letters such as A, B
or A1, A2.
Def: Sample space
The sample space of a random experiment is a mathematical abstraction used to represent all
possible outcomes of the experiment. We denote the sample space by S. Each outcome of the
experiment is represented by a point in S and is called a sample point. We use s (with or without
a subscript), to denote a sample point. An event on the sample space is represented by an
appropriate collection of sample point(s).
Def: Mutually exclusive (disjoint) events

PROBABILITY THEORY AND STOCHASTIC PROCESSES

Unit-I

Two events A and B are said to be mutually exclusive if they have no common elements (or
outcomes).Hence if A and B are mutually exclusive, they cannot occur together.
Def: Occurrence of an event
An event A of a random experiment is said to have occurred if the experiment terminates in an
outcome that belongs to A.
Probability of an event
The probability of an event has been defined in several ways. Some of the most popular
definitions are: i) the relative frequency definition ii) the classical definition and iii) Axiomatic
definition.

Def 1: The relative frequency definition

Suppose that a random experiment is repeated n times. If the event A occurs nA times, then the
probability of A, denoted by P(A), is defined as
n
P( A) lim( A )
n n
(

nA
)
n

represents the fraction of occurrence of A in n trails.

n
( A)
n
For small values of n, it is likely that
will fluctuate quite badly. But as n becomes larger and
n
( A)
n
larger, we expect,
to tend to a definite limiting value. For example, let the experiment be
n
( A)
n
that of tossing a coin and A the event 'outcome of a toss is Head'. If n is the order of 100,
may not deviate from by more than, say ten percent and as n becomes larger and larger, we
n
( A)
n
expect
to converge to .
Def 2: The classical definition
The relative frequency definition given above has empirical flavor. In the classical approach, the
probability of the event A is found without experimentation. This is done by counting the total
number N of the possible outcomes of the experiment. If NA of those outcomes are favorable to
the occurrence of the event A, then

PROBABILITY THEORY AND STOCHASTIC PROCESSES

P( A)

Unit-I

NA
N

where it is assumed that all outcomes are equally likely!

Def 3: Axiomatic definition
To define each event on sample space S, we shall assign a nonnegative number called
Probability. Probability is therefore a function; it is a function of the events defined. We adopt the
notation P(A) for the probability of event A. The assigned probabilities are chosen so as to satisfy
three axioms. Let A be any event defined on a sample space S.
P( A) 0

Axiom 1:

P(S ) 1

Axiom 2:
N

n 1

n 1

P (U An ) P ( An )

Axiom 3:
infinite.

Am An
if

for all

mn

=1,2,N, with possibly