Chapter 2

Basic Concepts of Probability

Theory
ENCS6161 - Probability and Stochastic
Processes
Concordia University
Specifying Random Experiments
Examples of random experiments: tossing a coin,
rolling a dice, the lifetime of a harddisk.
Sample space: the set of all possible outcomes of a
random experiment.
Sample point: an element of the sample space S
Examples: S = {H, T }
S = {1, 2, 3, 4, 5, 6}
S = {t|1 < t < 10}
Event: a subset of a sample space A ⊆ S
A = {H}
A = {2, 4, 6}
A=∅

ENCS6161 – p.1/16
The Axioms of Probability
A probability measure is a set function P (·) that
satisfies the following axioms.
1. P (A) ≥ 0
2. P (S) = 1
3. If A ∩ B = ∅, then P (A ∪ B) = P (A) + P (B)
4. If A1 , A2 , · · · are events s.t Ai ∩ Aj = ∅ for all i 6= j
 ∞  P∞
then P ∪k=1 Ak = k=1 P (Ak )
Corollary 1: P (Ac ) = 1 − P (A)
Corollary 2: P (A) ≤ 1 ∀A
Corollary 3: P (∅) = 0

ENCS6161 – p.2/16
The Axioms of Probability
Corollary 4: If A1 · · · An are mutually exclusive, i.e.
Ai ∩ Aj = ∅, ∀i 6= j then P (∪nk=1 Ak ) = nk=1 P (Ak )
P

Corollary 5: P (A ∪ B) = P (A) + P (B) − P (A ∩ B)

A B

A∩B

Union bound: P (A ∪ B) ≤ P (A) + P (B)

ENCS6161 – p.3/16
Counting Sample Points
Permutation: n distinct objects, how many ways can
we arrange them?
n · (n − 1) · (n − 2) · · · 2 · 1 = n!
Selection with order: select k objects from n ≥ k
objects
n!
n · (n − 1) · · · (n − k + 1) = = n Pk
(n − k)!
Combination (selection without order): select k
objects from n ≥ k objects  
n n! n
Ck = =
(n − k)!k! k

ENCS6161 – p.4/16
Counting Sample Points
Example: choose a president and a treasurer from 50
students
1. no restrictions
2. A will serve only if he is president
3. B & C will serve together or not at all
4. D & E will not serve together

ENCS6161 – p.5/16
Conditional Probability
The conditional prob. of A given B has occurred is
defined by :

P (A ∩ B)
P (A|B) = , P (B) > 0
P (B)

Similarly,

P (A ∩ B)
P (B|A) = , P (A) > 0
P (A)

So

P (A ∩ B) = P (B)P (A|B) = P (A)P (B|A)

ENCS6161 – p.6/16
Bayes’ Rule
If B1 · · · Bn are mutually exclusive and ∪Bi = S, we
call these sets a partition of S.
Theorem of total probability
For any event A, if B1 , · · · , Bn is a partition of S,

P (A) = P (A ∩ B1 ) + P (A ∩ B2 ) + · · · + P (A ∩ Bn )
= P (A|B1 )P (B1 ) + P (A|B2 )P (B2 ) + · · · + P (A|Bn )P (Bn )

Bayes’ Rule
Let B1 , · · · , Bn be a partition of S , and P (A) ≥ 0, then
P (A ∩ Bj ) P (A|Bj )P (Bj )
P (Bj |A) = = n
P
P (A) k=1 P (A|Bk )P (Bk )

ENCS6161 – p.7/16
Bayes’ Rule
Example: binary communication system
1−ε
0 0
ε
P (X = 0) = P
X Y
P (X = 1) = 1 − P
ε
1 1
1−ε

1. Find P (Y = 0)
2. Find P (X = 0|Y = 0) and P (X = 1|Y = 0)

ENCS6161 – p.8/16
Bayes’ Rule
Solution: let A be event Y = 0, B0 be X = 0, B1 be
X=1
P (Y = 0) = P (A) = P (A|B0 )P (B0 ) + P (A|B1 )P (B1 )
= (1 − ε)P + ε(1 − P )

P (A ∩ B0 )
P (X = 0|Y = 0) = P (B0 |A) =
P (A)
P (A|B1 )P (B1 ) (1 − ε)P
= =
P (A) (1 − ε)P + ε(1 − P )
Similarly:
ε(1 − P )
P (X = 1|Y = 0) =
(1 − ε)P + ε(1 − P )

ENCS6161 – p.9/16
Bayes’ Rule
If P (X = 0|Y = 0) > P (X = 1|Y = 0), we decide
X = 0, i.e., if

(1 − ε)P > ε(1 − P ) ⇒ P > ε,

we decide X = 0.
Similarly, if P < ε, we decide X = 1.
A special case: if P = 0.5 and ε < 0.5 then

Y = 0 ⇒ we decide X = 0
Y = 1 ⇒ we decide X = 1

ENCS6161 – p.10/16
Bayes’ Rule
Example: 2% of people have on type of blood
disease. If a person has the disease and take a
blood test, with 96% probability, the result is positive
and with 4% probability, the result is negtive. If a
person without the disease takes a blood test, then
94% negative and 6% positive. Find

P (have disease|positive)

ENCS6161 – p.11/16
Bayes’ Rule
Solution: Let
D: a person has the disease D0 : no disease
B : blood test positive B 0 : blood test negative
Then P (D) = 2% and
P (B|D) = 0.96 P (B 0 |D) = 0.04
P (B|D0 ) = 0.06 P (B 0 |D0 ) = 0.94
Apply Bayes Rule:

P (D|B) = 0.246 = 24.6%

ENCS6161 – p.12/16
Bayes’ Rule
We can also use tree diagram.

0.96 B 0.02 ∗ 0.96 = 0.0192 (D ∩ B)

D
0.02 0.078 B
0.04 B’ 0.02 ∗ 0.04 = 0.0008 (D ∩ B0)

0.06 B 0.98 ∗ 0.06 = 0.0588 (D 0 ∩ B)

0.98 0.922 B’

D’ 0 0
0.94 B’ 0.98 ∗ 0.94 = 0.9212 (D ∩ B )

P (D ∩ B) 0.0192
P (D|B) = = = 0.246
P (B) 0.078

ENCS6161 – p.13/16
Bayes’ Rule
How to find P (D|BB)?

P (D ∩ BB)
P (D|BB) =
P (BB)
0.02 ∗ 0.96 ∗ 0.96
=
0.02 ∗ 0.96 ∗ 0.96 + 0.98 ∗ 0.06 ∗ 0.06
= 83.9%

ENCS6161 – p.14/16
Independence of Events
A and B are independent iff

P (A ∩ B) = P (A)P (B)

If A and B are independent

P (A ∩ B)
P (A|B) =
P (B)
P (A)P (B)
=
P (B)
= P (A)

Similarly
P (B|A) = P (B)

ENCS6161 – p.15/16
Independence of Events
Events A1 , A2 , · · · , An are independent if

P (Ai1 ∩ Ai2 · · · ∩ Aik ) = P (Ai1 )P (Ai2 ) · · · P (Aik )

where1 ≤ i1 < i2 · · · < ik ≤ n

Qn
A special case P (A1 ∩ · · · ∩ An ) = i=1 P (An )
In general if events are not necessarily independent

P (A1 ∩ A2 · · · ∩ An ) = P (A1 )P (A2 |A1 )P (A3 |A1 ∩ A2 ) · · ·

· · · P (An |A1 ∩ · · · ∩ An−1 )

ENCS6161 – p.16/16