Lecture01 02 Intro Probability Theory

CSE801 Analysis of Stochastic
Systems
Welcome and Introduction
Dr. Muhammad Usman Ilyas

School of Electrical Engineering & Computer Science (SEECS)
National University of Sciences & Technology (NUST)
Copyright Syed Ali Khayam 2009
Course Information
Lecture Timings:
My Office:
Room # A-312
Office Hours
Tuesday: 5:30pm-7:20pm
Thursday: 5:30pm-6:20pm
Thursdays, 5:00-5:30pm, or by appointment.

usman.ilyas@seecs.edu.pk
The course will be managed through LMS
www.lms.nust.edu.pk
Facebook group
The lecture notes are designed and developed by Dr. Ali Khayam.
Timetable (MS EE-Telecom & Comp Networks)

Class Room # 5
Monday
TIME / DAYS
5:30pm-6:20pm
6:30pm-7:20pm
7:30pm-8:20pm
8:30pm-9:20pm
course Code
Adv. Computer
Networks
Adv. Digital
Commnication
Tuesday
Wednesday
Thursday
Stochastic Systems
Stochastic Systems
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Subject
Instructor
Adv. Computer
Networks
Friday
Saturday
Adv. Digital
Commnication
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Credit Hours
MS EE(Digital System and Signal Processing)-5

CSE 801
Stochastic Systems
Dr. Shahzad Younis
3+0
EE 831
Advanced Digital
Signal Processing
Dr. Amir Ali Khan
3+0
EE 823
Advanced Digital
System Design
Dr. Rehan Hafiz
3+0
MS EE(Telecommnication & Computer Networks)-5

CSE-801
Stochastic Systems
Dr. Usman Illyas
3+0
CSE-820
Adv. Computer
Networks
Dr. Junaid Qadir
3+0
Adv. Digital
Commnication
Dr. Rizwan Ahmed
3+0
EE-851
Manager-PG
Academic Coord Branch
(Iftikhar Ahmed)
Sep 2013
Timetable (MS EE-Power & Control)

MS EE(Power & Control )-1
First Semester (9 Sep 2013 - 10 Jan 2014)
Class Room # 7
TIME / DAYS
5:30pm-6:20pm
Monday
Tuesday
Wednesday
Power Electronics
and
Power Electronics and Stochastic Systems Electric Drives
Electric Drives
CR #5
6:30pm-7:20pm
Linear Control
Systems
Thursday
Stochastic
Systems
CR #5
Linear Control
Systems
Friday
Saturday
Library/Make-up Library/Make-up
Class/Seminar Class/Seminar
7:30pm-8:20pm
Library/Make-up
Class/Seminar
8:30pm-9:20pm
course Code
Subject
Library/Make-up
Library/Make-up
Class/Seminar Library/Make-up Class/Seminar
Class/Seminar
Instructor
Credit Hours
MS-EE(RF & MW )-4
EE-871
Power Electronics and Dr. Syed Raza

Electric Drives
Kazmi
Linear Control
Dr. Ammar
Systems
Hassan
CSE-801
Stochastic Systems
EE-901
Dr. Usman ilyas
3+0
3+0
3+0
4
Textbooks
Probability & Random Processes

for Electrical Engineers, 2nd or 3rd
ed.
Albert Leon-Garcia
Introduction to Probability Models,

9th ed.
Sheldon M. Ross
Elements of Information Theory
Thomas M. Cover and Joy

A. Thomas
Chaos Theory Tamed
Garnett P. Williams
Course Outline
Syllabus
Introduction to Probability Theory

Random Variables
Limits and Inequalities
Central Limit Theorem
Application Area: Information Theory
Stochastic Processes
Prediction and Estimation
Markov Chain and Processes (time permitting)
Application Area: Chaos Theory
Grading
(subject to change)
Final Exam:
Midterm Exam:
Quizzes:
Homework Assignments:
45%
30%
15%
10%
Policies
Quizzes will be unannounced
Late homework submissions will be accepted for up to 24 hours

with a 50% penalty.
Strong disciplinary action will be taken in case of plagiarism or

cheating in exams, homework or quizzes.
Attendance:
Will be taken at the beginning of the class.
The current rules of the school will be followed (75%
minimum requirement).
What will we cover in this lecture?
This lecture is intended to be an introduction to elementary

probability theory
We will cover:
Random Experiments and Random Variables

Axioms of Probability
Mutual Exclusivity
Conditional Probability
Independence
Law of Total Probability
Bayes Theorem
Definition of Probability
Probability:
1 : the quality or state of being possible
2 : something (as an event or circumstance) that is possible
3 : the ratio of the number of outcomes in an exhaustive set of

equally likely outcomes that produce a given event to the
total number of possible outcomes, the chance that a given
event will occur
We will revisit these definitions in a little bit
10
Definition of a Random Experiment
A random experiment comprises of:
A procedure
An outcome
Procedure
(e.g., flipping a coin)
Outcome
(e.g., the value
observed [head, tail] after
flipping the coin)
Sample Space
(Set of All Possible
Outcomes)
11
Definition of a Random Experiment:

Outcomes, Events and the Sample Space
An outcome cannot be further decomposed into other outcomes

{s1 = the value 1}, , {s6 = the value 6}
An event is a set of outcomes that are of interest to us

A = {s: such that s is an even number}
The set of all possible outcomes, S, is called the sample space

S = {s1, s2, s3, s4, s5, s6}
12

s1
s5
s6
s4
s2
s3
13

Example of a Random Experiment:
Experiment: Roll a fair dice once and record the

number of dots on the top face
S = {1, 2, 3, 4, 5, 6}
A = the outcome is even = {2, 4, 6}
B = the outcome is greater than 4 = {5, 6}
14
Axioms of Probability
Probability of any event A is non-negative:

Pr{A} 0
The probability that an outcome belongs to the sample space is 1:

Pr{S} = 1
The probability of the union of mutually exclusive events is equal

to the sum of their probabilities:
If A1 A2=,
=> Pr{A1 U A2} = Pr{A1} + Pr{A2}
15
Mutual Exclusivity
Are A1 and A2 mututally exclusive?

For mutually exclusive events A1, A2 AN, we have:
A1
s5
s1
s6
A2
s2
s4
s3
Find Pr{A1 U A2}

and Pr{A1}+Pr{A2}
in the fair dice
example
16
Mutual Exclusivity
Discarding the condition of exclusivity, in general, we have:
Pr{A1 U A2} = ??
s5
s1
s6
s4
s2
s3
S
17
Mutual Exclusivity
Discarding the condition of exclusivity, in general, we have:
Pr{A1 U A2} = Pr{A1} + Pr{A2} Pr{A1 A2}

s5
s1
s6
s4
s2
s3
S
18
Given that event B has already occurred, what is the probability

that event A will occur?
Given that event B has already occurred, reduces the sample
space of A
s5
s1
Event B has
already occurred
s6
s2
s4
s3
=> s2, s4, s3

cannot occur
s5
s1
s6
s2
s4
s3
19
Given that event B has already occurred, we define a new

conditional sample space that only contains Bs outcomes
The new event space for A is the intersection of A and B:
Event space -> EA|B = A B
s5
s1
s6
s2
s4
Event B has
already
occurred
s3
Whats missing here?

s5
s1
s6
s4
s2
s3
S|B = {s1, s5, s6}

EA|B= A B = {s6}
20
The probability of an event A in the conditional sample space is:
Pr
s5
s1
s4
s3
s5
s1
s6
s2
Pr
Pr =
Pr
Pr{6 }
1/6
1
=
=
=
Pr{}
3/6
3
Event B has
already
occurred
s6
s4
s2
s3
S|B = {s1, s5, s6}

21
Independence
Two events are independent if they do not provide any

information about each other:
(|) = ()
In other words, the fact that B has already happened does not
affect the probability of As outcomes
Implications:
(|) = ()

= ()
()
( ) = ()()
22
Independence: Example
Are events A and C independent?
Assume that all outcomes are equally likely
s2
s6
s4
s5
s1
s3
23
s2
s6
s4
s5
s1
s3
S
24

Pr{A C} = Pr{s5} = 1/6
Pr{A}Pr{C} = (3/6)x(2/6) = 1/6
Yes!
s2
s6
s4
s5
s1
s3
S
25
Are events A and B independent?
Assume that all outcomes are equally likely
s2
s6
s4
s5
s1
s3
S
26
Are events A and B independent?

Pr{A B} = Pr{s5} = 1/6
Pr{A}Pr{B} = (3/6)x(3/6) =
No!
s2
s6
s4
s5
s1
s3
S
27
Mutual Exclusivity and Independence

Experiment:
Roll a fair dice twice and record the dots on the top face:
= {(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6)}
28

Define three events:
1 = first roll gives an odd number
2 = second roll gives an odd number
= the sum of the two rolls is odd
Find the probability of using probability of 1 and 2
29

A2
S={
(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
A1
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),

(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
30
31
32
33
34
35
36
Recap
1.
Outcomes, events and sample space:
2.
For mutually exclusive events A1, A2,, AN, we have:
3.
In general, we have:
37
Recap
4.
Conditional probability reduces the sample space:
5.
Two events A and B are independent only if
6.
For independent events:
38
Four Rules of Thumb

1. Whenever you see two events which have an OR relationship (i.e., event A or
event B), their joint event will be their union, {A U B}
Example: On a binary channel, find the probability of error?
An error occurs when
A: a 0 is transmitted and a 1 is received OR
B: a 1 is transmitted and a 0 is received
Thus probability of error is: Pr{A U B}
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
39
Four Rules of Thumb

2. Whenever you see two events which have an AND relationship (i.e., both
event A and event B), their joint event will be their intersection, {A B}
Example: On a binary channel, find the probability that a 0 is transmitted and a
1 is received?
A: a 0 is transmitted AND
B: a 1 is received
Thus probability of above event is: Pr{A B}
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
40
Four Rules of Thumb

3. Whenever you see two events which have an OR relationship (i.e., A U B),
check if they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
Thus probability of error is: Pr{error} = Pr{A U B}
Are A and B are mutually exclusive?
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
41
Four Rules of Thumb

3. Whenever you see two events which have an OR relationship (i.e., A U B), check if
they are mutually exclusive. If so, set Pr{A U B} = Pr{A} + Pr{B}
Thus probability of error is: Pr{error} = Pr{A U B}
YES!
A and B are mutually exclusive; transmission of a 0 precludes the possibility of
transmission of a 1, and vice versa. Therefore, we can set
Pr{error} = Pr{A U B} = Pr{A} + Pr{B}
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
42
Four Rules of Thumb

4. Whenever you see two events which have an AND relationship (i.e., A B), check if
they are independent. If so, set Pr{A B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?
B: a 1 is received
Probability of above event is: Pr{A B}
Are A and B independent?
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
43
Four Rules of Thumb

4. Whenever you see two events which have an AND relationship (i.e., A B), check if
they are independent. If so, set Pr{A B} = Pr{A}Pr{B}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?
B: a 1 is received
Probability of above event is: Pr{A B}
Are A and B independent?
No.
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
1 01
Pr = Pr Pr = 01 =
2
2
1 1 1
Pr Pr = =
2 2 4
44
Total Probability
B1, B2,, BN form a partition of a sample space we have:
S = B 1 U B2 U U BN
Bi Bj = , i j
B2
B1
s1
B3
B4
s2
s5
s4
s6
s3
45
Total Probability
If B1, B2,, BN form a mutually exclusive partition:

What does this imply?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
46
Total Probability

B1 B2 .. Bn = and B1 U B2 U .. U Bn = S
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
47
Total Probability

How to express A in term of Bi?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
48
Total Probability

A = (A B1) U (A B2) U U (A BN)
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
49
Total Probability

A = (A B1) U (A B2) U U (A BN)
What is the probability of A?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
50
Total Probability

A = (A B1) U (A B2) U U (A BN)
What is the probability of A?
Pr{A} = Pr{A B1} + Pr{A B2} + + Pr{A BN}
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
51
Total Probability
Using the definition of conditional probability:

Pr{A| Bi} = Pr{A Bi} / Pr{Bi}
=> Pr{A Bi} = Pr{A| Bi} Pr{Bi}
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
52
The Law of Total Probability

The Law of Total Probability states:
If B1, B2,, BN form a partition then for any event A
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + + Pr{A|BN} Pr{BN}
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
53
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event, the
so-called inverse probability.
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
54
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event
Pr{Bi|A} = Pr{A Bi} / Pr{A}
=> Pr{A Bi} = Pr{A|Bi} Pr{Bi}
=> Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
55
Bayes Theorem
Pr{Bi|A} = Pr{A|Bi} Pr{Bi} / Pr{A}
From the Law of Total Probability, we have:
Pr{A} = Pr{A|B1} Pr{B1} + Pr{A|B2} Pr{B2} + + Pr{A|BN} Pr{BN}
Bayes Rule
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
56
Bayes Theorem
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
57
A Fishy Problem
Azeem (Iqbal) is a fisherman.
Azeem is an educated man.
Azeem builds a fishing robot that will do his work for him.
B2
B3
B1
B4
58
A Fishy Problem
Question: If Azeems robot catches a fish that is detected red, what species is it?
Answer: It could be any of four species in Azeems part of the sea.
B2
B3
B1
B4
59
A Fishy Problem
Lets change the question:
What is the chance that a red fish is a species B1, B2, B3
and B4?
B2
B3
B1
B4
60
A Fishy Problem
B2
B3
B1
B4
61
62

Lecture01 02 Intro Probability Theory

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture01 02 Intro Probability Theory

Uploaded by

Copyright:

Available Formats

CSE801 Analysis of Stochastic

Dr. Muhammad Usman Ilyas

Copyright Syed Ali Khayam 2009

Thursdays, 5:00-5:30pm, or by appointment.

The course will be managed through LMS

Copyright Syed Ali Khayam 2008

Timetable (MS EE-Telecom & Comp Networks)

MS EE(Digital System and Signal Processing)-5

Dr. Shahzad Younis

Dr. Amir Ali Khan

Dr. Rehan Hafiz

MS EE(Telecommnication & Computer Networks)-5

Dr. Usman Illyas

Dr. Junaid Qadir

Dr. Rizwan Ahmed

Copyright Syed Ali Khayam 2008

Timetable (MS EE-Power & Control)

MS-EE(RF & MW )-4

Power Electronics and Dr. Syed Raza

Copyright Syed Ali Khayam 2008

Dr. Usman ilyas

Copyright Syed Ali Khayam 2008

Probability & Random Processes

Introduction to Probability Models,

Elements of Information Theory

Thomas M. Cover and Joy

Chaos Theory Tamed

Introduction to Probability Theory

Application Area: Information Theory

Application Area: Chaos Theory

Copyright Syed Ali Khayam 2008

Copyright Syed Ali Khayam 2008

Quizzes will be unannounced

Late homework submissions will be accepted for up to 24 hours

Strong disciplinary action will be taken in case of plagiarism or

Copyright Syed Ali Khayam 2008

What will we cover in this lecture?

This lecture is intended to be an introduction to elementary

Random Experiments and Random Variables

Copyright Syed Ali Khayam 2008

3 : the ratio of the number of outcomes in an exhaustive set of

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment

A random experiment comprises of:

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

An outcome cannot be further decomposed into other outcomes

An event is a set of outcomes that are of interest to us

The set of all possible outcomes, S, is called the sample space

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

Example of a Random Experiment:

Experiment: Roll a fair dice once and record the

Copyright Syed Ali Khayam 2008

Probability of any event A is non-negative:

The probability that an outcome belongs to the sample space is 1:

The probability of the union of mutually exclusive events is equal

Copyright Syed Ali Khayam 2008

Are A1 and A2 mututally exclusive?

Find Pr{A1 U A2}

Copyright Syed Ali Khayam 2008

Discarding the condition of exclusivity, in general, we have: