ENCS 6161

Probability and Stochastic Processes

Concordia University
Course Material 6

Pairs of random variables

Examples of multiple random variables
1. The voltage signals at several points in a circuit, where
the voltage of a given point is a random variable.
2. Amplitude samples of voltage waveform that is random.

Objectives:
1. Develop techniques for determining the probabilities of
events that involve the joint behavior of two or more
random variables.
2. Determine when a set of random variables are
independent.
3. Quantify the degree of correlation when they are not
independent.

Vector random variable

A vector random variable is a function that
assigns a vector of real numbers to each
outcome , the sample space of the
random experiment.

Example 1
A random experiment consists of selecting a
student name from an urn, let be the outcome
of the experiment. Define the following three
functions:
(): age of the student in years
(): height of the student in inches
(): weight of the student in lbs

Then, the vector ( , , ()) is a vector

random variable.
4

Example 2
A random experiment consists of finding the number of
defects in a semiconductor chip and identifying their
locations. The outcome of the experiment is a vector
= , 1 , 2 , , , where the first component specifies
the total number of defects and the remaining components
specify the coordinates of their locations.
The chip consists of regions. Let 1 , 2 ,
, be the number of defects in each of these
regions, i.e., is the number of that fall in the
region . Thus, the vector = 1 , 2 , , is a
vector random variable. Here, both the outcome and the
random variable are vectors.
5

Example 3
Let the outcome of some random experiment
be a voltage waveform (). Let the random
variable = () be the sample of the
voltage taken at time .
The vector consisting of the first samples
= 1 , 2 , , , is then a vector random
variable.

Event and probabilities

Each event involving an -dimensional
random variable = 1 , 2 , , has a
corresponding region in the -dimensional real
space.

Example 4
Let = , be a two-dimensional random
variable. Find the region in the plane
corresponding to the events:
(a) = + 10
(b) = 2 + 2 100
(c) = min , 5

(a) = + 10

(b) = 2 + 2 100

(c) = min , 5

min , 5 = 5 5
that is, minimum of and is less than or equal to 5,
if and/or is less than or equal to 5.
9

 For an -dimensional random variable

= 1 , 2 , , , we are interested in
events that have product form
= 1 1 2 2
where is the one-dimensional event (i.e. a subset
of the real line) that involves only . the event
occurs when all the events occur jointly.

None of events , and above has product

form.

10

Examples of events having product

form

11

 A fundamental of a vector random variable

= 1 , 2 , , is to find the probability
of product-form events.
= 1 1 2 2
1 1 , 2 2 , ,

The probability as given by the above

equation is obtained by finding the probability
of the outcome in
=

12

 Many of the events of interest are not of product form.

However, the non-product form events that we are
interested in can be expressed either exactly or can be
approximated arbitrarily closely by the union of productform events.
The example of the event above can be written exactly as
union of disjoint events:
= 5 < > 5 5

13

 A non-product-form event can be approximated as a union of

rectangles, where each rectangle can be described as a product-form
event.
Thus, the non-product form events can be approximated as the union
of disjoint product-form events.
The approximations of the events and by rectangles of
infinitesimal width also suggest that we can express the probabilities
of the events as integrals of probability density over the regions that
correspond to the events

14

Probability of a non-product-form event

1. Approximate the event by the union of disjoint
product-form events, 1 , 2 , , .

=1

2.

=
=1

The approximation becomes exact in the limit as

become arbitrarily fine.
15

Independence of random variables

If one-dimensional random variables and are
independent, then the events that involve only are
independent of the events that involve only . If 1 is
any event that involves only and 2 any event that
involves only , then
1 , 2 = 1 2
In general,
1 1 , 2 2 , , = 1 1 2 2

Thus, the probabilities of the random variables

individually are sufficient to specify the probability of
joint event.
16

Pairs of discrete random variables

Let the discrete vector random variable = (, ) assume values from some
countable set = , , = 1,2 ; = 1,2 .
The joint probability mass function of and specifies probabilities of the
product form event = = :
, = = =
= , = ,
= 1,2, ; = 1,2,
(1)
Thus, the joint pmf gives the probabilities of occurrence of the pairs , .
The probability of any event is the sum of the pmf over the outcomes in .
=

(2)

Since the probability of the sample space is 1

, = 1
=1 =1
17

Marginal probability mass function

These are the probabilities of events involving each of the random
variables in isolation.
= = = = , =
= = , = 1 = , = 2

,
=1

Similarly,

= = =

,
=1

Knowledge of the marginal pmfs is not sufficient to specify the joint

pmf.

18

Example 5
The number of bytes in a message have a
geometric distribution with parameter 1 and
range = {0,1, }. Suppose that messages are
broken into packets of maximum length bytes.
Let be the number of full packets in a
message, and be the number of bytes left over.
Find the joint pmf and the marginal pmfs of
and .
19

 If a message has bytes then, the number of full

packets in the message equal to the quotient when
is divided by , and the number of remaining bytes is
the remainder of such division.
takes on the values in the range {0,1,2, } and the
range of is 0,1, , 1 .
has a geometric distribution with parameter 1 .
For a geometric random variable with parameter :
= 1 , = 0,1,
Thus, the probability of the elementary event , :
, , = = , =
= = + = 1 + , = 0, 1, ; = 0, 1, , -1

20

 The marginal pmf of :

= = = , + 1, , + 1
1

1 + = 1

=
=0

= 1

1
= 1
1

=0

= 0, 1,

Thus, the marginal pm of is a geometric distribution with parameter

1 .

The marginal pmf of :

= = = , + , 2 + ,

1 + = 1

=
=0

1
=

=0

= 0, 1, , 1
21

Joint cdf of and

For a single random variable we saw that semi-infinite intervals of the form
(, ] are the basic building blocks from which other one-dimensional
events can be built. By defining the cdf () as the probability of
(, ], we were then able to express the probabilities of other events in
terms of the cdf. We now extend this development to 2-D random variables.
The joint cdf of and is defined as probability of the product-form event
1 1
, 1 , 1 = 1 , 1

, 1 , 1 represents the long-term

proportion of times in which the outcome of
the random experiment yields a point that
falls in the rectangular region shown. In
terms of probability mass, , 1 , 1
represents the amount of mass contained in
the rectangular region.
22

Properties of joint cdf

1. The joint cdf is a nondecreasing in the northeast
direction:
, 1 , 1 , 2 , 2
1 < 2 1 < 2

since the semi-infinite rectangle defined by 1 , 1 is contained in

that defined by 2 , 2 .

2.
, , 1 = , 1 , = 0

since it is impossible for either and to assume a value less than .

3.
, , = 1

since it is certain that and will assume values less than infinity.

23

4. If one of the variables approaches infinity

while keeping the other fixed, we obtain the
marginal cdf
, , = , < = =
, , = < , = =

24

5. The joint cdf is continuous from the north

and from the east
lim+ , , = , ,

lim+ , , = , ,

25

Example 6
The joint cdf for the vector random variable
= (, ) is given by
,

1 1
, =
0

0, 0

Find the marginal cdfs

26

 = lim , , = 1

= lim , , = 1

Thus, and individually have exponential

distributions with parameters and ,
respectively

27

Probability of events expressed as a finite

rectangle

Events specified by the semi-infinite region defined by 1 , 1 and 2 , 1 :

1 = 1 , 1
2 = 2 , 1
Let
= 1 < 2 , 1
Therefore,
2 = 1
Since 1 = ,
2 = 1 + []

28

, 2 , 1 = , 1 , 1 + 1 < 2 , 1

Events specified by the semi-infinite region defined by 1 , 2 and 2 , 2 :

3 = 1 , 2
4 = 2 , 2
Let
= 1 < 2 , 1 < 2
Since

4 = 3
3 =
we have
4 = + + [3 ]
, 2 , 2 = 1 < 2 , 1 < 2 + , 2 , 1 , 1 , 1 + , 1 , 2
< , < = , , , , , , + , ,

29

Example 7
The joint cdf of the random variables and is
given by
,

1 1
, =
0

0, 0

Find the probability of the events

(a) = 1, 1
(b) = > , > , > 0, > 0
(c) = 1 < 2, 2 < 5
30

(a) = 1, 1
= 1, 1

= , 1, 1 = 1 1

(b) = > , > , > 0, > 0

= > , > = }{
= + }{
= 1 + 1 1 1
= 1 +

Therefore,
= 1 =

(c) = 1 < 2, 2 < 5

1 < 2, 2 < 5

= , 2, 5 , 2, 2 , 1, 5 + , 1, 2
= 1 2 1 5 1 2 1 2
1 1 5 + 1 1 2

31

Joint pdf of two jointly continuous

random variables

To compute the probability of events corresponding to regions other than

rectangles, we approximate the region as a union of disjoint rectangles. The
probability of such events can, therefore, be approximated by the sum of the
probabilities of rectangles. The sum approaches an integral of the probability
density function.
Two random variables and comprising a vector random variable (, ) are
said to be jointly continuous if the probability of events involving (, ) can be
expressed as an integral of a pdf . In other words, there is a nonnegative function
, (, ), called the joint pdf that is defined on the real plane, such that for every
event (i.e., a subset of the real plane)
=

, ,

Note the similarity between this equation and that for the discrete case
=

, ,
,

32

 When is the entire plane,

, , = 1

The joint cdf can be obtained in terms of the joint

pdf of jointly continuous random variables by
integrating over the semi-infinite rectangle
defined by (, ):

, , =

, ,

It then follows that if and are jointly

continuous random variables, then
2 , ,
, , =

33

 Probability of a rectangular region:

1 < 2 , 1 < 2
=

, ,

The probability of an infinitesimal rectangle

+

< + , < + =

, ,

= , ,

Thus, the joint pdf specifies the probability of

the product-form events:

< + }{ < +
34

 Marginal pdfs
= , , =

=
=

Similarly,

, ,

, ,

, ,

Thus, the marginal pdfs are obtained by integrating out

the variables that are not of interest.
35

Example 8
A randomly selected point (, ) in the unit
square has a uniform joint pdf given by
,

1
, =
0

0 1, 0 1

Find the joint cdf

36

 We must be careful with the limits of the integral: The

limits should define the region consisting of the
intersection of the semi-infinite rectangle defined by
(, ) and the region where the pdf is nonzero. There
are five cases

37

1. If < 0, or < 0 (Region I), the pdf is zero. Therefore,

, , = 0
2. If (, ) is inside the unit interval (Region II),

, , =

(1) =

3. If 0 1 and > 1 (Region III),

, , =

(1) =

4. If > 1 and 0 1 (Region IV),

, , =

(1) =

5. If > 1 and > 1 (Region V),

1

, , =

(1) = 1

0
38

Example 9
Given the joint pdf
,

, =
0

(a) Find the normalization constant .

(b) Find the marginal pdfs.
(c) Find + 1 .

39

(a)

1=

=0

=0

=0

1 =

=0

1
= 2
2

=0
1

=0

1
1 0

0
2
2

=
0

=2
(b)

= 2

, , =

= 2

, , =

= 2

1 ,

=
=

0<

= 2 0 + = 2 2 ,

0<

40

(c)

The region corresponding to the intersection of the event + 1 and the region where the
pdf is non-zero is shown in the figure.
=

+ 1 =
=
=

1
2

=1

=0 =
1
=
2

=0
1
=
2

2 2 1

=0

1
= 2 2 1
2

1
2

=0

1
1
1
1 1 0
2
2
2
= 1 2 1
=2

41

Example 10
The joint pdf of and is given by
, , =

1
2 1

2 2+ 2 )/2(12 )

< , <

Find the marginal pdfs.

42

 The marginal pdf of is found by integrating , ,

over :
2 /2(12 )

( 2 2)/2(12 )
=

2 1 2
Note that
2 2 + 2 2 2 2 = 2 2 2
=

2 /2(12 )

2 1 2

( 2 2 2 )/2(12 )

2 2 2 )/2(12 )
(

2 1 2

2 )/2(12 )
(

43

 Note that

1
2

2 )/2(12 )
(

= 1

2 1
since the integral I is that of a Gaussian pdf with mean
= and variance 2 = 1 2 .

Thus,

2 )/2
(

2
The marginal pdf of is therefore a 1-D Gaussian pdf with
mean = 0 and variance 2 = 1.
Because of the symmetry, the marginal pdf of :
2 )/2
(

=
2
44

Independence of two random variables

and are independent random variables if any event 1 defined
in terms of is independent of any event 2 defined in terms of ,
i.e.,
1 , 2 = 1 2
Let be the event defined by = 1 2 , where 1 involves only
and 2 involves only .
If and are independent, then 1 and 2 are independent events.
Let 1 = = and 2 = = .
, , = = , =
= = =
= (1)
Therefore, if and are independent discrete random variables,
then the joint pmf is equal to the product of marginal pmfs.
45

 Now suppose that we do not know, if and are

independent, but we do know that the joint pmf is product
separable, i.e., Equation (1) above is satisfied. Then,
=

, , =
1 2

1 2

= 1 [2 ]
2

i.e., 1 and 2 are independent events.

Therefore, the discrete random variables and are

independent if and only if the joint pmf of and is equal
to the product of the marginal pmfs for all and .

46

Example 11
In Example 5, we obtained
, , = 1 + ,

= 1
1
=
1

= 1

= 0, 1, ; = 0, 1, , 1

= 0,1,
= 0,1, , 1
1
+ =
=
1

,
1

Therefore, and are independent.

47

 In general, two random variables and are

independent if and only if their joint cdf is
equal to the product of the marginal cdfs.
, , =

(1)

If and are jointly continuous, then and

are independent if and only if their joint pdf is
equal to product of marginal pdfs
, , =

(2)

Equation (2) can be obtained by differentiating

(1), and (1) can be obtained by integrating (2).
48

Example 12
In Example 9, given the joint pdf:
,

, =
0

0<

we obtained the marginal pdfs:

= 2 1 ,

0<

= 2 2 ,

0<

Are the random variables and independent?

49

 and are nonzero for > 0 and

> 0, hence is nonzero in the
entire positive - quadrant. However,
, (, ) is nonzero only in the region <
inside the positive - quadrant. Therefore,
equation (2) does not hold for all and , and
the random variables and are not
independent. Note that even though , (, )
is product separable, it is not the product of the
marginal pdfs.
50

Example 13
In Example 6, given the joint cdf:
1 1
, , =
0
we obtained the marginal cdfs:

0, 0

= 1 ,

= 1 ,

1 1 ,
0,

, , =

0, 0

51

 If and are independent random variables, then

the random variables defined by any pair of
functions () and () are also independent.
Let be the set of all values of such that if
, then ; let be the set of all
values of such that if , then .
Then, since and are equivalent to the events
and ,
, = ,
Since and are independent random variables,
, =
=

52

 Many random variables of practical interest are

not independent
(i) The output of a communication channel must
be dependent on the input in order to convey
information.
(ii) The consecutive samples of a waveform that
varies slowly are very likely to be close in value
and hence, are not independent.

The conditional probability of the events

involving random variables that are not
independent are extremely useful.
53

Conditional probability
The probability that , given that we know the exact value of , i.e.
= :
, =
| = =
(1)
=
(i) When is discrete, (1) can be used to obtain the conditional cdf of
given = :
|

, =
=
=

= > 0 (2)

The conditional pdf of given = , if the derivative exists, is given by

| =

(3)

The probability of an event given = is obtained by integrating the

conditional pdf

| = =

|
54

If and are independent,

, = = = = () =
Thus, from (2)
| =
| =
(ii)

When and are discrete random variables, the pdf will consist of delta functions:
| =

where the conditional pmf of given = is given by

= = | =
= , =
, ( , )
=
=
=
( )

> 0

(4)

The probability of any event given = is found by summing the pmf over the event

= =

If and independent

= , =
= [ = ]
=
= = = ( )
=
=

55

(iii)

and are jointly continuous random variables.

Since is a continuous random variable, = = 0. Thus, (1) is undefined.
Assume that and are jointly continuous random variables with a joint pdf that is continuous and
nonzero over some region of the real plane. Define the conditional pdf of given = as limiting
case:

As 0

= lim | < < +

0
, < < +
< < + =
< <+

,
,

,
=
=
+

, (, )
=
=

()
Similarly,

, (, )
=
()

(5)

(5)

56

 Note that if and are independent:

, , =

Therefore, from (5a):

|
|
Similarly,
|
|

=
=
=
=
57

Example 14
In Example 9, given the joint pdf,
,
,
0
=
0,

we obtained the normalization constant = 2 and

the marginal pdfs as
= 2 1 ,
0<
= 2 2 ,
0<
Find the conditional probabilities (a) (|) and
(b) (|)
58

(a)

, (, ) 2
=
=
()
2 2
= 0

(b)

, (, )
2
=
=
()
2 1

=
0

1
The conditional pdf of is an exponential pdf shifted
by to the right.
The conditional pdf of is an exponential pdf that has
been truncated to the interval [0, ].
59

 From conditional probability:

= , =
= | = =
=
= , = = = | = =
Therefore,
, =
Suppose we are interested in the probability that
=

, , =

(6)

Therefore,
=

| =

(6)

60

| =

(6)

Equation (6) is a re-statement of the theorem on total

probability. The above result states that to compute
[ ], we can first compute | = and then
average over . The above equation also holds when is
continuous and is discrete.
If and both are continuous, then from (5a)
, , = ()
Then,

| =

(7)

Equation (7) can be thought of as the continuous version of

the theorem on total probability. This equation is also valid
when is a discrete random variable and is continuous.
61

Example 15
The total number of defects on a chip is a
Poisson random variable with mean . Suppose
that each defect has a probability of falling in a
specific region and the location of each defect
is independent of the locations of other defects.
Find the pmf of the number of defects that fall
in the region .

62

 We have, from (6b)

= =

= | = [ = ]
=0

Identify occurrence of defect as a Bernoulli trial with a

success when the defect falls in the region . The
probability of success of this single trial is .
The total number of defects that fall in the region is a
binomial random variable with parameters and :
= | =
0
>
=
1
0

63

 The total number of defects in a chip is Poisson

random variable with mean

= =

= 0,1,
!
Since ,

= =
1

!
=

=
!

1
!

(1)

=

=
!
!
Thus, is a Poisson random variable with mean .
64

Example 16
The random variable is selected at random
from the unit interval 0,1 . The random variable
is then selected at random from the interval
[0, ]. Find the cdf of Y.

65

 From (7), we have

= =

| =

When = , is uniformly distributed in the interval [0, ]. Thus, the

conditional pdf of given = is

0
| = =
1
>

1
0

01

1 1 +
0

= + ln 1
= + ln 1 ln = ln

=
=
ln = ln

01
66

Conditional expectation
The conditional expectation of , given = , is
defined by

In the case where and are both discrete random

variables:

(2)

is simply the center of mass associated with the

conditional pdf or pmf.
67

 The conditional expectation can be considered as defining a

function of :
=
Thus, we can consider
= [|]
as a random variable.
(a) For a continuous
=

() =

(b) For a discrete

()

(3)

=
=

( ) =

( )

(4)

68

 We will now show that

(5)

using the case when and are jointly continuous random variables.

(|) ()

Since

, (, )
=
()

= []

69

 Making use of (5) into (3), we have

()

(6)

Making use of (5) into (4), we have

=

( )

(7)

The result given by (5) also holds for the expected

value of a function of , ():
= ()
For example, the kth moment of is given by

70

Example 17
In Example 15, out of = defects in the chips,
the total number of defects that fall in the
region is a binominal distribution.
= | = =

0

1

>

The mean of this distribution is , i.e.

| = =
71

 From (7):

| = =
=0

= =
=0

=
=0

= []
Since is a Poisson random variable with mean

=
,
= 0,1,
!
and
=
Therefore,
=
72

Expected value of functions of a pair

random variables
= ,
(i) and jointly continuous:

, ,

(ii) and discrete

=

, ( , )

73

Example 18
Let = + . Find [].
= +

+ ,

, +

= + []

In general
1 + 2 + + = 1 + 2 + + [ ]

where 1 , 2 , , need not be independent.

74

Example 19
Let and be independent random variables and , =
1 2 (). Find , .
Since and are independent random variables,
, , = ()

1 2 , ,

= 1 2
In general, if 1 , 2 , , are independent random variables, then

1 1 2 2
= 1 1 2 2

75

-th joint moment of two random

variables and
The -th joint moment of and is defined as
follows:
(i) and jointly continuous

=
(ii) and discrete

Correlation of and

The = 1, = 1 joint moment of the random variables and

, , is called the correlation of and . If = 0,
then we say that and are orthogonal random variables.
76

-th central moment of and

The -th central moment of the random
variables and is defined as the -th moment
of the centered random variables ( ) and
( []):

Note that
= 2, = 0 gives []
= 0, = 2 gives []
77

Covariance of and
The covariance of and is defined as = = 1 central moment
, = [] []
The covariance of and as defined above can also be expressed
as
, = +
= +
=
Note that if either of the random variables has a zero mean, then
, = []
If and are independent random variable then ( []) and
( []) are also independent . Therefore,
, =
= =0
Therefore, pairs of independent random variables have a zero
covariance.
78

Correlation coefficient of and

The correlation coefficient of and is defined as
,

, =
=


We will show that
1 , 1
Since the expected value of the square of a random variable
is non-negative,
2

i.e.,

0
79

i.e.,
i.e.,

2
2

,
2

2
2

2
2

i.e., 1 2, + 1 0
Therefore,
1 , 1
The extreme values (i.e. +1 and -1) of are achieved
when and are related linearly, that is = + .
, = 1,
> 0
, = 1,
< 0
and are said to be uncorrelated, if , = 0
80

 If and are independent, , = 0,

and therefore, and are uncorrelated.
We have seen that when and are
jointly Gaussian and , = 0, then
and are independent. Thus, jointly
Gaussian uncorrelated random variables
and are also independent.
Non-Gaussian random variables that are
uncorrelated are not necessarily
independent.
81

Example 20
Given the joint pdf
2
, , =
0
obtain the (, )

82

 From the joint pdf, we first obtain the marginal pdfs () and
()

, , =

= 2

0
2

0<

1 ,

=
2 0

, , =

= 2

= 2

7
3
=
2
2

= 2 2

2
2

0<

3
=
2

7
=
2

5
=
4

83

=
2 =

2 =

1
=
2

2 2 2 =

1
1
=
2
2

1
2

1
=
4

, ,

= =

=
=

2 2

=0
=0
=
2

(2 )
1

=0

=1

,

=


3 1
1
1
2 2
=
=
5
5 1
4 4

, =

84

Jointly Gaussian random variables

The random variables, 1 , 2 , , , are said to be jointly Gaussian, if
their joint pdf is given by
1, 2, , 1 , 2 , ,
1
exp 1 ( )
2
=
2 /2 1/2
where
1
1
[1 ]
2
2
[2 ]
= ,
= =

[ ]
and is the covariance matrix defined by
[1 ] (1 , 2 ) (1 , )
(2 , 1 ) [2 ] (2 , )
=

( , 1 ) ( , 2 )
85

 Note that the covariance matrix is

symmetric, since , = ,
The pdf of jointly Gaussian random variables
is completely specified by the individual
means, variances and pairwise covariances.

86

Jointly Gaussian random variables

with , = 0,
In this case,

= [ ] = 2 , = 1,2, ,

Therefore, = 1 2
1 =

and

1
2

=
=1

Thus,
1,2,, 1 , 2 , , =

1
exp
2

=1

2
1

1
2
2
2
2

=
( )
2

=1
=1
Thus, for jointly Gaussian random variables, 1 , 2 , , , if the covariance matrix is a diagonal matrix
(i.e., the pairwise covariances , , , are zero), then the individual random variables are
independent Gaussian random variables.
87

Two-dimensional Gaussian pdf

The covariance matrix for the two-dimensional case:
12
, 1 2
=
, 1 2
22
2
= 12 22 (1 ,
)

1
2

2
= 1 2 1 ,

The inverse of the covariance matrix is given by

2
1

, 1 2
2
1
= 2 2
2
1 2 (1 ,
) , 1 2
12
88

= 1

1
2
12 22 1 ,

1
1 =
2

, , =

1
1

1
1

1
2
12 22 1 ,

2,

22
, 1 2

, 1 2 1
2
12

22 1 , 1 2 2
, 1 2 1 + 12 2

2
1
1
2
2
2 1 ,

1
2,
1

2
2
+
2
2

2
2

2
2 1

2
21 2 1 ,
89