1

Robust Video Stabilization Based

on Particle Filter Tracking of
Projected Camera Motion
(IEEE 2009)
Junlan Yang
University of Illinois,Chicago
2
Reference
[1]A tutorial on particle filters for online nonlinear non-
Gaussian Bayesian tracking
[4]probabilistic video stabilization using kalman filtering
and mosaicking
[5]Fast electronic digital image stabilization for off-road
navigation
[18]condensation conditional density propagation for
visual tracking
3
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


4
Introduction
Video Stabilization
Camera motion estimation
Particle filter
Tracking projected affine model of camera
motion
SIFT algorithm ()
Detect feature points in both images
Removing undesired (unintended) motion
Kalman filter
5
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


6
Example of camera motion
Motion
Camera
X
Y
Z
( x
0
, y
0
, z
0
)
at time t
0
P
Camera
X
Y
Z
( x
1
, y
1
, z
1
)
at time t
1
P
7
Generating Camera model
Related of two vectors
3 3 3 1
where R ,T are the transform of camera's
3-D rotation and translation, repectively

8
Building 2-D affine model
Projection of P in time t0 and t1
where is the image plane-to-lens distance of the camera
Z
X
Y
(x
0
,y
0
,z
0
)
(u
0
,v
0
,)
9
Building 2-D affine model
Rewriting the related of two projected vectors



2-D affine model
y 1 23 y
x 1 13 x 1 0
)T /z ( sR t
)T /z ( sR t , /z z s define we where
+ =
+ = =
10
Building 2-D affine model


3 3
R is orthonrmal matrix

Global motion estimation is to determine the six

parameters for every successive frame
11

Why do she use 2-D affine model to
represent camera motion?

A pure 2-D model
2-D translation vector and one rotation angle

3-D model
Giant complexity

12
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


13
Particle Filtering Estimation
Markov discrete-time state-space model
state vector at time k

observations z, and the posterior density is

k 1:k
p(x |z )
{ }
{ }
i
k
i
k
Give a set of particles x ,i = 1,...,N ,and weights
w ,i = 1,...,N ,where N is the number of particles
and k is the time step
T
k k k yk xk k k
R R R t t s x ] , , , , , [
21 12 11
=
14
To approximate the posterior
i i
k k
Where w are Normalized weights , particles x ~q()
are random vectors drawn from a proposal q() ,and the
q() refered as an importance density
i = 3 2 1 N
(.) ~ q x
i
k
...
15
Estimation of current state

N
1
rate e convergenc and sense
square mean in ) z | p(x density posterior
true to converge ion approximat , N As
:k 1 k

16
Importance density q(.)
Traditionally prior density

This paper takes into account the current
observation z
k
. The proposed important density
whose mean vector obtained from the
current observation z
k


Why do she use the particle filtering estimation ?



) x | p(x
1 - k k
k
x
17
Advantage of particle filtering
estimation
With Low error variance

Proof : In large particle numbers condition, the
estimation gives lower error variance than

k

x
k
x

2
k
k
k
1 1
k
i
k
matrix covariance diagonal with x state true of estimation
unbiased an is x that estimate fine a consider We
. estimation motion based - feature from obtained is x
and diagonal be set to is where , ) , x q( ~ x
18
Covariance matrix of errors




assumption ss unbiasedne given
) e , Cov(e as error covariance , x x e
) , Cov( as error covariance , x x

2
k k k k k
k k k k k

= =
=
mean zero have them of Both . x e and x

origin in the as x state true
set the she , prove he simplify t order to In
k k k k
k
= =
19
Lemma 1:







where
20
T
k k
1
k
i
k
i
k
1
k
T
i
k k
i
k k
i
k
i
k
1
k
i
k
x x ] x | x E[x
] x | ]E[x x | E[x ] x | x E[x
) , x G( ~ x

+ =
=
T
T
21
Lemma 1:
k with varing , variance and m mean
with variables random i.i.d as regarded and particles, i.i.d
for computed likelihood the are where , / w
2

i
k
N
1 i
i
k
i
k
i
k
=
=
Strong law of large number
k
1 2
k k
2

2
k
c ) (
N
1
) , Cov( , )/m (m c Denote

+ = + =
22

23

24
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


25
Complete system of video
stabilization
At time k
Frame k
Frame k-1
SIFT algorithm
Match feature points
PFME
(Particle filtering-
based motion
estimation)
k
x
Accumulative
motion
}

, { x
k k k k
T R s =
Kalman filter
} T , R , {s
A
k
A
k
A
k
Compensate
undesired motion
Stailized output
26
Getting six parameters
SIFT algorithm Find corresponding pairs
At time k
It needs three pairs to determine a unique solution
T
21k 12k 11k yk xk k k
T 1 T
] R , R , R , t , t , s [ x A
Y X X] [X A
= ==>
=

Y X A
27
(a) SIFT correspondence from frame 200,201 in outdoor sequence STREET
28
Generate particles
Important density q(.) is a six-dimensional
Gaussian distribution
Particles



In experience , N set to only 30 with better
quality than prior distribution set N = 300

29
Quality of the particles
N particles have N proposals of transformation
matrix ,and N Inverse transform to frame k have N
candidate image A
i

Compare these images with k-1 frame A
0
Point P at k-1frame
match
Inverse
transform
Point P at k frame
frame 1 k at P

Point
30
Similar with A
0
and A
i
Mean square error
Difference of gray-scale from pixel to pixel


Feature likelihood
Distance of all corresponding feature points
31
Particle filtering for global motion
estimation
Weight for each particle


Estimation of current state


where

32
Accumulative motion
At time k-1 to k


At time 0 to k





Where s is scaling factor , R is rotation matrix and T is
translation displacement
(

=
(
(

= =
22k 21k
12k 11k
k
y
x
k k k
R

,
t

, s

33
A
k
0
0
A
k
A
k
k
A
1 k k
0
0
A
1 k k
A
1 k k
1 - k
1 - k
k
k
k
A
1 k
0
0
A
1 k
A
1 k
1 k
1 k
T
v
u
R s
T

T R

v
u
R R

v
u
R

v
u
T
v
u
R s
v
u
+
(

=
+ +
(

= +
(

=
(

+
(

=
(

k k k
s s s
34
Intentional Motion estimation and
motion compensation



Compensate for the unwanted motion
D
k
D
k
D
k
s factor scaling and , T n vector translatio
, R matrix rotation l intentiona get filter to Kalman ng Implementi
35
Complete system of video
stabilization
At time k
Frame k
Frame k-1
SIFT algorithm
Match feature points
PFME
(Particle filtering-
based motion
estimation)
k
x
Accumulative
motion
}

, { x
k k k k
T R s =
Kalman filter
} T , R , {s
A
k
A
k
A
k
Compensate
undesired motion
Stailized output
36
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


37
(a) Original image , (b) Matched-feature-based motion estimation (MFME)
(c) p-norm cost function-based motion estimation (CFME) (d) proposed method
(PFME)
38
(a) Original image , (b) MFME (c) CFME (d) PFME
39
(a) Original image , (b) MFME (c) CFME (d) PFME
40
(a) Original video sequence (ground truth) (b) unstable video sequence (c) PFME
41
(a) Motion in horizontal direction (b) Motion in vertical direction
T
y
?
42
Comparison of average MSE and
PSNR for stabilized output




PSNR = peak signal to noise ratio
Large PSNR has low distortion
43
Outline
Introduction
Camera Model
Particle Filtering Estimation
Complete System of Video Stabilization
Simulation and Results
Conclusion


44
Conclusion
We demonstrated experimentally that the
proposed particle filtering scheme can be used
to obtain an efficient and accurate motion
estimation in video sequences.
45
Contributed of this paper
Constraining rotation matrix projected onto the
plane ?(depth change)
Show using particle filtering can reduce the error
variance compared to estimation without
particle filtering
Using both Intensity-based motion estimation
method (PFME) and feature-based motion
estimation (SIFT) method