Chapter 05b Image Restoration (Linear Restoration)

Digital Image Processing
Image Restoration and Reconstruction

(Linear Restoration Methods)
Christophoros Nikou
cnikou@cs.uoi.gr
University of Ioannina - Department of Computer Science
Contents
In this lecture we will look at linear image

restoration techniques
Differentiation of matrices and vectors
Linear space invariant degradation
Restoration in absence of noise
Inverse filter
Pseudo-inverse filter
Restoration in presence of noise

Inverse filter
Wiener filter
Constrained least squares filter
C. Nikou Digital Image Processing (E12)
Differentiation of Matrices and

Vectors
Notation:
A is a MxN matrix with elements aij.
x is a Nx1 vector with elements xi.
f(x) is a scalar function of vector x.
g(x) is a Mx1 vector valued function of vector x.

Vectors (cont...)
Scalar derivative of a matrix.
a11
K
t
A
M O
t
aM 1
L
t
a1N
M
aMN

Vectors (cont...)
Vector derivative of a function (gradient).
f(x) is a scalar function of vector x.
f
f
x f
K
x
x1
xN

Vectors (cont...)
Vector derivative of a vector (Jacobian):

g(x) is a Mx1 vector valued function of vector x.
g1
x
1
M
g M
x
1
K
O
L
g1
xN
g M
xN

Vectors (cont...)
Some useful derivatives.

b is a Nx1 vector with elements bi.
T
b x b
It is the derivative of the scalar valued function

bTx with respect to vector x.

Vectors (cont...)

T
T
x
Ax
A
x
x
If A is symmetric:
T
x Ax 2Ax
x

Vectors (cont...)

b is a Nx1 vector with elements bi.
2
T
Ax b 2A Ax b
x
It may be proved using the previous properties.

10
Linear, Position-Invariant
Degradation
We now consider a degraded image to be
modelled by:
g ( x, y ) h ( x, y ) f ( x, y ) ( x, y )
where h(x, y) is the impulse response of the
degradation function ( i.e. point spread function
blurring the image).
The convolution implies that the degradation
mechanism is linear and position invariant (it
depends only on image values and not on
location).
11
Degradation (cont...)
In the Fourier domain:
G (k , l ) H (k , l ) F (k , l ) N (k , l )
where multiplication is element-wise.
In matrix-vector form:
g = Hf +
where H is a doubly block circulant matrix and
f,g, and are vectors (lexicographic ordering).
12
Degradation (cont...)
g ( x , y ) h ( x , y ) f ( x , y ) ( x, y )
G (k , l ) H (k , l ) F (k , l ) N ( k , l )
g = Hf +
If the degradation function is unknown the
problem of simultaneously recovering f(x,y)
and h(x,y) is called blind deconvolution.
13
Linear Restoration
Using the imaging system
g = Hf +
we want to estimate the true image from the
degraded observation with known
degradation H.
A linear method applies an operator (a
matrix) P to the observation g to estimate
the unobserved noise-free image f:
f = Pg
14
Restoration in Absence of Noise

The Inverse Filter
When there is no noise:
g = Hf
an obvious solution would be to use the
inverse filter:
P=H
-1
yielding
f = Pg = H -1g = H -1Hf = f
15

The Inverse Filter (cont...)
f = H -1g
For a NxN image, H is a N2xN2 matrix!
To tackle the problem we transform it to the
Fourier domain.
H is doubly block circulant and therefore it
may be diagonalized by the 2D DFT matrix
W:
-1
H = WW

16
-1
where
H = WW
diag{H (1,1),..., H ( N ,1), H (1, 2),...H ( N , N )}

Therefore:
-1
-1
f = Pg f = H -1g f = (WW)
g
-1 -1
-1 -1
f = W Wg Wf = WW Wg
-1
F = G
17

Which is the vectorized form of the DFT of
the image:
G
(
k
,
l
)
F = G F (k , l ) =
H (k , l )
-1
Take the inverse DFT and obtain f(m,n).

Problem: what happens if H(k,l) has zero
values?
Cannot perform inverse filtering!
18

The Pseudo-inverse Filter
A solution is to set:
G (k , l )
, H (k , l ) 0
F (k , l ) = H (k , l )
0
, H (k , l ) 0
which is a type of pseudo-inversion.

Notice that the signal cannot be restored at
locations where H(k,l)=0.
19

The Pseudo-inverse Filter (cont...)
A pseudo-inverse filter also arises by the
unconstrained least squares approach.
Find the image f, that, when it is blurred by
H, it will provide an observation as close as
possible to g, i.e. It minimizes the distance
between Hf and g.
20

The Pseudo-inverse Filter (cont...)
This distance is expressed by the norm:
J (f ) Hf - g
J
min J (f )
0
f
f
f
Hf - g 0
2HT Hf - g 0 2HT Hf 2HT g
f H H HT g
T
21
Restoration in Presence of Noise

The Inverse Filter
Recall the imaging model with spatially
invariant degradation and noise
g ( x , y ) h ( x , y ) f ( x , y ) ( x, y )
G (k , l ) H (k , l ) F (k , l ) N ( k , l )
g = Hf +
22

Applying the inverse filter in the Fourier
domain:
G (k , l ) H (k , l ) F (k , l ) N (k , l )
N
(
k
,
l
)
F (k , l ) F (k , l )
H (k , l )
Even if we know H(k,l) we cannot recover F(k,l)
due to the second term.
If H(k,l) has small values the second term
dominates (it goes to infinity if H(k,l)=0!).
23

One approach to get around the problem is
to limit the ratio G(k,l) / H(k,l) to frequencies
near the origin that have lower probability of
being zero.
We know that H(0,0) is usually the highest
value of the DFT.
Thus, by limiting the analysis to frequencies
near the origin we reduce the probability of
encountering zero values.
24

Blurring degradation
25

Inverse filter with cut-off
26

Wiener Filter
So far we assumed nothing about the statistical
properties of the image and noise.
We now consider image and noise as random
variables and the objective is to find an estimate of
the uncorrupted image f such that the mean square
error between the estimate and the image is
minimized:
(f f ) 2
min
E
where E[x] is the expected value of vector x.

27

Wiener Filter (cont...)
Recall also the definition of the correlation
matrix between two vectors x and y:
R xy E xy
E x1 y1
E xN y1
E x1 y2
M
E xN y2
E x1 y N
M
M
K
K E xN y N
K
We assume that the image and the noise are

uncorrelated:
R f R f 0
28

We are looking for the best estimate:
min
E
(
f
f
)
Lets confine our estimate to be obtainable by

a linear operator on the observation:
f Pg
and the goal is to find the best matrix P.
29

J (f ) E (f f )
2
2
E f f
E f Pg
T
n
Denoting by p the n-th row of P:
J (f ) E
p g
T
n
2
T
E f n p n g

30
J (f ) E f n p g f n p g
n
T
n
T
n
T
T
T
T
T
T
E f nf n p n gf n f n g p n p n gg p n
n
T
T
T
T
T
T
E f nf n p n E
gf n
E f n g p n p n E
gg p n
n
R fnfn 2pTn R gfn pTn R gg p n

n
31

We can now minimize the sum with respect
to each term:
T
T
R fnfn 2p n R gfn p n R gg p n 0
p n
1
R gfn
2R gfn 2R gg p n 0 p n R gg
p R fn g R
T
n
1
gg
32

Assembling the rows together:
P R fg R
1
gg
We have to compute the two matrices:

T
(Hf Hf
R gg E gg E
)(
E Hff H Hf f H
T
HR ff HT HR f R f HT R

33
Assumming noise is uncorrelated with image:

R gg HR ff
H R
T
Also,
f (Hf )T ...R Hff
R fg E fgT E
Finally the matrix we are looking for is

1
gg
P R fg R R ff H
HR
H R
T
ff
34

The estimated uncorrupted image is
f Pg f R HT HR HT R 1 g
ff
ff
which may be also expressed as
f H R H R
T
1 1
ff
HT R 1 g
This result is known as the Wiener filter or

the minimum mean square error (MMSE)
filter.
35

Special cases:
No blur (H=I, g=f+): f R ff R ff R
g
No noise (R=0, g=f):
This is the inverse filter.
No blur, no noise (H=I, R=0): f g

Do nothing on the observation.
36

The size of the matrix to be inverted poses
difficulties and Wiener filter is implemented in
the Fourier domain.
This occurs when H is doubly block circulant
(represents convolution) and the image f and
noise are wide-sense stationary (w.s.s).
Definition of a w.s.s. signal:
1) E[f(m,n)]=, independent of m,n.
2) E[f(m,n)f(k,l)]=r(m-k,n-l), independent of
location.
37

Reminder: the inverse DFT complex exponential
matrix diagonalizes any circulant matrix:
1
H W W
H
The columns of W-1 are the eigenvectors of any
circulant matrix H.
The corresponding eigenvalues are the DFT values
of the signal producing the circulant matrix.
Remember also that WT W
1 T
1
and that
W
W

38
We will employ the following relations:

1
H W W
H
H W W
H
W W
H
If H is real:
H H
T
T *
( NW)
*
H
1
W
N
1 *
W
*
W
W
*
H
*
H
1 *

39
The Wiener solution is now transformed to

the Fourier domain:
f R ff H T HR ff
HT R
g
1
( W W
)(
W W ) W
( W
)(
W W
)(
W W ) W
( W ) g
ff
H
ff
1
W
ff W W
1
*
H
*
H
(
H
ff
Wf ff
H ( H ff
*
H
*
H
*
H
) g
Wg
)
Notice that the matrices are diagonal.

40

*

Wf
ff
H ( H ff
*
H
Wg
) 1
F (k , l )
S ff (k , l ) H (k , l )
2
S ff (k , l ) H (k , l ) S (k , l )
G (k , l )
Sff (k,l)=DFT(Rff (m,n)) is the power spectrum of

the image f (m,n).
S(k,l)=DFT(R (m,n)) is the power spectrum
of the noise (m,n).
41

If Sff (k,l) is not zero we may define the Signal
to Noise Ratio in the frequency domain:
SNR(k , l )
S ff (k , l )
S (k , l )
and the Wiener filter becomes:
F (k , l )
H (k , l )
2
H (k , l ) SNR -1 (k , l )
G (k , l )
42

A well known estimate of Sff (k,l) is the
periodogram (the ML estimate of Rff when f ia
assumed Gaussian):
1
2
S ff (k , l )
F (k , l )
MN
In practice, as F(k,l) is unknown, we use

1
2
S ff (k , l )
G (k , l )
MN
43

44

Degraded image
Inverse
Noise variance one order

of magnitude less.
Noise variance ten orders

of magnitude less.
Wiener
45

Constrained Least Squares Filter
When we do not have information on the power
spectra the Wiener filter is not optimal.
Another idea is to introduce a smoothness term in
our criterion.
2
We define smoothness by the quantity Qf
where Q is a high pass filter operator, e.g. the
Laplacian, (Q is a doubly block circulant matrix
representing convolution by the Laplacian).
2
We look for smooth solutions minimizing Qf
46

Constrained Least Squares Filter (cont...)
We have the following constrained least

squares (CLS) optimization problem:
2
Minimize Qf subject to Hf g
yielding the Lagrange multiplier minimization of
the function:
2
J (f , ) Hf g Qf
Data fidelity term
Smoothness term
47

J (f , ) 0 2HT (Hf g ) 2QT Qf 0

f
f H H Q Q H g
T
Parameter controls the degree of smothness:

0, f H H HT g
T
, f 0
pseudo-inverse, ultra rough solution
ultra smooth solution

48

In the Fourier domain, the constrained least
squares filter becomes:
F (k , l )
H * (k , l )
2
H (k , l ) Q(k , l )
G (k , l )
Keep always in mind to zero-pad the images

properly.
49

Low noise: Wiener and CLS generate equal results.

High noise: CLS outperforms Wiener if is properly selected.
It is easier to select the scalar value for than to approximate the SNR
which is seldom constant.
50
Restoration Performance Measures
Original image f and restored image f

Mean square error (MSE):
1
MSE
MN
M 1 N 1
( m, n)
f
(
m
,
n
)
m 0 n 0
51
Restoration Performance Measures

(cont...)
The Signal to Noise Ratio (SNR) considers the

difference between the two images as noise:
M 1 N 1
SNR
M 1 N 1
f (m, n) 2
m 0 n 0
( m, n)
f
(
m
,
n
)
m 0 n 0

Chapter 05b Image Restoration (Linear Restoration)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 05b Image Restoration (Linear Restoration)

Uploaded by

Copyright:

Available Formats

Digital Image Processing

Image Restoration and Reconstruction

University of Ioannina - Department of Computer Science

In this lecture we will look at linear image

Restoration in presence of noise

C. Nikou Digital Image Processing (E12)

Differentiation of Matrices and

C. Nikou Digital Image Processing (E12)

Differentiation of Matrices and

C. Nikou Digital Image Processing (E12)

Differentiation of Matrices and

C. Nikou Digital Image Processing (E12)

Differentiation of Matrices and

Vector derivative of a vector (Jacobian):

C. Nikou Digital Image Processing (E12)

Differentiation of Matrices and

Some useful derivatives.

It is the derivative of the scalar valued function

Differentiation of Matrices and

Some useful derivatives.

Differentiation of Matrices and

Some useful derivatives.

It may be proved using the previous properties.

Restoration in Absence of Noise

Restoration in Absence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Absence of Noise

diag{H (1,1),..., H ( N ,1), H (1, 2),...H ( N , N )}

Restoration in Absence of Noise

Take the inverse DFT and obtain f(m,n).

Restoration in Absence of Noise

which is a type of pseudo-inversion.

Restoration in Absence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Absence of Noise

2HT Hf - g 0 2HT Hf 2HT g

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

Restoration in Presence of Noise

Restoration in Presence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

where E[x] is the expected value of vector x.

Restoration in Presence of Noise

We assume that the image and the noise are

Restoration in Presence of Noise

Lets confine our estimate to be obtainable by

Restoration in Presence of Noise

Denoting by p the n-th row of P:

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

R fnfn 2pTn R gfn pTn R gg p n

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

C. Nikou Digital Image Processing (E12)

Restoration in Presence of Noise

We have to compute the two matrices:

Restoration in Presence of Noise

Assumming noise is uncorrelated with image:

Finally the matrix we are looking for is