You are on page 1of 4

3136

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 46, NO. 11, NOVEMBER 1998

On the Second-Order Statistics of the Eigenvectors of Sample Covariance Matrices


B. Friedlander and A. J. Weiss

E ui uj u u

H g = i
N

M k=1 k6=j
(i

0 k )2 ukuk ij 0 ij ):

k

(4)

E ui uj u u

Tg =0

i j T uj u i (1 N (i j )2

(5)

Abstract Eigenvectors of sample covariance matrices are used in a variety of statistical signal processing problems. The second-order statistics of these eigenvectors are needed to compute the variance of estimates based on these eigenvectors. Formulas for the secondorder statistics of the eigenvectors have been derived in the statistical literature and are widely used. In this correspondence, we point out a discrepancy between the statistics observed in numerical simulations and the theoretical formulas, due to the nonuniqueness of the denition of eigenvectors. We present two ways to resolve this discrepancy. The rst involves modifying the theoretical formulas to match the computational results. The second involved a simple modication of the computations to make them match existing formulas. Index TermsAsymptotic, covariance, eigenvalue, eigenvector, statistics.

o(N 01 ):

In the above, ij is the Kronecker delta, and the approximations are

An important issue that does not seem to be addressed in the literature is the nonuniqueness of the eigenvector of a complex matrix. From the denition of an eigenvector
Ru = u u

(6)

I. INTRODUCTION Eigenvectors of sample covariance matrices are used in the solution of a wide range of statistical problems, such as frequency estimation and array processing. Eigendecomposition also plays an important role in signal processing applications such as spatial and temporal spectral analysis [4]. The sample covariance matrix is usually dened as
^ R= 1

N n=1

y (n)y (n)

(1)

where y (n) are M 2 1 random vectors taken from a complex multivariate Gaussian distribution with zero mean and covariance matrix R: The eigenvectors of the sample covariance matrix are perturbed versions of the true eigenvectors, and therefore, whatever parameters are estimated based on these eigenvectors will be perturbed versions of the true parameter values. To analyze the variance of the estimation errors, it is necessary to know the second-order statistics of the sample eigenvectors. The statistics of the sample eigenvalues and eigenvectors have been studied in the statistical literature and are quoted in standard texts such as [1] and [2]. The following is a summary of the results presented in these references. ^ Let i and ui denote the ith eigenvalue and the corresponding ^ ^ eigenvector of a sample covariance matrix. Dene i = i 0 i and u i = u i 0 ui : Then u ^
^ ^ ^^ Rui = i ui

E i j

2 i = ij N

(2) (3)

Manuscript received November 5, 1997; revised March 26, 1998. This work was supported by the United States Army Research Ofce under Contract DAAL03-91-C-0022, sponsored by U.S. Army Communications Electronics Command, Center for Signals Warfare. The associate editor coordinating the review of this paper and approving it for publication was Prof. James A. Bucklew. B. Friedlander is with the Department of Electrical and Computer Engineering, University of California, Davis, CA 95616 USA. A. J. Weiss is with the Department of Electrical Engineering Systems, Tel Aviv University, Tel Aviv, Israel. Publisher Item Identier S 1053-587X(98)07826-X.

it is straightforward to conclude that if u is an eigenvector, u will u also be an eigenvector for any complex number : It is common practice to require that the eigenvector be normalized to have unit norm. In that case, if u is an eigenvector, uej will also be an eigenvector for an arbitrary angle : (Note that in the case of real eigenvectors, the unit norm requirement uniquely denes the eigenvector up to a sign.) The nonuniqueness due to the rotation by  can be eliminated, for example, by requiring that the rst element of the eigenvector be real (or real and non-negative to also eliminate the sign ambiguity). The perturbation analysis in [1] and [2] does not seem to take into account this nonuniqueness. Note, for example, that the norm of the perturbed eignevector u i = ui + u i is different from the norm of u ^ the true eigenvector ui : In other words, the norm of the eigenvectors are not preserved in this analysis. The unit-norm normalization is considered in [3], where it is shown that the covariance of the normalized and unnormalized eigenvectors is the same to rst order approximation. However, the issue of nonuniqueness due to rotation by  does not seem to have been addressed in the literature. The nonuniqueness is not only a theoretical problem but a practical one as well. Consider for example what happens if we attempt to verify these formulas by a Monte Carlo simulation. Such a simulation will involve generating multiple sample covariance matrices, computing their eigenvectors, and then computing the sample covariance of the ensemble. This yields what we will refer to as the experimental covariance matrix of the sample eigenvectors to distinguish it from the theoretical covariance matrix presented above. This experiment, which can be easily carried out by the reader, yields the disturbing result that the experimental and theoretical covariance matrices are completely different! This is contrary to the expectation that the experimental result will converge to the theoretical result as the number of independent Monte Carlo runs increases. This discrepancy is due to the fact that the computed eigenvectors correspond to a choice of , which is generally different from the choice used in the analysis. To see this more clearly, let us dene the different eigenvectors ^ involved as follows. Given a sample covariance matrix R , let ui ^ denote the unique eigenvector (with real rst element) corresponding to the ith eigenvalue. Then, ui ej will denote the eigenvector ^ considered in the analysis and ui ej the eigenvector produced by ^ the computational algorithm. When we attempt to evaluate E fui uj g experimentally, we u uH compute the eigenvectors for many independent random realizations

1053587X/98$10.00 1998 IEEE

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 46, NO. 11, NOVEMBER 1998

3137

of the sample covariance matrix and then evaluate


^ C=

Collecting terms related to

K k=1 K k=1
1

K K

ui (k)ej (k)uH (k)e0j (k) 0 is H ^ ^j ^ ^js ui (k)^i (k)ej ( (k)0 (k)) 0 is H ^ uH ^ ^js
is the sample mean (8) (7)

u` , we get u u R (R 0 ` I )u ` ' 0Ru` + ` u ` :


M
(k

(17)

Note that
(R

where

k is the experiment number, and is ^ is = ^ K k=1


1

ui (k)ej (k) : ^

The analytical results, on the other hand, were derived by evaluating H (9) C = E fuiuj ej ( 0 ) g 0 ia H ^ ^ ja where

(18) k=1 k6=` where V ` is a matrix of the eigenvectors not including u` , and 6 ` is a diagonal matrix with the diagonal entries k 0 ` for all values of k not including k = `: 0 ` Multiplying (17) by V `6 ` 1V H yields 1 0 (19) V `V H u` ' 0V `6 ` 1V H Ru` = q ` : ` u ` R

0 `I ) = V `6 `V H = `

H 0 ` )ukuk

V `.

Equation (19) provides the projection of We can therefore conclude that

u` on the column space of u


(20)

ia = E fui ej ^

g:

(10)

Clearly, the two results will be different unless the experimental and analytical phases are the same or are highly correlated. However, there is no reason to expect that this will be the case. A similar situation holds when we attempt to evaluate E fu i uT g: u uj In estimation problems involving eigendecomposition of sample covariance matrices, the quantity of interest often involves functions of the eigenvectors that are insensitive to the multiplication of the eigenvectors by a unit-modulus complex number. Thus, as long as we consider only functions of the eigenvectors that have this invariance property, the experimental results will match the theoretical results. However, when using arbitrary functions of the eigenvectors, the experimental results will be very different from what the theory predicts due to the issue discussed above. In principle, there are two ways in which this discrepancy can be eliminated. The rst is to derive a new set of theoretical formulas that take into account the particular phase generated by the computational algorithm. The second is to multiply the eigenvectors computed in the simulation by the appropriate ej , which will make them consistent with the existing theoretical formulas. In this correspondence, we consider both solutions. II. FIXING
THE

where ` can be determined by using two additional constraints that make the eigendecomposition unique. The rst is the common H requirement that the eigenvectors have unit norm, i.e., u` u` = 1, which leads to the rst-order perturbation equation H u (21) u` u` + u` u` ' 0: uH Substituting (20) in (21) and recalling that q ` is orthogonal to we get

u ` ' q ` + `u` u

u` ,
(22)

We need a second constraint that will pin down =f ` g: Derivations of the existing results [2], [3] make the choice ` = 0, thus forcing u` to be orthogonal to u` : However, in practice, the error-free u` u is not known, and a different constraint is used. For example, the svd function of MATLAB forces all eigenvectors to have a real rst element, i.e., =fu` (1)g = 0: We will now nd ` based on this constraint. We have

<f ` g ' 0:

u` (1) 0 u3 (1) = 0: u u` Substituting (20) and recalling that 3 = 0 ` `


we get

3 and u` (1) = u ` (1),


(24) (25) (26)

(23)

THEORY

In this section, we develop formulas for the covariance matrix of the sample eigenvectors that are produced by numerical simulations. A. Mathematical Preliminaries Consider the covariance of zero mean, complex Gaussian vectors y (n) given by R: The eigendecomposition of R is given by

3 q ` (1) + `u` (1)0 q ` (1) + `u` (1) = 0 3 2 ` u` (1) = q ` (1) 0 q ` (1) 1 3 ` = (q ` (1) 0 q ` (1)) 2u` (1)

B. Derivation of the Covariance Expressions The eigenvector covariance is given by H H H E fu` u` g = E fq `q ` g + E fj ` j2 gu `u` + u` E f `q` g u uH 3 H (27) + E f ` q ` gu` : In order to obtain closed-form expression, we use the result [2]

R = U 3U H

M
=

1 U = [u1 ; u2 1 1 1 uM ] 1 3 = diag f1 ; 2 1 1 1 M g:

k=1

H k uk uk

(11) (12) (13)

We assume that the eigenvalues are distinct and that the eigenvectors are orthonormal. If R is perturbed, we get H ^ ^^^ (14) R =U0U
^ ^ ^^ u Ru` = `u` = (` + ` )(u` + u` ):

(15)

Keeping only rst-order terms of (15) and recalling that Ru` = `u` , we get

Ru` + Ru` ' ` u ` + `u ` : u R u

(16)

H H R (28) (y Ry )(y Ry 2 ) N 1 4 3 where y i are any complex vectors, and N is the number of realizations used for estimating R : This equality also holds if y 1 ; y 4 are H matrices. We rst evaluate E fq ` q ` g by substituting (19) H 0 H R 0 E fq `q ` g = E fV `6 ` 1V H Ru`u` RV `6 ` 1V H g ` R ` 1 H Ru )(V 6 01V H RV 6 01V H ) (u = ` ` ` N ` ` ` ` ` ` V 6 02  V H = N ` ` ` ` ` M k 1 = u uH = Q ` : (29) N k=1 (k 0 ` )2 k k k=` 6
1

E f(yH Ry 2 )(y H Ry 4 )g = 3 R 1 R

3138

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 46, NO. 11, NOVEMBER 1998

` = u H R u ` (30) ` ` = V H RV ` (31) ` where ` is an (M 0 1) 2 (M 0 1) diagonal matrix with fk g; k = 1; 1 1 1 ` 0 1; ` + 1 1 1 1 M on its main diagonal.
Now

We used the identities

We have

where w = [1; 0 1 1 1 0]T , and we used Efw T q `q ` w g = 0, which is w T obtained by following the same steps leading to (29). Now

H H T Efj ` j2 gu`u` = 4u`2u(1) Ef2wT q `q ` w 0 w T q `q` w u H u ` ` H 0 w T q 3q ` w g ` Q (1; 1) u`uH = `2 ` 2u (1) `

(32) (33)

H where the last equation is based on uk Ru` = 0, which is true for k = ` and 6 1 3 3 Ef ` k g = 4u` (1)u` (1) Efq `(1)qk (1) + q ` (1)qk (1) q 3 3 3 0 q ` (1)qk (1) 0 q ` (1)qk (1)g: (43)
However

Efq`qk g = EfV `6 ` 1V H Ru`uk RV k6 k 1V H g q H V 0 ` R HR 0 k 1 H 01 H 01 H = N (uk Ru` )(V `6 ` V ` RV k6 k V k ) = 0

(42)

H H w 3H u` Ef `q` g = 2uu` EfwT q ` q ` 0 w T q `q ` g ` (1) u` EfwT q q H g =0 w `` 2u` (1) u` w T Q : =0 ` 2u` (1) H ` Efu` u` g = Q` + 2u`1(1) Qu`(1; 1) u`u` u uH (1) H 0 u ` w T Q ` 0 Q ` wu` :

(34) (35) (36)

Efq` (1)qk (1)g = EfwT q `q k w g = 0 q 3 w H Efq` (1)qk (1)g q = Efw q ` wq k g wT T 01 H R T 01 H R = Ef(w V ` 6 ` V ` Ru` )(w k 6 k k Ru k )g 1 T 01 H T 01 H = N (w V `6 ` V ` Ruk )(w k6 k k Ru` ) ` k u` (1)uk (1) : =0 (` 0 k )2 N
Substituting (44) and (45) in (43), we get

(44)

(45)

Collecting all these results, we nally obtain

(37) In addition

 3 Ef ` k g = 2N (`` k k )2 : 0
H H Ef `qk g = 2u`1(1) EfwT q ` q k 0 w T q `q k g w 3H

(46)

Following the same considerations, we also obtain

Q T Efu` u` g = 2u`1(1) 0 <fQ` (1; 1)g u`u` u uT u` (1) T T T + Q ` wu` + u ` w Q ` :

(38) and

1 wT 3 H u` (1) Efw q ` q k g

(47)

Note that under the assumption of [2], that also (4) and (29)]

` = 0, we get [see

M  H Efu` u` g = Efq`q` g = ` (k 0k`)2 ukuk = Q` u uH q H N k=1 k6=`


(39)

i; j entry in the matrix Efu` u` g is given by u uT eiT Efu` u` gej u uT e = Efei q ` q ` e j g eT T T 01 H R T RT 3 01 T = Ef(ei V `6 ` V ` Ru` )(u` R V ` 6 ` V ` e j )g T V `6 01V H Ru` )(ej V `6 01V H Ru` )g T = Ef(ei ` ` R ` ` R 1 T 01 H T 01 H = N2 (ei V `6 ` V ` Ru` )(ej V `6 ` V ` Ru`) ` (eT V `6 01V H u`)(eT V `6 01V H u` ) = 0: (40) = j ` ` ` ` N i In the last equation, we used V H u` = 0 based on the denition of ` V ` : These results correspond to (4) and (5).
and the C. Deriving the Cross-Covariance Expressions Throughout this section, we assume that expression for Efu` uk g: u uH

EfwT q ` q k g w 3H = Efq ` w q k g qH H H R 01 H H R 01 H = Ef(u` RV ` 6 ` V ` w )(uk RV k 6 k k )g 1 H 01 H H 01 H = N (u` RV k6 k V k )(uk RV `6 ` V ` w ) H ` k uk (1)u` : =0 2 (k 0 ` ) N
Collecting all the expressions, we nally get

(48)

 (1) H H Efu` uk g = 2N (`k 0` k )2 u`uk 0 uk(1) u`u` u uH u` u` H 0 uk(1) ukuk : (1)
Following similar considerations, we obtain

(49)

` 6= k: We now nd an

3 u H H Efu` uk g = Efq`qk g + Ef ` k gu`uk + u` Ef `qk g u uH q H (41) + Efq ` k guk : q 3 uH

[see also (42)]

 T T u` Efu` uk g = 2N (`k 0` k )2 02uku` 0 u`uk + uk(1) uk u uT (1) uk (1) u`uH : T 1 uk + u` (1) ` (50) Note that under the assumption of [2], that ` = k = 0, we get Efu` uk g = Efq`q k g = 0 u uH q H
(51)

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 46, NO. 11, NOVEMBER 1998

3139

and the

i; j element of Efu` uT g is given by u uk eiT Efu` uk gej u uT e = Efei q ` q k e j g eT T T 01 H R T RT 3 01 T = Ef(ei V `6 ` l Ru` )(uk R V k 6 k V k e j )g T V 6 01V H Ru )(eT V 6 01V H R ]u )g = Ef(ei ` ` ` R ` j k k k R k 1 T 01 H T 01 H = N (ei V `6 ` V ` Ruk )(ej V k6 k V k Ru` ) `k (eT V `6 01V H uk )(eT V k6 01V H u`) = ` j ` k k N i T u uT e k ` ei k ` j : =0 (52) N (k 0 `)2
T u ` k Efu`uk g = Efq `q k g = 0 N` (k ku` )2 u uT q T 0
(53)

instead of the conventional method of computing the sample covariance matrix K 1 ^ ^ u ^ ^j C = K ui (k)^iT 0 i H (58) k=1 where
1 i = K ^

K k=1

ui (k): ^

(59)

IV. CONCLUSIONS Formulas for the statistics of the eigenvectors of sample covariances are widely used in the statistical signal processing literature. In this correspondence, we point out a potential difculty in reconciling theoretical and experimental results due to the nonuniqueness of eigenvectors of complex matrices. We present two solutions to this problem. First, we derive a new set of theoretical formulas for the covariance of the eigenvectors produced by numerical simulations. Next, we present a method for adjusting the eigenvectors produced by numerical simulations so that their statistics match the existing formulas. Both solutions have been veried by numerical simulations, which are not presented here. ACKNOWLEDGMENT The authors would like to thank Dr. N. Yuen for asking the question that led to the work described in this correspondence. REFERENCES
[1] J. H. Wilkinson, The Algebraic Eigenvalue Problem. New York: Oxford Univ. Press, 1965. [2] D. R. Brillinger, Time Series Data Analysis and Theory. San Francisco, CA: Holden-Day, 1981. [3] M. Kaveh and A. J. Barabell, The statistical performance of MUSIC and the minimum-norm algorithms in resolving plane waves in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-34, pp. 331341, Apr. 1986. [4] R. O. Schmidt, Multiple emitter location and signal parameter estimation, in Proc. RADC Spectrum Estimation Workshop, Grifths AFB, NY, 1979, pp. 243258. Reprinted in IEEE Trans. Antennas Propagat., vol. AP-34, pp. 276280, Mar. 1986.

In matrix form, we get

which corresponds to (5). III. FIXING


THE

SIMULATION

In this section, we show how to modify slightly the numerical computations so that the results will match the existing formulas of [1] and [2]. First, we note that MATLABs svd function, for example, produces a set of singular vectors (which can be used in lieu of eigenvectors for the Hermitian covariance matrix) that are normalized to unit norm and have a real rst entry. Thus, the eigenvectors computed by MATLAB simulations correspond to the case where is = 0: Next, we note that the derivation of [2] is making use of an implicit assumption that the perturbed eigenvector has the property that the perturbation is orthogonal to the eigenvector itselfnote that the perturbation term include all but the ith eigenvector [see (19) and (20) with ` = 0]. Using this property, we have a mechanism for identifying the phase ia , which is needed in order to rotate the computed eigenvector into the eigenvector, which is consistent with the analytical results. Given the unperturbed eigenvector ui and the ^ perturbed eigenvector ui , we want to nd the rotated eigenvector ui ej such that the difference ui ej 0ui will be orthogonal to ui : ^ ^ u Given the well-known orthogonality property of linear least-squares methods, it follows that ej must be the least-squares solution of the equation ui = u i x or ^ H ^ ej = juiH ui j : (54) ui ui ^ H Note that the division by jui ui j is necessary to ensure that the result ^ will have unit modulus. To summarize, we propose to evaluate the experimental covariance matrix of the eigenvectors by K 1 ~ ~ u ~ ~j (55) C = K ui (k)~iT 0 i H k=1 where

H ^ ui = ui juiH ui j ~ ^ u u ^ i i
and
1 i = K ~

(56)

K k=1

ui (k) ~

(57)

You might also like