Professional Documents
Culture Documents
Point Estimation
Estimator (formula) function of data to give estimates Method of Moments Least Squares Maximum Likelihood Estimates
Data X, Y
Note: The Estimator is a random variable, as the value of the estimate changes as dierent samples are drawn. Thus, the estimator has a probability distribution. The samples are regarded as constants. Measuring the Preference of an Estimator: Mean Square Error (MSE) - Measures Tradeo Between Bias and Eciency
MSE = E
= E E + E = = Var( ) + Bias( )
) (
Bias = E
Therefore, if an estimator is unbiased:
() ()
Bias = 0 E =
Disadvantages: No equal weights, magni es values that >1 and miniaturizes values that < 1 Mean Absolute Deviations (MAD)
()
()
MAD = E
Method of Moments Estimator (MME):
Kth Sample Moment (non-centered) k=1
1 n Xi = X n i=1 1 n 2 Xi n i=1
...
E ( X)
E X 2 = Var ( X ) + E ( X )
...
k=2
( )
1 n k Xi n i=1
Computable from Data
E ( Xk )
Not Computable Depends on unknown parameters
n n l ( ) = ln L ( ) = ln f ( X i ; ) = ln f ( X i ; ) i=1 i=1
Step 2: Maximize Likelihood Function Maximize log-likelihood function by dierentiation or by observation
dl ( ) =0 d =
Note: In likelihood function, Xs (sample data) are treated as constants, and (parameter) is a variable First order conditions are only application if the function is continuous dierentiable Properties: Under suitable conditions, MLE and MME have nice large sample properties Consistent -
n , MSE 0
Asymptotically Normal As n approaches in nity, under some regularity conditions,
()
1 n N 0, I ( )
1 1 N 0, N , nI ( ) nI ( )
2 2 2 I ( ) = E 2 ln f ( X; ) = f ( X; ) i 2 ln f ( X; ) = f ( X; ) i 2 ln f ( X; )
On top of this, MLE is also: Invariant
MLE ( ) = MLE g ( ) = g
More Ecient than MME
()
Conceptually, MLE uses more information in calculation than MME Hence, we always use MLE when possible
two. Con dence Intervals Con dence Interval for Parameter at 90/95/99% interval
C SE
()
Length of Con dence Interval = 2 x SE X% Con dence Interval: If one repeats the interval estimation a large number of times, about x% of times the interval estimator covers the true Con dence Intervals for Population Mean
Population Normally Distributed Variance Known Variance Unknown Population Not Normal If N is large, By Central Limit Theorem For Proportion Simpli cation after CLT
X N ( 0,1) n
Use Z-table
X t n1 s n
Use t-table
X N ( 0,1) n
approximately Use Z-table
X N ( 0,1) (1 ) n
Use Z-table
Chi-Squared Distribution
( n 1) s2
2 n1
( n 1) s 2 ( n 1) s 2 is 2 , 2 1 /2,n1 /2,n1
Proof
i = 1, 2, ..., n
1 n Xi 2 = 2 (X i )2 n i=1 i=1
n n 2 2 2 1 n 1 n 1 n (X i )2 = 2 ( X i X ) + ( X ) = 2 ( X i X ) + ( X ) + 2 ( X ) ( X i X ) 2 i=1 i=1 i=1 i=1 i=1
2 2 1 n X ) = 2 (X ) 2 ( i=1
X 12 = n
2 1 n ( n 1) 1 n X X 2 = ( n 1) s2 2 2 = 2 2 ( Xi X ) = ) 2 ( n 1 n1 i=1 2 n 1 i=1 i
t-Distribution
T=
Z tk Xk
t is a continuous and symmetric distribution on (- ,), with heavier tails than the normal distribution. As k approaches in nity, the t distribution converges to the standard normal distribution. Test for Population Mean:
X t n1 s n
s X t /2,n1 n
Proof
n
2
N ( 0,1) and
( n 1) s2
2 n1
2 n
( n 1) s2
2 n
n 1
2 X = t n1 s s2 n
F-Distribution
U~
2 p
V~
2 k
W= Fp,k = [ 0, )
2 Y s2 s2 X X = 2 2 = r 2 ~ Fn1,m1 X sY sY
H 0 : = k vs H1 : k
two three
H 0 : = k vs H1 : > k / H 0 : = k vs H1 : < k
Compute the Test Statistic (T)
Look Up Critical Values at Level of Signi cance (1%, 5%, 10%) One-tail Test:
Compute the p-value p-value is the smallest level of signi cance that Ho can be rejected. Let t be the calculated test statistic
P(T > C ) =
If T > C , reject at level of significance
Two-tail Test:
P( T > C /2 ) =
If T > C /2 , reject at level of significance
four
Conclude Reject even at the 1% level of signi cance / Do not reject even at the 10% level of signi cance
Errors in Hypothesis Testing Type 1 Error: Type 2 Error: Power: Reject the Null Hypothesis when H0 is true Not rejecting H0 when H1 is true P ( Rejecting H0 when H1 is true ) Note: P ( Type 2 Error ) + Power = 1
For Example:
H 0 : = 0 vs H1 : 0 T=
Type 1 Error
X 0 Under H 0 , T N(0,1) n
Type 2 Error
1 0 T ~ N ,1 n
P(Type 2 Error) = P(T, under H1, lies within critical values) Power = P(T, under H1, lies outside critical values)
Properties
Trade o between Type 1 and Type 2 error As falls, P( Type 2 Error) Increases If variance increases, P( Type 2 Error) Increases
As the distance 0 - 1 increases, P( Type 2 Error) Falls If no. of samples increases, variance falls, P( Type 2 Error) Increases
2 2 X Y X N X , Y N Y , nX nY
X k = a
Matched Pairs Two distributions can be logically linked, same sample size
nX = nY = n
2 2 X Y Z = X Y N X Y , + n n
nX nY T=
( X Y) a SE ( X Y)
T=
Za
+ n n
2 X
2 Y
~ N(0,1)
SE ( X Y) = T=
2 2 X Y + nX nY
( X Y) a ~ N ( 0,1)
2 2 X Y + nX nY
T=
Za 1 ~ t n1 where s 2 = n 1 s n
( Z
2 i
nZ 2
2 2 X Y Z /2 X + Y nX nY
If Variance unknown, but equal Test for Correlation E ( X EX ) ( Y EY) Cov ( X, Y) Corr ( X, Y) = = Var ( X ) Var ( Y) ( X EX )2 ( Y EY)2 Correlation measures the linear relationship between X and Y. When p=0, X and Y are linearly independent.
2 2 X = Y = 2
Pooled Variance = s 2 = P
( n x 1) s2X + ( n Y 1) s2Y ( n x 1) + ( n Y 1)
H 0 : = 0 vs H1 : > 0 / < 0 / 0
Sample Correlation Coecient:
=
( X X ) ( Y Y ) ( X X ) ( Y Y)
2
Xi Yi nXY
(n 1)s Xs Y
T=
( X Y) a =
s s + nX nY
2 P 2 P
2 2 1 1 ( n x 1) s X + ( n Y 1) s Y n + n ( n 1) + ( n 1) X Y x Y
( X Y) a
~ t n+m2
T=
n2 = 1 2
n2 ~t 1 2 1 n2
Goodness of Fit Test 1. H0: r.v. X follows a certain distribution H1: r.v. X does not follow a certain distribution
Note: in cases where the parameters of the distribution are not given, use MLE/MME to get a point estimate 2. Construct the Table and calculate the Test Statistic
1 Observed Frequency, Zi Probability, p Expected Frequency, Ei Dierence Zi - Ei (Zi - Ei)2 / Ei np1 np2 2 ... Total n 1 n 0 T
T=
i=1
( Z i E i )2 ~ 2
Ei
n
T=
i=1
( Z i E i )2 =
Ei
Note: If any category has expected cell count < 5, then merge groups so that all groups have expected counts that are more than or equal to 5. For some cases, intervals or groups can be self created. For example, in a test for normality, one can divide the line into 10 intervals with the probability at each interval being 10%. Contingency Tables / Tests of Association
Y Z 1 1 2 X ... r ... ... ... ... ... Zrc
r
E 2 ... ... ... c Z1c Z2c X Zr1 Zr2 Z11 Z12 Z21 Z22
T =
i=1 j=1
( Z i E i )2 ~ 2
Ei
pd
p - number of free counts among Z d - no. of the estimated free parameters for most cases, p-d = (r-1)(c-1)
Z Z ii , p i j = i j p ij = p ii p i j E ij = n p ij n n Z + Z12 + ...+ Z1n 11 Test for several Binomial Distributions: H 0 : p11 = p12 = ... = p1n = p p= Z1 + Z2 + ...+ Zn
Test of independence: H 0 : p ij = p ii p i j p ii =
four. Simple Linear Regression Model ALL FORMULAS BELOW ARE FOUND IN THE FORMULA SHEET
Testing of B0 and B1
i i.i.d. N ( 0, 2 )
y i ~ N ( 0 + 1x i , 2 )
2 =
1 n y i 0 + 1x i n 2 i=1
( )
( )
= 0 T= 0 0 = H0 : 0 SE 0
( )
~ t n2
H 0 : 1 = 0 T= 1 1 = SE 1
( )
1 1 / (x i x)
2 i=1 n 2
~ t n2
n 2 2 1 t /2,n2 / (x i x) i=1
( n 2 ) 2
1 n y i 0 + 1x i 2 i=1
2 ~ n2
ANOVA
Total SS = (y i y) = y 2 ny 2 i
i=1 i=1
( )
~ F1,n2
R=
R adj =
Regression SS / ( n 2 ) Total SS / ( n 1)
STATISTICS (ST102) MICHAELMAS TERM MATERIAL Notepad for Probability Basic axioms, independence, mutual exclusion, pairwise disjoint/partitions, total probability, Bayes theorem, permutations and combinations
c.d.f
E(X)
Var(X)
m.g.f.
Misc
Discrete Random Variables 1: Discrete Uniform Distribution pdf: cdf: mgf: mean:
variance:
variance:
variance:
variance:
variance:
variance:
variance:
Notepad for Multivariate Random Variables Joint distributions, Marginal Distributions, Conditional Distributions, Covariance and Correlation