Slides Seminaire Validation

Introduction
Backtesting Principles
Testing strategies
Recommandations
Backtesting Value-at-Risk Models
Christophe Hurlin
University of Orléans
Séminaire Validation des Modèles Financiers. 29 Avril 2013
Christophe Hurlin Backtesting

Introduction
Testing strategies
Recommandations
Introduction
The Value-at-Risk (VaR) and more generally the Distortion Risk

Measures (Expected Shortfall, etc.) are standard risk measures
used in the current regulations introduced in Finance (Basel 2), or
Insurance (Solvency 2) to …x the required capital (Pillar 1), or to
monitor the risk by means of internal risk models (Pillar 2).

Introduction
Testing strategies
Recommandations
Introduction
De…nition
Let frt gTt=1 be a given P&L series. The daily (conditional) VaR for
a nominal coverage rate α is de…ned as
Pr[ rt < VaR t jt 1 (α) Ft 1] =α
where Ft 1 denotes the set of information available at time t 1.

Introduction
Testing strategies
Recommandations
Introduction
Who does use VaR? What for?

Bank risk manager Measure …rm-level market, credit, op. risk
Bank executives Set limits (management)
Banking regulators Determine capital requirements
Exchanges Compute margins
Regulators Forecast systemic risk (CoVaR)
Industry Ex: EDF, spot prices of electricity

Introduction
Testing strategies
Recommandations
"Disclosure of quantitative measures of market risk, such

as value-at-risk, is enligthening only when accompanied
by a thorough discussion of how the risk measures were
calculated and how they related to actual performance",
Alan Greenspan (1996)

Introduction
Testing strategies
Recommandations
Introduction
De…nition
Backtesting is a set of statistical procedures designed to check if
the real losses are in line with VaR forecasts (Jorion, 2007).

Introduction
Testing strategies
Recommandations
Introduction
Whatever the type of use of VaR, the VaR forecasts are

generated by an internal risk model.
This model is used to produced a sequence of pseudo out-of

sample VaR forecasts for a past period (typically one year)
The backtesting is based on the comparison of the observed

P&L to these VaR forecasts.

Introduction
Testing strategies
Recommandations
Outlines
1 How to test the validity of a VaR model?

2 What are the backtesting strategies?
3 What are the good practices?

Introduction
Testing strategies
Recommandations

Introduction
Testing strategies
Recommandations
Remark 1: Ex-post VaR is not observable, so it is impossible to

compute traditional statistics or criteria such as MSFE.
Remark 2: There is no proxy for the VaR contrary to the volatility

(realized volatility, Andersen and Bollerslev 1998)
Patton, A.J. (2011) Volatility forecast comparison using imperfect volatility
proxies, Journal of Econometrics, 260, 246-256.

Introduction
Testing strategies
Recommandations
Backtesting procedures are based on VaR exceptions
De…nition
We denote It (α) the hit variable associated to the ex-post
observation of an α% VaR exception at time t :
(
1 if rt < VaR t jt 1 (α)
It (α) =
0 else

Introduction
Testing strategies
Recommandations
Christo¤ersen (1998) : VaR forecasts are valid if and only if the

violation process It (α) satis…es the following two assumptions:
1 The unconditional coverage (UC) hypothesis.

2 The independence (IND) hypothesis.
Christo¤ersen P.F. (1998), Evaluating interval forecasts, International Economic

Review, 39, pp. 841-862.

Introduction
Testing strategies
Recommandations
De…nition (unconditional coverage hypothesis)

The unconditionnal probability of a violation must be equal to the
α coverage rate
Pr [It (α) = 1] = E [It (α)] = α.
If Pr [It (α) = 1] > α, the risk is under-estimated
If Pr [It (α) = 1] < α, the risk is over-estimated

Introduction
Testing strategies
Recommandations
De…nition (independence hypothesis)

VaR violations observed at two di¤erent dates must be
independently distributed.
It (α) and Is (α) are independently distributed for t 6= s

Introduction
Testing strategies
Recommandations
Figure: Illustration: violations’cluster
8
VaR(95%)
P&L
6
-2
-4
-6
0 50 100 150 200 250

Introduction
Testing strategies
Recommandations
Figure: Illustration: violations’cluster
8
VaR(95%)
P&L
6
-2
-4
-6
0 50 100 150 200 250

Introduction
Testing strategies
Recommandations
De…nition (conditional coverage hypothesis)

The violation process satis…es a di¤erence martingale assumption.
E [ It (α) j Ft 1] =α

Introduction
Testing strategies
Recommandations
Remark: These assumptions can be expressed as distributional

assumptions.
Under the UC assumption, each variable It (α) has a Bernouilli
distribution with a probability α.
Itt (α) Bernouilli (α)
Under the IND assumption, these variables are independent, and

the number of violations has a Binomial distribution.
T
∑ It (α) B (T , α )
t =1

Frequency-based tests
Introduction
Magnitude-based tests
Multivariate tests
Testing strategies
Independence tests
Recommandations
Duration-based tests
Testing strategies
What are the backtesting strategies?

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Let us consider a sequence of daily VaR out-of-sample forecasts

T
VaR t jt 1 (α) t =1 and the corresponding observed P&L.
How to test the validity of the internal risk model?
Hurlin C. and Pérignon C. (2012), Margin Backtesting,

Review of Futures Market, 20, pp. 179-194

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Testing strategies:
1 Frequency-based tests
2 Magnitude-based tests
3 Multivariate tests
4 Independence tests
5 Duration-based tests

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies: frequency-based tests (1/5)
Figure: BIS "Tra¢ c Light" System
Note: VaR(1%, 1 day), 250 daily observations

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition
Christo¤ersen (1998) proposes a Likelihood Ratio statistic for UC
de…ned as:
h i
LRUC = 2 ln (1 α)T H αH
h i
d
+2 ln (1 H/T )T H (H/T )H ! χ2 (1)
T !∞
where H = ∑Tt=1 It (α) denotes the total number of exceedances.
For a nominal risk of 5%, the null of UC can not be rejected if

and only if H < 7 for T = 250 and α = 1%.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Example
Berkowitz and O-Brien (2002) consider the VaR forecasts of six US
commercial banks
Berkowitz, J., and O-Brien J. (2002), How Accurate are the

Value-at-Risk Models at Commercial Banks, Journal of
Finance.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Bank Daily VaR Models

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Violations of Banks’99% VaR

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Testing strategies:

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies: magnitude-based tests (2/5)
All these tests do not take into account the magnitude of the
losses beyond the VaR
Example
Consider two banks that both have a one-day 1%-VaR of $100
million. Assume each bank reports three VaR exceptions, but the
average VaR exceedance is $1 million for bank A and $500 million
for bank B.
In this case, standard backtesting methodologies would indicate
that the performance of both models is equal and acceptable.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Daily VaR and P/L for SocGen 2008

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Daily VaR and P/L for SocGen 2008

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
The Risk Map
Colletaz G., Hurlin C. and Perignon C. (2013), The Risk

Map: a new tool for Risk Management, forthcoming in
Journal of Banking and Finance

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
We propose a VaR backtesting methodology based on the number

and the severity of VaR exceptions: this approach exploits the
concept of "super exception".
De…nition
We de…ne a super exception using a VaR with a much smaller
coverage probability α0 , with α0 < α. In this case, a super
exception is de…ned as a loss greater than VaRt (α0 ).

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: VaR Exception vs. VaR Super Exception

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Solution
Given VaR exceptions It (α) and VaR super exception It (α0 ), we
de…ne a Risk Map that jointly accounts for the number and the
magnitude of the VaR exceptions
Let us consider a given UC test with a statistic Z (α) based on the

violations sequence fIt (α)gTt=1 .
H0 : E [It (α)] = α
H1 : E [It (α)] 6= α.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Number of VaR Exceptions (N)
Non-rejection area for test

on VaR exceptions

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Based on the same UC test, it is possible to test for the

magnitude of VaR exceptions, via the VaR super exceptions
fIt (α0 )gTt=1
H0 : E It α0 = α0
H1 : E It α0 6= α0

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Non-rejection
area for test
on VaR super
exceptions
Number of VaR Super Exceptions (N’)

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
We can also test jointly for both magnitude and frequency of VaR
exceptions:
H0 : E [It (α)] = α and E It α0 = α0
Multivariate approach
Perignon C. and Smith, D. (2008), A New Approach to
Comparing VaR Estimation Methods, Journal of
Derivatives

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
15
14
13
12
11
10
9
8
7
6
5
4
3 Nominal risk 5%
2
1 Nominal risk 1%
0
0 1 2 3 4 5 6 7 8
Number of VaR Super Exceptions (N')

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Backtesting Bank VaR: La Caixa (2007-2008)

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
15
14
13
12
11
10
9
8
7
6
5
4
3 Nominal risk 5%
2
1 Nominal risk 1%
0
0 1 2 3 4 5 6 7 8
Number of VaR Super Exceptions (N')

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Testing strategies:

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies: multivariate tests (3/5)
Intuition: Testing the validity of the VaR model for M coverage

rates, with M > 1.
Perignon C. and Smith, D. (2008), A New Approach to

Comparing VaR Estimation Methods, Journal of
Derivatives
Hurlin C. and Tokpavi, S. (2006), ”Backtesting

Value-at-Risk Accuracy: A Simple New Test”, Journal of
Risk

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Perignon and Smith (2008) consider the null:
H0,MUC : E [It (α)] = α and E It α0 = α0 .

Let us denote:
J0,t = 1 J1,t J2,t

0)
1 if VaR t jt 1 (α < rt < VaR t jt 1 (α)
J1,t =
0 otherwise
1 if rt < VaR t jt 0)
1 (α
J2,t = .
0 otherwise

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition (Perignon and Smith, 2008)

The multivariate unconditional coverage test is a LR test given by:
h i
H H
LRMUC = 2 ln (1 α)H 0 α α0 1 α0 2
" #
H 0 H 0 H0 H1 H 1 H 2 H 2
+2 ln 1 .
T T T T
where Hi = ∑Tt=1 Ji ,t , for i = 0, 1, 2, denote the count variable

associated with each of the Bernoulli variables.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Hurlin and Tokpavi (2006):

A natural test of the CC is the univariate Ljung-Box test of
H0,CC : r1 = ... = rK = 0
where rk denotes the k th autocorrelation:

K
rk2
b
∑
d
LB (K ) = T (T + 2) ! χ2 (K )
k =1
T k T !∞

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition (Hurlin and Tokpavi, 2006)

Let Θ = fθ 1 , .., θ m g be a discrete set of m di¤erent coverage rates
0
and Hitt = [Hitt (θ 1 ) : Hitt (θ 2 ) : ... : Hitt (θ m )]
(
1 θ i if rt < VaR t jt 1 (θ i )
Hitt (θ i ) =
θi else
Under the null of CC (martingale di¤erence):

0
H0,CC : E [Hitt Hitt k = 0m 8k = 1, ..., K

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Testing strategies:

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies: independence tests (4/5)

LR tests
Christo¤ersen (1998) assumes that the violation process It (α) can
be represented as a Markov chaine with two states:
1 π 01 π 01
Π=
1 π 11 π 11
π ij = Pr [ It (α) = j j It 1 (α) = i ]
De…nition
The null of CC can be de…ned as follows:
1 α α
H0,CC : Π = Πα =
1 α α

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

LR tests
Christo¤ersen (1998) assumes that the violation process It (α) can
be represented as a Markov chaine with two states:
1 π 01 π 01
Π=
1 π 11 π 11
π ij = Pr [ It (α) = j j It 1 (α) = i ]
De…nition
The null of IND can be de…ned as follows:
1 β β
H0,IND : Π = Π β =
1 β β

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

The corresponding LR statistics are de…ned by:
h i
LRIND = 2 ln (1 H/T )T H (H/T )H
d
+2 ln [(1 π b n0101 (1
b 01 )n00 π π b n1111 ]
b 11 )n10 π ! χ2 (1)
T !∞
h i
T H H
LRCC = 2 ln (1 α) (α)
d
+2 ln [(1 π b n0101 (1
b 01 )n00 π π b n1111 ]
b 11 )n10 π ! χ2 (2)
T !∞
By de…nition:
LRCC = LRUC + LRIND

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Figure: Violations of Banks’99% VaR

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Regression based tests
Engle and Manganelli (2004) suggest another approach based

on a linear regression model. This model links current margin
exceedances to past exceedances and/or past information.
Let Hit (α) = It (α) α be the demeaned process associated
with It (α):
1 α if rt < VaR t jt 1 (α)

Hitt (α) = .
α otherwise

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Consider the following linear regression model:
K K
Hitt (α) = δ + ∑ βk Hitt k (α) + ∑ γk zt k + εt
k =1 k =1
where the zt k variables belong to the information set Ωt 1

(lagged P&L, squared past P&L, past margins, etc.)

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

The null hypothesis test of CC corresponds to testing the joint
nullity of all the regression coe¢ cients:
H0,CC : δ = βk = γk = 0, 8k = 1, ..., K .
since under the null :
E [Hitt (α)] = E [It (α) α] = 0 () Pr [It (α) = 1] = α

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition (Engle and Manganelli, 2004)

Denote Ψ = (δ β1 ...βK γ1 ...γK )0 the vector of the 2K + 1
parameters in this model and Z the matrix of explanatory variables
of model, the Wald statistic, denoted DQCC , then veri…es:
b 0Z 0Z Ψ
Ψ b d
DQCC = ! χ2 (2K + 1)
α (1 α ) T !∞
b is the OLS estimate of Ψ.

where Ψ

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Extension: A natural extension of the test of Engle and Manganelli
(2004) consists in considering a (probit or logit) binary model
linking current violations to past ones
Dumitrescu E., Hurlin C. and Pham V. (2012),

Backtesting Value-at-Risk: From Dynamic Quantile to
Dynamic Binary Tests, Finance

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition (Dumitrescu et al., 2012)

We consider a dichotomic model:
Pr [ It (α) = 1 j Ft 1] = F (π t ) .
where F (.) denotes a c.d.f. and the index π t satis…es the

following autoregressive representation:
q1 q2 q3
πt = c + ∑ βj π t j + ∑ δj It j (α) + ∑ γj xt j ,
j =1 j =1 j =1
where l (.) is a function of a …nite number of lagged values of

observables, and xt is a vector of explicative variables.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

1
H0 : β = 0, δ = 0, γ = 0 and c = F (α) .
since under the null of CC:
1
Pr(It = 1 j Ft 1) = F (F (α)) = α.
The Dynamic Binary (DB) LR test statistic is:

1
DBLR CC = 2 ln L(0, F (α); It (α), Zt ) ln L(θ̂, ĉ; It (α), Zt )
d
! χ2 (dim(Zt ))
T !∞

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies
Testing strategies:

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Testing strategies: duration-based tests (5/5)

The UC, IND, and CC hypotheses also have some implications
on the time between two consecutive VaR margin exceedances.
Let us denote by dv the duration between two consecutive

VaR margin violations:
dv = tv tv 1
where tv denotes the date of the v th exceedance.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Under CC hypothesis, the duration process dv has a geometric

distribution:
Pr [dv = k ] = α (1 α )k 1
k2N .
This distribution characterizes the memory-free property of

the violation process It (α) with E (dv ) = 1/α

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition
Christo¤ersen and Pelletier (2004) use under the null hypothesis
the exponential distribution:
g (dv ; α) = α exp ( αdv ) .
Under the alternative hypothesis, they postulate a Weibull

distribution for the duration variable:
h i
h (dv ; a, b ) = ab bdvb 1 exp (adv )b .
H0,IND : b = 1 H0,CC : b = 1, a = α

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Drawback: we have to postulate a distribution for the duration

under the alternative (misspeci…cation of the VaR model).
Solution: Candelon et al. (2001) propose a J-test based on
orthonormal polynomials associated to the geometric distribution.
Candelon B., Colletaz G., Hurlin C. et Tokpavi S. (2011),

"Backtesting Value-at-Risk: a GMM duration-based
test", Journal of Financial Econometrics,

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Candelon et al. (2001)

In the case of continuous distributions, the Pearson distributions
(Normal, Student, etc.) are associated to some particular
orthonormal polynomials whose expectation is equal to zero.
These polynomials can be used as special moments to test for
a distributional assumption (see. Bontemps and Meddahi,
Journal of Econometrics, 2005).
In the discrete case, orthonormal polynomials are de…ned for
distributions belonging to the Ord’s family (Poisson, Pascal,
hypergeometric, etc.).

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

De…nition
The orthonormal polynomials associated to a geometric
distribution with a success probability β are de…ned by the
following recursive relationship, 8d 2 N :
(1 β) (2j + 1) + β (j d + 1)
Mj +1 (d; β) = p Mj (d; β)
(j + 1) 1 β
j
Mj 1 (d; β) ,
j +1
for any order j 2 N , with M 1 (d; β) = 0 and M0 (d; β) = 1 and:
E [Mj (d; β)] = 0 8j 2 N , 8d 2 N .

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
Example
We can show that if d follows a geometric distribution of
parameter β, then:
p
M1 (d; β) = (1 βd ) / 1 β
with
E [M1 (d; β)] = 0

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

Our duration-based backtest procedure exploits these moment
conditions.
More precisely, let us de…ne fd1 ; ...; dN g a sequence of N
durations between VaR violations, computed from the
sequence of the hit variables fIt (α)gTt=1 .
Under the CC assumption, the durations di , i = 1, .., N, are
i.i.d. geometric(α). Hence, the null of CC can be expressed
as follows:
H0,CC : E [Mj (di ; α)] = 0, j = f1, .., p g ,
where p denotes the number of moment conditions.

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations

De…nition
The null hypothesis of CC can be expressed as
H0,CC : E [M (di ; α)] = 0,
where M (di ; α) denotes a (p, 1) vector whose components are the

orthonormal polynomials Mj (di ; α) , for j = 1, .., p. Under some
regularity conditions:
!| !
1 N 1 N
JCC (p ) = p ∑ M (di ; α) p ∑ M (di ; α)
d
! χ2 (p )
N i =1 N i =1 N !∞

Introduction
Multivariate tests
Testing strategies
Independence tests
Recommandations
De…nition
Under UC, the mean of durations between two violations is equal
to 1/α, and the null hypothesis is
H0,UC : E [M1 (di ; α)] = 0.
with a test statistic equal to

!2
N
1
∑ M1 (di ; α)
d
JUC = p ! χ2 (1)
N N !∞
i =1

Introduction Test, test and test
Backtesting Principles Check the P&L data
Testing strategies The power of your tests may be low...
Recommandations Estimation risk
Backtesting
Recommandations

Backtesting
Recommandation 1: Test, test and test

Recommandation 2: Check the P&L data
Recommandation 3: The power of your tests may be low..
Recommandation 4: Take into account the estimation risk

Backtesting
Recommandation 1: Test, test and test
Each type of test (frequency, severity, independence,

conditional coverage, multivariate test etc..) captures one
type of potential misspeci…cation of the VaR model.
It is important to use a variety of tests

Backtesting

Backtesting

Backtesting
Recommandation 2: Check the P&L data
Frésard, L., C. Perignon, and W., Anders (2011), The

Pernicious E¤ects of Contaminated Data in Risk Management,
Journal of Banking and Finance.
1 A large fraction of US and international banks validate their
market risk model using P&L data that include fees and
commissions and intraday trading revenues.
2 Distinction between dirty P/L and hypothetical P/L (JP.
Morgan, Romain Berry 2011).

Backtesting
Recommandation 3: The power of your tests may be low..
De…nition
The power of a backtesting test corresponds to its capacity to
detect misspeci…ed VaR model.
Pr [ Rejection H0 j H1 ]
Example
Berkowitz, J., Christo¤ersen, P. F., and Pelletier, D., 2013,
Evaluating Value-at-Risk Models with Desk-Level Data.
Management Science.

Backtesting

Backtesting

Backtesting

Backtesting
Hurlin C. et Tokpavi S. (2008), ”Une Evaluation des

Procédures de Backtesting : Tout va pour le Mieux dans le
Meilleur des Mondes", Finance
Idea: we use 6 di¤erent methods (GARCH, RiskMetrics, HS,
CaviaR, Hybride, Delta Normale) to forecast a VaR(5%) on the
same asset (GM, Nasdaq), and we apply the backtests (LR, DQ,
Duration based tests) on a set of 500 samples (rolling window) of
T = 250 forecasts.

Backtesting
Example
LRCC tests: for 47% of the samples, we don’t reject (at 5%) the
null for any of the six VaR forecats. In 71% of the samples, we
reject at the most one VaR.
Example
DQCC tests: for 20% of the samples, we don’t reject (at 5%) the
null for any of the six VaR forecats. In 51% of the samples, we
reject at the most one VaR.

Backtesting
The power of a consistent test tends to 1 when the sample

size tends to ini…nity.
Recommandation: increase at the maximum the sample size

of your backtest.. (T = 500, 750 or more.)

Backtesting
Recommandation 4: Take into account the estimation risk
The risk dynamic is usually represented by a parametric or

semi-parametric model, which has to be estimated in a
preliminary step. However, the estimated counterparts of risk
measures are subject to estimation uncertainty.
Replacing, in the theoretical formulas, the true parameter

value by an estimator induces a bias in the coverage
probabilities.

Backtesting

Backtesting
Escanciano and Olmo (2010, 2011) studied the e¤ects of

estimation risk on backtesting procedures. They showed how
to correct the critical values in standard tests used to assess
VaR models.
Escanciano, J.C. and J. Olmo (2010) Backtesting Parametric

Value-at-Risk with Estimation Risk, Journal of Business and
Economics Statistics.
Escanciano, J.C. and J. Olmo (2011) Robust Backtesting Tests

for Value-at-Risk Models. Journal of Financial Econometrics.

Backtesting
Estimation Adjusted VaR

Gouriéroux and Zakoian (2013) a method to directly adjust the
VaR to estimation risk ensuring the right conditional coverage
probability at order 1/T :
Pr rt < EVaR t jt 1 (α) = α + oP (1/T )
Gouriéroux C. and Zakoian J.M. (2013), Estimation Adjusted

VaR, forthcoming in Econometric Theory.

Thank you for your attention

Slides Seminaire Validation

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Slides Seminaire Validation

Uploaded by

Copyright:

Available Formats

Introduction

Backtesting Value-at-Risk Models

Séminaire Validation des Modèles Financiers. 29 Avril 2013

Christophe Hurlin Backtesting

The Value-at-Risk (VaR) and more generally the Distortion Risk

Christophe Hurlin Backtesting

Pr[ rt < VaR t jt 1 (α) Ft 1] =α

where Ft 1 denotes the set of information available at time t 1.

Christophe Hurlin Backtesting

Who does use VaR? What for?

Christophe Hurlin Backtesting

"Disclosure of quantitative measures of market risk, such

Christophe Hurlin Backtesting

Christophe Hurlin Backtesting

Whatever the type of use of VaR, the VaR forecasts are

This model is used to produced a sequence of pseudo out-of

The backtesting is based on the comparison of the observed

Christophe Hurlin Backtesting

1 How to test the validity of a VaR model?

Christophe Hurlin Backtesting

Christophe Hurlin Backtesting

Remark 1: Ex-post VaR is not observable, so it is impossible to

Remark 2: There is no proxy for the VaR contrary to the volatility

Christophe Hurlin Backtesting

Backtesting procedures are based on VaR exceptions

Christophe Hurlin Backtesting

Christo¤ersen (1998) : VaR forecasts are valid if and only if the

1 The unconditional coverage (UC) hypothesis.

Christo¤ersen P.F. (1998), Evaluating interval forecasts, International Economic

Christophe Hurlin Backtesting

De…nition (unconditional coverage hypothesis)

Pr [It (α) = 1] = E [It (α)] = α.

If Pr [It (α) = 1] > α, the risk is under-estimated

If Pr [It (α) = 1] < α, the risk is over-estimated

Christophe Hurlin Backtesting

De…nition (independence hypothesis)

It (α) and Is (α) are independently distributed for t 6= s

Christophe Hurlin Backtesting

Figure: Illustration: violations’cluster

Christophe Hurlin Backtesting

Figure: Illustration: violations’cluster

Christophe Hurlin Backtesting

De…nition (conditional coverage hypothesis)

Christophe Hurlin Backtesting

Remark: These assumptions can be expressed as distributional

Itt (α) Bernouilli (α)

Under the IND assumption, these variables are independent, and

Christophe Hurlin Backtesting

Christophe Hurlin Backtesting

Let us consider a sequence of daily VaR out-of-sample forecasts

Hurlin C. and Pérignon C. (2012), Margin Backtesting,

Christophe Hurlin Backtesting

Christophe Hurlin Backtesting

Testing strategies: frequency-based tests (1/5)

Figure: BIS "Tra¢ c Light" System

Note: VaR(1%, 1 day), 250 daily observations

Christophe Hurlin Backtesting

Testing strategies: frequency-based tests (1/5)

where H = ∑Tt=1 It (α) denotes the total number of exceedances.

For a nominal risk of 5%, the null of UC can not be rejected if

Christophe Hurlin Backtesting

Testing strategies: frequency-based tests (1/5)

Berkowitz, J., and O-Brien J. (2002), How Accurate are the

Christophe Hurlin Backtesting