Model Validation Policy

Solvency II Internal Model

XYZ
Table of content

Expected business outcomes .................................................................................................... 3

1 Purpose and scope ........................................................................................................ 1
1.1 Introduction and objectives ........................................................................................ 1
1.2 Scope of application .................................................................................................. 2
1.3 Validation principles ................................................................................................... 2
1.4 Roles and responsibilities .......................................................................................... 3
1.4.1 Board and executive management ........................................................................ 3
1.4.2 Model developers .................................................................................................. 3
1.4.3 Risk management function .................................................................................... 3
1.4.4 Role of internal audit .............................................................................................. 4
2 Standards and requirements ......................................................................................... 4
2.1 Independent review ................................................................................................... 4
2.2 Regulation and related requirements ........................................................................ 4
2.3 Frequency of the review ............................................................................................ 5
2.4 Risk-based validation................................................................................................. 5
2.5 Reporting requirements ............................................................................................. 5
2.6 Model documentation ................................................................................................ 6
2.7 Local considerations and procedures ........................................................................ 6
3 Validation process ......................................................................................................... 6
3.1 Validation methods .................................................................................................... 6
3.2 Validation cycle .......................................................................................................... 7
3.3 Validation process ..................................................................................................... 7
3.3.1 Data ....................................................................................................................... 7
3.3.2 Methods and assumptions ..................................................................................... 8
3.3.3 Expert judgement ................................................................................................... 8
3.3.4 Systems (model design) ........................................................................................ 8
3.3.5 Validating the theoretical underpinnings of the model ........................................... 9
3.3.6 Validation tools ...................................................................................................... 9
3.3.7 Reporting and escalation ..................................................................................... 10
3.3.8 Tracking remedial actions .................................................................................... 10
3.3.9 Limitations ............................................................................................................ 10
4 References to supporting material ............................................................................... 10
 Expected business outcomes

The scope of the validation process does not only apply to the system used to calculate the
group’s solvency capital requirements, but encompasses aspects of related actuarial
systems, as well as the review of inputs, model results and use, IT systems, and governance
arrangements.

The following expected business outcomes drive the requirements for systems, processes
and controls outlined in this validation policy:
• The mandatory requirements for model validation (and reporting thereof) under
Solvency II,

• Roles and responsibilities across the group,

One single • The process to ensure the on-going appropriateness of the model (including
validation policy is requirements for reporting to the executive management and board); and
required. The
policy should
apply to all BUs
• The process whereby weaknesses and potential improvements are reported and can
that will use the be fed back into model design.
internal model.
The document sets out the group rules required for model validation and do not apply to local
business units.

The principal components of XYZ’s model validation policy are as follows:

• As a minimum, model validation should meet the requirements for validation set out in
RMF can be the Solvency II Directive, Level 2 implementing measures, Level 3 binding technical
supported by other standards and must consider non-binding Level 3 guidance.
functions, but not
replaced • Independent validation of the model should be led by the risk management function
and reviews should be performed by the risk management function replaced in
particular areas by other functions reviews,
Regular validation
cycle
• Validation should involve a regular cycle of model reviews, with periodic reporting to
executive management and the group board,
Proportionality
assessed from
• All validation work should be planned and performed to a level that is proportionate
the group from the group perspective. In adopting this approach XYZ is focusing validation
perspective only. efforts only on the most significant assumptions and processes, and ensuring that the
structure of the review corresponds to the perceived level of risk. Remaining areas do
Application of the
proportionality not require validation to be performed.
principle in such a
way that validation The completion of a satisfactory Model Validation Report produced under this policy is a
in some areas requirement for Solvency II internal model supervisory approval.
would never take Validation report
place. required
 1 Purpose and scope
1.1 Introduction and objectives
This validation policy (the ‘policy’) sets out the business requirements for validation of the
internal model and supporting risk systems (the ‘model’) for XYZ Group.
The policy has been written to be used by the management teams responsible for model
development and the model validation process only.
XYZ requires that there is an appropriate model validation process in place, in order to ensure
that the internal model and supporting risk systems are fit for purpose. Ensuring fitness for
purpose includes:

• Monitoring the performance of the model;

• Reviewing the ongoing appropriateness of its specification; and

• Testing its results against experience.

This document sets out the group- policy required for model validation and is not to be used
RMF responsibility by local undertakings.
This policy is owned by the group risk management function which is responsible for:

• Maintaining the integrity of the policy content.

• Providing advice, support and technical guidance in relation to this policy.

Board sign-off
The policy has been approved by the XYZ group board.
The policy will be reviewed when the risk management function finds it appropriate to ensure
Should be done at that it is still applicable and aligned with any new developments.
least annually
The policy is intended to meet both XYZ’s interpretation of best industry practice and to
comply with Solvency II requirements. The purpose of this policy is to:

• Set out the scope of the validation process;

• Specify the requirements and minimum standard for the validation process and
reporting for the XYZ group;
• Designate roles and responsibilities in respect of all the aspects of the model
validation;
• Set out the process to ensure the on-going appropriateness of the model;
• Set out the process whereby weaknesses and potential improvements are reported
and can be fed back into the model design;

• Set out the scope and limitations of the validation policy itself, and the process to
ensure the on-going appropriateness of the policy; and
Proportionality
assessed from the • To comply with the requirements of Articles 101, 112, 113 and Articles 120 to 126 of
group perspective the Solvency II Directive.
only.
Processes and controls should be designed and operated to satisfy the mandatory
requirements set out in this policy based on the nature, scale and complexity of the group
business and the nature of risks and challenges the group faces.

Model Validation Policy Page 1

The validation
process should
cover both 1.2 Scope of application
quantitative as
well as qualitative The scope of application of this policy is limited to the group’s definition of Solvency II internal
aspects of the model and supporting risk systems, and is in line with Solvency II requirements.
model and
supporting risk The validation process applies only to the calculation kernel used to calculate the group’s
systems including
verification if they solvency capital requirements.
fit local business
Validation is a set of tools and processes used by XYZ to gain confidence over the results,
design, workings and other processes within the model. These tools and processes used for
validation are quantitative only.

This policy applies to the group. It is the responsibility of the risk management function to
Risk management define the detailed scope and objectives of the validation at the outset of each review, in
function
responsibility
accordance with this policy, the Solvency II Directive, Level 2 Implementing Measures and
Level 3 binding technical standards, and must consider non-binding Level 3 guidance. The
scope includes methods and assumptions underlying the valuation of stressed balance
Independence sheets (e.g. assets, technical provisions and own funds).
from model 1.3 Validation principles
development as
well as model The principles guiding the policy are as follows:
operation and use
should be kept.
RMF should • Independence – the validation processes and reporting should be independent from
check tests and the development of the internal model and supporting risk systems, however in case
analysis of the validation of the local elements, independence is ensured by the revision of
performed and
tests and analysis conducted by local models developers performed by the group risk
make own ones.
management function.

• Responsibility – independent model validation will comprise of a combination of

reviews, performed by the risk management function replaced by other functions
RMF can be where appropriate. In addition, other functions will report to the relevant bodies on
supported by other
functions, but not the results of their assessment of the effectiveness of the validation activity.
replaced
• Mandatory regulatory requirements – model validation should meet the
requirements set out within the Solvency II Directive (to be specific: Articles 101, 112,
113 and Articles 120 to 126) and Level 2 Implementing Measures and Level 3 binding
The validation technical standards, and must consider non-binding Level 3 guidance.
process should
cover both • Materiality – the validation tools and processes should be proportionate to the
quantitative as nature, scale and complexity of the aspect under consideration. The validation
well as qualitative
aspects of the
process will be risk-based.
model.
• Quantitative - model validation should encompass quantitative elements only.

• Governance standards – there should be clear and unambiguous criteria and

process whereby issues must be escalated, for example where the model is deemed
not to have met the validation requirements or where there are requirements for this
policy to be revised.
Regular reviews – • Regular review – the model validation process, reporting and this policy should be
at least annually.
reviewed regularly for effectiveness, having in mind that in case of any change ad-
hoc validation occurs necessary.

Model Validation Policy Page 2

 1.4 Roles and responsibilities
1.4.1 Board and executive management
Board Under the Solvency II Directive the board has responsibility for ensuring the on-going
responsibilities appropriateness of the design and operations of the model, and that the model continues to
indicated
appropriately reflect the risk profile of the undertakings concerned. They have a further
responsibility to make sure that they then use this information to assist in determining any
changes that may be required to the internal model.

The group board is responsible for approving this model validation policy, and making sure it
is adhered to, regularly reviewed and updated as necessary. On a day to day basis the
responsibility for ensuring this policy is adhered to and remains fit for purpose has been
delegated to the group risk management function.

On a day-to-day basis the board responsibilities are delegated to the group's risk
management function which is responsible for demonstrating at the outset of each validation
cycle that the respective entities and the group are compliant with the applicable regulatory
standards and Implementing Measures pertaining to the scope of the validation review

1.4.2 Model developers

Model developers are responsible for performance of internal quantitative validation tasks to
provide group risk management function with information that the group is adhering to internal
standards and regulatory requirements and internal model is ready for independent validation.
Not all important
aspects have been This internal validation should include an assessment of the following:
indicated (e.g. • Monitoring the performance of the model.
model use is • Reviewing the ongoing appropriateness of the specification of the model.
missing) • Quantitative validation of the model, including testing the results of the model against
experience.
• Analyzing the stability of the model, in particular testing sensitivity of results to
changes in key assumptions.
• Reviewing the respective suitability and credentials of individuals who have acted as
experts relating to the use of expert judgement in the model.
Procedures and results should be documented and the documentation and results must be
Checklists should made available to the reviewers during the validation process. If appropriate and adequate
have only a testing has been performed in the businesses, then this may be relied upon to reduce the
supporting role. effort required in completing the independent validation. Model developers are requested to
prepare and fulfil the checklists which are the subject of the examination performed by the risk
management function.
RMF can be
1.4.3 Risk management function
supported by other
functions, but not The independent validation of the model will comprise a combination of reviews performed
replaced
primarily by the risk management function replaced where appropriate by other functions
reviews. Each function will report on the conclusion of their validation reviews to the relevant
bodies to which they report on the regular basis.
Checklists should
have only a The risk management function could fully rely on the checklists prepared and filled in by
supporting role. models developers and there is no obligation to perform additional tests or analysis.

The risk management function oversees each cycle of the model validation. The purpose is to
oversee the appropriateness and timeliness of the review and escalation, and to ensure
RMF responsibility
executive management involvement in the review.

Model Validation Policy Page 3

RMF responsibility It is a requirement that the risk management function prepares a detailed plan for the
with board approval validation process for each validation cycle. This plan should include areas to be tested,
responsibilities for individual validation tasks, materiality assessments, and the timeline for
field work and reporting. The methodology and plan must be approved by the board.

If the risk management function or other functions lacks the resources for effective
independent review, they may use external support in the validation process (subject to the
requirements set out in this policy).

Regular review of The specific timing of the validation activity should be agreed.
the validation
policy with board
approval It is the responsibility of the risk management function to ensure that this policy is reviewed at
least annually and more frequently if there is a significant change in regulatory environment,
to ensure that it remains fit for purpose and captures all relevant legislative requirements.
The conclusions of this review (and if applicable the proposed revisions) must be presented to
the XYZ group board.
Regular review of
the validation
1.4.4 Role of internal audit
process Model validation processes and validation reporting should be subject to periodic independent
review by internal and external auditors to ensure that implementation is effective and is
carried out in compliance with this policy and any relevant regulatory standards.

2 Standards and requirements

Independence from
2.1 Independent review
the operation of IM
is also required. The validation must be performed by personnel independent from the personnel (or firm) who
designed or built the models. Specifically, no individual who has been employed in designing
or building any related aspect of the model in the previous year is permitted to work on the
independent model validation. Personnel employed in the validation activity should be rotated
1
as often as practical, for example between areas of responsibility .

It is the responsibility of the group risk director to ensure that the tasks set out in the validation
process create and maintain independence over time. The independent review of the
validation process by the group’s internal audit must consider whether independence has
been implemented effectively.

2.2 Regulation and related requirements

In addition to adhering to this policy, the development and validation of the internal model
must as a minimum adhere to:
• Articles 101 and 112, 113 and Articles 120 to 126 of the Solvency II Directive, Level 2
Implementing Measures , Level 3 binding technical standards and must consider non-
binding Level 3 guidance.
• All relevant XYZ policies on model development and change control.
• All relevant aspects of methodological and operational manuals, such as technical
provisions manual, own funds manual and SCR manual.

1
In practice, the validation process should recognize that if model changes are implemented in
response to an independent review, then the subsequent review by the same reviewer in future
validation cycles may result in decreased independence over time.

Model Validation Policy Page 4

The validation
should be carried
out at least 2.3 Frequency of the review
annually.
The model review must be carried out regularly, especially if there is a material change in the
model or the model environment. A review may also be carried out at any time at the request
of the group CRO. If a review is carried out at the request of the local CRO, then the group
CRO must be asked for approval of the scope and objectives of this ad-hoc review at the
outset. For the avoidance of doubt, a local CRO may only trigger a review in respect of the
No freedom given
to the local entity. respective business entities and it cannot influence methodology provided by the group.

2.4 Risk-based validation

The group's validation approach should be risk-based. It is not practical to apply validation to
2
all aspects of the models at the most granular level . The risk management function must
assess the scope of a validation review and determine the material risks for all material
markets of the group, and to plan their validation process accordingly. In assessing
materiality thresholds for the validation process the risk management function should consider
the results of sensitivity testing. Materiality levels and the scope of each periodic review must
be agreed with the group board at the outset of each periodic review.

In adopting this approach XYZ is focusing validation efforts on the most significant
assumptions and processes, and ensuring that the structure of the review corresponds to the
perceived level of risk.

The nature and materiality of risks will change over time and this must be considered in
developing the validation plan. Reliance may be placed on review work carried out in a
3
previous validation cycle .

RMF should 2.5 Reporting requirements

prepare whole
report. The risk management function is responsible for preparing and presenting the validation
report to the board of the XYZ group only in relation to areas validated by risk management
function, where other functions are responsible for preparing and presenting to the respective
Recommendations
bodies the validation report concerning validation tasks performed by them. The model
required. validation report for each validation process performed must cover the results of the validation
and the resulting conclusions and consequences from the analysis of the validation results,
including identifying where changes to the model are required.

As a minimum the contents of this report must include:

Fit for use • Summary descriptions of the validation that has been performed.
mentioned.
• Whether the model is fit for the purpose for which it is used and compliant with
applicable regulatory requirements.

• An independent assessment on the reasonableness of the firm’s available and

required regulatory capital calculations, and analyses of movements thereof.

• An independent assessment of the material accuracy of any stress testing results

reviewed during the validation process.

Initial validation • An independent assessment of the completeness and accuracy of capital.

indicated.
2
However the first validation should cover all aspects of the models at the most granular level
3
For example, there is no need to repeat validation work carried out in a previous review if it is deemed
by the reviewing function that changes in the model and\or the risk profile of the business are not
significant enough to warrant a detailed review.

Model Validation Policy Page 5

 • Assessment of the strengths and weaknesses of the model and the conditions of its
applicability regarding the context in which the model is used.

• Improvements that must or should be made to the model.

2.6 Model documentation

An adequate record of the design, development, testing and inputs of the model must be
maintained. This is the responsibility of the functions responsible for model development and
assumption setting. Model documentation must be sufficiently detailed to adequately support
the independent review and should be made available to the risk management function and
other functions performing validation tasks at the outset of the validation process. Specifically,
at the group level the following elements must be available for the review during the validation
process:
• a catalogue of models and their application
• a catalogue of assumptions
• a specification of modelling approaches and the rationale for adopting these
techniques
• a catalogue of known model limitations
• a catalogue of all documentation related to the models
• a catalogue of model changes and the rationale pertaining to these changes

A single validation 2.7 Local considerations and procedures

policy is required.
Business units (to be specific: XYZ’s solo entities in the EU and business units in non-EU
countries) could develop their local validation policy and are not required to comply with this
policy.

Tools and 3 Validation process

processes used to
validate the model
may be qualitative 3.1 Validation methods
as well.
The tools and processes used to validate the model must be quantitative. The diagram sets
out the components of model validation that are required by XYZ for model validation. These
components include the requirements for regulatory approval for an internal model for
Solvency II.

Validation
methods

Stress and Model stability

Methods and
Data Systems \ IT P&L Attribution Scanerio Backtesting and sensitivity
Assumptions*
Testing analysis

The model validation process must employ each of the techniques set out above as a
minimum; however, additional techniques may be used, for example benchmarking.

Model Validation Policy Page 6

 In addition to considering the validation components the risk management function must
consider the validation of the model in its entirety. A model should only be implemented if it
receives a positive assessment from the group CRO.

3.2 Validation cycle

The following process flow illustrates the key steps and responsibilities in XYZ’s validation
cycle:

Planning
• Timing of review and terms of reference
• Terms of reference for third party
engagement agreed (e.g. role of Internal
Audit Function, scope and timing of peer
review) (Risk)
• Approach and materiality thresholds

Possible changes to the internal Preliminary Quantitative Validation

model(s) • Preliminary quantitative validation carried
• Model changes arising from the review are out using validation techniques prescribed
imlpemented (Finance) in this Standard (Finance)
• Preliminary validation results reviewed by
senior management (Finance)

Model Validation Report Analysis of Validation Results

• Draft model Validation Report prepapred • Independent review of preliminary
(Risk) quantitative validation (Risk)
• Draft report discussed with an appropriate
senior manager
• Report submitted to Group senior
management.

Lack of
documentation,
model governance,
model use. 3.3 Validation process
The minimum required techniques are prescribed in the regulatory requirements. A summary
of the validation tools that are mandated by this policy is given below

3.3.1 Data
Data comes from both internal sources and external sources. It includes policy data and
All data should be market data. For policy data the validation process must test to ensure that the data is
validated consistent with the information on the firm’s policy administration system. Externally provided
market data do not need to be checked as long as are widely used by other market
participants.

If data problems are identified during the validation process then the validation report must
appraise senior management and the board of the problems and the implications; specifically
the implication for reliance on the model outputs. In some cases it may be necessary for the
firm to devote more resources to the problem, and the Validation Report may recommend

Model Validation Policy Page 7

 this. It may also be necessary to recommend the inclusion of an adjustment for data quality
problems in the calculation of capital or other model uses.

3.3.2 Methods and assumptions

Statistical foundations of the model must comply with the statistical quality standards set out
in the Solvency II Directive and any relevant Level 2 Implementing Measures, Level 3 binding
technical standards and must consider non-binding Level 3 guidance. The aim of statistical
quality standards is to ensure that appropriate statistical modelling techniques have been
adopted.

Some inputs into the group’s Models may be determined by a separate model, which itself
has inputs, processes and outputs. These must be validated following the principles set out in
this policy. Appropriateness of the methods and assumptions must be considered when
validating the model. If assumptions are appropriately derived from publicly available data
then a sample do not need to be checked for accuracy.

Many of the model assumptions will be calibrated through financial market analysis or
analysis of industry or firm specific experience for insurance risk. Management’s view on
potential future events may also be relevant in setting assumptions. The modelling team must
be able to provide a clearly documented rationale for these assumptions.

Assumptions must be routinely compared to actual experience. If, over a period of several
years the experience is significantly different from that assumed in the model then the
assumptions should be changed to reflect experience. This must be reviewed in the validation
process.

Model outputs derived from the methods and assumptions must produce a capital
requirement which provides an equivalent level of protection to 99.5% value at risk over a
one-year period.

3.3.3 Expert judgement

Expert judgement models or assumptions are required where there is minimal or no data
available. Expert judgement uses the insights of experts in a given field to predict future
levels and interactions of variables in the model. The rationale and assumptions pertaining to
the use of expert judgement must be documented.

The suitability of an individual expert must be considered during the validation process.
Relevant factors would include whether the individual fulfils the criteria for an expert
Expert judgment in prescribed in the XYZ’s internal standards, and whether the individual is free from potential
the scope of the bias. As a minimum, the validation process must consider these factors and (a) the overall
validation process. reasonableness of the expert’s methods and rationale, (b) the track record of expert
judgement applied and (c) whether the use of expert judgement in the model is consistent
with the scope and documented rationale of the expert reviewer or expert panel.

The results of sensitivity analysis should be considered when assessing the materiality of
expert judgement and to focus the validation process accordingly. Expert judgement models
and assumptions should be benchmarked to the extent this is possible.

3.3.4 Systems (model design)

The model can be split into three components; (a) robust IT infrastructure and system
controls, (b) model design (the representation of real world events that are being modelled)
and (c) model implementation (the mathematics and computer coding).

Model Validation Policy Page 8

 3.3.5 Validating the theoretical underpinnings of the model
Validation of the theoretical underpinnings of the model must be performed by an experienced
No detailed team. Comparison with other models and experience of the team will be useful for
description (with
aim of using) of uncovering conceptual errors. Other models may include similar models already used in
the validation another part of the firm. The use of experts may include the use of personnel from other parts
techniques and of the group, who are independent of the model design, build and testing.
procedures.
3.3.6 Validation tools
3.3.6.1 Backtesting
The backtesting results shall be the basis for an analysis which identifies the reason for the
divergence between the modelled results and reality. In performing their independent model
validation the reviewer must consider whether deviations should lead to a recommended
change in the model, and whether the change will be to the model structure or only to some of
the parameter values.

Backtesting shall be applied at various levels of the business activity. The extent of
backtesting performed must be proportionate to the data available and materiality of the risks.
The XYZ should identify areas, targets and measures for backtesting.

Where significant concerns arise here, as in other areas of validation, during the validation
process these must be escalated in a timely manner.

3.3.6.2 Model stability and sensitivity analysis

The validation of model stability is concerned with assessing the model architecture,
structure, formulaic definition or numerical procedures, and the parameters used by the
model. Specifically sensitivity analysis must be used in the validation process:

• to assess alternative outcomes using reasonable alternatives for key assumptions.

• to assist in assessing parts of the internal model which place particular reliance on
expert judgement, or where the expert judgement has a material impact on the
results.
• informing the risk management function’s view on materiality of a component
methodology or risk in the Model.

3.3.6.3 Stress and scenario testing

As a validation technique stress and scenario testing shall permit a degree of confidence in
the model by providing more information about what results may look like under various
conditions. It may also identify possible limitations of the model, thereby increasing the
confidence of users in the outputs of the model. Specifically, traditional stress and scenario
testing and reverse testing results must be reviewed by the risk management function during
the validation process in order to assist in reaching an overall assessment of the model’s
fitness for purpose.

3.3.6.4 Profit & loss attribution

Profit and loss attribution provides information as to whether the risks in the internal model
are complete, and whether there are any material risks in the risk profile of the undertaking
which are not represented in the model. The profit and loss categories of the model must be
sufficient to explain the sources and causes of profit and loss arising on the business.

Model Validation Policy Page 9

 Critical analysis of the results here, as in other aspects of the quantitative validation, may
determine that changes are required to the model.
3.3.7 Reporting and escalation
A model validation report pertaining to the relevant aspect(s) of the model must be presented
to the board in advance of submitting a regulatory application for model approval or model
change, and in advance of external publication (or submitting to regulators) of results
generated from the model.

Escalation path The model validation report may include a response to issues raised in the report. Material
defined. issues arising in the process of completing the validation work must be escalated to the group
CEO in a timely manner by the team carrying out the validation work. In the first instance
issues identified during the risk management function validation work should be escalated to
the CRO (if applicable) who should discuss and agree remedial action plans.

3.3.8 Tracking remedial actions

The risk management function must track completion of remedial or improvement tasks
identified in the validation process and report on these at least twice annually to (as a
minimum) the board.

3.3.9 Limitations
RMF responsibility
for model In some cases it may be appropriate to rely on the validation work of other parties, for
validation
interfered. example as a result of internal resource constraints. Where this is the case, care should be
taken to ensure that third parties have the necessary skills and expertise to conduct the
review and that the reliance on the work of third parties is appropriate. Validation performed
by qualified third parties does not require risk management function to carry out any additional
activities. In such a case validation report is delivered by an external party directly to the
group board.

Limitations specific to each periodic review must be documented as part of the validation
process; for example time constraints or resource constraints in respect of the model
validation work.

The validation processes and procedures must constantly evolve as the risk profile of the solo
entities (and group) changes, as the models develop and as new validation tools become
available. This policy must be reviewed at least annually to ensure it remains fit for purpose
and complies with all relevant requirements.

4 References to supporting material

This policy is dependent on the requirements of the Solvency II Directive and Level 2
Implementing Measures and Level 3 binding technical standards. Changes in these
regulations or associated guidance may affect this policy. In the case of a conflict, the
Directive and Level 2 Implementing Measure and Level 3 binding technical standards have
precedence over this policy.

In addition, the following material relate to this policy:

• Internal model change business policy

• Solvency II data governance policy
• ORSA business standard

Model Validation Policy Page 10