You are on page 1of 28

“You Can’t Manage It If

You Can’t Measure It

ISACA
March 2006
Agenda

• Do you know how well your information security program is


working?

• Key Performance Indicator (KPI)

• Key Performance Index (KPX)

• Information Collection

• Examples

• Summary

© Deloitte & Touche LLP and affiliated entities.


What do we have to be worried about?

The time between


the discovery of a
vulnerability and
the potential
exploit is
diminishing from
months to days if
not hours

© Deloitte & Touche LLP and affiliated entities.


IT Security Governance Maturity Model

• The Maturity Model is sponsored by the IT Governance


Institute
• It is used to rank the maturity of an organization’s practices
and standards against industry best practices and standards
• It can be used to help guide an organization on the areas that
will improve their overall information security posture

© Deloitte & Touche LLP and affiliated entities.


How do you know if you have an information security
program that effectively manages risks?

• Obtain a high score on an ISO 17799 assessment?


• Complete regular, active penetration tests with no discovered
vulnerabilities?
• Have an acceptably low # of security incidents reported using
the Incident Response process?
• Have an effective virus program (few or no infections and any
infections are managed effectively with little interruption)?
• Have Measurable Service Level Expectations (SLE) that are
consistently being achieved?
• Have an effective IDS program (# and type of alerts are being
managed effectively, little impact on the business, in line or
better than industry benchmarks)?
• Obtain certification against an information security reference
standard (ISO 27001)?

© Deloitte & Touche LLP and affiliated entities.


There are several problems to avoid when establishing
an information security measurement program

• Lack of management commitment

• Measuring too much, too soon

• Measuring too little, too late

• Measuring the wrong things

• Imprecise metrics definitions

• Using metrics data to evaluate individuals

• Using metrics to motivate, rather than to understand

• Collecting data that is not used

• Lack of communication and training

• Misinterpreting metrics data


© Deloitte & Touche LLP and affiliated entities.
Key Performance Indicators (KPIs) can help
determine the current status of the information
security program

• A key performance indicator is a measure of a particular


organizational performance activity, or an important indicator
of a precise health condition of an organization

• Used as an indication of the current state of a component of


the business to take the “surprise” out of risk

• To be effective, the KPI must be defined as succinctly as


possible

• Can be measured as an “improvement” from a known state or


a reference standard

© Deloitte & Touche LLP and affiliated entities.


A Key Performance Indicator . . .

• Must be something that can be measured and continued to be


measured

• Must be precise, meaningful and understandable

• Must be relevant to the business

• May be required by legislation and/or Regulations

• Must have a measurement index that has meaning

• Must have an appropriate life (Stickiness)

• Should be tied to the organization’s vision and strategy

© Deloitte & Touche LLP and affiliated entities.


Types of Key Performance Indicators (KPIs)

• Threshold – when an index reaches set targets or falls into


set ranges
– e.g., ETS scores on defined risks

• Milestone – when a specific condition is reached


– e.g., certification

• Quantitative – measure of value (number, time, $, etc.)


– e.g., number of reported security incidents, lost time due to
viruses

• Qualitative – measure of acceptability or health


– e.g., survey ratings, rating of risks

© Deloitte & Touche LLP and affiliated entities.


Examples of Key Performance Indicators
• Awareness
• Knowledge of policies, standards and procedures (surveys and
tests)
• Risk Assessment
• Depth and breadth of regular risk assessments across the
enterprise (When was the last assessment? Qualitative
measure of the risks, risk index)
• Risk Management
• Number of incidents reported, amount of loss incurred,
number of situations managed
• Audit
• Noted deficiencies against the policy and standards (measured
year over year)
• Benchmarks and Certification
• Maintaining/following IT security certifications such as FIPS
140-1, ISO 27001, ISO 15408 (Common Criteria)

© Deloitte & Touche LLP and affiliated entities.


Possible Non-Risk Key Performance Indicators (KPIs)

• People
– Training & Certifications
– Competence Turnover

• Technology
– Currency
– Cost management
– Compliance / licensing

• Investment
– Trends per area

• Effectiveness & Return on Investment


– Key Risk Indicator experience vs. cost

• Productivity
– Missed Deadlines

© Deloitte & Touche LLP and affiliated entities.


KPIs can be used to measure the Effectiveness of
Investment (EOI)

• A Return on Investment (ROI) for information security is


difficult to measure since risk, and especially risk reduction, is
challenging to quantify in terms of dollars.

• The Effectiveness of Investment (EOI) could be the


comparison of the effectiveness of the security measures with
the value of the investment.

• For example, the number and impact of viruses and worms


can be compared with the investment in virus detection
technology and support programs.

• A collection of KPIs could be used to measure the EOI for


information security

© Deloitte & Touche LLP and affiliated entities.


A Key Performance Index (KPX) is a summary or
correlation of one or more KPIs that provides an
indication of the overall performance of a defined
area of the security program

• May prompt the organization to change strategic direction in


information security

• Levels may be triggered by a variety of factors

• Must be meaningful and understandable

• Must be relevant to the business

• Must have a measurement index that has meaning

• Must have an appropriate life (Stickiness) and

• Should be tied to the organization’s vision and strategy

© Deloitte & Touche LLP and affiliated entities.


Example KPI Format

KPI Name Short name or title for the KPI

Description Description of the KPI – what does it address?

Objective What are the objectives of the KPI – what is it measuring? Why is it
important?
Stakeholder Who is this KPI relevant to?

Type __ Quantitative ___ Qualitative ___ Milestone ___ Threshold

Effort __ Low __ Medium __ High

Unit/Dept What does it apply to?

Method Method used to measure the KPI

Tools Any potential tools used to support the measurement and reporting
process?
Frequency ___ Day ___ Week ___ Month
___ Quarter ___ Year ___ Year+
Comments Any additional information or comments? Is this a requirement from
legislation or regulations?

© Deloitte & Touche LLP and affiliated entities.


Example Key Performance Indicator (KPI)

KPI Name Weekly Reported Security Incidents

Description Provides a relative index on the current number of reported security


incidents/events at differing security levels for the recent reporting week
Objective A measure of the relative size and effectiveness of the organizations risk
management processes
Stakeholder CSIO, CIO, Operations Management, Technology Management

Type _X_ Quantitative ___ Qualitative ___ Milestone ___ Threshold

Effort __ Low _X_ Medium __ High

Unit/Dept Information Security

Method Count number of reported security incidents/events at low, medium and


high severity over the past week
Tools IDS and/or security management/reporting software

Frequency ___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+

Comments Need to have confidence in the detection and reporting mechanisms to be


able to measure changes to the index over time. A lower index will then
mean less risk

© Deloitte & Touche LLP and affiliated entities.


Example Key Performance Index (KPX)

KPI Name Information Security Risk Management Index

Description Provides a relative index on the current number of reported security


incidents/events at differing security levels within a specified time frame
Objective A measure of the relative size and effectiveness of the organizations risk
management processes
Stakeholder CSIO,CIO

Type _X_ Quantitative ___ Qualitative ___ Milestone ___ Threshold

Effort __ Low _X_ Medium __ High

Unit/Dept Core Systems

Method Count number of reported security incidents/events at low, medium and


high severity over a defined time frame
Tools IDS and/or security management/reporting software

Frequency ___ Day _X_ Week _X_ Month _X_ Quarter _X_ Year ___ Year+

Comments Need to have confidence in the detection and reporting mechanisms to be


able to measure changes to the index over time. A lower index will then
mean less risk

© Deloitte & Touche LLP and affiliated entities.


Several automated tools can provide a view of
security incidents and trends

© Deloitte & Touche LLP and affiliated entities.


Security Incidents - Advanced Forensic Tools

© Deloitte & Touche LLP and affiliated entities.


The Information Security Program should include a
reporting mechanism that provides a single point of
reference for concise, executive-level information for
business and technology owners.

Sample Security Dashboard Trend View


Operator Event View

Geographic Threat View


Advanced Forensic Tools

Incident Tracking
(Ticketing System)
Geographical Dashboard
View
Reports

The dashboard aims to transform data from operations to actionable information for
decision makers

© Deloitte & Touche LLP and affiliated entities.


An analysis of security incidents will contribute to the
current status of the Information Security Program

© Deloitte & Touche LLP and affiliated entities.


Keep track of each area of concern that is the object
of a KPI or KPX definition
Topic - <What is the KPI or area of concern?>
Vision/Mission What is the Vision and Mission statement that directs IT security?

Objective What is the main objective – how is it measured? – Why is it important?

Key Control What are the key control objectives and controls that should be in place for the
Objectives and organization? The controls should be based on international reference
Controls standards

Measurements What are the measurements that may be available to report on this area?

KPI(s) What Key Performance Indicators(s) should be defined for this objective?

KPX(s) What summary index(s) can be defined that is a high-level representation of


one or more KPIs that are vitally important to the organization?
Map KPI(s) to How does the KPI(s) map to the individual performance goals?
Performance Goals

Reporting Any required acknowledgement or reporting for this KPI?

Comments Any additional information or comments?

© Deloitte & Touche LLP and affiliated entities.


An example KPI for Inappropriate Use

Measurement - 1
KPI - 1 Number of inappropriate
use cases opened and
Number of verified verified
instances of inappropriate
use over a set time
period. (weekly or by
reporting period)

Inappropirate Use - KPX KPI - 2


The impact of recorded
inappropriate use events Impact of inappropriate use Measurement - 2
compared to the amount of events to the business in Amount of service lost to
terms of resources and or inappropriate use
IT security awareness loss over time (weekly or
training per person. by reporting time)

KPI - 3
Number of verified
inappropriate use events
compared with the number
of IT security awareness
training days per person Measurement -3
compared over time Number of IT security
awareness training days

Presentation Name (View / Header and Footer) © Deloitte & Touche LLP and affiliated entities.
An example KPX for Inappropriate Use

KPX

© Deloitte & Touche LLP and affiliated entities.


An example KPI for Intrusion Detection

Measurement - 1
Number of incidents of intrusions
detected and reported
KPI - 1
Average amount of Loss
(productivity time) per intrusion Measurement - 3
within a set time period (weekly Amount of downtime or productivity
or per reporting period). loss caused by intrusion incidents.

IDS KPX Measurement - 2


The measureable amount of KPI - 2 Number of incidents of intrusions impacting
productivity loss attributed Number of events caught and the organization that were not reported
to intrusions in relation to the prevented by the IDS within a
the number of events and the set time period
cost of the IDS program. Measurement - 4
The number of systems with
active monitoring capabilities
KPI - 3
Number of IDS
program failures Measurement - 5
Number of Sensors per
network segment
KPI - 4
Cost of the IDS program in
relation to the number and Measurement - 6
impact of detected events Cost of the hardware and/or
software to implement intrusion
detection sensors

© Deloitte & Touche LLP and affiliated entities.


An example KPX for Threat Management–
Intrusion Detection System (IDS)

1 2
Number of Resolved Major and Number of Major and Catastrophic
Catastrophic Incidents Over Time
Incidents Over Time

High Risk Incidents


# of # of Major
Critical Incidents
Resolved and
Major and Catastrophic
Catastrophic Incidents
Incidents

Time/ Reporting Period Time/ Reporting Period

3 Average Time to Resolve a Number


of Major and Catastrophic Incidents

>4<10 hrs/month/
# of system productivity
loss
Resolved
>10hrs/month/ system
Major and productivity loss
Catastrophic Major Incidents
Incidents
Catastrophic
Incidents

Number of Resolved
Major and
Average Time to Resolve Major and Catastrophic Incidents
Catastrophic Incidents

© Deloitte & Touche LLP and affiliated entities.


Summary

• A good collection of Key Performance Indicators will provide


an overview of the current status of risk management within
the organization
– Use the collection of KPIs as an information security dashboard

• The KPIs can be used to help comply with legislative or


regulatory requirements
– Provide the information that can be used for reporting purposes

• The KPIs must be carefully selected and defined to be useful


– Must be meaningful and measurable

• Effective KPIs can be used to demonstrate good management


of risk
– For example, KPIs may provide a financial institution the ability to
reduce the percentage of reserve required to offset operational risk
defined by the Basel II Accord

© Deloitte & Touche LLP and affiliated entities.


Questions?
Glen Bruce, glebruce@deloitte.ca
© Deloitte & Touche LLP and affiliated entities.

Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and
financial advisory services through more than 6,100 people in 47 offices. Deloitte operates in Québec
as Samson Bélair/Deloitte & Touche s.e.n.c.r.l. The firm is dedicated to helping its clients and its
people excel. Deloitte is the Canadian member firm of Deloitte Touche Tohmatsu.

Deloitte refers to one or more of Deloitte Touche Tohmatsu, a Swiss Verein, its member firms, and
their respective subsidiaries and affiliates. As a Swiss Verein (association), neither Deloitte Touche
Tohmatsu nor any of its member firms has any liability for each other's acts or omissions. Each of the
member firms is a separate and independent legal entity operating under the names "Deloitte,"
"Deloitte & Touche," "Deloitte Touche Tohmatsu," or other related names. Services are provided by Member of
the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein. Deloitte Touche Tohmatsu

You might also like