A Paisley White Paper

Risk Rating the

Audit Universe
A critical look at traditional audit
universe risk-rating factors

Prepared by:

Bruce McCuaig
Chief Risk Officer and Principal Consultant
 Risk Rating the Audit Universe

Risk Rating the Audit Universe

Table of Contents
INTRODUCTION
Introduction 2
Internal vs. External One outcome of the Sarbanes-Oxley Act, and the related Public Company
Auditor Performance 2
Accounting Oversight Board AS2, and more recently AS5, is more information
How Should Internal
Auditors Prioritize Audit 4
in the public domain about the performance (or failure) of internal controls

Guidance For over financial reporting. The information comes from the hundreds of internal
Improvement 9 control deficiencies reported by accelerated filers. Analyzing this data to deter-
About Paisley 10 mine what kinds of companies reported deficiencies, how deficiencies were
detected, what business processes the deficiencies related to, and what ac-
counts and assertions they impacted provides great insight into how controls
work in modern public companies. This information also provides insight into
the role and performance of internal auditors. Knowledge gained from these
deficiency disclosures may challenge internal auditors’ assumptions about
where risk lies and how to better prioritize an audit universe. Specifically,
can we learn more about how to risk rate an audit universe to better focus
resources on where the deficiencies lie? Big risks can lurk under small rocks,
and the indicators of big risks are often ignored in audit planning. Internal
audit has played an important role in finding and reporting SOX deficiencies,
however, external audit has played a far bigger role. This paper will identify
some areas for improvement.

INTERNAL VS EXTERNAL AUDITOR PERFORMANCE

Internal audit professionals are guided to establish a risk-based audit universe
by the Institute of Internal Auditors International Standards for the Profes-
sional Practice of Internal Auditing and related practice advisories. Currently
under revision, the proposed International Professional Practices Framework
(IPPF) Performance Standard 2010, Planning, states,

“The chief audit executive must establish risk-based plans to

determine the priorities of the internal audit activity, consistent
with the organization’s goals.”

The proposed standard is more explicit than its predecessor, making it

mandatory for the chief auditor to develop a risk-based plan.

There is room for improvement in the execution of a risk-based audit

approach. A recent study published by the Financial Executives Research
Foundation, Control Deficiency Reporting: Review and Analysis of Filings
2
 Risk Rating the Audit Universe

During 2004, analyzes the control deficiency disclosures made by 329 companies
in various SEC filings from November 1, 2003, to October 31, 2004. It analyzes
over 950 disclosures to identify trends to help users of financial statements better
understand the nature of control deficiency reporting made by SEC registrants.
Management and internal auditors appear to have performed poorly in detecting
Internal auditors
and reporting deficiencies. Evidence suggests that only about 28 percent of com-
either used risk panies were proactively bringing reportable deficiencies to the attention of their
audit committees or external auditors. This strongly suggests that internal audi-
prioritization
tors either used risk prioritization models that routinely scoped out high-risk areas
models that
for internal control deficiencies or did not detect or report deficiencies that were
routinely scoped found.

out high-risk
More recent statistics confirm this trend. A February 2007 trend alert from Glass
areas for internal
Lewis & Co, a leading investor analyst firm, reported: 2,931 U.S. companies,
control deficien- about 23 pecent, filed at least one restatement during the last four years; 683
companies restated two or more times.
cies or did not
detect or
There is little to suggest that either internal or external auditors are improving
report deficiencies their track record of looking in the right places or finding problems if they exist.
The February 27, 2007, Yellow Card Trend Alert produced by Glass Lewis & Co
that were found.
titled, The Errors of Their Ways, concluded:

“Companies take note: If you restated, you must have had material weaknesses.
We still have a hard time figuring out how so many companies that restated also
could have reasonably concluded that their internal controls are effective and that
they have no material weaknesses – or that no material weaknesses even existed
at the time of the errors.”

The trend in reported deficiencies is alarming. While individual companies

and their internal auditors may fail to detect or report some internal control defi-
ciencies in audits they conduct, the trend in the total number of restatements and
the number of companies reporting deficiencies, and their late and sudden disclo-
sure suggest a systemic problem. Material weaknesses and significant deficiencies
are simply not being found and reported by management. Restatements continue
at a high level.

Unless internal auditors are applying completely different risk-based standards to

planning audits of internal control over financial reporting, it is reasonable to sug-
gest that the method of prioritizing internal audit activity may be a problem. Is
the error rate experienced in audits of ICFR the same as the error rate in audits of
other areas?
3
 Risk Rating the Audit Universe

HOW SHOULD INTERNAL AUDITORS PRIORITIZE AUDITS?

Material The IIA provides practice advisories to assist in the interpretation and imple-
mentation of the Professional Standards. Practice Advisory 2010-2, Linking
weaknesses and
the Audit Plan to Risk and Exposures, suggests that the following risk
significant factors, among others, should be considered:

deficiencies are • Dollar materiality

• Asset liquidity
simply not being
• Quality of internal controls
found and
• Degree of change or stability
reported. • Complexity
• Management competence

Individual internal audit departments are free to establish their own prioriti-
zation frameworks, however, based on the last several years of publicly
disclosed information; company management and their internal auditors
may have missed the boat on finding and reporting internal control deficien-
cies. The alarming increase in reported deficiencies begs an evaluation of
how the risk factors suggested by the IIA correlate to reported disclosures.

Dollar Materiality as a Risk Factor

Internal audit departments frequently take into account the dollar material-
ity of auditable entities or processes in determining audit risk. If dollar
materiality was a significant factor in internal control deficiencies, one
should expect to see larger companies with more deficiencies or at least
more material weaknesses.

According to the FERF study, the average large cap company (>$1B) in the
sample reported 3.71 deficiencies and the average small cap (<$250M)
reported 2.51 deficiencies; the reporting rate is far less than the size ratio
would suggest. The relationship between dollar materiality and risk is dis-
proportionate to size. As a risk factor, dollar materiality seems to have an
inverse relationship. Entities or processes with low dollar materiality bear a
disproportionate amount of disclosure risk. Billion-dollar companies do not
report four times as many deficiencies as are reported by companies one
quarter as large. Clearly dollar materiality should be a factor, but its weight
should be determined by other factors.

4
 Risk Rating the Audit Universe

The relationship
between dollar
materiality and
risk is dispropor-
tionate to size. As
a risk factor, Asset Liquidity as a Risk Factor

dollar materiality Many internal audit departments are charged with ensuring the safeguarding
of assets and preventing fraud and theft. Liquid assets are perceived to be
seems to have an
particularly vulnerable to fraud and theft. If liquid assets were truly at risk,
inverse relation- one would expect to see a large number of deficiencies related to cash and
ship to risk. equivalents and certain inventories and one would expect to see a large
number of deficiencies related to cash and equivalents and certain invento-
ries and one would expect the existence assertion to be related to many
reported deficiencies. Neither has proven to be true.

According to the FERF study, the following accounts were most frequently
involved in internal control weaknesses: accounts receivable, sales, inven-
tory, cost of goods sold, accrued expenses/reserves, and selling, general
and administrative. Furthermore, according to an analysis of related asser-
tions, the
existence assertion was the one least likely to be attributed to a reported
deficiency in the sample. There is no doubt that liquid assets can be lost or
stolen. But on the whole they have not proven difficult to control and their
existence has not proven to be a significant risk factor for internal control
deficiencies. Internal audit departments may in fact be misdirecting re-
sources by focusing too much attention on liquid assets.

5
 Risk Rating the Audit Universe

Quality of Internal Controls as a Risk Factor

Internal auditors tend to consider the quality of internal controls as a
significant risk factor. In doing so, internal auditors often use the COSO
internal control framework component of control activities as their benchmark
in assessing the existence and quality of internal controls.

One would then expect that a significant number of control deficiencies could
be classified as to control activities. In other words, broken or missing control
activities, if they are truly important, should be behind a significant number
of reported control deficiencies in the FERF study sample. This has not proven
to be true. Where sufficient information made it possible, the authors of the
FERF study classified each control deficiency into its related COSO framework
component. Many deficiencies were so poorly reported as to defy
classification, but of those that were classified, control activities were a
relatively minor category.

… on the whole
[liquid assets]
have not proven
difficult to control
and their exis-
tence has not
proven to be a
significant [SOX]
risk factor
As can be seen in Exhibit 2, across the range of companies in the sample,
between 6 percent and 9 percent of reported deficiencies were attributable
to control activities. If the quality of internal control is an important risk
factor, one should expect missing or broken control activities to be associated
with a significant number of control deficiencies. If the lack of evidence of sig-
nificant absences of or breakdowns in control activities suggests they are,
in fact, present and working well in most companies, where are all the
deficiencies coming from? Just how important are control activities as a risk
factor? If internal auditors are using the existence or absence of control
6
 Risk Rating the Audit Universe

activities as evidence of the quality of internal control in risk rating their

audit universe, they may be placing more confidence on these controls
than evidence warrants.

Other COSO framework components seem to be much better predictors of

risk. It seems logical to attribute extra risk to a turbulent, rapidly changing
business environment, but the rate of business change or stability is not
among the deciding factors in determining whether a control deficiency
… where are all
exists or is reportable. Risk assessment is the COSO framework component
the deficiencies one would expect to see cited as a weakness if the degree of business
coming from? Just change was a factor. Change management is part of the risk assessment
component in COSO. Interestingly, risk assessment is the least cited attrib-
how important
ute when attributing deficiencies to COSO framework components.
are control
activities as a Change or Stability Risk Factors
risk factor? It is not clear if change or stability are reasonable factors. What is clear is
that risk assessment is not being performed adequately. A better factor than
stability or degree of change to consider is whether the auditable entity has
a risk assessment process and, if so, what are its results. Supporting this
argument is a table (Exhibit 3) from the Glass, Lewis & Co. study that
breaks down material weaknesses by type.

7
 Risk Rating the Audit Universe

According to the study, almost 60 percent of material weaknesses are attributable

to financial systems and procedures and personnel. Both categories are likely to
be impacted by rapid change in a business and both suggest a lack of change
management practices. Moreover, risk assessment, with one percent of reported
deficiencies, seems to contradict the notion that instability is a problem.

Business Complexity as a Risk Factor

Internal auditors often assess the complexity of their auditable locations. There is
no standard definition of complexity. Some industries have complex business mod-
els, some have complex technology, and others have complex, nonstandard trans-
actions. Size alone often infers complexity, particularly if it leads to complex corpo-
rate structures or multiple locations. But size has been assessed as a risk
factor and found to be a significant but not determining factor. In fact, one could
argue that disclosure risk decreases with size. Smaller companies tend to have
relatively more internal control deficiencies.

However, another picture emerges when one looks at the breakdown of control
deficiencies reported by business process in the FERF study, as partially excerpted
Whatever the
in Exhibit 4. Whatever the complexity of the industry, the vast majority of control
complexity of the deficiencies are concentrated in only a few business processes. Period-end report-
industry, the vast ing and revenue cycles account for 58 percent of the deficiencies in the FERF
sample. Are these two processes significantly impacted by technological or operat-
majority of
ing complexity? Paradoxically, information systems, often assigned high complexity
control scores, accounted for only 5 percent of deficiencies. There is little convincing
deficiencies are
concentrated in
only a few
business
processes.

8
 Risk Rating the Audit Universe

evidence in either study that suggests a subjective assessment of business

complexity, in itself, is a reliable risk factor in prioritizing an audit universe.

Management Competence as a Risk Factor

The control environment component of the COSO framework is the one closest
related to directly dealing with management competence. This COSO control

Management ethics environment component includes integrity, ethical values, competence and a
range of other factors likely to affect the organization as a whole. As the table in
may be the single
Exhibit 3 indicates, about 50 percent of all reported control deficiencies can be
best risk predictor: attributed to problems with the control environment, making it potentially the

• Does management single most significant risk factor in prioritizing the audit universe.

activity reflect
Clearly, of all the factors considered, an assessment of the control environment of
ethical behavior
a company or any of its auditable entities should play a major role in prioritizing
(think backdated
an audit universe. Internal control deficiencies are directly and strongly correlated
stock options)?
to control environment scores. Soft controls do count. Specifically, gaps in the
• Are earnings following elements of the control environment must be considered as specific risk
being managed? factors:
• Does management • Integrity and ethical values

stress the need for • A commitment to competence

ethics? • The board of directors or the audit committee

• Does a corporate • Management philosophy and operating style

code of conduct • The organizational structure

exist? • Assignment of authority and responsibility

GUIDANCE FOR IMPROVEMENT

The importance of accurately prioritizing the audit universe is obvious. Until now,
little empirical evidence has been available to test prioritization methodologies.
That is no longer true. Tested against the evidence of publicly reported internal
control deficiencies, many traditional risk factors look extremely questionable at
best. At worst they are causing valuable internal audit resources to be potentially
misdirected.

What is clear is that internal audit plays an integral part in an organization’s

governance, risk, and compliance initiatives and a critical role in providing assur-
ance to the integrality of the organizations governance framework. In an effort to
improve the effectiveness of internal audit processes, history would suggest that
changes need to be made. Recommended changes include:

9
 Risk Rating the Audit Universe

• A standards-based approach to internal audits will drive greater

consistency and integrity of audit data. Business process improvement is
achievable through a feedback loop of audit results. Financial process
performance should be monitored as a key factor in developing a risk
based plan. Below target performance suggests unidentified risks or
ineffective controls.

• Root cause analysis of internally reported deficiencies and insight into

how control deficiencies are detected and how they impact the entity
are essential if internal auditors want to refine their audit planning and
prioritization models. Root cause analysis is simply not required today
under AS5.

For more • Greater investment in internal audit processes and systems is a

information, pre-requisite to any effective governance, risk and compliance initiative.

call
320.286.5870, ABOUT PAISLEY
email
Paisley is an industry leading software vendor that provides solutions for
info@paisley.com
or visit governance, risk and compliance (GRC) including financial controls
www.paisley.com management, internal audit, operational risk management, compliance, IT
governance, and enterprise risk management. For more than a decade,
Paisley has delivered superior software and services to both large enterprise
and mid-market organizations. Governance, risk and compliance software has
always been and continues to be the company's focus.

Leveraging industry best practices, standards based technology, and a choice

of software platforms and deployment options, Paisley customers are
empowered to improve the accuracy, consistency and efficiency associated
with internal audit, financial controls management, enterprise risk
management, operational risk management, IT governance, and compliance
initiatives. Developed for companies of every size and across multiple
industries, Paisley’s solutions enable organizations to streamline governance,
risk, and compliance processes, reduces costs of compliance, manage and
mitigate risks, and provide visibility, oversight and assurance.

For more information, call 320.286.5870, email info@paisley.com or

visit www.paisley.com.

© 2008 Paisley. All Rights Reserved 10