http://lists101.his.com/pipermail/intelforum/2005-July/008358.

html

Date: Thu, 30 Jun 2005 16:51:16 -0700 (PDT)

From: Stephen Marrin <spm8p@xxxxxxxxx>
Subject: Understanding What Intell Analysts Do
To: intelforum at lists101.his.com

All:

For entertainment value on a long weekend, the short essay copied below is a slightly adapted
version of some thoughts describing what intelligence analysts do when they do analysis that I
posted to an internal CIA bulletin board--the DI Discussion Database--on 6 May 1997. I hope it
has relevance for those of you who do or have done intelligence analysis....

Since I am now an academic, I also believe the discussion has relevance to those who study
intelligence analysis from the academic side. Specifically, the underlying issue of the essay has
to do with the use of heuristics in foreign policy decisionmaking.

In terms of international relations, these cognitive shortcuts can be applied at 2 different levels of
analysis.

First, these cognitive shortcuts are used in the intelligence analysis process by analysts who play
a role in the decisionmaking process by providing their understandings of events (traditionally
those that occurred overseas, but now--for homeland security/law enforcement analysts--
domestically as well) to relevant decisionmakers. The focus on cognitive processes at this level
(ie. within bureaucracies) can be folded into an assessment of the influence that variations in
information flow within organizations have on decisionmaking...in other words, Graham
Allison's organizational process model.

The second level of analysis is that of the individual decisionmaker; focusing on how these kinds
of heuristics *as applied by individual decisionmakers rather than intelligence analysts* might
affect the final decision or policy. In other words, this approach would be akin to Robert Jervis'
(and others') focus on the influence of factors at the individual level of analysis (psychology,
history, perceptions, etc) and how they influence decisionmaking and ultimately state behavior.

When you get right down to it, I am addressing epistemology in this essay....epistemology of
international relations and the understandings—albeit flawed--of those who interpret
(intelligence analysts) and act (decisionmakers) based on their understandings.

Hope it is of interest, and all comments welcome...

Regards,
Steve Marrin

###
Modified Posting to DI Discussion Database
May 6, 1997

Complexity is in the Eye of the Beholder

The question of what is simple and what is complicated has been floating in the back of my mind
for a while, ever since I was told that the extremely complicated job that I was asked to do was in
reality exceedingly simple. Based on this thought--in combination with an article on complexity
in the New York Times today—let me throw out a hypothesis which may clarify the debate of
the above postings.

Reality is complicated. The human mind cannot comprehend randomness, so we look for
patterns. One kind of a pattern is a cause/effect relationship. When the causal linkages can be
established we become able to understand and predict outcomes. But nothing in the real world
consists of one cause/effect relationship. Instead we have to deal with outcomes that are the
result of the interactions of many causes. In trying to determine the impact of the independent
variables on the outcome we make frequent mistakes. We can overestimate, underestimate, or
miss altogether the influence of a particular independent variable on the end result. But as long
as we are close, its good enough. Humans are lazy, and do not want to do all this work all the
time. Once we find a causal relationship that approximates reality, it becomes a rule.

And to make things even easier, we will adopt the rules of others--conventional wisdom. These
rules are then applied the next time we think the cause shows up. The application of these rules is
assumptions at work. Some are accurate, some are partially accurate, and some are wrong. Those
who build correct frameworks of assumptions--those who apply accurate hypotheses--become
known as experts. But even experts can be wrong because their expertise is based on rules which
are at best blunt approximations of reality.

In addition, we can aggregate the rules by which most people act, and upon these rules are built
the social sciences. But individuals themselves remain unpredictable, hence error will always
remain a part of the enterprise.

Now to apply this to the work of intelligence.

International relations is an interdisciplinary social science. This institution uses many different
kinds of analysts (political, military, economic, imagery, transnational, etc) to derive and identify
the operative variables, their interactions, and subsequent impacts.

Understanding is nice, but prediction (or more accurately, estimating) is the name of the
intelligence game. Prediction in international relations is possible, but very difficult to do
accurately. Many times we are right because of all the time and effort to ensure that the
information at our disposal is such that it helps us to create more accurate interpretations of
reality. But we will also be wrong for any of innumerable reasons--including the use of
inappropriate assumptions--regardless of whether it is derivation of broad societal trends or spot
crisis prediction that we are tasked with. In the end any prediction will be an approximation of
the real world and therefore subject to some amount of error. The whole idea of this enterprise is
to reduce the margin of error so that when a judgement is asked for we can confidently give it.

The amount of work we are required to do all depends on what is needed. If all that is needed is
for us to identify threats--to say 'there could be a problem over there....maybe y'all should do
something about it' then crisis prediction may not be that difficult after all. But if you think
intelligence should be able to predict a crisis by identifying its cause exactly--and thereby giving
the policy folks a better idea of how to solve the problem--prediction becomes more difficult in a
hurry.

One possible way to rigorously approach the issue of prediction and reduce the possibility of
error is to use analytical methodologies to self-consciously break down the prediction into
variables and develop future scenarios based on our perceptions of their relative influence. In this
way we can productively use assumptions to weed through complexity.

My concluding thought is that the determination of whether something is easy or complicated

will depend upon the number of assumptions in play. If you assume a lot it will be easy to create
predictions. However, reality constantly reminds us that many of our assumptions are
inaccurately applied. But if you leave assumptions as open questions the project becomes more
complicated as it has to deal with a greater number of variables and outcomes. Different people
studying the same issue could find the task exceedingly easy or incredibly complicated
depending upon their own personal frameworks of assumptions.

It all hinges on assumptions and how you use them.

Stephen Marrin