Sensemaking: A Structure for an Intelligence Revolution
By David T. Moore – NATIONAL DEFENSE INTELLIGENCE COLLEGE WASHINGTON, DC
March 2011
FOREWORD
Gregory F. Treverton
Director
RAND Corporation Center for Global Risk and Security
We at NGA used to look for things and know what we were looking for. If we saw a Soviet T-72 tank, we knew we’d find a number of its brethren nearby. Now, though, we’re not looking for things. Instead, we’re looking for activities or transactions. And we don’t know what we’re looking for.
In fancier language, the paradigm of intelligence and intelligence analysis has changed, driven primarily by the shift in targets from the primacy of nation-states to trans-national groups or irregular forces. In the world of the national-state, I and others divided intelligence problems into puzzles and mysteries (or variants of those words).1 Puzzles are those questions that have a definitive answer in principle. How many nuclear missiles the Soviet Union had was a puzzle. So is whether Al Qaeda possesses fissile material. By contrast, mysteries are questions that cannot be answered with certainty. They are future and contingent.
For puzzles, intelligence tried to produce the answer.
For mysteries there was no answer. Instead, analysts sought to frame the mystery by providing a best estimate, along, perhaps, with excursions or scenarios to test the sensitivity of critical factors. If intelligence failed to understand the full picture of Soviet missiles, and puzzle became mystery, it at least knew something about where to look: there was experience and theory about missile building, plus historical experience of Soviet programs. The mystery came with some shape.
However, today’s transnational threats confront us with something more than mysteries. I call these shapeless mysteries-plus “complexities,” borrowing Dave Snowden’s term. They are sometimes called, as Moore notes, “wicked problems” or simply “messes.” The come without history or shape. Large numbers of relatively small actors respond to a shifting set of situational factors. Thus, they do not necessarily repeat in any established pattern and are not amenable to predictive analysis in the same way as mysteries. Those characteristics describe many transnational targets, like terrorists—small groups forming and reforming, seeking to find vulnerabilities, thus adapting constantly, and interacting in ways that may be new.
For complexities, especially, the challenge is to employ sensemaking— the term is from Michigan psychologist, Karl Weick. Exactly how to accomplish sensemaking is a task that still mostly lies before us, which makes this book such an important contribution. Sensemaking departs, as Moore notes, from the postwar tradition of Sherman Kent, in which analysis meant, in the dictionary’s language, “the process of separating something into its constituent elements.” Sensemaking also blurs America’s bright white line between intelligence and policy, for, ideally, the two would try to make sense together, some- times disaggregating events, sometimes aggregating multiple perspectives, always entertaining new hypotheses, all against the recognition that dramatic failure (or success) might occur at any moment.
he [David More] is very careful about classification. That means the visible trails of his practice in his scholarship are sparse, and his cases are mostly familiar ones, albeit ones often spun in new directions.
The new paradigm makes the use of machines and method imperative, letting machines do what they do best—searching large amounts of data, remembering old patterns, and the like—while letting humans use the judgment they alone can apply. Yet the tests by Moore and his colleagues remind us that methods are critical but only if they have been tested. It turns out, for instance, that ACH, analysis of competing hypotheses, a method more frequently used now and one that has been tested, isn’t all that valuable, at least not for analysts beyond the novice level.
COMMENTARY
Anthony Olcott, PhD
Associate, Institute for the Study of Diplomacy Georgetown University
David Moore is right to talk of the need for an intelligence revolution. However, as Lenin learned in the 18 years that passed between publication of The Development of Capitalism in Russia and taking over the Winter Palace, it takes more than a diagnosis and a prescription to make a revolution. Although his is among the best, Moore’s book is also but the latest addition to a groaning shelf of books devoted to intelligence and analytic reform while the companion shelf, for books on how to improve the policy process, sits dusty and all but empty. In that regard, even though Moore’s discussion of the processes of analysis and how the ways we answer questions might be improved is one of the strongest in recent memory, the most valuable part of the book could well be the somewhat smaller amount of attention it devotes to the problem of how we formulate our questions in the first place.
Kendall did not share Kent’s conviction that the job of the analyst was “to stand behind [the policymakers] with the book opened at the right page, to call their attention to the stubborn fact they may be neglecting.”4
Moore has done a deep and convincing job of diagnosing the ills of the IC, and has proposed a rich and promising cure. This, as Hilton points out, is an extended act of cognition. What lies between this book and Moore’s revolution, however, is the need to have others come to the same conclusion— which, as Hilton points out, requires communication, not cognition.
Sixty years ago a small group of analysts—dubbed “Talmudists” for their pains—worked out a complex, sophisticated method of deriving action- able intelligence from the tightly controlled propaganda outlets of the USSR and Mao’s China. This let IC sinologists spot the first signs of the Sino-Soviet split as early as April 1952, and by 1955 Khrushchev had been tagged as the likely winner in the struggle to consolidate power in the Kremlin after Stalin’s death. Those early indicators, however, remained scoffed at and un-acted upon precisely because the methodology—which a colleague in the CIA compared to studying “invisible writing on slugs”6—was too complex and too weird to be easily explained to policymakers—who, in any case, already believed other hypotheses, and had their own “facts.” 7
(6) Richard Shryock, “For An Eclectic Sovietology,” Studies in Intelligence, vol. 8, no. 1 (Win- ter 1964).
COMMENTARY
Emily S. Patterson, PhD Assistant Professor College of Medicine
The Ohio State University
it is an achievable goal for the vast majority of United States policy to be directly informed by evidence that is systematically validated, collated, and synthesized by teams of professional intelligence analysts.
This book is a critical milestone in attaining the goal of analysis directly supporting evidence-based policymaking. This book’s primary contribution is to conduct sensemaking on the label sensemaking. Decades of relevant academic literatures have been synthesized into one framework that illustrates how disparate research streams relate to each other and to the framework. Until now, there has not been such an extensive effort to pull together related research on sensemaking from such diverse disciplines as psychology, political science, philosophy, organizational science, business, education, economics, design, human-computer interaction, naturalistic decisionmaking, and macrocognition.
The contributions of this book go beyond a literature review, how- ever, in that an action-oriented stance is taken toward capturing nuggets of insight on how to improve aspects of analysis. The categories themselves are useful in putting some shape and structure to the amorphous value that expertise brings to creating a solid analytic product in an uncertain world: planning, foraging, marshaling, understanding, and communicating. Of par- ticular value is describing different aspects of validation that are relevant to intelligence sensemaking, and distinguishing processes for predicting future events (foresight) from processes for describing past events and assessing their impacts (hindsight).
look at what is measured [to] operationally (to) determine how people truly define a concept.
Even if high rigor is not possible under extreme time pressure, data overload, and workload conditions, the measure has potential value in supporting negotiations for what aspects are most important to do well for a given task, as well as communicating the strengths and weaknesses of the process behind an analytic conclusion.
COMMENTARY
Christian P. Westermann
Senior Analyst
Bureau of Intelligence and Research U.S. Department of State
History will tell us if current intelligence reforms are evolutionary or revolutionary, but the Intelligence Community is responding to mandated change brought about by the 2004 Intelligence Reform and Terrorism Prevention Act (IRTPA).8 In particular, the analytic and collector communities are adjusting to one of IRTPA’s pillars—improved information sharing. As reforms unfold, the collector and analyst must adapt to new rules and new analytic standards, and incorporate more methodologies, techniques, and alternatives in their analysis, in collaboration with managers and tradecraft cells in the national intelligence organizations. These new structures and guidelines present an intellectual challenge as well as a bureaucratic maze for the collector and analyst struggling not only to “produce” intelligence in a timely fashion but also to improve their product. This is not easy for the intelligence professional because time is not on their side. This is why improving the way in which all analysts think is so important and why an understanding of sensemaking will help advance the profession beyond the “established analytic paradigm” for complex problems and create greater possibilities for the application of imagination in the IC. The failure to properly assess Saddam Hussein’s WMD programs during the lead-up to Operation Iraqi Freedom is the preferred example of this failure to imagine alternatives. The corporate solution to this problem is increased collaboration and information sharing; David Moore is not in disagreement but has suggested that it must go beyond new methodologies or techniques—it must be done with a strong sense of rigor and individualism in one’s thinking.
David Moore has written for the Intelligence Community a revolutionary epistemology. His novel construct for intelligence professionals is the foundation for a philosophy of intelligence.
Moore’s prescription is to take the disaggregation of data, commonly referred to as analysis, synthesize it, and then apply to it one’s interpretation and communication skills to make sense of the information. Sensemaking therefore is a theory of knowledge for the intelligence professional and also a practice to aid the difficult art of intelligence reasoning.
His attention to revolutionary change in the art of intelligence thinking grows from his recognition that organizational reform has been ongoing for decades and despite those changes attendant failures have occurred and continue to occur. Therefore the only hope for achieving positive reform rests with changing the practice of intelligence whereby the individual collector and analyst, working together, and accepting the responsibility to think critically but also independently and across the Community, make sense of the 21st century national security environment.
COMMENTARY
Phil Williams, PhD
Director, Matthew B. Ridgway Center for International Security Studies Wesley W. Posvar Chair of International Security
Graduate School of Public and International Affairs
University of Pittsburgh
Moore’s Law for Intelligence
Any book that discusses amongst other things, red brains and blue brains, kayaking, information foraging, flashlights as blindfolds, space-time envelopes, and intellectual audit trails, is out of the ordinary. When you throw in the contention by the author that intelligence as currently practiced is akin to medicine in the 14th Century you have a book that will raise hackles, blood pressure, and voices.
This is not an easy read. But the overall thesis is straightforward and compelling: the environment within which the U.S. intelligence community now finds itself is not only highly complex but also full of wicked problems. To provide the kind of intelligence that is useful, relevant, and helpful to policy makers who have to anticipate and respond to these problems and challenges, Moore argues that the traditional paradigm developed largely by Sherman Kent has to be superseded by a new paradigm based largely on ideas initially outlined by Willmoore Kendall, a contemporary critic of Kent. The original Moore’s Law11 was narrowly technical; David Moore in contrast argues that a complex environment full of mysteries, not puzzles, requires holistic thinking (as opposed to simply disaggregation of problems), mindfulness (as opposed to mindlessness which he also elucidates), and a dynamic willing- ness to change paradigms, shift perspectives, and abandon strongly held perceptions. The book also develops the notion of sensemaking rigor and shows how metrics of rigor can be applied to several studies examining the rise and impact of non-state actors.
David Moore’s analysis is important and deserves to be widely read in the intelligence community and in the academic world.
it would have been helpful, for example, if David Moore had considered more explicitly the argument by David Snowden that making sense of a complex environment requires probing the environment. Further thought about this suggests that law enforcement is particularly good at this form of knowledge elicitation and sensemaking: sting operations, controlled deliveries, infiltration of criminal organizations, are all probing mechanisms that can contribute significantly to an increased level of understanding and, concomitantly, to an enhanced capacity for effective action. For many intelligence professionals, especially those who have had a dismissive view of law enforcement, the idea that law enforcement approaches to sensemaking might be ahead of those in the intelligence community, is likely to be as uncomfortable as most of the arguments in David Moore’s book. Certainly Moore’s volume is designed to shake and to stir. It is a manifesto for an intellectual revolution in the approach to intelligence and, as such, is likely to be both acclaimed and reviled.
PREFACE
On Being Mindful
What Is Mindlessness?
We are surrounded by errors and they are ours. Intelligence officials at the national level repeatedly use the same excuses for professional errors and for the systemic failures that follow. Despite directives to “fix” the structures, and most recently the means, by which intelligence is created, we insistently fail at our obligation to make early sense of vital threats and opportunities.
Ellen Langer, summarizing her pioneering social psychology research, finds mindlessness to arise from an over-reliance on “categories and distinctions created in the past.”12 She holds that such categories “take on a life of their own.”13
Langer also sees mindlessness arising from “automatic behavior.” Here, people rely on automatic responses as the basis for their behavior, as when one writes “a check in January with the previous year’s date.”14 By extension, intelligence professionals, in assessing sources, may develop a habit of discounting human intelligence sources because some are untrustworthy. As a result, they may miss novel insights because they use certain sources to the exclusion of others.
Finally, mindlessness can result from a failure to take into account alternative information that transcends our comfortable worldview. Langer observes that “[highly] specific instructions…encourage mindlessness” because they define what is acceptable and limit the viability of alternative signals that could lead to more accurate understanding of a phenomenon.
There were in fact 100 nuclear- tipped tactical missiles deployed on the island months before the arrival of the more infamous strategic missiles.17 A rigid notion of what constituted a nuclear missile, usually conceived as an offensive weapon, appears to have contributed to the case officers’ mindless disregard of the witnesses.
With respect to intelligence consumers, two faculty members at the International Institute of Management Development (IMD), corporate strategy expert Cyril Bouquet and corporate leadership and organization expert Ben Bryant, suggest that “decision makers often suffer from poor attention management, being obsessed with the wrong types of signals and ignoring possibilities that could significantly improve the fate of their undertakings.”18 They characterize these behaviors as fixation and relaxation. People who fixate “become so preoccupied with a few central signals that they largely ignore things at the periphery.”1
Bouquet and Bryant identify relaxation as when, after a “sustained period of high concentration,” people become unfocused on the task at hand and look to the ultimate goal.23
Sometimes ascribed to intelligence professionals’ and national consumers’ falling prey to “creeping normalcy,” relaxation was also a contributor to Israel’s failure to anticipate the attacks by Egypt and Syria in 1973.
In sum, mindlessness too often guides the assessment of affairs in too many domains, leading to errors, failures, and catastrophes. Mindless- ness is deemed unacceptable within the larger American society only when the resulting errors do lead to accidents and disasters. However, mindlessness is completely unacceptable within the domain of intelligence. One can never be certain in foresight whether errors will occur, so intelligence professionals must seek to anticipate, recognize and avoid them at all costs.
Attaining Mindfulness
The antithesis of mindlessness is mindfulness.
For Langer, a mind- ful state corresponds with: “(1) [aptitude for the] creation of new categories; (2) openness to new information; and (3) awareness of more than one perspective.”26 For example, as an intelligence professional considers who might be a member of Al Qaeda, a mindful attitude would involve constant reassessment and categorization of who might hold such membership— leaving the path open to new information for making sense of the organization and its membership. Thus, as we apply the idea that “[a] steer is a steak to a rancher, a sacred object to a Hindu, and a collection of genes and proteins to a molecular biologist,”27 the notion of a Nigerian male or even a blonde woman from Pennsylvania as a possible Al Qaeda affiliate would emerge from a mindful perspective.
Leadership scholar Deepak Sethi sees mindfulness as a “form of meditation” teaching “three simple-on-the-surface yet revolutionary skills: Focus, Awareness, and Living in the Moment.”28 This definition descends from millennia of Buddhist tradition. He argues that rather than an esoteric method it is “very practical, action oriented, and transformational.” Sethi believes that one practical way to bring about mindfulness is through the use of daily meditation, first using one’s breathing as a focus, and then using “specific daily activities such as meetings with another colleague.” However, the “real challenge [of employing mindfulness] is to take it from the meditation chair to the office chair and the real world.”29 Intelligence journeymen face this dilemma from a different perspective. They confront the real world and are challenged to contemplate their own thought processes as they engage it.
Ben Bryant and IMD research associate Jeanny Wildi write that mindfulness “involves the ability to accurately recognize where one is in one’s emotional landscape and allows…understanding, empathy, and capacity for accurate analysis and problem-solving.”34 They identify a process of detaching, noticing, and developing “here and now awareness.”35 Detachment, for example, allows a viewer to remember that a movie is really merely a “beam of light passing through a piece of moving celluloid projecting onto a screen with some sound and music that are designed to generate particular emotions.”36
In intelligence work, detachment involves stepping back from the full sensual experience of an issue to consider the actors involved, their motives, the larger context. Critical thinking as it is taught in the Intelligence Community attempts to make sense of the overall purpose or goal of a phenomenon, the points of view and assumptions of the actors involved, the implications of their acting in certain fashions, and other aspects of the larger context sur- rounding the issue.37 Questioning the available evidence and the inferences arising from it brings further detachment from the issue.
Noticing involves remaining open to both internal and external stimuli. Ultimately, situational information is conveyed from external sources through sight, sound, touch, smell, and taste. People can think consciously about these but they tend to process them using more autonomic brain structures, often without noticing they are doing so. The unease one feels about getting into a taxi or onto an elevator in an unfamiliar setting are examples of such input. In intelligence work this might be represented as a hunch about what an adversary will do. As Daniel Kahneman and Gary Klein note, in certain environments—where one can learn the cues—these intuitions may be quite accurate.38 However, in domains where one has not developed expertise, such intuitions can be inaccurate.39 The challenge is determining which of these situations one is in. This brings us back to the imperative of applying mindful detachment from the situation.
34 Ben Bryant and Jeanny Wildi, “Mindfulness,” Perspectives for Managers, no. 162 (September 2008), 1, URL: <http://www.imd.ch/research/publications/upload/PFM162_ LR_Bryant_Wildi.pdf>, accessed 14 January 2010. Cited hereafter as Bryant and Wildi, “Mindfulness.”
as Warren Fishbein and Gregory Treverton note, as “[mindful- ness] is the result of a never-ending effort to challenge expectations and to consider alternative possibilities.”43
“executives need to meditate in their own way, find ways to step back and reflect on their thoughts, actions, and motivations, and decide which ones are really supportive of their strategic agendas.”
Definitions for Making Sense of Sensemaking
Intelligence is a “specialized form of knowledge…[that] informs leaders, uniquely aiding their judgment and decision-making.” It is a type of knowledge created through organized activity that adds unique value to the policy- or decisionmaker’s deliberations. In the U.S. context, it makes sense of phenomena of interest to national leaders, warfighters, and those that directly and indirectly support them. Intelligence makes sense of phenomena related to the social behaviors of others. It reflects interest in what anyone will do to, and with, others that could affect the national interests of the United States as well as the prosperity and security of its citizens. Intelligence maintains an interest in external phenomena, such as epidemic or pandemic diseases, that impact U.S. national interests. In contrast to some popular portrayals, it really is not voyeuristic: what others do privately and alone is generally of little interest or value except as it affects how they relate to, and behave toward others. In other words, when private behaviors reveal either vulnerabilities or preferences, they may become of value to intelligence practitioners.
Sensemaking as it is used here refers to “a set of philosophical assumptions, substantive propositions, methodological framings, and methods.” As Mark Stefik notes (referring to work done with col- leagues Stuart Card and Peter Pirolli), it “is how we gain a necessary understanding of relevant parts of our world. Everyone does it.” Sensemaking goes beyond analysis, a disaggregative process, and also beyond synthesis, which meaningfully integrates factors relevant to an issue. It includes an interpretation of the results of that analysis and synthesis. It is sometimes referred to as an approach to creating situational awareness “in situations of uncertainty.” Gary Klein, Brian Moon, and Robert Hoffman consider the elements of sense- making and conclude that it “is a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively.”
These authors conclude that “the phenomena of sensemaking remain ripe for further empirical investigation and [warn] that the common view of sensemaking might suffer from the tendency toward reductive explanation.” By reductive explanation Klein, Moon, and Hoffman refer to a tendency to overly simplify explanations—to “reduce” complex phenomena to simplistic models facilitating an apparently needed but shallow understanding.
Intelligence sensemaking encompasses the processes by which specialized knowledge about ambiguous, complex, and uncertain issues is created. This knowledge is generated by professionals who in this context become known as Intelligence Sensemakers.
These terms are used as defined here throughout this book.
Sensemaking: A Structure for an Intelligence Revolution
CHAPTER 1 Introduction
Where We Are
How people notice and make sense of phenomena are core issues in assessing intelligence successes and failures. Members of the U.S. Intelligence Community (IC) became adept at responding to certain sets of phenomena and “analyzing” their significance (not always correctly) during the Cold War. The paradigm was one of “hard, formalized and centralized processes, involving planned searches, scrupulously sticking with a cycle of gathering, analyzing, estimating and disseminating supposed enriched information.”
A growing professional literature by intelligence practitioners discusses these trends and their implications for advising and warning policymakers.58
The literature by practitioners embodies a trust that national intelligence producers can overcome the “inherent” enemies of intelligence to prevent strategic intelligence failure.59 The disparity between this approach and accepting the inevitability of intelligence failure has grown sharp enough to warrant the identification of separate camps or schools of “skeptics” and “meliorists.”60 As a leading skeptic, Richard Betts charitably plants the hopeful note that in ambiguous situations, “the intelligence officer may perform most usefully by not offering the answer sought by authorities but by forcing questions on them, acting as a Socratic agnostic.”61 However, he completes this thought by declaring, fatalistically, that most leaders will neither appreciate nor accept this approach.
Jervis asserts that policymakers and decisionmakers “need confidence and political support, and honest intelligence unfortunately often diminishes rather than increases these goods by pointing to ambiguities, uncertainties, and the costs and risks of policies.”63 The antagonism is exacerbated when policy is revealed to be flawed and to have ignored intelligence knowledge.
Jervis’ article on intelligence and policy relations, while it correctly notes the tensions arising from the differing roles of intelligence and policy, over-generalizes the homogeneity of the policy community. It is the author’s experience that outside of the highest levels, there are many levels of policymaking that both encourage and welcome the contributions of intelligence. Indeed, some parts of the policy community, beyond the Department of Defense (DoD) where it is the norm to do so, rely strongly on intelligence. Further, disagreements (which Jervis consistently labels conflict) are inherent and typically welcome in the process. Hard questions about the accuracy of judgments must be asked. If we are doomed to such “disagreements,” then it is a doom we should be eager to embrace.
The other perspective is that of the meliorists—those who feel intelligence processes can be improved. The present authors reside in this camp, preferring to believe that the application of well-informed, mindful exper- tise, as developed in the present work, can bring positive and substantive value to the fulfillment of the IC’s obligations.
intense attention within and outside the IC has focused on the means by which pertinent phenomena are to be under- stood. So-called intelligence “analytic” methods are being unshelved or developed and taught to novice and experienced intelligence professionals alike. However, less fully considered are the appropriateness and validity of these methods as well as the underlying assumptions they enshrine. Even less well understood is what happens when specific methods are combined and how those combinations may be made. Several ways exist to characterize these methods in terms of their purpose. However, to date, there is no readily avail- able way to characterize methodological appropriateness or effectiveness, nor the limitations of individual methods. We also lack sound guidance on the use of combined methodologies, despite some recent, promising literature.
Before these deficiencies can be remedied, however, we need to reframe the way in which intelligence is created. Such a re-conceptualization involves critically examining what intelligence practitioners actually do, and why. The examination demands methodological rigor with particular attention to how we might ensure the validity of our approach to the work of intelligence. If the examination indicates that the existing paradigm for intelligence creation is inadequate, then a revolutionary shift in IC habits will be justified.
the intelligence-creation process remains largely a product of Cold War-era institutions and thinking, using the same cognitive frameworks that have been employed for decades. Some argue that what worked in the past is still appropriate. However, as numerous executive and legislative reports confirm, intelligence targets have in fact evolved: adversaries’ goals have changed, and their methods have evolved, even if the threats they pose seem very familiar. In sum, the old national intelligence paradigm is woefully out of date.
Intelligence issues are not the same as the issues framed separately by policymakers. To partner successfully with policymakers, intelligence professionals must consider issues from multiple perspectives. This is the role of sensemaking. Yes, the sensemaking process includes “analysis” or attacking issues by “taking them apart.” The process also includes synthesis—putting the pieces back together; interpretation—making sense of what the evidence means; and communication—sharing the findings with interested consumers. Essential to these processes is another, that of sound planning or “design.”69 While it could be said that this is what intelligence analysts do, such a statement is epistemologically false. Strictly speaking, intelligence analysts only take issues apart.
Why should we be concerned with a matter of semantics? In short, because the terms we use within the Intelligence Community shape and reflect our practice. If we are to change the culture of intelligence, and be changed by it, our practice of intelligence must also change. New language encourages a new paradigm, and paradigm shifts are revolutionary, not evolutionary.
Kent’s Imperative70
When much of the tradecraft of intelligence was put in place sixty or more years ago, the dominant framework was that of the historian as scientist. The primary intellectual framework for Cold War intelligence at the national level grew from Sherman Kent’s seminal work, Strategic Intelligence for Ameri- can World Policy.71 Kent’s legacy remains active in the National Intelligence Council and the Community at large.72 Although decision theory and other social science thinking began to influence the creation of intelligence in the 1960s and 1970s, these inputs languished until the reform efforts of recent years. More recently, advances in cognitive science, anthropology, decision theory, knowledge theory, and methods and operations research have brought us to the brink of informed, mindful intelligence sensemaking.
Sherman Kent argues that in creating predictive intelligence about its adversaries “the United States should know two things. These are: (1)…strategic stature, (2)…specific vulnerabilities.”73 These objectives focus on capabilities and draw heavily from the “descriptive and reportorial elements” of intelligence for basic data.74
The Failure of an Analytic Paradigm…
Kent’s preference for gathering and disaggregating more and more data to find answers fails today in the face of information volume, velocity, and volatility. Marshaling and disaggregating ever more data does not equate to contextual understanding. Further, the assumption that larger pipes to collect data and larger arrays to store it will then allow us to uncover the hid- den, clarifying nuggets, is misleading.
Consider what actually happens when intelligence professionals look for an answer to a problem or question. They do not just disaggregate data. Instead, people inquisitively (and selectively) interpret patterns by comparing observed, newly emergent phenomena to what they already “understand.” They make sense of phenomena by asking questions; foraging for information; marshaling it into evidence; analyzing, synthesizing, and interpreting that evidence, and communicating their evidence-based understanding of issues to others. Something makes sense because, based on their experience, its pattern is similar to something they previously have seen and that made sense to them. They may even employ a new, self-generated pattern based on previously learned and remembered patterns if they do not get a good match to an ostensible pattern.78
In other words, one must be able to convincingly correlate ostensible patterns to the data or information for which one is attempting to “make sense.” This is not always possible, especially if the phenomenon or issue is broad, novel, or poorly understood; that is, not easily subject to confirmation by universal human sensory apparatii.
For practitioners to create intelligence knowledge—even with an acknowledged degree of uncertainty—therefore requires much more than mere “analysis.” One alternative framework is embodied in the concept of sensemaking. Sensemaking begins with a mindful planning and questioning that leads to foraging for answers. It is true that along the way the resulting relevant assemblage of information—or evidence—is disaggregated into its constituent elements. However, it is also synthesized or combined to form a theory or systematic interpretation of the issue that subsequently must be explained, and convincingly. Throughout sensemaking, a continuous assessment is demanded of both the processes by which the intelligence is created and of the intelligence knowledge itself.81 Mindfulness—as discussed above in the Preface—coupled with a critical thinking-based approach, pro- vide the vigilance, awareness, and self-reflection needed to assess an issue rigorously. This is a central point: Intelligence does not exist in a vacuum. It must contribute to the understanding of an issue by informing the concerned parties of a perspective or information they did not already know. Ultimately, if no one is concerned about the knowledge sensemakers create, it is not intelligence.
IARPA, the Intelligence Advanced Research Projects Activity, employs a definition of sensemaking that is complementary to that developed here.86They propose that sensemaking is “a core human cognitive ability [that] underlies intelligence analysts’ ability to recognize and explain relationships among sparse and ambiguous data.”87 This book accepts that perspective and develops the psychological, behavioral, and social levels of sensemaking as they apply to intelligence creation. By contrast, IARPA’s own program on sensemaking seeks to build upon advances in computational cognitive neuroscience that reveal “the underlying neuro-cognitive mechanisms of sensemaking.”88
IARPA, BAA-10-04, 4. On the emerging discipline of cognitive neuroscience, see The 4th Computational Cognitive Neuroscience Conference, URL: <http://ccnconference.org/>, accessed 7 June 2010.
As characterized by Peter Pirolli, the process of sensemaking is highly iterative, involving a foraging loop and a sensemaking loop.89 In the former the sensemaker seeks information, “searching and filtering it,” while in the latter an iteratively developed mental model or schema is developed “that best fits the evidence.”90 While the overall flow is “from raw information to reportable results,” top-down and bottom-up processes act in concert to reframe issues: information either does or does not fit the hypotheses being considered; hypotheses are refuted or refined, and the larger issue and its context are also reframed, as it comes to be more thoroughly understood.91 How this can occur within the context of intelligence creation is developed in the following chapters.
To sum up, this book argues that intelligence built around a model of disaggregation as it originated with and developed under Kent, and is still largely practiced today, is at best insufficient. A paradigm based on the concept of sensemaking and employing insights from other knowledge-creation disciplines provides a more appropriate means of skillfully creating intelligence. This book draws a general picture of 21st Century intelligence under a revolutionary paradigm, although it does not explain how all its contours can be fleshed out. We believe that intelligence could be a true profession and moving toward that goal is our desire.
CHAPTER 2
The Failure of “Normal Intelligence”
Intelligence Challenges
Our understanding of everyday phenomena is confounded by every- day strategies employed to mitigate cognitive dissonance, a stressful condition arising when reality clashes with one’s perceptions. Two broad strategies, selective exposure and selective perception, can prevent dissonance, but at the expense of sound, mindful reasoning. Through the former, we limit the evidence to that which agrees with or otherwise supports our positions; in the latter, we interpret what we experience in terms of our pre-existing world-view.
the differences between “intelligence error” and “intelligence failure.” Anthropologist Rob Johnston defines intelligence error in terms of “factual inaccuracies in analysis resulting from poor or missing data.”98 Conversely, intelligence failures are “systemic organizational surprise resulting from incorrect, missing, discarded, or inadequate hypotheses.”99 Thus, the term “failure of imagination” makes sense as a synonym for intelligence failure, where members of an intelligence creating organization fail to imagine in advance the essential outlines of an incident that subsequently occurs.
The etymology of “imagination”—generating images—reminds us of the contemporary critic of Kent, Willmoore Kendall, who suggested that the job of national intelligence is to communicate with decisionmakers in a “holistic” way so as to generate the “pictures [mental models] that they have in their heads of the world to which their decisions relate.”112
Considering Standard Models
Intelligence failures occur as practitioners employ a “standard model”113 of intelligence: In it, analysts “separate something into its constituent elements114 so as to find out their nature, proportion, function, relationship, etc.115 and “produce reports” based on “collected” information and data. There is a definitional presumption that disaggregation will lead to answers. However, this model incompletely describes what the intelligence professional does and its underlying presumption about finding answers may be false.
One problem is that in Kent’s data-based analytic framework, analysts need to have all the data available so they can be marshaled into a coherent account. “Dots”—if they exist at all—can be connected in more than one way.116 In foresight it is difficult at best to determine which combination and order is valid. Such determinations can be further complicated by the fact that adversaries may change their actions if they suspect we have arrived at a certain conclusion.
An additional problem is that with an increased number of signals there is also an increased level of noise. Which signals, which facts, or which inferences the intelligence professional should consider valid becomes a very important consideration. At best, warning of a pending incident is a problem of assembling and making sense of the details of a specific incident in advance. However, many intelligence problems inherently defy such linear characterization. They are in fact “wicked” problems—a formal designation of a complex issue with myriad linkages. We turn next to an exploration of problem types to see how their nature directs our making sense of them.
Types of Problems
In order to understand “wicked problems,” one must first understand the nature of “tame problems.”
Tame Problems
In a tame problem there is general agreement as to what or who an adversary is, what the “battlefield area” is, and what an attack is. Such problems, while difficult, exhibit specific characteristics: They are clearly defined and it is obvious when they are solved. Solutions to these problems arise from a limited set of alternatives that can be tested; the correct solution can be objectively assessed. Finally, solving one tame problem can facilitate creating valid solutions to other, similar tame problems.
It is important to note that analysis protocols for tame problems con- tain little or no room for “emergent” properties. One may not know that the analytic protocol is insufficient until the puzzle has been incorrectly defined, characterized, and solved, if it is in fact solvable. One arrives at one solution that at first appears to have resolved the issue, but in fact, the issue reemerges elsewhere. For example, the implementation of a linear, intelligence-driven solution to crack down on insurgents and their improvised explosive devices (IEDs) in one area may lead to an emergence of IED-caused explosions some- where else. In such a case, the application of “tame problem protocols” may in fact have been inappropriate—the problem is in fact not tame.
Admittedly, many 21st Century intelligence issues remain puzzles or tame problems. This occurs when the events surrounding the issues have already occurred, appropriate questions are readily identifiable, and answers exist, even if they are difficult to find.
Wicked Problems
However, seen in a larger context, are such puzzles truly tame? Or are they components—as Russell Ackoff suggests—of something larger: a mystery in Treverton’s terms, or a “mess” according to Ackoff.119 Treverton’s intelligence mysteries defy easy definition. They belong to a class of problems defined by social researchers Horst Rittel and Melvin Webber as “Wicked Problems.”
The adaptive nature of adversaries makes seemingly tame puzzles wicked, moving them into the realm of “unknown unknowables.”
By definition, wicked problems are “incomplete, contradictory, and changing.”121 They do not have single answers and in fact, are never truly answered. In the context of intelligence, the sensemaker may never realize a problem has been resolved. This is because “the solution of one of its aspects may reveal or create another, even more complex problem.”122 The emergent complexity of the problem itself, its adaptive nature, efforts at denial and deception by adversarial actors, as well as cognitive frailties on the part of sensemakers, compound the problem, confounding sensemaking, leading in some cases to disastrous courses of action or consequences.
A Wicked Look at Wicked Problems in Intelligence
Characterizing intelligence issues in terms of their problem type— admittedly somewhat vaguely (in keeping with their nature)—reveals just how prevalent wicked problems are within the domains of intelligence.
Wicked problems have no definite formulation. To Rittel and Webber, “the process of solving the problem is identical with the process of understanding its nature, because there are no criteria for sufficient understanding.”124 In other words, making sense of problems deemed sufficiently complex so as to be considered wicked is equivalent to characterizing them in the first place; the description encompasses all possible solutions.
one wicked problem example could be “how best to stem the growth of terrorism in the Middle East.” An assumption in considering this problem is that if intelligence professionals can understand what motivates people to become terrorists in the first place, intervention might be possible. Mitigating the creation of new terrorists could aid in reducing both their numbers and by extension, their attacks. Do people become terrorists because they are dissatisfied with what they see as contradictions and hypocrisies in their lives? If so, what then are the specific roots of dissatisfaction and contradiction? One commonly cited is a lack of economic opportunity for males within societies. In that light, Rittel and Webber ask, “where within the…system does the real problem lie? Is it deficiency of the national and regional economies, or is it deficiencies of cognitive and occupational skills within the labor force?”125 The possible solutions to this problem extend the domain of questions, spreading ever outward.126
as Nicholas Taleb notes, [Our] track record in predicting those events is dismal; yet by some mechanism called the hindsight bias we think that we understand them. We have a bad habit of finding “laws” in history (by fitting stories to events and detecting false patterns); we are drivers looking through the rear view mirror while convinced we are looking ahead.
Even with the addressal of major assumptions, there remain additional underlying factors that do not get questioned—almost an endless succession of assumptions that must be peeled off the problem much as one peels layers off an onion. There is an added complication that individual layers are not sequential and in fact may lead (to continue the analogy) to other onions or other vegetables, or even fruit. In intelligence, such assumptions are themselves a mess: a complex system of interrelated experience, knowledge, and even ignorance that affects reasoning at multiple levels sequentially and simultaneously.
Wicked problems have no clear end-point.
With tame and well-structured problems one knows when the solution is reached. In wicked problems this is not so, as Rittel and Webber make clear:
There are no criteria for sufficient understanding and because there are no ends to the causal chains that link interacting open systems, the would-be planner can always try to do better. Some additional investment of effort might increase the chances of finding a better solution.
We have a bad habit of finding “laws” in history (by fitting stories to events and detecting false patterns); we are drivers looking through the rear view mirror while convinced we are looking ahead.127
This is not a new consideration. Writing in the 1930s, John Dewey observed that “the ‘settlement’ of a particular situation by a particular inquiry is no guarantee that that settled conclusion will always remain settled. The attainment of settled beliefs is a progressive matter; there is no belief so set- tled as not to be exposed to further inquiry.”131 Intelligence sensemakers routinely confront this challenge. Reports and assessments often update or revise previous conclusions. Often the previous reporting is consulted before the new report is written so that the author can determine the preexisting point of view on the issue. Such consultations at best determine whether the current situation deviates from the norm. Unfortunately, sometimes such consultations lead to the rejection of the new evidence, opening the way to intelligence errors and failures. One goal of an adversary’s denial and decep- tion activities is to facilitate rejection of the novel deviation. It was in this way that the possibility of nuclear missiles deployed to Cuba was rejected amid outlandish noise during the summer of 1962, and military exercises along the Suez Canal lulled Israel into a sense of creeping normalcy prior to October 1973.
Solutions to problems may be implemented for “considerations that are external to the problem” itself: problem solvers “run out of time, or money, or patience.”132 In intelligence, sensemakers may only be able to work for a given time on a problem before they have to issue their report. Changes in funding may mean that an effort to understand a phenomenon has to be dis- continued. The practicalities of resource limitations force changes in sense- makers’ foci. However, this does not mean that the problem does not continue to exist and perhaps, threaten. Rather, an answer has been developed to a dis- tilled problem, communicated, and now other things must be done.
Solutions to wicked problems are at best good or bad.
Some problems have true or false, yes or no answers. These are not wicked problems. Wicked problems have no such answers. Differing perspectives applied by different problem solvers, differing sets of assumptions, and differing sources of evidence are several of the factors that lead separate groups to come to different judgments about wicked problems. The impossibility of exhaustively considering all the factors and solutions of the problem also contributes to a multiplicity of solutions.
Focusing on the economics sur- rounding the growth of terrorism leads to different proposed solutions than does focusing on the demographics involved in the issue. Religious consider- ations or broader cultural considerations also create different solutions. Each of these perspectives in turn optimizes multiple points of view with differing, good and bad solutions. Overlap is possible and even desired. Good solutions encompass multiple domains.
Tests of solutions to wicked problems may not demonstrate their validity and may provoke undesired consequences. Implemented solutions to wicked problems “generate waves of consequences over an extended— virtually an unbounded—period of time.”
Further, these consequences may themselves prove so undesirable as to negate any and all benefits of the original decision—and this cannot be determined in advance. Thus an intelligence-based decision to invade a country’s possessions may create circum- stances that offset any gains initially won, as the Argentineans discovered in 1982 when they—unwisely in retrospect—seized the British-owned Falkland Islands.
Implementing solutions to wicked problems can change the problem. In intelligence problems, real solutions cannot be practiced; there are no “dry runs.” True, sensemakers and their policy-making customers can (and should) consider what might happen or the “implications” of the decisions or solutions of the problem at hand. Doing so might increase the likelihood that the decision selected is the best or the less bad of a set of bad alternatives.
Modeling the situation is one common means of assessing the implications of a potential action. However, models must by their very nature limit the factors considered. This raises the question of how one might know in advance if the eliminated factors are in fact significant.
As Rittel and Webber note, “every attempt to reverse a decision or to correct for…undesired consequences poses another set of wicked problems,”135 as sensemakers and planners involved in the U.S.- led “war on terror” have discovered. Actions, once taken, may mitigate the threat, or may not, which leads to the next facet of wicked problems.
Sensemakers can never know if they have determined all the solutions to wicked problems. They can expect, however, that they almost certainly have not determined all the solutions. In developing the range of alternatives within scenarios, two goals predominate: mutual exclusivity and collective exhaustion.
In other words, each alternative must preclude the simultaneous possibility of the others, and the entire set of known alter- natives must be considered. In practical terms, this is much more difficult to achieve than it sounds. Intellectual frameworks and so-called “biases” such as vividness, anchoring, confirmation, and others combine to prevent people from being able to consider all the alternatives. Adding to this is the fact that issues evolve in unpredictable ways. All the solutions simply are not knowable because they lie in the future
Each wicked problem is unique. While it is true that common elements can be found between problems, there remain additional and unique properties of “overriding importance.”136 In other words, wicked problems cannot be characterized into “classes…in the sense that principles of solu- tion can be developed to fit all members of a class.”
Every wicked problem is embodied in another one. Rittel and Webber describe problems as
[discrepancies] between the state of affairs as it is and the state as it ought to be. The process of resolving the problem starts with the search for causal explanation of the discrepancy. Removal of that cause poses another problem of which the original problem is a “symptom.” In turn, it can be considered the symptom of still another, “higher level” problem.139
What policies and actions, for example, are necessary to “fix intelligence?” Answering this involves asking what is causing intelligence to fail. One place to start is to consider why analysts are wrong and how intelligence errors lead to intelligence failure.140 Yet such considerations lead one to consider how consumers may ignore intelligence, and how adversaries may in fact be “more capable” than expected. These in turn lead to what Jeffrey Cooper considers “analytic pathologies” that decrement both individual and corporate efforts to make sense of issues (table 2). Each of Cooper’s specific pathologies is furthermore at least partially embodied in the others, giving rise to error-producing systems.141 For example, Cooper argues that intelligence professionals’ pathological focus on both “the ‘dots’ analogy and the model of ‘evidence-based’ analysis…understate significantly the need for imagination and curiosity.”142 Related to this is what he calls the myth of “Scientific Methodology.”
Analysis is not [hard] science and is not about proof. Rather it is about discovery.143 These are embodied in the protocols he refers to as the flawed “Tradecraft Culture,”—a guild system of potential sensemakers and their historically unchanging ways of working.144
How wicked problems are resolved is determined by the means and methods used to make sense of them. In other words, how problems are perceived determines the kinds of solutions that are proposed. Point of view becomes essential in defining what a problem is and how it is to be resolved. Complex, wicked problems (as well as many “tame” ones) cannot be defined from one point of view.
| How Are Wicked Problems Disruptive? |
| Disruption, as developed by Clayton Christensen, emerges from technologies that, while they may under-perform established tech- nologies, open new markets and change the ways people do things. Enlarging the definition, disruptive intelligence problems threaten to change the way people interact. They proffer or impose new para- digms—both “good” and “bad”—for non-governments and govern- ments alike. The disruption occurs because the incumbent is doing the most rational thing it can do given its circumstances. Doing the right thing generates the opportunity for disruption.
Clayton Christensen, The Innovator’s Dilemma (Cambridge, MA: Harvard University Press, 1997). |
Sensemakers have no right to be wrong.
[T]he primary function of the Central Intelligence Agency is to seek the truth regarding what is going on abroad and be able to report that truth without fear or favor. In other words, the CIA at its best is the one place in Washington that a President can turn to for an unvarnished truthful answer to a delicate policy problem.148
Will Pitt, “Interview: 27-Year CIA Veteran,” Truthout, 26 June 2003,
This aphorism may have validity in the domain of tame problems where the truth is known or knowable. However, it has much less (if any) validity in the world of wicked problems where many truths can coexist, depending on the point of view expressed, the context can be simultaneously true and contradictory, and may in fact be unknowable.
The goal of assessing wicked problems may be to “improve some characteristics of the world where people live. Planners are liable for the con- sequences of the actions they generate; the effects can matter a great deal to those people that are touched by those actions.”
An Intelligence Example: Pandemics as Wicked Problems
One of the threats faced by intelligence organizations and their professionals is that of an emergent global pandemic. What kind of a threat is a pandemic? Is it a tame or wicked problem, or something in between? Such considerations matter because they define what approaches are suitable for alleviating or mitigating the threats to national security that pandemics pose.
When the stakes are the lives of many people, sensemakers and policymakers who miscalculate or underestimate or are otherwise wrong about a pandemic and its impact on their countries or region can expect vilification at best. A fear of such vilification from the public and the media might contribute to the situation whereby pandemic-tracking organizations such as the U.N. World Health Organization or the CDC overestimate the severity and threat posed by a pandemic such as the 2009-2010 Swine Flu pandemic.158 For intelligence professionals this phenomenon is not unknown. Common wisdom among intelligence sensemakers is that it is far better to warn and be mistaken (and nothing hap- pens) than to not warn and be mistaken (something happens).
Complexity
Rittel and Webber’s notions of wicked problems can also be characterized through the lens of complexity theory. As developed by Jonathan Rosenhead, “systems of interest to complexity theory, under certain conditions, perform in regular, predictable ways; under other conditions, they exhibit behaviour [sic] in which regularity and predictability is lost.”159 This is exceptionally true of intelligence. Certain kinds of issues, including the interpretable indications of a build-up to armed conflict, can be extremely predictable.
However, in other situations, there may be a number of unknowable, unpredictable, and unanticipatable outcomes. Thus, reliable prognostication is simply not possible.160 For instance, if a coalition of nations removes an oligarch in another nation from power, the specific outcomes of that action cannot be known in foresight. While alternative outcomes can be modeled and simulated, they remain valuable only as discussion points: there is no guarantee in advance that they have captured the reality that will occur. Modeling and simulation are feasible because complexity science shows that the “indeterminate meanderings of these systems, plotted over time, show there is pattern in the movements…the pattern stays within a pattern, a family of trajectories.”161 Unfortunately, because intelligence must address the “real” world, rather than its modeled or simulated semblance, events often are unique and therefore their patterns also are unique.
Thus, there exists an inability to guarantee a future reality; even probabilities may be suspect.
Analysis as here defined is insufficient to address complexity. Disaggregation simply does not reveal future alternatives. That this is so becomes obvious if one finds that it is the emergence of unique and novel behaviors arising from different and minutely differing initial conditions that characterize many 21st Century intelligence issues. In these circumstances, the whole of an issue is greater than its parts. But, in analysis, the issue is by definition and practice the sum of its parts.
Given these complex issues, the concept of “analysis” is simply insufficient for sensemaking. Instead, greater conceptual accuracy and precision of terminology is required.
To achieve the needed accuracy and precision requires more than semantic invention. It also demands that underlying concepts, known as assumptions or premises, be identified and accounted for. Therefore, in developing the case for considering new paradigms for intelligence, certain terms require explicit (re)definition.
Implications of Complexity
Viewed from a larger context, complexity stymies the entire “standard model” of intelligence creation. With regard to Kent’s concept of knowledge, or how intelligence is created, complexity—as viewed from the framework of wicked problems—confounds the consideration and mitigation of such problems. Kent’s model of predictive and specific warning seems more miss than hit. Complexity further confounds the collaborative processes contained within Kent’s notions of Activity and Organization, by which intelligence pro- fessionals are tasked to interact. How does the intelligence professional know in advance whose imagination will be most helpful in making sense of the problem at hand in time to prevent a catastrophe or even imagine one?
Given the challenges of both tame and wicked 21st Century intelligence problems and their inherent complexity, what are intelligence professionals to do? One avenue open to them, and presented below, is the development and validation of methods of reasoning about key evidence.
CHAPTER 3
From Normal to Revolutionary Intelligence
Evidence-Based Intelligence Creation
Intelligence sensemakers use more than context-less data and information. They employ assemblages of evidence—at a minimum, collections of data and information determined through marshaling to be relevant to the issue under consideration—in other words, contextualized to specific issues. Evidence reveals alternative explanations through pattern-primed, induced inferences about what is going to happen or what has happened already in the past.
While the inferences are typically uncertain, they do justify beliefs about phenomena. Justifying beliefs (or theories or hypotheses) presents a case for their accuracy but does not guarantee ground (or any other) “truth.” Rather, as Peter Kosso notes, justifying beliefs is “about meeting the standards of evidence and reason [to] indicate [the] likelihood of accuracy.”168 Sensemakers go further and seek to demonstrate that the knowledge of tendencies they establish provides for “a correlation between being more justified and being true.”
It is arguable whether greater evidentiary justification demonstrates the likelihood of a strongly accurate correlation with truth. As Kosso makes clear, even with abundant justification, there is no certainty of truth.
As figure 1 illustrates, intelligence sensemaking is conducted in ser- vice of a number of goals, including describing states of affairs, explaining phenomena, interpreting events and actions, and estimating the likelihood and impact of a foe’s future actions. As intelligence professionals move from describing events, explaining patterns of behavior, and grasping underlying factors and intentions, ever more justification of beliefs about the phenomena under scrutiny is required. Yet, as intelligence professionals attempt to apply greater scrutiny in this sequence, their capability to do so decreases as they face greater ambiguity.
intelligence evidence, while it may appear to remain “haphazard,” is the result of systematic foraging, gathering and interpretation. The past tells intelligence practitioners what to look for in the future. This poses dangers when those indicators are no longer (if indeed they ever were) valid.
If using the past to gain wisdom about what the future holds is not feasible, what about studying the past to avoid folly? Tversky and Kahneman’s work on availability leads one to suspect (as Fischoff also notes) that focusing on misfortunes “disproportionately enhance[s] their perceived frequency.”181 Another challenge to considering the past as a teacher of what not to do is that one may not properly understand the problem.
With the intention to improve evidence-based intelligence creation, recent legislation “reforming” intelligence goes so far as to require that “alternative analysis” be conducted.182 The IC, at least through its schools, interprets this to mean that multiple hypotheses be considered. The relevant act mentions “red teaming: a means by which another group of intelligence professionals consider alternative explanations for an issue being scrutinized.183 The legislation leaves unexamined the question of whether the criteria for sensemaking will be met in examining tame problems and especially wicked problems arising from consideration of adversarial intentions.
If, for example, one estimates that a particular country whose policies one’s own government generally opposes will develop both a long-range missile capability and a nuclear weapons capability and then marry the two together, one has to have already imagined, in the context of the target country’s political and technological environment, what a long-range missile capability is, what a nuclear weapon is, what a weapon of mass destruction is, and a strong sense of the will to combine these threat elements. Policymakers may challenge the target country’s actions, making their leaders more adversarial. Thus, at a minimum, intelligence and policy create the future—or a version of it. Done poorly, this can lead to unintended and dangerous implications.
In a tense bilateral or even multilateral environment, rhetoric and actions can precipitate events so as to create a future consistent with those pattern-derived conclusions, driving the target country to produce the weapons. Each side then blames the other nation’s government for having “caused” the crisis.
when interpretations of the evidence lead to coherent alternative inferential conclusions, then the existing or accepted theories require changing. What must not happen is to reinterpret the evidence to support the prevailing pre- existing theory,
People are often unwilling to abandon their cherished positions. This occurs in part because they are not dispassionate as they reason about evidence. In other words, positions are influenced by various worldviews or cognitive approaches, particularly selective perception and selective exposure. These combine to steer how people recognize issues, the phenomena that comprise them, and how they go about making sense of them.184 These influences or theoretical frameworks shape the patterns people use to interpret new phenomena. The benefit is that these frameworks make people smart and do so quickly.
In an information-rich environment brought about by technical collection, intelligence professionals can select inappropriate patterns to use in making sense of new phenomena. In intelligence work, if such patterns conspire to affect the search for and the selection of the evidence sensemakers use, and that they and their consumers then accept, selective perception and selective exposure set the stage for intelligence error and failure.
Evidence always requires a context, and as the missile example illustrates, there may be more than one explanatory context that makes sense. In intelligence, “evidence is [particularly] rarely self-sufficient in information or credibility.”
unless the correct context is known, evidence—if its constituent information can even meet that threshold— is subject to many different interpretations. Without context the person assessing the evidence has no way of knowing which interpretation is correct. Multiple contexts further confound the situation, for different contexts often lead to alternative conclusions as was illustrated in the missile development scenario just described. Finally, as Hampson’s essay reveals, the political context of the policymaker may skew the actual context conveyed by intelligence.
Considering the Normal
The process described in the preceding section can be thought of as “normal intelligence.” As conceived by Thomas Kuhn, “normal” refers to “the relatively routine work…within a paradigm, slowly accumulating detail in accord with established broad theory, not actually challenging or attempting to test the underlying assumptions of that theory.”195 We can thus see that “normal intelligence” is an activity of expanding knowledge in which most intelligence professionals engage and which incrementally increases knowledge about targeted phenomena.
The perceived and recalled successes of the past contribute to the repeat use of unvalidated tradecraft. The paradigm presumes state-level adversaries—eventually with mutually destructive capabilities.
As used in this context, “normal intelligence” is to “intelligence” as Thomas Kuhn’s “normal science” is to “science.” In both domains newly created knowledge incrementally adds to an increasingly established paradigm; new knowledge does not easily redefine the paradigm.
Normal science or normal intelligence does not seek to revise significantly the paradigm by which new phenomena are known and understood. This may be seen in the way new intelligence personnel adopt existing job accounts. A common practice involves their reviewing previous reporting on the account, with a tendency for new reporting to stay within the conceptual boundaries of what has gone before. Knowledge increases only incrementally.
Normal paradigms prevail until previously unnoticed and unnoticeable discrepancies create sufficient inconsistencies in explaining and understanding phenomena so as to cause errors that cannot be ignored.
In the cultural environment of human interaction, the new perceptions of reality can be enough to force a reconsideration of the old. In social scientific terms, a new paradigm not only explains the new, it does better at explaining the old.
The existence of particular intelligence errors does not necessarily indicate a paradigm has changed. However, repeated intelligence errors do. As is the case with science, small errors in adequately characterizing phenomena lead to the emergence of “corrective constants.”
The state-as-adversary paradigm for intelligence creation is obsolete. Two decades now separate the interpretable intelligence context from that of the Cold War: the adversaries and issues are now strikingly different.202 The power of the Soviet Union waned dramatically after 1990 as that of China increased. But even more central to the intelligence context, novel phenom- ena also appeared that were non-state based: emerging non-state actors posed new challenges by threatening traditional state structures.
The anomalies these new phenomena have created illustrate how and why normal intelligence is no longer adequate: it could no longer characterize these phenomena within the threat and opportunity framework of strategic intelligence. The “normal” means by which error is explained remain inadequate. As documented in the various Congressional and independent commission reports, intelligence no longer adequately describes, explains, or predicts with respect to the phenomena its consumers need to understand. Thus, intelligence change is necessary—revolutionary change.
Paradigm Shift
Revolutions in science, politics, and military affairs occur because crises reveal the insufficiency of the reigning paradigm.
periodic revolutions change how phenomena are perceived and understood.204 Crises are a precursor of such paradigm shifts.
here are repeated attempts to impose methods of “[social] scientific study…to analysis of complex ongoing situations and estimates of likely future events.”206 What is lacking is any sort of a systematic approach across the Intelligence Community. As long-time practitioner and observer Jack Davis noted a decade ago, no corporate standards for how intelligence is created, including the methods employed, exist.207 Although sound practice does not ensure that intelligence assessments will be correct, its absence, by definition, contributes to flawed conclusions.
In short, U.S. intelligence professionals operate in an environment similar to an unfolding Kuhnian revolution: the epistemology of normal intelligence is insufficient and new knowledge is needed. The recent failures highlight the necessity for change, as does the graying of the intelligence sensemaking workforce—new people faced with new and emerging issues should be comfortable with finding new ways to systematize their work.
Not all “old school” intelligence practices are without continuing value. Several significant state-level adversaries remain as threats to the security of the American nation although they too are challenged by the new non-state actors and issues that populate the paradigm of the new intelligence—something that compounds any estimate of how they are likely to engage the United States. Further, in many circumstances and in dealing with certain issues, the tacit expertise of highly experienced intelligence professionals is appropriately tapped for “recognition-primed” sensemaking.209 These “old hands” possess both current knowledge and a highly evolved skill set. Years of innovative and critical thinking mean they are skilled in looking at issues from a variety of perspectives and have the wisdom of deep context.
The challenges involve knowing when such expertise is valuable and needed in the first place, and encouraging the intelligence enterprise to develop and retain the cognitive and organizational flexibility that such thinking requires.
Indeed, a part of successful and revolutionized intelligence work involves gleaning new meanings from old patterns that have remained hidden to those who have stopped short of sensemaking. One challenge is that the “fresh” eyes lack the knowledge of potentially relevant patterns while the “old” eyes cannot see things as new. Each lacks the other’s strength. Experience acquired by newer professionals who engage in the practice of traditional “analysis” jaundices their once-fresh viewpoints even as they start to acquire the relevant and necessary experience.
One solution may be to adopt a model of core competencies broken out according to task analyses of existing intelligence missions and functions. Such a model identifies what is needed and has been at least partially imple- mented in the IC’s Analytic Resource Catalog developed during the tenure of former Director of Central Intelligence George Tenet.
Mark M. Lowenthal, “Foreword,” in Moore, Critical Thinking, xi.
Rewarding the successful use of some of the most important competencies may also encourage their retention in the catalog. Among these are curiosity, perseverance, and pattern recognition.
Simply put, intelligence practitioners create knowledge to support their customers. As used here, intelligence practitioners are presumed to be contributors to government plans and policies at a variety of levels where they have the opportunity to share broad strategic perspectives with national leaders as well as ensure that deployed warfighters have at hand the fruits of technical collection and marshaling of tactical data.
Finally, it should be noted that intelligence Knowledge is only one component of a strategic, operational, and tactical intelligence triumvirate.213 Activity and Organization are the other two. It is the author’s belief that Activity and Organization also are in need of new paradigms. However, such a discussion hinges on what intelligence Knowledge is and how it is created—in short, the sensemaking involved. Insofar as Activity describes how the precursors of intelligence are hunted, gathered, made sense of, and transformed into knowledge, it is considered here. However, the uses of intelligence (also an activity) and how intelligence professionals are grouped, led, and managed to act and create knowledge—the realm of Organization—lie beyond the scope of this book.
CHAPTER 4:
The Shape of Intelligence Sensemaking
Intelligence Sensemaking involves a number of overlapping high- level activities. First, intelligence professionals engage in planning or design and then hunt for and gather the materials they require in order to under- stand issues, answer questions, or explore new ideas. They can be externally motivated by the needs of a customer or they can be self-motivated as a result of an observation, or both. Second, these professionals disaggregate and then reassemble relevant information, trying to determine what it means.
At every stage in their work they assess critically their processes and results, seeking to validate both how they are engaged and the outcomes of their engagements. These overlapping activities can be characterized as Planning, Foraging, Marshaling, Understanding, and Communication. They are supported by Questioning and Assessing.
Planning for Tame and Wicked Intelligence Problems
Making sense of either tame or wicked problems is predicated upon planning. Plans, according to Gary Klein, are “prescriptions or roadmaps for procedures that can be followed to reach some goal, with perhaps some modification based on monitoring outcomes.”214 Creating plans requires “choosing and organizing courses of action on the basis of assumptions about what will happen in the future.”215 Known as planning, this process characterizes the “contingencies and interdependencies such as actions that must occur first as a precondition for later actions.”216 When we add the concept that practitioners—through critical thinking—also engage in reflective thinking and learning, both singularly and collaboratively, we may similarly label this process “the art of intelligence design.”
With tame problems, where answers and solutions can be anticipated, algorithms can calculate actionable probabilities and repeatedly make sense of the problem.217 The design of useful algorithms may be complex and they operate well only in finite and specific environments. How, on the other hand does one plan or design for wicked problems? One answer is to re-imagine the wicked problem as a tame one. However, the repeated occurrence of “unintended consequences” in past scenarios suggests that this is not a good option. Disaggregating what are assumed to be tame problems into their component parts, regardless of the actual problem type, often proves inadequate, as unintended and unforeseen consequences make clear.
Yet planning must occur regardless of problem type.
Klein considers—along with Rittel and Webber—that planning is an emer- gent process: Goals are clarified and revised as understanding of the problem grows.218 He notes,
Goals can be dynamic and can change completely as a function of changing circumstances. Goals can conflict with other goals in ways we can’t anticipate or resolve in advance. Goals can carry implications we can’t perceive or anticipate until events transpire.219
In terms of intelligence creation, this means that larger, strategic goals can—and perhaps must—emerge as sense is made of the problems under scrutiny. Thus, a tasking from an intelligence consumer changes as mindful sense is made of the tasking itself, of the resources that are available for understanding it, and the mix of actors involved.
Klein refers to this reflective problem planning as “flexible execution” or “Flexecution.”220 Within the framework of intelligence sensemaking, it provides a self-reflective process—at the individual and organizational levels—that monitors the goals and whether what is understood or being done is consistent with those goals, modifying those goals as understanding emerges.
Foraging
Hunting and Gathering
If, as Baumard asserts, “Intelligence, a continuous human activity, gives sense to the stimuli received from the environment [then] these stimuli [must] be passively or actively sought.”223 This requires hunting and gathering. They comprise foraging, which in turn refers to “a wide search over an area in order to obtain something.”
An apt analogy for the foraging activities of intelligence professionals can be drawn from anthropology, where the activities of the hunter-gatherer have been immortalized. In nonagricultural societies people both hunt for specific game and take advantage of what the local environment provides.225 Similarly, intelligence professionals may seek specific information, often tasking collection systems as part of the search. They also take advantage of existing repositories of information. Neither approach is wholly satisfying nor provides for all of the sensemaker’s needs all of the time. However, without the basic act of foraging there can be no sensemaking as there is nothing from which to make sense.
Information foraging is a rich subject about which Peter Pirolli has done extensive work, some of it sponsored by the Intelligence Community’s Novel Intelligence from Massive Data research project funded by IARPA’s predecessor, ARDA (Advanced Research and Development Activity).
Underlying information foraging theory is the idea that “humans actively seek, gather, share, and consume information to a degree unapproached by other organisms” and therefore, “when feasible, natural information systems evolve toward stable states that maximize gains of valuable information per unit cost.”
Efficiency derives from optimizing the time necessary to achieve a goal, the quality of the achievement, and the satisfaction obtained in doing so.227In application to the human information foraging scene, the theory becomes “a rational analysis of the task and information environment that draws on optimal foraging theory from biology and…a production system model of the cognitive structure of [the] task.”
Peter Pirolli and Stuart Card, “Information Foraging,” Psychological Review, vol. 106, no. 4
Toward a Practice of Intelligence Foraging
Developing an optimal foraging model for information acquisition requires the subject to consider whether to remain at a source that provides a superabundance of information of questionable value, or to seek another, more valuable, source.229 Pirolli and colleague Stuart Card observe that for human foragers, this involves “a tradeoff among three kinds of processes”: exploring, enriching, and exploiting.230
These three foraging steps will not seem foreign to traditional intelligence practitioners. “Exploring” is a breadth activity whereby a sensemaker broadly examines a wide variety of information that may or may not be relevant to the issue. The premise is that when one considers a broad variety and volume of data, there is less opportunity to miss “something novel in the data.”231 Speaking in traditional intelligence terms, exploring is like reconnaissance. By contrast, “enriching” is a depth activity. Here the sensemaker identifies areas of interest and focuses attention on those areas. As Pirolli and Card note, this is “a process in which smaller, higher-precision sets of documents are created.”232 Reconnaissance has become more narrowly focused, but highly targeted “surveillance” is not yet in play. Finally, the practitioner “exploits” the results of foraging by thoroughly examining what is found and extracting information as needed. This activity extrapolates from tacit sensemaker behaviors and information-based patterns to create hypotheses about what the information means. At this point, foraging evolves to sensemaking.
The appropriate amount of exploration depends on the context. However, there appears to be a limit after which more information does not increase accuracy although it does increase the sensemaker’s overall confidence.
The discussion of how much exploration is needed is germane because in the Pirolli-Card framework the sensemaker may believe she controls the amount of foraging. In a sense this is true. The sensemaker will stop foraging once she believes she has what she needs. But how much information is sufficient? Slovic’s results and Heuer’s subsequent discussion suggest that practical sufficiency is achieved at lower levels of exploration than expected.235
Further, contrary to the beliefs of the sensemaker, it is often information itself that controls the processes. Intelligence professionals are over- whelmed with more and more information that arrives faster and faster and may be valuable for shorter and shorter periods of time. This information flood challenges the sensemaker to efficiently find the information needed in order to make sense of the phenomena or issue under scrutiny in a timely manner. Does one need to peruse all the information received? Would a different source be more productive by providing more focused information? These are examples of the difficult questions the sensemaker must consider. Compounding her deliberations further is the fact that she must answer this question in foresight; hindsight is too late.
Foraging practice begins with an understanding of what it is the sensemaker seeks to know, the foraging resources available, and the urgency of the issue. But how can an intelligence sensemaker know what to look for? To what degree does she need to explore, enrich, or exploit the information? Further, how does she know if she is getting what she needs?
A first step is to think critically about the issue itself and the resources needed. Using a metacognitive, process-focused critical-thinking model such as that adapted from Richard Paul, Linda Elder, and Gerald Nosich, the practitioner can dissect the issue and her thinking on the issue.
She makes assumptions explicit, explores relevant points of view, starts to con- sider the ramifications of the issue, and she asks important questions about what resources will best inform her about the issue; she considers the con- text in which she is working, and ponders the alternatives to her reasoning about where and how to forage. This critical thinking defines her foraging activities. She may engage in all three strategies at once or at different times as she engages in the analyses that produce her syntheses and the interpretations necessary to generate knowledge. A well-developed understanding of exploited information may direct her back to do additional exploring or enriching (or both).
In intelligence foraging as it traditionally has been practiced, there is a tendency to linger at a fruitful source rather than to explore elsewhere for the required information.
The danger, of course, is that the practitioner may limit the information she acquires and the relevant perspectives it informs. If, for example, the practitioner has a belief that two parties in whom she has an interest communicate via one means and she can acquire technical collection that captures the communications via those means, she may ignore the fact that they also use other methods to communicate. She may subsequently fail to task systems that collect those other communications in the belief that what she is getting suffices. Should the parties suspect that their communications are being targeted, they may engage in deceptive practices over that collected means and use the other non-collected methods for their real exchanges.
Foraging Challenges
Ongoing research at University College London offers a view of how younger sensemakers likely search for information. The researchers report that people foraging for information spend four to eight minutes viewing each resource.237 Thus, a great many resources may be consulted but none of them very deeply. Within the context of intelligence, such foraging strategies may facilitate broad searches but leave open the question of whether deeper searches are also accomplished.
An additional challenge with information foraging is that if the practitioner misses the opportunity to acquire something, it may never again be obtainable. Like the elements of a fleeting interpersonal conversation, the original foraging behavior, if left un-captured, can never be recaptured. Indeed, there may be no indications that such a conversation even occurred.
A further consideration for the practitioner is the “self-marketing” of the information. Vivid stories market themselves much better than do flat ones. Exploited information that supports a favored hypothesis may be preferred over information that does not; an unfortunate reality is that little motivation remains for further exploration. The practitioner is human—she will not likely have a truly agnostic attitude about what she seeks and why.
Compounding this is the idea that sources and means for foraging are self-protective. For example, there is a presumption that sources will continue to communicate via specific means. The methods that capture those communications and the people that support them tend to seek justification. Assets may be kept active after their usefulness expires. The practitioner returns to the same sources over and over because they have been useful in the past and such attention helps keep those sources actively collecting.
Critically assessing what she is doing is one way the sensemaking practitioner may be able to overcome these limiting tendencies. By constantly asking herself how she is thinking about the issue, what she seeks, other perspectives, her assumptions, as well as relevant concepts such as self-deception or adversarial deception, the practitioner may diminish the impact that her preferences play on her foraging decisions.
Another part of this critical assessment is the consideration of the costs of foraging. As Herbert Simon notes,
What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.
The superabundance of information available on the Internet (and elsewhere) creates a “poverty of attention” to any one source. Rather, people skim across a great many sources.
A predator may go out seeking one type of prey and find none of it but there may be an abundance of some other kind of game. Within the context of intelligence such opportunism may or may not be appropriate (or even legal) when a technical system or an asset is tasked to provide information for intelligence. Lacking the desired information, a human source might opportunistically substitute what might be perceived as desired or desirable, even if it is not closely related to the issue at hand—or for that matter, even “true.” It is used because it satisfices for the immediate term.
Harvesting
A special case of foraging involves “harvested” information. Technical agencies that field systems to gather information also can be characterized by a different model, that of harvesting. The systems employed simply harvest that which lies within their purview, then process it and store it in silos— data repositories from which sensemakers subsequently must forage.
Such systems are efficient at creating broad collections; they tend to be inefficient and unreliable when very narrowly focused. Thus, directed rather than broad collection of specific phenomena is needed.
Technical collection systems tend to provide—even in the negative—what sensemakers want to find. This can create a potentially dangerous confirmation of an idea that may be invalid.
Automated retrievals from information repositories typically pro- vide sensemakers with what they believe to be the needed and relevant information—in short, their evidence. The evidence pertaining to specific issues arrives at the sensemaker’s desk and reports are issued. At first, the evidence is carefully scrutinized and the system that provides it assessed. As the process repeats, however, as it certainly did in the Cold War era, complacency may set in. Critical assessment of quality and quantity may cease.
Finally, no amount of foraging can discover valuable information if it has not been collected by some system—human or technical—in the first place.
Marshaling
What can be done to revolutionize the way information foraging is accomplished so as to overcome or at least mitigate these problems? Some answers lie in an understanding of marshaling. Part of the sensemaker’s practice is to turn foraged and gleaned information into evidence.
This is a broad activity, for if the issue has multiple explanations or future possibilities, then evidence will be information relevant to any, many, or even all of those explanations or possible outcomes. In order to make that determination, the sensemaker will need to have identified what those alternatives are and to have collected information (both dis-confirming and confirming) about them.243 This may require foraging from additional resources with all the attendant challenges discussed above.
Understanding
If we presume that foraging has yielded relevant and valuable information—evidence—on the issue under study, the next step is to determine what it means. This is the heart of sensemaking: evidence is dissected, reassembled
is the sensemaker focusing on individual actors, the actions of a collection of actors, the beliefs that guide the activity, or the processes that determine the actions of the collective?
The disaggregation of each of these perspectives and their associated stories provides a rich brew for sensemaking.
Synthesizing
Synthesizing is “the combination of ideas to form a theory or system.”245 Even as the intelligence professional analyzes the individual pieces of information, they are synthesized into a mental picture of the larger issue. Pieces of information are implicitly combined even when the sense- maker works within the yield of a particular foraging discipline or within a frame or reference. Such synthesis drives further foraging and analysis.
Synthesis needs to be explicit. In the example developed above, the intelligence professional is required to synthesize the differing trajectories
of the three principal actors, considering how their beliefs harden or soften their positions and how they are vulnerable to the actions, influences and processes of the groups. Doing so in a systematic fashion leads the intelligence professional to new insights about the situation: what is going on… and (from the U.S. perspective) what to do about it.
Interpreting
Issues can be dissected and reconstructed in a variety of ways, creating different meanings. Sense must be made of these different meanings. Interpreting, or “the action of explaining the meaning of something,” is another component of sensemaking.246 We may say that whereas analysis and synthesis establish the what, interpretation establishes the so what.
A revolutionary approach to sensemaking now being undertaken by analysts from DIA, State, and CIA, is to engage in “adversarial briefing” of principals, where briefers adopt opposing perspectives for a thorough airing of the issue, complete with the participation of the principals themselves.
Communicating
New models of knowledge transfer recognize change in both message and medium. Social networking, peer-reviewed shared multimedia, and interactively blogged communications are examples of these new mediums. The message is short and subject to change by different contributors. Authority is based on consensus. The distinction—if it exists at all—between formal and informal communication is blurred. There are dangers here as authority and truth are no longer necessarily linked. One risk is that the “wisdom of a crowd” can in fact be the “madness of a mob”—a phenomenon occur- ring in both the public arena and within the IC’s blogosphere. In both arenas the loudest voices strive to bludgeon into silence those who would disagree, all the while advancing their egocentric or sociocentric positions. Scientific knowledge and empirical facts matter little in such cases.
In summarizing the “introspective works responding to…intelligence failures,” Charles Weiss agrees that intelligence practitioners’ failures include a lack of proper attention to hypotheses and data collection efforts that are contrary to what they regard as the most likely interpreta- tion of available information.”248 One danger is that the very judgment about which the sensemaker is least confident might be the one that turns out to be correct. The fallacy of depending on the communication of confidence levels relates to the fact that each assessment or report only fills in some unknown portion of the gaps in the sensemaker’s and policymaker’s knowledge.
a carefully considered, standardized metric of uncertainty could pro- vide one means of assessing and communicating confidence independently from the sensemaker. Weiss suggests that either Kent’s scale250 or its more recent instantiation by the Office of the Director of National Intelligence offers an appropriate means by which the uncertainty could be systematically captured.251 The challenges inherent in such metrics are twofold. First, the evidentiary statistics necessary for their use are “typically unavailable to intelligence analysts—or it they are available, must be based on small samples of past events.”252 Additionally, scoring the conclusions from such small samples across production lines and even from day to day by a single intelligence professional can be observed to be inconsistent. Steve Rieber discusses calibrating sensemakers as a solution.253 To date no such strategy has been implemented.
CHAPTER 5
A Practice of Understanding
Judgment in intelligence sensemaking, as in a number of other domains, likely improves as one progresses to higher levels of proficiency and expertise. However, prediction is both difficult and inherently unreliable because events of different kinds vary considerably in their inherent predictability.
Intuition
It is difficult to wrap appropriate words around the concepts that are at hand, thus care should be taken to make certain distinctions. Sometimes, judgments can be rapid, non-conscious, non-deliberative, and almost seem as if they are immediate perceptions and feelings rather than judgments.
Intuitive, or, as it is sometimes called, automatic thinking forms the basis for much of our personal sensemaking.260 It allows us to process complex inputs that far exceed the “span of immediate apprehension” of approximately seven “chunks” or elements that individuals can consciously process in working (or short-term) memory.261 Such thinking, for example, explains how people (mostly) drive successfully. Recent research revealing a correlation between cell phone use (especially texting) and accidents leads one to extrapolate that attention-requiring activities such as engaging in professional-level office conversation or dialing and conversing while mobile respectively impede otherwise effective automatic activities such as ten-finger typing or absent-mindedly but safely driving an automobile.
There is another side of intuitive reasoning that sometimes works in opposition to one’s survival. Intuitively reasoned responses to stress are often highly focused and narrow. Laurence Gonzales notes that in such reactions “the amygdala…in concert with numerous other structures in the brain and body, help to trigger a staggeringly complex sequence of events, all aimed at producing a behavior to promote survival.”266 But, in many cases, behavior is also locked down.
Intuitive or automatic thinking is a survival mechanism.268 Gonzales notes that such mechanisms “work across a large number of trials to keep the species alive. The individual may live or die.”269 But over time—and in reference to humans—generally more live than die, leading to evolution and the genetic transmission of the “reflex.” If a particular set of behaviors confers a survival value, that set can become more widespread in the population. Seen in this light, unease at entering an elevator at night could be a modern instance of sensing shadows in the tall grass.
On a shorter time horizon, people use experience-based intuitive patterns or mental models. These patterns or models direct how situations are perceived and how they are responded to.
We turn next to an examination of types of judgment, distinguishing between those that are skill-based and those that rely on “heuristics,” or learning through personal discovery whether rules of thumb are valid shortcuts to understanding an issue. We then link this dissection of judgment to the work of intelligence professionals.
Types of Judgment
First we must clarify the meaning of “judgment.” A judgment can be an observer’s belief, evaluation or conclusion about anything—one can form a judgment about anything of interest, including one’s own reasoning.275 Judgment also describes a process, surely more than one kind of mental process, by which one reaches a decision.
Judgment can be expressed as affective evaluation (example: That is a good thing), objective evaluation (It looks like a cat, but it is just a stuffed cat), or categorical assignment (My judgment is that this is a case of highway robbery). Judgment as process can also be described as apodictic, modal, or oral, among others.276
In terms of intelligence sensemaking, successful intuitive judgment arises from the tacit knowledge of experts who assess “normal” (in Kuhnian terms) situations, or as has been discussed above, the tame, familiar or regularly occurring kinds of problems (although they may be quite complex).
However, it should be noted that the combination of skill-based and heuristic-based intuition confers a benefit to mindful experts: a sense of when a case seems typical at first glance, yet there is something not quite right about it. While the less experienced person may be lulled into believing the case fits a certain type, expert decision makers react differently. They note some worrisome clue that raises questions. “Maybe this is not a typical case,” they venture. Eventually they may come to see that the case is in fact atypical. So informed, they make a different judgment, sometimes in disagreement with other experts. As we will now see, intuition certainly becomes a part of the intelligence process when practitioners make, or fail to make, useful and accurate predictions.
The Question of Predictability
Some in the IC argue that the pertinent aspects of all intelligence problems can be adduced, “if we only knew more” or “had the right algorithm or method.” However, we mislead ourselves if we believe that any messy problem can be resolved with a probability-juggling program. The authors are reminded of the observation, “There are the hard sciences and then there are the difficult sciences.” It is both impossible and inappropriate to attempt to remake social and cognitive sciences into emulations of calculational physical sciences. If the reduction were possible, someone would have achieved it, or would have at least made demonstrable progress toward its realization, in the 200-plus years during which psychology and the other “social sciences” have called themselves “sciences.” If reduction was appropriate, and we could get “the right information to the right person at the right time,” we would not need that right person—“truth” would be self-evident.
Some examples of mindful, heuristic-based decision making, especially pertinent because they involve the thinking habits of senior U.S. civilian and military officials as well as of their strategic advisors, are discussed in Neustadt and May’s Thinking in Time.292 The authors point out that a sensemaker’s awareness of historical decision making in even loosely analogous situations helps to keep at bay the further unsettling idea that the present circumstances constitute a “crisis.”
Anchoring,” which is the biasing of a judgment because of the framing of the initial question, and “attribute substitution,” arising from replacing a difficult question with an easier one, are two contributors to such flawed intuitive judgments.
Thinking About Anticipating
Jurisprudence, clinical psychology, and economic forecasting are all examples of domains where accurate prediction is difficult or impossible, and it is not terribly clear what it means for a person to be an expert in any of those fields. In the realm of jurisprudence, studies of the low rate of successful intuitive predictions about future recidivism among paroled offenders serves as one of many pieces of evidence showing that even highly experienced professionals in certain domains may be no better than laypersons at making intuitive judgments.
Anticipating Deception: Applying the Space-Time Envelope
A recurring worry within alert intelligence services is whether they are being deceived by their adversaries. From Troy, in the second millennium BCE, through to mis-direction schemes in World War II, and on to the lead-up to the Iraq invasion of 2003, adversarial deception has played a strong or decisive role in final outcomes.307
Implications of Visualizing Anticipation
Diagramming using Concept Maps (and related kinds of diagrams called causal maps and cognitive maps) has been used as a de-biasing technique for analysis under uncertainty. This use is well known in the field of business and strategic management:
in the pages of The Journal of Strategic Management, Gerard Hodgkinson and his colleagues added:
In addition to providing a useful means for gaining insights into the nature and significance of cognitive processes underpinning strategic decision making, this dynamic emphasis on antecedents, behaviors and consequences, renders causal cognitive mapping techniques particularly attractive as a potential means for overcoming the effects of framing (and possibly other cognitive biases) in situations involving relatively complex decision scenarios.313
Hodgkinson et alia investigated “the extent to which judgmental biases arising from the framing of risky decision problems [could] indeed be eliminated through the use of this particular cognitive mapping technique” and found cognitive mapping to be “an effective means of limiting the damage accruing from this bias.”314
The Roles of Intuitive Thinking in Intelligence Sensemaking
Given these considerations, what are (or should be) the roles of skills- based intuitive and heuristic-based intuitive thinking in intelligence sensemaking? Many, if not most, intelligence professionals have had a “feeling” about an issue and what is going to happen. Sometimes those intuitions are correct, particularly if the requirement entails real-time observation and situational awareness. When it comes to anticipatory sensemaking, however, the authors suspect that intelligence professionals may fare no better than does the average citizen in predictive situations.315
There are a number of reasons for this, not the least of which has to do with the availability of evidence, or relevant information. A somewhat persistent myth about intelligence is that its professionals have access to all the information they need and that they can get any and all other necessary information. This not only simply is not true but is likely highly undesirable. While it is true that intelligence professionals must make their assessments based on incomplete, often faulty, and sometimes deceptive information, at least they can do so. Forcing them to try to make sense of all the relevant information relating to an issue would likely burden them sufficiently so as to preclude anything but the most general findings being issued—if anything can be issued at all. Finally, as was discussed above in relation to figure 1 (see Chapter 3), complexity, ambiguity, and uncertainty increase as one moves from Descriptive to Estimative (or Anticipatory) Intelligence.316
intelligence professionals compete against other foreign intelligence organizations whose professionals may be as skilled at obfuscating what their factions, groups, or nations are doing as we are at making sense of what they are doing. Sometimes that other side is, in fact, better. And, like a closely matched sports event, the difference between valid and true sense- making versus invalid or untrue sensemaking—or even no sensemaking at all—might be the result of luck.
Since such Type 2 domains are ones in which the primary task goals involve the understanding and prediction of the activities of individuals or groups, accuracy and precision are elusive. Consequently, as has been noted, Type 2 domains are also characterized by tasks involving a lack of timely feedback and a paucity of robust decision aids. Further, if the object of study does not know what she or they will do, how can someone else predict it reliably? Therefore, is it any surprise that in such domains, intuition is limited in its usefulness? What are intelligence professionals to do?
It should be reiterated that although over-estimative errors in intelligence sensemaking, as has been noted, are unacceptable, under-estimative errors are even less tolerated. It is better to have warned and been wrong than not to have warned and been wrong. False alarms are better than misses. A warning about an attempt by another individual to set off a bomb on a subway system—which subsequently does not occur—generates far less uproar (if any at all) than does a failure to warn of an individual who in fact plans to blow up an airliner, and through anticipatory sensemaking, being able to catch him preemptively.
It is by no means obvious that simply throwing more information at a problem will make solving it any easier.
Future Vision: Red Brains, Blue Brains?
Darren Schreiber et alia used functional MRIs (magnetic resonance imaging tests) to assess how people associated with the U.S. Republican and Democratic political parties deal with risk. The researchers discovered that members of the two groups used distinctly different portions of their brains when making “winning risky versus winning safe decisions.”324 The authors note that the different portions of the brain play different roles in human cognition and conclude that
it appears in our experiment that Republican participants, when making a risky choice, are predominantly externally oriented, reacting to the fear-related processes with a tangible potential external consequence. In comparison, risky decisions made by Democratic participants appear to be associated with monitoring how the selection of a risky response might feel internally.325
While neurocognitive paradigms for intelligence sensemaking have not yet formally been identified or established, implications of this work—to the degree that intelligence professionals can speak to the concerns of decisionmakers who are, after all, particular political partisans—may be significant. The research to date shows that the cognitive mechanisms and especially the emotion-based attitudes of partisan sensemakers shape their reasoning as they assess uncertain and risky phenomena.
Looking Ahead
Intuitive reasoning is something that we do naturally, all the time. It cannot be prevented, is not easily neutralized, and it is sometimes useful and necessary in sensemaking. While it can be reliable when employed as the sole basis for actionably valid, predictive intelligence creation in Type 1 domains, it is highly fallible when used for intelligence creation in Type 2 domains.
What can be done to challenge and validate the surety one has about an intuitive intelligence judgment? Employing approaches to reasoning such as those found in critical thinking seminars and courses, especially as currently offered across the IC’s educational institutions, and developing skills that aid mindfulness (as discussed in the Preface), offer possible means of accomplishing calibrated reasoning.
CHAPTER 6 Considering Validation
How does one know if the knowledge that intelligence sensemakers create is itself valid? Does accuracy alone ensure validity? What was accurate when findings were communicated may not be accurate subsequently. This flux suggests a strong procedural basis for validation. For example, were steps followed to avoid perceptual errors and cognitive traps? Was the process documented? Were alternatives adequately explored? Given the inherent uncertainty in intelligence judgments, it remains possible that all the appropriate processes may be sufficiently applied and yet the judgment is wrong
Analogies from Other Fields
Medicine
Medical practice is at times presented as having notable similarities to intelligence practice. For example, with respect to validation, an ultimate metric for failure in medicine is that the patient dies. But is medicine successful if the patient lives? At what quality of life and for how long are two additional questions. Perhaps death with a minimum of suffering is the most favorable medical outcome—is this a success? Depending on the specifics of the case, maybe it is.
Jurisprudence
Jurisprudence is an adversarial system in which the ultimate confrontation is a trial wherein two advocates make inferences about evidence to argue opposite sides of a case before an impartial third entity or body (often of non-experts).
So, how does one measure success? There are at least five points of view involved in jurisprudence: That of each of the advocates, the judging entity (a jury or judge) the accused person, and the community or government. Each party, depending on the verdict, has a different metric for success. In certain types of cases such as those involving child molestation or alleged terrorism, the accused person tends to be deemed guilty by the community, prosecuting advocate, and government even if exonerated. In all cases where an opinion—particularly in the media—runs counter to the majority’s views, the conclusion may be made that the court failed to render the “right” verdict
Science
Science involves a number of metrics that include a sound method of documenting both process and results, as well as replication. Work is considered preliminary and non-definitive if it has not been replicated.
despite the intellectual appeal of theories about genetic links to specific behaviors, “there are few replicated studies to give them heft.”331 In other words, the underlying theories may not be sound.
Science depends on refutation of alternative hypotheses, and replication studies attempt to refute that which has been shown. It is quite acceptable to be wrong so long as one admits it when the fact becomes apparent. Indeed, one model for science is that of competitive cooperation. Scientists attempt to tear down the new work of colleagues—without resorting to personal attacks. This dialectic approach may last generations or longer. In the process new knowledge is discovered and—if it cannot be refuted—validated.
Replication in Intelligence
The inability to replicate much of the process of sensemaking in intelligence limits the application of this indispensable practice of science. The pressures of real-time production inhibit the re-visitation of past judgments, although with at least one recent National Intelligence Estimate, the repetition of “alternative analysis” led to the questioning and revision of the original conclusions. Essentially, any meaningful replication of intelligence phenomena can only be accurately made in foresight, as in a National Intelligence Estimate.
when intelligence students faced with a scenario involving three fictitious nations at odds with each other develop a common set of hypotheses regarding who will initiate a war and with whom, and are then given a finite set of evidence and a common method such as the Analysis of Competing Hypotheses, they come to similar conclusions as to which hypotheses are the least likely and therefore which eventualities can be expected.334 Unfortunately there does not yet exist a similar body of results for real intelligence problems interpreted through the lenses of different intelligence disciplines and sources.
Yet, replication remains an important metric of the intelligence sensemaking process. As Caroline Park notes, “[the] basic reason research must be replicated is because the findings of a lone researcher might not be correct.”
Lynn Hasher, David Goldstein, and Thomas Toppino concluded that confidence in assertions increases through repetition of the assertions in situations when it is impossible to independently determine their truth or falsity.338
Validation in Foresight and Hindsight
People—and intelligence practitioners and their customers are merely people—evaluate judgments they have made in hindsight. They believe, according to Mark and Stephanie Pezzo, “that one could have more accurately predicted past events than is actually the case.”340 Thus, hindsight occurs at least in part because as people make sense of “surprising or negative” events, “the reasons in favor of the outcome [are] strengthened, and reasons for alternative outcomes [are] weakened.” Further, in hindsight all the relevant facts may be known whereas in foresight this is not the case. But evaluating “mistakes” in hindsight obscures an important point made clear by Taleb: Mistakes can only be determined as such by what was known at the time they were made and then only by the person making the mistake.341 In other words, mistakes need to be evaluated from the points of view held in fore- sight. And seen from that perspective they may not be mistakes at all.
Applied to intelligence sensemaking, this means that many so-called intelligence errors and failures may, in fact, be well-reasoned and reasonable judgments based on what is known prior to the decision. Certainly, when viewed in hindsight they were wrong. But in foresight they were accurate and valid to the best of the sensemaker’s abilities. How can this enduring problem be mitigated?
Validating the Practice of Intelligence Sensemaking
What else contributes to bringing about validated sensemaking? If a method does not do what it is commonly purported to do, is it invalid? This is one question that has been raised with regard to the Analysis of Competing Hypotheses, or as it is commonly known, ACH. Richards Heuer, Jr. initially developed the method for the detection and mitigation of attempts at adversarial denial and deception.343 ACH forces consideration of alternative explanations for, or predictions about, phenomena.344 It forces consideration of the entire set of evidence, not “cherry-picked” trifles that support a favored hypothesis.
a study by MITRE failed to show that it does eliminate the confirmation bias.345 Both the MITRE study and an earlier one by NDIC student Robert Folker do suggest that ACH is of value when used by novice intelligence professionals. However, Folker tentatively concluded that experts seem not to be aided by the method.346 Is it still a valid method for intelligence sensemaking?
Perhaps it is. The method provokes detailed consideration of the issue and the associated evidence through the generation of alternative explanations or predictions and the marshaling of the evidence. It asks the sensemaker to establish the diagnosticity of each piece of evidence.
ACH further makes explicit the fact that evidence may be consistent with more than one hypothesis. Since the most likely hypothesis is deemed to be the one with the least evidence against it, honest consideration may reveal that an alternative explanation is as likely or even more likely than that which is favored. The synthesis of the evidence and the subsequent interpretations in light of the multiple hypotheses is also more thorough than when no such formalized method is employed.
Indeed, Robert Folker’s “modest” experiment in applying qualitative structured methods—specifically ACH—to intelligence issues showed that “analysts who apply a structured method—hypothesis testing, in this case— to an intelligence problem, outperform those who rely on “analysis-as-art,” or the intuitive approach.”347 Simply put, Folker experimentally showed that method improves the quality of practitioners’ findings. Folker’s study offers evidence that “intelligence value may be added to information by investing some pointed time and effort in analysis, rather than expecting such value to arise as a by-product of ‘normal’ office activity.”348
One variation on ACH implementation provides a structured means of developing issues. As applied by the faculty and students of the Institute for Intelligence Studies at Mercyhurst College, practitioners begin with a high-level question and use sequential iterations of ACH to eliminate alternative explanations.349 The next phase takes the “non-losers” and develops them further. Another round is conducted and again the “non-losers” are selected and further developed. While this could generate a plethora of branching explanations, in reality it is the author’s experience that it tends to disambiguate the issue fairly efficiently. At worst the structuring inherent in the method leaves the sensemaker with an in-depth understanding of the issue; at best, a couple of eventualities and their likely indicators are determined. Assets can then be tasked, foraging conducted, and more exact determinations made as the issue develops.
Johnston observed that the IC has at its disposal “at least 160” [ana- lytic methods]…but it lacks “a standardized analytic doctrine. That is, there is no body of research across the Intelligence Community asserting that method X is the most effective method for solving case one and that method Y is the most effective method for solving case two.”353 As referenced here, such a doctrine arises out of knowledge that the specific methods are valid, in other words it has been demonstrated empirically that they actually do what they claim to do. Such a doctrine proffers a menu of sensemaking options dependent on the goals of the sensemaker. The current model, where validity is presumed by intelligence professionals because they are taught the method(s) in the community’s training schools, is insufficient because—at the most basic level such a metric is insufficient for determining validity. Further, there is little sense of what methods are appropriate in what situations. A claim of “we always do it that way,” is known to be insufficient but remains part of the “corporate analytic tradecraft.”
Heuer noted that “intelligence [error] and failures must be expected.”354 One implication of this assertion is that intelligence leadership cannot fall back on a “lack of skills” excuse when the next major intelligence failure occurs. However, without validating a canon of method and a taxonomy to characterize its use, intelligence professionals will remain hamstrung in their efforts to make fuller sense of threatening phenomena, increasing the likelihood of error and failure.
Seeking Validation: Toward Multiple Methods
Within the canon of social science method lies an approach to sense- making that may offer intelligence practitioners a means of disambiguating the wicked mysteries as well as the hard puzzles they face daily. Even in current practice, intelligence practitioners employ this approach when they do not rely on merely one method for sensemaking. Multi-method intelligence sensemaking explores complex issues from multiple perspectives. Each method used—such as ACH—provides an incomplete understanding of the issue, leaving the intelligence professional the task of making sense of the differing sensemaking conclusions.
intelligence professionals who engage in a “multiframe” sensemaking approach consider issues from multiple points of view created from the intersections of action- and process-focused vantage points and the perspectives of the individual and the collective. As developed by Monitor 360 for the National Security Agency’s Institute for Analysis, it facilitates sensemakers’ developing different answers to the intelligence question at hand.356 They must combine the differing results, in other words, synthesize and interpret partial answers, in order to better understand the issue underlying the question and to determine a best (at the time) understanding of the issue.
The lexicon of multi-methodology provides a term for this combinatorial activity: triangulation, or pinpointing “the values of a phenome- non more accurately by sighting in on it from [the] different methodological viewpoints employed.357 This is a process of measurement—which to be useful (accurate) “must give both consistent results and measure the phenomenon it purports to measure.”358 In other words, triangulation requires that the methods employed are repeatable and valid. Intelligence creation requires that those methods be applied with rigor.
CHAPTER 7
Making Sense of Non-State Actors: A Multimethod Case Study of a Wicked Problem
we find that a diversely practiced, multimethod approach that does incorporate a specific process, organizing principles, and an operational structure can fulfill the need for 21st century intelligence sensemaking. Such an approach reflects a Kendallian approach to intelligence sensemaking: It collaboratively paints a picture for a decision-maker rather than presenting a “scientific fact.”
Introducing the Wicked Problem of Non-State Actors
It is commonly thought that non-state actors are emerging as a dominant global force in the realm of national and international security, yet conclusive evidence confirming this belief is lacking. Part of the challenge is that non-state actors fit the profile of a wicked problem. While it is true they can be identified (the name accomplishes this—they are non-state versus state actors), there is no commonly accepted definition.361 In other words, non- state actors are defined by what they are not, leaving room for disagreement as to what they are. Further, some non-state actors can be characterized as “good” and others as “bad.” Differing points of view about whether a particular non-state actor is “good” or “bad” leads to varying characterizations of their activities.
National Intelligence Council, “Nonstate Actors: Impact on International Relations and Implications for the United States,” Conference Report, August 2007, URL: <http://www.dni. gov/nic/confreports_nonstate_actors.html>, accessed 10 May 2010, The Conference Report suggested that “Nonstate actors are non-sovereign entities that exercise significant economic, political, or social power and influence at a national, and in some cases international, level. There is no consensus on the members of this category, and some definitions include trade unions, community organizations, religious institutions, ethnic groupings, and universities in addition to the players outlined above”
Issues involving non-state actors lack clear definitions and are resistant to traditional intelligence approaches due to their open-ended nature; potential solutions to problems are neither clearly right or wrong; and difficult-to-discern and complex inter-linkages exist, although drivers for issues involving non-state actors can be identified
Three Approaches to Making Sense of Non-State Actors
The starting point for this case study was a 2007 National Intelligence Council (NIC) Desktop Memorandum that analyzed key findings from a series of seminars co-hosted with the Eurasia Group, a global political risk research and consulting firm.
The Memorandum observes that non-state actors are of interest “because they have international clout, but are often overlooked in geopolitical analysis.” The implicit but demanding questions of why and how much non-state actor “power” and “influence” have increased worldwide was not answered.
363 See the Eurasia Group web site, URL: <http://www.eurasiagroup.net/about-eurasia- group>, accessed 14 May 2010.
364 National Intelligence Council, “Nonstate Actors: Impact on International Relations and Implications for the United States,” Conference Report, August 2007, URL: <http://www.dni. gov/nic/confreports_nonstate_actors.html>, accessed 27 April 2010, 2. Cited hereafter as NIC, “Nonstate Actors.”
Key Findings of the Mercyhurst Study on Non-State Actors
Students in the Mercyhurst College Institute of Intelligence Studies (MCIIS) focused on the roles non-state actors play and their expected impact in Sub-Saharan Africa over the next five years (results), and on building a multi-methodological paradigm for considering the issue (process).369 Within this context, three additional questions were raised:
- What is the likely importance of [Non-State Actors] vs. State Actors, Supra-State Actors and other relevant categories of actors in Sub- Saharan Africa?
- What are the roles of these actors in key countries, such as Niger?
- Are there geographic, cultural, economic or other patterns of activity along which the roles of these actors are either very different or strikingly similar?370
The students developed a scoring system for both lawful and unlawful non-state actors, in terms of the socio-political environment, and applied this index to all 42 Sub-Saharan African countries (figure 7). The scoring characterized the roles of non-state actors vis-à-vis government and non-government interactions based on four drivers: An “ease of doing business” variable and a contrasting “corruption perception” variable; a democracy variable and a contrasting failed states variable.373 Stable and failing states were revealed to have differing interactions with non-state actors. In the former, non-state actors were lawful actors who tended to have government-sanctioned role potentials, whereas in the latter they were typically unlawful actors engaged in anti-government roles.
Mapping significant multinational corporations, NGOs, and terrorist organizations to specific countries as representative of non-state actor activity revealed correlations between role potential spectra and geospatial data, whereby each generally supported the other.375 Thus, geospatial sense- making tended to confirm the conclusions derived from the non-state actor role spectra.
Key Findings of the Least Squares Study on Non-State Actors
The Least Squares study of non-state actors began with the hypothesis that “non-state actors emerge in vacuums and voids.”376 Their study focused on the issue of violent and non-violent non-state actors but also explored a set of contingent methodological approaches. The inquiry sought to contribute novel understanding of non-state actors by
synthesizing available data and disparate taxonomies,…by generating and testing hypotheses concerning the key dynamics driving the transfer of power from states to [non-state actors] and favoring the emergence of novel [non-state actors] under globalization; and…by investigating the development of methodologies that might be most useful for future research.377
Two key findings revealed the critical role of environmental knowledge and of public expectations in motivating non-state actors, both as individuals and as members of the collective. Such findings were found to be significant to efforts aimed at mitigating the recruitment of specific Al Qaeda- associated individuals to assail the United States. Additionally, the team found that an approach based on critical thinking led to reasoning pathways that likely would not have been noticed or explored had a more intuitive and less rigorous approach been employed.378
Approaches and Methodologies Thinking Critically about the Issue
In order to impose structured thinking on a highly unstructured problem, the NIC advisor to, and the members of LSS first inventoried their own understanding of the non-state actor issue using the “eight elements of reasoning” developed and espoused by the Foundation for Critical Thinking and used throughout much of the IC.379 These elements include:
- Question at issue (What is the issue at hand?)
- Purpose of thinking (why examine the issue?)
- Points of view (What other perspectives need consideration?)
- Assumptions (What presuppositions are being taken for granted?)
- Implications and consequences (What might happen? What does happen?)
- Evidence (What relevant data, information, or experiences are needed for assessment?)
- Inferences and interpretations (What can be inferred from the evidence?)
- Concepts (What theories, definitions, axioms, laws, principles, or models underlie the issue?
Literature Consultation
Concurrent with their critical thinking, MCIIS, LSS, and others examined key academic and applied-academic works related to the assessment of non-state actors. Notable among them, work by Bas Arts and Piet Verschuren describes a qualitative method for assessing the influence of stakeholders in political decision-making.380 The “triangulation” referred to in their title encompasses “(1) political players’ own perception of their influence; (2) other players’ perceptions of the influence brought to bear; and (3) a process analysis by the researcher.”
Another contribution in the applied realm came from recent work by a new generation of military (and ex-military) authors who see the rise of non-state actors as a seminal event that will drive U.S. national security strategy. Among these sources is Warlords Rising: Confronting Violent Non-State Actors, whose authors anchor their work in open systems theory (the concept that actors and organizations are strongly influenced by their environment).382 In particular, they ask what environments give rise to violent383 non-state actors, what sustains them, and how changes to those environments might disrupt them.
Application: Indicators of Non-State Actor Power in Africa
The students were able to validate their findings employing three different methods as well as different evidence sets and also assess their methodological validity. This kind of meta- sensemaking could constitute a bridge between now-traditional IC efforts and a revolutionary approach to building a sensemaking argument in official circles.
Of note is a remark by project supervisor Professor Wheaton: “The big advantage [of the multimethodological approach] was the ability to see similar patterns crop up again and again by looking at the data in different ways. This increased their [the students’] confidence enormously.”384 Additionally, given the temporal context (short) and the scale of the project (large) a multimethodological approach was perhaps the only means of tackling the problem.
Application: A Multi-Disciplinary Workshop on Non-State Actors
Participants used three frameworks to consider the environment within which the three groups exist and operate.
- Points of segmentation are the boundaries or borders between and among groups of people, where the degree of disagreement on issues is indicated numerically.386 Points of segmentation can track inherent characteristics such as gender or ascribed cultural differentiators such as Sunni or Shiite. The set of points distinguishes one individual or group from another and identifies possible points of cooperation and conflict that can be exploited. Specific values for points of segmentation are derived from an expert assessment of the strength of the actors’ expressed attitudes, reinforced by observable behavior. They distinguish one individual or group from another, and identify the points most suitable for exploitation by the protagonist.
- Prospect theory, originally developed by Kahneman and Tversky, posits that “people tend to be risk-preferring when facing long shot risks involving significant gains, such as betting on race horses, and are risk averse when facing significant losses: [in other words, when] buying a home or car insurance” respectively.”388
- Institutional interactions is the name associated with a systematic model that allowed workshop participants to explore the complex roles non-state actors play as they influence (and are influenced by) overlapping institutional capabilities and needs.389 The participants concluded that even a simple model of institutional networks has enormous complexity—or high entropy—making it a good candidate for a subsequent in-depth modeling project. Due to imposed time constraints, development and application of the modeling was not completed.
- Morphological Analysis was identified as an additional approach through the institutional interactions method. Morphological analysis considers an entire space of possible implications opening the way for follow-on disambiguation (perhaps using additional multimethodological approaches) in order to abductively and soundly derive the kind of judgments that become useful knowledge.390
Their discussions and modeling, however, supported the Warlords Rising thesis: that environment is a critical factor in understanding the emergence and roles of non-state actors.
Critical Assessment: Lessons Learned from the Study of Non-State Actors
No matter what the methodological approach, project participants emphasized that close attention to environmental factors remains a key to understanding non-state actors. Nonetheless, even those approaches that emphasized environmental factors fell prey to certain inadequacies.
Changes in the Roles of Non-State Actors: An Alternative View
A systematic review of what was done and not done in the three non-state actor studies provides insights into how critical thinking can com- bine with multimethodological, mindful sensemaking, to provide a paradigm for 21st Century intelligence creation and its active communication to policymakers in a fashion that transcends the Sherman Kent tradition. This review is facilitated by employing the ten elements of reasoning noted above.
- Question: the beginning question of the NIC-Eurasia Group seminars was, “If non-state actors are emerging as a dominant global force, where is the evidence?” In other words, while there appears to be a consensus that they are a dominant global force, where is the formal evidence?
- Purpose: Determine whether or not there is evidence that non-state actors are emerging as a dominant global force. This problem is one of basic research to determine if the evidence in fact exists. However, the underlying issue of how we might measure relative power must first be conceptualized and addressed.
- Points of View: As we consider the original and complementary studies, there are two predominant points of view at issue: first, that of the NIC and its customers—who may believe that non-state actors are an emerging global force and want to quantify this shift in influence and power. The other, unavoidable point of view is that of non-state actors—some of whom would believe they are an emerging force and some who would not believe they are.
- Assumptions: The use of the term “non-state actor” as an apparent all-encompassing term in the initial problem question and statement presumes an initial understanding and consensus about what is or is not a non-state actor. This is actually inaccurate as the differing foci of the three groups make clear. However, the differences in this case become evident in hindsight although measures could be taken in foresight to at least check the understanding of different groups engaged in collaborative assessments.
Greater precision of the term non-state actors is needed. Differentiating between benign and non-benign non-state actors is a first step. Subsequent refinements of “benign non-state actors” into non- governmental organizations, multinational corporations, and super- empowered individuals is also useful. A similar set of distinctions within the set of violent non-state actors is also necessary. Then, a crosscheck among the teams must be accomplished so that consensus on the meaning and use of these terms is achieved.
Another assumption involves what is meant by the term “dominant global force.” Again, both greater precision and clarity is needed in coping with this assumption. One key question is, “Exactly what does dominant global force mean?” One answer to this could be that everywhere on the planet non-state actors are the force affecting politics and life. Such a simplified and simplistic view is likely inaccurate, and a range of political process models—among them those of the “rational actor,” of “bureaucratic politics,” and of “organizational process,” need to be parsed.
- Implications and Consequences: The consideration of implications and consequences means to anticipate and explore the events that follow a decision, and to put in play especially the interpretive aspect of sensemaking. In the context of non-state actors, it means to explore what happens if non-state actors are (or are not) emerging as dominant global forces and we are right or wrong about their power. Regardless of whether non-state actors are a dominant global force, if their influence is underestimated then surprises can be expected: Some non-state actor is likely to act in a fashion that is completely unexpected and with unanticipated results. On the other hand, overestimating the influence of non-state actors might create self-fulfilling prophecies. If, though, the influence of non-state actors is accurately measured it may be possible to mitigate that influence (where the non-state actors are acting on interests at odds with those of the United States). Alternately, where non-state actors are acting in consonance with the interests of the United States or are able to exploit opportunities put in place to get them to be helpful, the United States fulfills its goals.
Finally, the sensemakers’ interpretation of likely actions or events allows the implications and consequences of those actions to be considered, even if absolute prediction is elusive. Here, in the con- text of collaborative sensemaking through the communication of intelligence to a policymaker, we understand the admonition of Sherman’s Kent’s contemporary critic, Willmoore Kendall, that intelligence most critically “concerns the communication to the politically responsible laymen of the knowledge which…deter- mines the ‘pictures’ they have in their heads of the world to which their decisions relate.”392 This vision suggests communication of intelligence as an “insider” rather than offering “intelligence input” at arms length in the Kent paradigm.
- Evidence: What evidence is needed to determine that non-state actors are, and as importantly, are not an emerging dominant global force? As we have seen, each group gathered and sifted consider- able information on non-state actors, some of it highly relevant to the central question and some not. To the best knowledge of the authors, each of the three groups chose and evaluated evidence only with inductive logic. They did not take advantage of a means, avail- able in particular to a community with robust intelligence capabilities, to deductively eliminate one of the two possibilities.
- Inferences and Conclusions: With three different and independent efforts, the challenge lies in ensuring a useful triangulation of the results of those potentially disparate efforts. The Mercyhurst approach (internally triangulated) found that non-state actors, both legal and extralegal, are least effective in authoritarian states.
the Least Squares workshop demonstrated that within the context of either failed or failing states, expectations and perceptions of the public, or the political environment, are key drivers in anticipating the likelihood of actions by (violent) non-state actors. Strident or acrimonious expression of dissent that arises when domestic and international political/economic issues reinforce each other within the United States and Europe suggest a possible correlation in post-industrial states. This leads to a general conclusion that when expectations are at odds with situational reality, non-state actor activity increases.
- Concepts: Not only the assumptions, but other concepts as well were in play at multiple levels in the non-state actor case studies. The very notions of “non-state actor” and the ideas of democracy, authoritarianism, and anarchy needed clarification, ideally through well-grounded, empirical as well as theoretical research, to ensure common understanding.
- Alternatives: If non-state actors are not emerging as a dominant global force, then what can we say about their global role? Is their influence staying the same? Is it diminishing? Given a credible means of measuring change in the influence and power of non-state actors, the next step in this study of non-state actors would be to examine hypotheses generated from these alternative questions.
- Context: As has been repeatedly noted, non-state actors present both a challenge to U.S. interests and an opportunity for advancing those interests. The U.S. would like to mitigate the challenges and take advantage of the opportunities. How to make that happen in domains and regions of little existing U.S. influence or of waning U.S. and Western influence becomes a key concern as the United States strives to carry out a meaningful global role. Future attempts to make sense of the role of non-state actors may benefit from tap- ping into the larger context of recent policy-relevant literature on the problem of fragile states in applied academic journals.
Moving Beyond a Proto-Revolution
Microcognition and Macrocognition in the Study of Non-State Actors
There emerge two very general domains of which intelligence professionals must make sense: that of the relatively static, state-based system and that of the much more dynamic non-state actor. Of course, these do not exist in isolation from one another. There are boundaries, interstices, and points of segmentation; there is considerable overlap when one usurps or adopts the actions of the other. Further, the separate domains of domestic and foreign areas of interest and action, embraced by the Kent model of intelligence creation and communication, have been superseded by an indivisible, world- wide web of personal and organizational relationships. Broadly speaking, the “classic” model of intelligence sensemaking largely sufficed and perhaps continues to suffice when issues remain clearly tied to the political entities associated with the Westphalian system of state-based power.
However when dealing with non-state actors, a new, revolutionary paradigm becomes essential for making sense of issues as well as their interactions with the states of the other paradigm. In the former, a traditional, intuitive and expert-supported approach was largely adequate. In the case of the latter, as is glimpsed in this case study, a more rigorous approach is required.
In national intelligence terms, practitioners and their customers work in a macrocognitive envi- ronment as they manage the uncertainty they face in dealing with wicked problems.
Macrocognition, then, includes a focus on process as well as results—what we have labeled mindful, self-reflective sensemaking.
Klein et alia observe that intelligence professionals and decision- makers traditionally are “microcognitively” focused. That is, like those who follow in the Sherman Kent tradition, they are concerned with solving puzzles, searching, and “estimating probabilities or uncertainty values” for different phenomena of interest.403 As has been discussed, such an approach still may be suitable for solving tame problems or those of the Type 1 domain. Thus, microcognition describes the reductionist foci of the current intelligence “analysis” paradigm. However, this is not sensemaking, which requires another approach.
The transition or shift to macrocognition requires a focus on “planning and problem detection, using leverage points to construct options and attention management.”
Elements of the foregoing case study exemplify this strategy. Both the Mercyhurst Role Spectrum Analysis and the Least Squares Points of Segmentation identified potential leverage points that revealed truths about non-state actors, leading to more robust problem detection. A next step would have been to take the triangulated results from all the deployed sensemaking methods and use the results to construct options for dealing with nonstate actors in multiple environments. Such a macrocognitive approach would allow more persistent attention to the anti ipation of the broad course of events (in this case involving non-state actors), in contrast to a microcognitive focus on predicting more isolated and specific future incidents.
Next Steps in Revolutionary Sensemaking about Non-State Actors
The foregoing elaboration of non-coordinated sensemaking activities, even with its limitations, moved beyond the traditional model of intelligence creation. It specifically identified the multiple approaches taken by independent teams who used alternative schema and methods that, perhaps unsurprisingly, resulted in a broader understanding of the problem.
Triangulation was largely informal both within and between the groups. Thus, the work met the criteria for a transitional intelligence sensemaking project. The participants in all three efforts engaged in critical thinking to one degree or another. All were also mindful of the wicked issue of non-state actors and its significance.
the adoption of the new paradigm for sensemaking depends on bringing into play a cooperative spirit of science and scientific inquiry to the process of intelligence creation and communication. Mindful, critical thinking-based, multimethodological approaches to analysis, synthesis and interpretation are one means of doing this. Additionally, a means needs to be found to ensure that this approach to sensemaking remains rigorous. This becomes the subject of the next chapter.
CHAPTER 8
Establishing Metrics of Rigor
Defining Intelligence Rigor
I know the distinction between inductive and deductive reasoning. An intelligence officer is inherently inductive. We begin with the particular and we draw generalized conclusions. Policymakers are generally deductive. They start with the vision or general principle and then apply it to specific situations. That creates a fascinating dynamic, when the intelligence guy, who I call the fact guy, has to have a conversation with the policymaker, who I tend to call the vision guy. You get into the same room, but you clearly come into the room from different doors. The task of the intelligence officer is to be true to his base, which is true to the facts, and yet at the same time be relevant to the policymaker and his vision. That’s a fairly narrow sweet spot, but the task of the intelligence officer is to operate in that spot.
— GEN Michael V. Hayden (Ret.), former Director of the Central Intelligence Agency and the National Security Agency
Michael Hayden’s view that the intelligence officer needs to operate in the “sweet spot” linking intelligence and policymaker cognitive worlds coincides with the aim of the sensemaking paradigm. To bring these two worlds together, intelligence professionals can take advantage of the opportunity to meld their fact-based inductive tendencies with the visionary, deductive model of policymakers through the application of collective rigor to well-conceived questions. This approach allows intelligence professionals to embrace a triangulation on wicked problems from their professional perspective, and to improve their chance to communicate with policymakers whose circumscribed comfort zone may accept or even welcome wicked problems as opportunities to apply their vision to bring about politically rewarding solutions.
At present most tradecraft for sensemaking triangulation remains intuitive, operating in the realm of tacit knowledge. Thus, part of a revolution in intelligence requires that more formal and explicit means of triangulation be developed. It may be that some existing analytic tradecraft, when conscientiously applied, will improve synthesis and interpretation. Another option is to explore and experiment with new tools for conceptualizing rigor in information analysis, synthesis and interpretation.
Rigor in sensemaking can refer to inflexible adherence to a process or, alternatively, to flexibility and adaptation “to highly dynamic environments.”408 As proponents of the latter approach, Daniel Zelik, Emily Patterson, and David Woods recently reframed the idea of rigor into a more manageable concept of “sufficiency.”409 In the applied world of sensemakers, then, an apt question is: “Were sufficient considerations made or precautions taken in the process of making sense of the issue?” Zelik et alia observe that this requires a “deliberate process of collecting data, reflecting upon it, and aggregating those findings into knowledge, understanding, and the potential for action.”410 In order to achieve answers to this question Zelik et alia developed an eight-element taxonomy of sufficiency and a trinomial measurement of rigor: Each element was calibrated in terms of high, medium or low rigor. In their examination of information products, an overall score could be computed that, in intelligence terms, would communicate to both the practitioner’s management and to consumers the rigor of the crafted intelligence product.
Attributes of the Rigor Metric
Hypothesis Exploration describes the extent to which multiple hypotheses were considered in explaining data. In a low-rigor process there is minimal weighing of alternatives. A high-rigor process, in contrast, involves broadening of the hypothesis set beyond an initial framing and incorporating multiple perspectives to identify the best, most probable explanations.
Information Search relates to the depth and breadth of the search process used in collecting data. A low rigor analysis process does not go beyond routine and readily available data sources, whereas a high rigor process attempts to exhaustively explore all data potentially available in the relevant sample space.
Information Validation details the levels at which information sources are corroborated and cross-validated. In a low-rigor process little effort is made to use converging evidence to verify source accuracy, while a high-rigor process includes a systematic approach for verifying information and, when possible, ensures the use of sources closest to the areas of interest.
Stance Analysis is the evaluation of data with the goal of identifying the stance or perspective of the source and placing it into a broader context of understanding. At the low-rigor level an analyst may notice a clear bias in a source, while a high-rigor process involves research into source backgrounds with the intent of gaining a more subtle understanding of how their perspective might influence their stance toward analysis- relevant issues.
Sensitivity Analysis considers the extent to which the analyst considers and understands the assumptions and limitations of their analysis. In a low-rigor process, explanations seem appropriate and valid on a surface level. In a high-rigor process the analyst employs a strategy to consider the strength of explanations if individual supporting sources were to prove invalid.
Specialist Collaboration describes the degree to which an analyst incorporates the perspectives of domain experts into their assessments. In a low-rigor process little effort is made to seek out such expertise, while in a high-rigor process the analyst has talked to, or may be, a leading expert in the key content areas of the analysis.
Information Synthesis refers to how far beyond simply collecting and listing data an analyst went in their process. In the low rigor process an analyst simply compiles the relevant information in a unified form, whereas a high- rigor process has extracted and integrated information with a thorough consideration of diverse interpretations of relevant data.
Explanation Critique is a different form of collaboration that captures how many different perspectives were incorporated in examining the primary hypotheses. In a low-rigor process, there is little use of other analysts to give input on explanation quality. In a high-rigor process peers and experts have examined the chain of reasoning and explicitly identified which inferences are stronger and which are weaker.
The tradecraft underlying these attributes assesses the domains of intelligence foraging and intelligence sensemaking (analyzing, synthesizing, and interpreting) as described here. The study by Zelik et alia reveals that what might at first glance be considered the application of a low level of rigor really is merely a function of a varying distribution of rigor applied among the attributes.
This distinction suggests a profound insight, namely that information search is perceived as more highly valued than information synthesis. In the case above, a definitive judgment about this insight cannot be made, as the other attributes of the two assessments were not identical. Still, it seems clear that at least the participants in the study were still wrestling with a consideration formally discussed by Richards Heuer, Jr. (and many others before and since—including Moore and Hoffman above): How much information is necessary for effective sensemaking?
Assessing Sensemaking Rigor in Studies of Non-State Actors
Rigor and the NIC-Eurasia Group Effort
The NIC-Eurasia Group effort (summarized in figure 12 and column one of table 7) garnered the fewest points of the three groups. Hypothesis Exploration — Low: The NIC-Eurasia Group memorandum noted that non-state actors are of interest “because they have international clout, but are often overlooked in geopolitical analysis.” The implicit but demanding questions of why and how much non-state actor “power” and “influence” have increased worldwide were not answered, nor was a time frame established. This failure to broaden the hypotheses beyond the initial framing of the issue led to a lack of incorporation of multiple perspectives to identify at least “best,” and perhaps most probable answers to these questions.
Rigor and the Mercyhurst Effort
The Mercyhurst students relied on published data. No evidence of consultation with external experts was evident. While this was to be expected given the demographics of a student team, it nevertheless led to a score of Low for the Specialist Collaboration metric.
Rigor and the LSS Effort
The LSS social science study of non-state actors scored the highest of all three groups, earning a high in each metric save one, Information Validation, where they scored a Medium. In this case, while converging information was employed to cross-validate source accuracy for the evidence closest to the areas of interest, a systematic approach for doing this was not evi- dent, resulting in the lower score.
Since the entire team was made up of specialists, their score in Specialist Collaboration should come as no surprise. Similarly, the involvement of diverse social scientists as well as the involvement of external peers foreordained that the Explanation Critique would be rigorous. All the individuals brought differing perspectives as they identified the strengths and weaknesses of each other’s inferences and conclusions. Finally, an explicit multi-methodological approach forced consideration of diverse interpretations of the evidence—a highly rigorous example of Information Synthesis.
Observations and Discussion
It is no accident that the traditional means by which assessments of such issues are created, as evidenced by the NIC-Eurasia Group effort, resulted in a relatively weak score, whereas the highly rigorous, critical-thinking based, multimethodological effort by a collaborative team of diverse experts led to a relatively high score (a comparison highlighted by figure 15).
Another advantage of graphic analysis using Zelik et alia’s metric is that more information can be clearly conveyed. For example, in examining the composite efforts of all three groups of participants in the non-state actor study, it is evident that Information Validation could have been improved through the use of a more rigorous systematic approach, ensuring that the sources were deemed valid and “closest to the areas of interest.”415
there appear to be several rea- sons why information is often not fully validated in intelligence work. First, validation is difficult and the intelligence professional may decide the result is not worth the effort, or that initial conditions suggest validity. “Information hubris”—arising when similar information has without negative con- sequence been presumed or found to be valid, may compound this effort. Wishful thinking and belief in the infallibility of the source are other factors that may contribute to this pathology. Finally, information uncertainty may allow it to resist validation. Unfortunately any or all of these can lead to intelligence errors and failures, suggesting that information validation, despite its inclusion in the rigor metric itself, may require a transcendent application of rigor.
In applying the metric it becomes apparent that some disambiguation is necessary between several of the individual considerations. The differences between high rigor assessments involving Stance Analysis and Sensitivity Analysis at first glance appear to be unusually subtle, suggesting a need for an explanatory critique as part of the standard process assessment. Additionally, because several of the specific metrics are process-related, assessors need to be present to observe the process or otherwise have access to appropriate and sometimes-scarce process-associated materials. Alternately, a formal means for capturing applied (and omitted) sensemaking processes needs to be developed.
In developing the rigor metric, Zelik et alia note that it is “grounded largely in the domain of intelligence analysis.”417 Looking at the metric from a generalizing point of view, Zelik et alia are interested in whether it can be broadened to other disciplines such “as information search by students working on educational projects, medical diagnosis using automated algorithms for x-ray interpretation, and accident investigation analyses.”418 For those of us within the domain of intelligence, however, that this model “emerged from studies of how experts ‘critique’ analysis (rather than how experts ‘perform’ analysis.” is a strength.419 The rigor metric has been empirically, if tentatively, shown (according to Daniel Zelik) to “reveal [some of the] “critical attributes to consider in judging analytical rigor” in intelligence sensemaking.420 In so doing, Zelik and his colleagues also validated the usefulness of the model within the intelligence domain.
A danger inherent in any process of making sense of an issue exists when the “process is prematurely concluded and is subsequently of inadequate depth relative to the demands of [the] situation.”
As Zelik et alia conclude, “the concept of analytical rigor in information analysis warrants continued exploration and diverse application as a macrocognitive measure of analytical sensemaking activity.”424 But what does it mean if such a model is adopted, or not adopted? The final chapter examines the implications of either outcome.
CHAPTER 9
In Search of Foresight: Implications, Limitations, and Conclusions
Considering Foresight
We turn in conclusion to a discussion of the purpose of mindful, critical sensemaking for intelligence. The discussion may be best framed by pertinent questions: To what end is intelligence intended? In other words, intelligence professionals and their overseers critically ask “knowledge of what?” and, secondly, “knowledge for whom?” One answer to these questions is embodied in the concept of foresight: Intelligence knowledge advises policymakers and decisionmakers about what phenomena are likely precursors of events of interest before they occur. Such foresight—in light of the discussions in this book—does not entail specific predictions. Rather, it allows us to anticipate a range of alternative event sequences.
Foresight informs policy and decisionmakers about what could happen so that those individuals can improve the quality of their decisions. Done mindfully, its vision shifts and evolves apace with the phenomena about which it makes sense. Done wisely in such a manner as presented here, it augments the vision of leaders, enabling mobilization and discouraging two traits that often handicap visionaries: recklessness and intolerance. Done rigorously, it cannot be accused of failing to be imaginative. This prospective approach contrasts with the current practice and paradigm for intelligence production.
In the Kent tradition, as has been noted (and is summarized in figure 16), intelligence knowledge of “analyzed” issues becomes and tends to remain disaggregated into constituent parts—oriented, as Treverton notes, toward solving isolated “puzzles” rather than the more holistic “mysteries” of the intelligence world. On the other hand, we have discussed how Kendall’s competing idea, that intelligence knowledge should “paint a picture” (by way of a macrocognitive, holistic approach) for the policy maker as a fellow “insider,” is consistent with a model of intelligence where the predominant method and motive of intelligence sensemaking is through aggregation and the articulation of a fact-based “vision” recognizable by national-level policymakers (figure 16). Even in an operational military scenario, where isolated, specific facts are essential to successful employment of mission knowledge, a larger intelligence sense of who the ultimate commanders are and why they are doing what they’re doing, remains the essence of useful, foresightful strategic knowledge—also known as national intelligence.
Implications
Creating intelligence as presented here is a mindful process of sensemaking, encompassing the activities of planning, foraging, marshaling, understanding, and communicating. It is critical of itself and the means that are employed in bringing it about. It lies within the largely overlooked Kendallian vision of what national intelligence ought to accomplish. This approach allows a focus on better and worse solutions, and anticipation of likely futures, instead of a more narrow focus on right and wrong answers in an intellectual environment trained on predictive and specific warning. It can make sense of wicked problems.
By contrast, intelligence as it is currently practiced is still somewhat akin to the practice of medicine in the 14th Century.
Intelligence practitioners find themselves in a similar situation. They often do not know why they do what they do, only that the last time, it “worked”—or that it is an “accepted practice.” They do not acknowledge that they have “forgotten” all the times it did not work. Yet, intelligence practitioners who would wear the “professional” label need to know what they are doing and why.427
One means set forth for “improving intelligence” is to capture the processes by which sense is made of an issue. It is certainly true that imposing audit trails is a critical step because they encourage process improvement in the light of serious errors, and stimulate repetitive analysis, synthesis, and interpretation for validation in the full course of sensemaking. However, auditing trails remain inadequate when the Community cannot understand from an epistemological point of view what does and does not work and in what situations.
The major intelligence failures of the first years of the present decade, as well as repeated failures over at least six decades, demonstrate what hap- pens when there is a formal failure to synthesize and interpret beyond what is popularly believed or even to recognize that a situation exists that requires new synthesis and interpretation. A popular hypothesis is that tradecraft can minimize the likelihood of such failures of imagination. Yet this hypothesis remains untested except in some anecdotal cases which, given the Type 2, wicked nature of the intelligence issues now often faced by the Community, is inadequate.
Limitations
It should be noted that a tradecraft of mindful understanding does not guarantee accurate findings. Any of the components of sensemaking can be done poorly yet “correct” answers can be reached. Disaggregating phenomena can be done well yet yield faulty results. Synthesis and interpretation of analyzed phenomena can still lead to faulty conclusions. However, analysis, synthesis and interpretation within the framework of appropriately applied, multimethod tradecraft does guarantee more rigorous sensemaking.
Conclusions
As has been noted repeatedly in this book, many 21st-Century intelligence issues are wicked problems: They are ill-defined and poorly understood issues with multiple goals that must be made sense of within severe time constraints; the stakes and risks are high and there exists no tolerance for failure. As a means of increasing situational awareness, merely creating mindfulness about such complex issues falls short. On the other hand, a mindful sense- making approach to situational awareness accomplishes more by enabling the intense, holistic scrutiny of a complex developing scenario, as suggested in the case study in chapter 7. This macrocognitive approach ensures that the knowledge created also evolves.
If intelligence is to rise above the noise and get the attention of policy and then be acted upon it must be both. A critical, mindful process of sensemaking offers a means for this to occur. As we have seen, it covers the issue broadly, takes into account its complexity, is systematic and rigorous. It offers the best means currently understood for making sense of what is known and knowable.
Aggressive, mindful sensemaking is one path- way to this new paradigm, and may require a different mix of skills and abilities than is currently present. It certainly requires greater, authentic diversity. Considering the present community, one is reminded of Kent’s quip, “When an intelligence staff has been screened through [too fine a mesh], its members will be as alike as tiles on a bathroom floor—and about as capable of meaningful and original thought.”439 In contrast, making sense of the 21st Century’s intelligence challenges requires as much rigorous, “meaningful and original thought” as we can muster. Sensemaking, as it has been developed here, offers us a means of creating that desperately needed thought.