Author: Ariel

Bringing Intelligence About: Practitioners Reflect on Best Practices

Russell G. Swenson, Editor

With a Foreword by Mark M. Lowenthal, Assistant Director of Central Intelligence

INTRODUCTION

Russell G. Swenson with David T. Moore and Lisa Krizan

This book is the product of studious self-reflection by currently serving intelligence professionals, as well as by those who are in a position, with recent experience and continuing contacts, to influence the development of succeeding generations of intelligence personnel. Contributors to this book represent eight of the fourteen organizations that make up the National Foreign Intelligence Community. A positive image of a community of professionals, engaged in public service, and concerned about continuous self- improvement through “best practices,” emerges from these pages.

Community partners, such as the Central Intelligence Agency (CIA), Defense Intelligence Agency (DIA), the National Security Agency (NSA), and the State Department’s Bureau of Intelligence and Research (INR), for example, share responsibilities for national security issues that allow individual collectors, analysts, issue managers and offices to work together on interagency task forces.

with its strategic focus, the Intelligence Community expects to be forward-looking, envisioning future developments and their repercussions, whereas law enforcement intelligence efforts have typically focused on exploiting pattern analysis to link together the extralegal behavior of individuals and organizations with clear and legally acceptable evidence.

a facile claim of significant differences between law enforcement and national security intelligence may hold up to scrutiny only in terms of the scale of operations supported rather than professional intelligence techniques employed.  We may infer from these observations that the principles of intelligence collection and analysis addressed in this book will apply to intelligence creation in the broadly overlapping cultures of law enforcement and national security intelligence.

The U.S. Intelligence Community was subject during the 1990s to a congressionally mandated reduction in Intelligence Community personnel levels.2 This reduction occurred despite numerous small wars and the continuation of international criminal activity during the decade. When dissenters, such as former Director of Central Intelligence James Woolsey, “talked about the proliferators, traffickers, terrorists, and rogue states as the serpents that came in the wake of the slain Soviet dragon, [they were] accused of ‘creating threats’ to jus- tify an inflated intelligence budget.”3 Even government reports such as that of the United States Commission on National Security (commonly referred to as the Hart-Rudman Report), which warned of catastrophic attacks against the American homeland and a need for vigilance, were dismissed.4

even though collection methods are often arcane, methods of analysis are not very esoteric. Analytic methods used by intelligence analysts are readily available to specialists in the academic world.6 The commonalities that do exist among collectors and analysts across the Community have rarely been noted in intelligence literature. The essays in this book will help fill that gap, and should illuminate for non-specialists the important role of self-reflection among intelligence professionals who remain in government service.

6 Many if not most analysts have been exposed, by training or experimentation, to such techniques as link analysis, the Delphi technique, and analysis of competing hypotheses. Morgan D. Jones, former CIA analyst, has distilled the less structured techniques that intelligence analysts may employ in The Thinker’s Toolkit: 14 Powerful Techniques for Problem Solving (New York: Times Business, Random House, 1995 and 1998)

It may now be true that the value of intelligence to consumers is more dependent on the evaluation of information (grappling with mysteries) than on discovering “secrets.”7 If so, then the evaluation of social trends in various regions might best begin with systematic exploitation of authentic or “grass-roots” reporting from newspapers and other mass media.

language capabilities are indispensable for any country’s intelligence personnel who seek insights through indigenous mass media. Language capabilities must mirror those tongues used across the electronic media that represent the target entities

Garin defines the intelligence corporation in terms of a “learning organization” and then applies external standards from the Baldrige National Quality Program to selected intelligence-producing offices within the Defense Intelligence Agency. This benchmarking study not only identifies best practices, but also shows how such professional standards could be used to identify exemplary offices or individuals across the entire Intelligence Community.

If a communitarian ethos distinguishes intelligence professionals from their more individualistic and self-absorbed brethren in academia, then self- reflection among intelligence practitioners can also easily become a communal good. Tension between a communitarian and individualistic ethos can resolve itself among intelligence professionals through the strength of their bureaucratic (Weberian), nonmonastic tradition. The essays in this volume illustrate how, through self-reflection, that tension may be resolved. For example, individual professionals can easily spell out connections among these essays that would quickly move the discussion to a classified realm—into their “culture.”

That culture is typically characterized by fast-moving events and requirements that preclude introspection about the phenomena of intelligence collection and production.

Self-reflection not only allows the various agency sub-cultures to be displayed, as portrayed here, but also allows “insiders” to realize the subtle connections of their individual work to the overall enterprise. As a further illustration of this principle, the intense intellectual effort that characterized earlier eras of intelligence production and that continues as a part of the enduring culture, is evoked by the observations of William Millward, a World War II intelligence analyst at the UK’s Bletchley Park:

[Analysis] means reviewing the known facts, sorting out significant from insignificant, assessing them severally and jointly, and arriving at a conclusion by the exercise of judgment: part induction, part deduction. Absolute intellectual honesty is essential. The process must not be muddied by emotion or prejudice, nor by a desire to please.9

National intelligence collection management and intelligence analysis remain inherently government functions, and privatized intelligence—with its prospect of reduced congressional oversight—is even more antagonistic to the communal sharing of information than are the more stringently overseen bureaucratic fiefdoms.

to “bring intelligence about” from the point of view of the American people requires peeling back some of the thick mantle of secrecy that has shrouded individual initiatives and management approaches—Community best practices—employed in the execution of ordinary and extraordinary tasks. Readers who look closely at the observations set down by the authors here will find a serviceable tool for unwrapping some of the otherwise enigmatic enthusiasms and motivations of government intelligence professionals.

THE INTELLIGENCE PRO AND THE PROFESSOR: TOWARD AN ALCHEMY OF APPLIED ARTS AND SCIENCES

Pauletta Otis

Recent events have led to an increasing realization that all of the resources of the American public must be engaged in support of national security. The U.S. academic community and the U.S. National Foreign Intelligence Community are institutional managers of “information as power.” Yet their potential for useful cooperation and collaboration is unrealized and the relationship between the communities continues to be somewhat strained. This relationship has been the topic of discussions for many years in a somewhat benign security environment. With dramatic changes in that environment in late 2001, maximizing resources through enhanced cooperation and collaboration assumes critical importance.

From the academic’s point of view, the U.S. Government’s stability and its concern for the common welfare derive from the First Amendment. If a stable democracy depends on an informed public, the public cannot make good decisions without accurate and timely information. Without the “self-correcting” mechanism of free speech, a government tends to be subject to centrifugal forces, to spin on its own axis, and to become an entity quite separate from the citizenry.

Yet, on a practical level, we need to “keep secrets” from U.S. enemies, however they may be defined. A certain level of secrecy is required to protect national interests. How much secrecy is “best” in a democracy is subject to ongoing debate.11 The pendulum swings over time and in response to perceived threats to national security.

COMMONALITIES

Both the IC and the academic community work in the world of word-power. Both are populated by dedicated and committed Americans. In most cases, cooperative efforts have yielded the “best” for the United States in terms of public policy and decisionmaking. Yet, at other times, the communities have been at serious odds. The ensuing problems have been of significant concern to the attentive public and detrimental to the common good. These problems emanate not only from the theory of democracy but from competition over the nature of truth and value for the country. They can also be traced to concern about the nature and processes of intelligence gathering.

The IC and the academic community have much in common: both realize that information is power and that power comes in the form of words. These words form the basis of cultural assumptions, common concepts, and grand theory. In terms of information handling minutiae, intelligence specialists and academic scholars do much the same thing. They research, write, explain, and predict with excruciating self-consciousness and realization of the power of words. Intelligence professionals and academics contribute to pub- lic discussion in both oral forums and in written format. Intelligence professionals and academics are valued and participant citizens: They attend common religious services, participate in the education of children, contribute to community service organizations, and talk to neighbors. At this level, both are truly in the same “game.” In the public domain, both intelligence professionals and academics contribute information, analysis, and opinions. These are read and integrated by the public, by policymakers, and by the “other” community. This common body of public information and analysis is at the core of intelligent decisionmaking in the United States.

For 50-plus years, relations between the IC and academia have wavered between cooperation and competition. The basic tension inherent in keeping “secrets in a democracy” has been played out in specific activities and attitudes.

Between 1939 and 1961, Yale University was overtly supportive of the IC.15 Students and faculty were participants in intelligence collection and analysis. Study centers were financially supported and professional journals published. Information was collected and archived (including the Human Relations Area Files). The money was channeled to individuals or to institutions but seemed to raise few concerns.16 At the political level, the existence of a “community” of intelligence-related agencies and offices has been noted in legislation, suggesting wide acceptance of the idea of government intelligence operations as regularized institutions with special taskings in support of U.S. national security.17

CONTINUITY, CHANGE, AND THE DEVELOPMENT OF DIFFERENCES

The “position descriptions” of an intelligence specialist and of an academic are polar opposites: The basic job of an academic is to find information, collect factoids whether they are useful or not–and scatter the information to the winds hoping and praying to make more sense of the ensuing chaos. Intelligence specialists collect similar factoids for very specific productive purposes. Whereas the academic is spreading information, the intelligence professional is collecting, narrowing, and refining data to produce reports, even if at the same time continuing an academic interest in the subject. The academic may learn for the fun of it; the intelligence professional prefers, or learns to be able to actually do something with the information. In the long run, the academic must write, produce, and contribute to the body of professional literature, but there is not the pressure for immediate production that the intelligence professional must face. The academic can say “that is interesting”; the intelligence professional must certify that “it is important.”

During the Vietnam era, the question was raised in another context: Information concerning populations in Laos, Cambodia, and Vietnam, collected by anthropologists, was subsequently used in support of the war effort.

during the Cold War, the strain between academics and the IC contributed to a number of very ugly scenes. Distinctions were made between those academics who were “patriotic, loyal, and anti-communist,” and those who challenged authority in any context. Simple questioning became a sign of disloyalty.

Administrative penalties for individual scholars included blacklisting, failure to get tenure, lack of research support, and even dismissal from teaching positions. The public presentation of “truth” was simple: teaching loyalty to the country was the real job of the teacher/ aca- demic. Anything contrary was a serious and significant challenge to the future of the nation. And, if the public supported the university both financially and with its “children,” it in turn should be able to expect loyalty on the part of its faculty employees.

Both the intelligence professional and the academic are, to some extent, prisoners of their own bureaucracies and are habitually resistant to change.

Each community has a body of common knowledge which it assumes everyone working within the institutional framework knows and accepts. Because common knowledge is “common,” it is not self-conscious and most individuals do not even realize that basic assumptions are seldom challenged. For example: the academic community assumes that books are good and thus libraries should have them. Individuals in the IC often assume that books are inherently historical and that current information and intelligence must be “near-real-time” and continually refreshed. There is a distinct attitude of arrogance when these individuals discuss the currency of information with academics. What is not at issue, of course, is that the intelligence professional has to produce and is charged with an indications and warning responsibility mandating inclusion of current information.

Academics are known for jargon and obfuscation; the IC is known for producing acronyms seemingly at leisure. The tribal languages are difficult to learn without a willing translator.

Academics are notoriously fractious and democratic. Anyone can challenge anyone on any idea at any time–and do so. Sometimes it is not very polite or courteous even when the comment begins with: “I understand your premise, but, have you considered…..?” Bloodletting is the rule rather than the exception. Members of the IC appear to be more polite although “outsiders” may assume that they derive enjoyment from more discreet disagreement. The various forms of interaction are notable when individuals from the IC quietly participate in sections of the International Studies Association or the Political Science Association, or when academics vociferously contribute to conferences held under the auspices of the IC.

There is further the problem of “group-think” on both sides. Each assumes not infrequently that they have a lock on reality and often simply refuse to be open-minded. This occurred during the Vietnam era when the IC felt itself misunderstood and abused, and the academic community believed that its information and personnel were being used by intelligence agencies in ways of which academe did not approve. The complexity of the story is well-known but the mythology remains extant.19 Part of the mythology is that intelligence agencies will collect information from academics but not return or share information in a collegial manner.

The academic community believes that it is not afflicted with the IC’s tunnel vision and can contribute to lateral thinking that supports enhanced problem-solving capabilities.

THE ACADEMIC’S DECISION

The motivations for an academic to work for the IC are, of course, individualistic and mixed. He or she may simply want a “steady” job that pays well–even if only working over the summer break. There can also be an ego boost when intelligence experts value the information and assessment done by an academic in a particular field. The value attached to being “appreciated” for the information and analysis cannot be overstated as there is very little of that inherent in the academic community.

A number of problems may emerge if an academic goes to work for the IC, becomes a contractor, or produces information that can be used by that community. He may “go native,” that is, become more “intelligence-centric” than the IC needs or desires, all in an attempt to “belong.” There is a tendency for self-delusion and thinking that the information contributed is going to “make a difference.” Sometimes it does; sometimes it is just part of the package provided for a specific tasking. The academic really has no way of knowing what impact his or her work is having.

There are special areas in which the academic can contribute to the IC. A primary contribution is in the collection of facts and information using sources that the IC is precluded from using. Academics can generally travel more easily than intelligence analysts and can establish human networks across international boundaries that can be very useful especially in times of crisis. Academics generally are trained in the development of analytical frameworks and employ them with a degree of ease and clarity. Many academics contribute a unique ability to critique or evaluate projects based on a wide reading of the literature combined with “on the ground” experience. The good academic is always a per- verse thinker–looking to examine the null hypothesis or suggesting alternative explanations. They tend not to accept the “generally accepted explanation.” This habit can make them a bit difficult as conversationalists especially if “critique” is translated as “criticism.” A good academic scholar will insist on “rigor,” as in following the requirement to test basic assumptions, validate criteria used in hypothesis building, monitor the internal validity of hypotheses, and pinpoint the public implications of certain theories. Academic rigor mandates “elegance”; that is, a hypothesis must be important, clear, and predictive. Even in descriptive analysis, an academic insists on looking at a range of possible alternative choices and at the internal coherence of the final product.

There is another caveat: most academics believe that intelligence professionals are better paid and have more job security for doing similar work.

The ultimate caveat: it is hard to explain to professional colleagues that “I am not a spy.” The mystique outlives the reality. It can be overcome with honesty and some degree of good humor.

There are specific benefits to the IC if academics and scholars are employed wisely. One of the often-heard criticisms of the IC is that it has a special susceptibility to “group- think” because analysts seldom have internal challenges. Projects, after all, are group-produced and tend to reduce an individual’s unique contribution in favor of the final product. The academic, simply by the nature of the job, challenges the norms, supports individual initiative and creativity, and values novel or unconventional thinking.

SUGGESTIONS FOR MUTUAL SUPPORT AND MUTUAL BENEFIT

For all of the reasons just noted, it is important that the IC encourage the participation of academics. Academics can make significant contributions in creative thinking, rigor of analysis, and professional presentation of products.

But the choice of which academic to invite as a participant and the venue for participation are not givens. The IC might do well to choose individual academics not for their geographic convenience, near to the Washington or New York areas, but for the written scholarly work produced in their “home environment.”

CONCLUSION

Academics and intelligence professionals are concerned with much the same set of problems. The approaches to problem solving may differ but certainly the practices inherent in American democratic tradition have constructed the intellectual environment so as to provide common definitions of contemporary threats and challenges. Both communities agree that liberty, democracy, freedom, equality, law, and property are defensible American values. Cooperation between the IC and academics, and specific contributions that academics can make to the production of good intelligence, can and must be further supported. It would be foolish to waste available skills and talent. There is too much at stake.

VIA THE INTERNET:
NEWS AND INFORMATION FOR THE ANALYST FROM NORTH AFRICAN ELECTRONIC MEDIA

John Turner

The remarkable growth of electronic media over the past decade in Francophone North Africa provides specialists in the region with a wealth of news and other information. This survey and analysis of the French-language media of Algeria, Morocco, and Tunisia examines how these media go about providing information on North African politics, economics, and culture in a way that does take account of expatriates, who make up an important sector of the politically active population. For “outsiders’’ who specialize in the interpretation of the region, news media offer a “grass-roots’’-based prism through which future developments may be anticipated.1 Observable trends in the North Africa media, when carefully documented and placed in context, validate the contention of various authors, from Robert Steele and Gregory Treverton to Arthur Hulnick, who have addressed the promises and perils of devoting a greater share of intelligence resources to the exploitation open-source information. They have found much more promise than peril.2

MEDIA ISSUES: OBJECTIVITY AND ACCURACY

The political heritage of North Africa and its impact on the region’s media culture together raise the inevitable questions of editorial freedom, objectivity and accuracy. The ideal standard for North African publications, as in other countries, is one of adherence to strict standards of reporting and analyzing events in a climate of universal press freedom. However, “red lines’’ exist in editorial freedom, objectivity, and accuracy that are perilous to cross. North African media have suffered negative sanctions imposed by government officials as the result of reporting that has broken taboos.19

Objectivity is a concern for analysts and researchers in a media culture where most papers, not only those owned by manifestly political groups, but also the independent press, are controlled by interests that have a marked political agenda. Nevertheless, in both Algeria and Morocco a system of checks and balances exists, with daily and weekly print media and their electronic counterparts in vigorous competition for the domestic and foreign market. These publications maintain high standards of journalism, and are normally quite open about their particular biases or journalistic objectives. Such competition and declared interests ensure a degree of confidence in political and economic analysis that allows a high degree of confidence among business and government consumers who seek information for decisionmaking. Tunisia, with its dearth of news dailies and strong government influence in the media, remains problematic for the area specialist seeking in- depth analysis of some events, although most economic and business reporting remains of high quality and supports its citizens’ decisionmaking effectively.20

Accuracy has historically been less of a concern. A flourishing media culture and rivalry among news dailies and weeklies in Algeria and Morocco ensures a degree of competition that normally means news items are reliable; that is, consistently reported in different publications with a sufficient degree of detail and verification of factual data to make them satisfactory records of events. In such an information-rich climate, attempts by government or private parties to plant stories would be subject to immediate scrutiny and detection by the press. The latter frequently analyzes government reporting on an issue and takes it to task for various shortcomings, thereby ensuring, through the resulting reliability, that validity or accuracy is also addressed.

For observers of North African affairs, trends in the French-language electronic news media are reassuring for the near future. Area specialists and other interested professionals will have a steadily growing body of information from which they can draw for a reasonably authentic representation of social trends. If political, economic, and social information remains readily available without the geographic limitations associated with print media distribution, the confidence level in information associated with this region among government and business interests worldwide will continue to improve. The ability of North African intelligence specialists to take advantage of multiple rather than single-source information to provide analysis to decisionmakers will, however, increasingly have to take account of Arabic-language media as well as independent Internet news sites in order to ensure a more complete picture of events.

IMPROVING CIA ANALYSIS BY OVERCOMING INSTITUTIONAL OBSTACLES

Stephen Marrin

The accuracy of CIA intelligence analysis depends in part upon an individual analyst’s expertise, yet programs implemented to increase this expertise may not be sufficient to increase the accuracy of either an individual’s analysis or the institution’s output as a whole. Improving analytic accuracy by increasing the expertise of the analyst is not easy to achieve. Even if expertise development programs were to result in greater regional expertise, language capability, or an improved application of methodological tools, the production process itself still takes place within an institutional context that sets parameters for this expertise. The agency’s bureaucratic processes and structure can impede an analyst’s acquisition and application of additional expertise, preventing the full realization of the potential inherent in expertise development programs. Therefore, any new reform or program intended to improve analytic accuracy by increasing the expertise of its analysts should be supplemented with complementary reforms to bureaucratic processes — and perhaps even organizational structure — so as to increase the likelihood that individual or institutional improvement will occur.

CIA’s intelligence production may be subject to improvement, but making that a reality requires a sophisticated understanding of how an analyst operates within the Directorate for Intelligence’s [DI’s] institutional context. However, empirical verification of this hypothesis is impossible since, as Jervis notes, “[r]igorous measures of the quality of intelligence are lacking” and are insurmountably difficult to create.

this essay is a conceptual “what-if” exploration into the interplay between the acquisition and application of expertise on three levels: that of the individual, bureaucratic processes, and organizational structure.

THE GOAL: IMPROVING CIA’S ANALYSIS

Improving the accuracy of CIA’s finished intelligence products could make an immediate and direct improvement to national security policymaking as well as reduce the frequency and severity of intelligence failures.

In the author’s experience and observation, a DI analyst interprets the international environment through an information-processing methodology approximating the scientific method to convert raw intelligence data into finished analysis. The traditional “intelligence cycle” describes how an analyst integrates information collected by numerous entities and disseminates this information to policymakers. As William Colby—former Director of Cen- tral Intelligence (DCI) and veteran operations officer—notes, “at the center of the intelligence machine lies the analyst, and he is the fellow to whom all the information goes so that he can review it and think about it and determine what it means.4 Although this model depicts the process in sequential terms, more accurately the analyst is engaged in never-ending conversations with collectors and policymakers over the status of international events and their implications for U.S. policy. As part of this process, intelligence analysts “take the usually fragmentary and inconclusive evidence gathered by the collectors and processors, study it, and write it up in short reports or long studies that meaning- fully synthesize and interpret the findings,” according to intelligence scholar Loch Johnson.5

Intelligence failures of every stripe from the trivial to the vitally important occur every day for a variety of reasons, including the mis-prioritization of collection systems, hasty analysis, and inappropriately applied assumptions.

Administrators at the Joint Military Intelligence College note that “analysis is subject to many pitfalls — biases, stereotypes, mirror-imaging, simplistic thinking, confusion between cause and effect, bureaucratic politics, group-think, and a host of other human failings.”7 Yet most intelligence failures do not lead to direct negative consequences for the U.S. primarily because the stakes of everyday policymaking are not high, and errors in fact and interpretation can be corrected as the iterative process between intelligence and policy develops. As a result, most failures or inaccuracies are eventually corrected and usually never even noticed. However, sometimes intelligence failure is accompanied by either great policymaker surprise or serious negative consequences for U.S. national security, or both.

The CIA’s May 1998 failure to warn American policymakers of India’s intentions to test nuclear weapons is an illustration of both kinds of failure. This lapse — widely criti- cized by foreign policy experts and the press — highlighted intelligence limitations such as the DI’s inability to add together all indications of a possible nuclear test and warn top policymakers. According to New York Times correspondent Tim Weiner, these indicators included “the announced intentions of the new Hindu nationalist government to make nuclear weapons part of its arsenal, the published pronouncements of India’s atomic weapons commissioner, who said…that he was ready to test if political leaders gave the go-ahead, and …missile tests by Pakistan that all but dared New Delhi to respond.”8 CIA’s inability to integrate these indicators — a failure of analysis — led to charges of “lack of critical thinking and analytic rigor.”9 Admiral David Jeremiah—who headed the official investigation into the failure—concluded that intelligence failed to provide warning in part because analysts “had a mindset that said everybody else is going to work like we work,” otherwise known as mirror-imaging.

CIA’s search for ways to improve analytic accuracy and prevent intelligence failure— if successful—could have a positive impact on national security policymaking. The CIA is arguably the centerpiece of the United States’ fourteen-agency intelligence community (IC) and “has primary responsibility for all-source intelligence analysis in the [IC] and the preparation of finished national intelligence for the President and his top policymakers,” according to former CIA Inspector General Fred Hitz.12 If CIA analysis does have this kind of central role in influencing policy, improving its accuracy should provide policy- makers with the opportunity to create or implement policies that more effectively protect national security and advance national interests.

THE METHOD: INCREASING ANALYTIC EXPERTISE

Improving the capabilities and knowledge of the individual CIA analyst through pro- grams reflecting Admiral Jeremiah’s recommendation, is one way to improve the accuracy of intelligence analysis. An analyst’s expertise, defined as “the skill of an expert,” 13 is a crucial component for the production of accurate intelligence analysis. “In the lexicon of US intelligence professionals, ‘analysis’ refers to the interpretation by experts of unevaluated [‘raw’] information collected by the [IC],” according to Loch Johnson.14 The presumption is the more “expert” an analyst is the more accurate the resulting interpretation will be.

Analytic expertise is a multi-faceted concept because CIA uses a complex web of ana- lytic specialties to produce multi-disciplinary analysis. Not hired directly by the CIA or even the DI, most analysts are hired by the individual DI offices and assigned to “groups” that cover specific geographic areas, and are then assigned a functional specialty—”disci- pline” or “occupation” in DI terminology—such as political, military, economic, leader- ship, or scientific, technical, and weapons intelligence, according to CIA’s website.16 An analyst’s expertise can vary depending on his or her relative degree of regional knowl- edge, familiarity with disciplinary theory, and with intelligence methods in general:

■ Regional expertise is essentially area studies: a combination of the geography, history, sociology, and political structures of a defined geographic region. The DI’s regional offices are responsible for an analyst’s regional expertise and develop it by providing access to language training, regional familiarization through university courses, or in- house seminars.

■ Disciplinary expertise relates to the theory and practice that underlies the individual analytic occupations. For example, economic, military, political and leadership analysis are built on a bed of theory derived from the academic disciplines of economics, military science, political science, and political psychology, respectively. Disciplinary expertise can be acquired through previous academic coursework, on-the-job experience, or supplementary training.

For the most part each CIA analyst possesses a very small area of direct responsibility defined by a combination of regional area and discipline as they work in country teams with analysts of other disciplines and interact with other regional or disciplinary special- ists as the need arises. CIA’s small analytic niches create specialists, but their specialties must be re-integrated in order to provide high-level policymakers with a bigger picture that is more accurate and balanced than that arising from the limited perspective or knowledge of the niche analyst. This process of re-integration—known as “coordination” in DI parlance—allows analysts of all kinds to weigh in with their niche expertise on pieces of finished intelligence before they are disseminated. According to CIA analyst Frank Watanabe: “We coordinate to ensure a corporate product and to bring the substantive expertise of others to bear.”

former CIA officer Robert Steele observed that “[t]he average analyst has 2 to 5 years’ experience. They haven’t been to the countries they’re analyzing. They don’t have the language, the historical knowledge, the in-country residence time or the respect of their private-sector peers,” as reported by Tim Weiner.

Increasing expertise may not be sufficient to produce accuracy or prevent failure. As Jervis notes, “experts will [not] necessarily get the right answers. Indeed, the parochialism of those who know all the facts about a particular country that they consider to be unique, but lack the conceptual tools for making sense of much of what they see, is well known.”24 In addition, “[e]ven if the organizational problems…and perceptual impediments to accu- rate perception were remedied or removed, we could not expect an enormous increase in our ability to predict events” because “(t)he impediments to understanding our world are so great that… intelligence will often reach incorrect conclusions.”25 That is because human cognitive limitations require analysts to simplify reality through the analytic process, but reality simplified is no longer reality. As a result, “even experts can be wrong because their expertise is based on rules which are at best blunt approximations of reality. In the end any analytic judgment will be an approximation of the real world and therefore subject to some amount of error”26 and analytic inaccuracies—and sometimes intelligence failure—will be inevitable. Therefore, although increasing expertise is a goal, it cannot be the only goal for increasing DI capabilities.

FIRST INTERVENING FACTOR: BUREAUCRATIC PROCESSES

Bureaucratic processes can impede the acquisition or application of expertise gained in expertise development programs, thereby limiting any potential improvement in overall analytic accuracy. The DI is a bureaucracy, and like every bureaucracy creates “standard operating procedures” (SOPs) which are necessary for efficient functioning but over time usually prove to be rigid in implementation.

Analysts must have the opportunity to apply newly acquired expertise back at their desks in the DI for any improvement to result. However, if analysts do not have the opportunity to apply this expertise, it will likely wither for lack of practice. The DI produces many types of finished intelligence—some reportorial, some analytical, and some estima- tive—to meet policymakers’ varying needs for information. In addition to daily updates on developments worldwide, policymakers and their staffs also use information and analyses when crafting policy and monitoring its implementation. Obstacles to the development of expertise appear when shorter product types are emphasized over longer more research-oriented ones. Short turnaround products—including daily updates known as “current” intelligence—have at times been emphasized over other products for their greater relevancy to policymakers, but this emphasis at the same time has reduced the expertise of the DI writ large because they require different kinds of mental operations that reduce the scope and scale of an analyst’s research and knowledge.

Robert Jervis once observed that “the informal norms and incentives of the intelligence community often form what Charles Perrow has called ‘an error-inducing system.’ That is, interlocking and supporting habits of the community systematically decrease the likelihood that careful and penetrating intelligence analyses will be produced and therefore make errors extremely likely.”30 Bureaucratic processes can contribute to the creation of this kind of “error-inducing system.”

The Swinging Pendulum

For much of the DI’s history, analysts acquired expertise by writing long reports. As former CIA officer Arthur Hulnick notes: “a great deal of research was being done, but …much of it was done to enable the analyst to speak authoritatively on a current issue, rather than for publication.”

In a 1993 article, former analyst Jay Young argued that “the needs of the policy-maker too often get lost in the time-consuming, self-absorbed and corrosive intelligence research production process. The DI’s research program is run with the same rigid attention to production quotas as any five-year plan in the old USSR. … This fixation on numerical production leads managers to keep analysts working away on papers whose relevance may be increasingly questionable. …

In short, too much of the Agency’s longer-term research is untimely, on subjects of marginal importance and chock-full of fuzzy judgments.”

Losing Expertise to Gain Relevance

The swing of the pendulum that emphasizes policymaker relevance over analytic depth causes the DI to produce a large amount of current intelligence that prevents the acquisition and application of analytic expertise. Many analysts are facile data interpreters able to integrate large volumes of new information with existing knowledge, and interpret— based on underlying conceptual constructs — the significance of the new data in terms of its implications for U.S. foreign policymaking. This process provides policymakers with exactly what they are looking for from intelligence analysis. However, if provided with minimal time in which to integrate and process the information, the intelligence analyst by necessity cuts corners. When the DI emphasizes intelligence “on-demand,” analysts meet the much shorter deadlines by reducing the scope and scale of their research as well as sidestepping the more laborious tradecraft procedures by not rigorously scrutinizing assumptions or comparing working hypotheses to competing explanations.

Many times current intelligence analysis consists of a single hypothesis — derived within the first hour of the tasking — that the analyst intuitively believes provides the best explanation for the data. Current intelligence as a product has a lesser chance of being accurate because it lacks the self-conscious rigor that tradecraft entails even though it is the best that can be done under the press of quick deadlines.

The production of a single piece of current intelligence has limited effect on expertise because it draws on the knowledge and tools that an analyst has developed through training and prior analysis. However, if over time the analyst does not have the time to think, learn, or integrate new information with old to create new understandings, knowledge of facts and events may increase but the ability to interpret these events accurately decreases.

Intelligence scholar Robert Folker observed that in his interviews of intelligence analysts a “repeated complaint was the analyst’s lack of time to devote to thoughtful intelligence analysis. In a separate interview at CIA it was revealed that [intelligence analysts]… spend little time or effort conducting analysis.”39 Therefore, the effectiveness of expertise development programs in improving analytic accuracy may depend in part on whether the CIA is able to redress the over- emphasis on short-term analysis.

A Structural Fix?

The DI appears to be remarkably blind to differentiation in both analysis and analysts, perhaps because it assigns tasks to “analysts” and equates the output with “analysis.” As a result, “[w]e do not and never have used the term ‘analysis’ rigorously in the [IC]” according to Robert Bovey, formerly a special assistant to a DCI.41 Robert Jervis illustrated the problems of equating the two in 1988 by arguing that “most political analysis would better be described as political reporting” and that instead of following an analytical approach “the analyst is expected to summarize the recent reports from the field—“cable-gisting.” … [T]he reporting style is not analytical—there are few attempts to dig much beneath the surface of events, to look beyond the next few weeks, to con- sider alternative explanations for the events, or to carefully marshal evidence that could support alternative views.”42 Jervis correctly differentiated the analytical intelligence product from its non-analytic cousin, but failed to distinguish between the analysts best suited for each. The DI — instead of differentiating between analysts — uses a one-size- fits-all recruitment, placement, training, and promotion strategy, and for the most part views analysts as interchangeable.

As a result it has perennially had difficulty creating an appropriate mix of analytic abilities and skills for intelligence production when an issue or crisis develops. In particular, over time the shift in expertise corresponding to a preference for longer papers or shorter more current pieces is especially noticeable.

SECOND INTERVENING FACTOR: ORGANIZATIONAL STRUCTURE

The DI’s organizational structure also influences which kind of analyst expertise is acquired and applied in finished intelligence products. Political scientist Thomas Hammond argues that organizational structure impacts analytic output. He employs information-flow models to demonstrate that—given the same information—one group of intelligence analysts organized by discipline would produce different analytic output than another group organized by region. He also concludes that it is impossible to design a structure that does not impact output.44 If we accept his argument, then organizational structure always affects internal information flows, and likely outputs as well. Theoretically, therefore, an organizational structure divided along disciplinary lines will accentuate learning of political, economic, military, or leadership methodologies through constant, interactive contact and awareness of the projects of other analysts.

From approximately the early 1960s to 1981, the DI was structured primarily by discipline with political, economic, military, and leadership offices each subdivided by geography. In 1981 “(a) (DI)-wide reorganization…shuffled most analysts and created geographically based, or “regional” offices out of the previous organization.”47 According to former CIA officer Arthur Hulnick: “These [new] offices combined political, economic, military, and other kinds of research under “one roof,” thus making more detailed analytic research feasible. After some grumbling, the new offices began to turn out the in- depth products consumers had been seeking, while still providing current intelligence materials.”48 This integration of analytic disciplines in regional offices provided a more productive interpretation of the forces at work within a target country or region but also negatively affected the DI’s ability to maintain disciplinary knowledge.

The distribution of what had previously been centralized knowledge of analytic tools, specialized product formats, and disciplinary theory throughout the DI meant that new leadership analysts were not well trained in their discipline. These new analysts relied solely on the fast-dissipating knowledge of the handful of former LDA officers who happened to be assigned to their team or issue. In addition, actual physical co-location did not occur for months — and in some cases years — due to lack of space in CIA’s overcrowded headquarters building. As a result of being “out of sight, out of mind,” leadership analysts were frequently not informed of ongoing projects, briefings, and meetings, and such incidents had a negative impact on the finished analytical product. When leadership analysts were not included in briefings, the DI risked failing to keep its consumers fully informed of both leadership dynamics and changes within the country or region. In addition, products published and disseminated without coordination at times contained factual errors such as the wrong names or positions for foreign government officials, or distortions in analysis due to the lack of leadership analyst input.

Therefore, the elimination of a leadership analysis-based office resulted in both increased incorporation of leadership analysis and insight into the regional teams’ products, and decreased corporate knowledge and expertise in leadership analysis as a discipline.

PUTTING THE PIECES TOGETHER AGAIN

By the mid-1990s DI analysts were realizing that while multi-disciplinary analysis on country teams made for better integration of disciplines, it also led to the dissipation of disciplinary knowledge. Former LDA analysts’ efforts to sustain their hold on disciplinary knowledge triggered similar efforts by political, economic and military analysts to both sustain and reconstruct occupational-specific knowledge. Without the framework of structural organization to bind each discipline together, over time they had each grown apart.

in 1997 the DI created the “senior-level” Council of Intelligence Occupations (CIOC) with the initial intent of disciplinary “work- force strategic planning…in the areas of recruitment, assignments, and training” as well as “identify[ing] core skills and standards for expertise in the [DI].”

In practice, CIOC became a home for senior analysts interested in learning and teaching their discipline’s methodologies. They “establish[ed] a professional development program for all DI employees that provides explicit proficiency criteria at each level so that everyone can see the knowledge, skills, and experiences required for advancement within each occupation or across occupations.”55 All this was done—according to the CIA website — “so that the DI has the expertise needed to provide value-added all-source analysis to its customers.”

There may be no easy solution to the expertise trade-offs inherent in organizational structure. By definition, the structure will create synergies in the areas emphasized but the opportunity cost of doing so means losing the synergies in other areas.

TO THE FUTURE

Expertise-development programs that create the potential to improve overall analytic accuracy—such as formal training in methodologies, regions, disciplines, or languages, or informal training resulting from greater overseas experience — do not provide much in the way of improvement if in fact the DI’s business practices prevent application of this hard-earned expertise. CIA’s leaders should continue to pursue ways to increase analyst expertise, for they could contribute to increased analytic accuracy. Yet at the same time the DI must adapt its practices to leverage the potential improvements of these programs if the CIA is to provide its policymaking customers with the intelligence they need now and in the future.

APPRAISING BEST PRACTICES IN DEFENSE INTELLIGENCE ANALYSIS

Thomas A. Garin, Lt Col, USAF

ABOUT THE AUTHOR

Lt Col Tom Garin, an Air Force officer assigned to the National Reconnaissance Office (NRO), arrived at the Joint Military Intelligence College in September 1999 to occupy the General Thomas S. Moorman, Jr. Chair for National Reconnaissance Systems. He has taught graduate-level core and elective courses on missiles and space systems, structured techniques for intelligence analysis, and research methodologies, as well as an undergraduate core course on space systems. Prior to this post, Lt Col Garin was Chief, Organizational Development at the NRO. There, he used a balanced scorecard approach to link human resource policies with the overall NRO mission.

An adaptable organization is one that has the capacity for internal change in response to external conditions. To be adaptable, organizations must be able to learn. Organizations can learn by monitoring their environment, relating this information to internal norms, detecting any deviations from these norms, and correcting discrepancies.

A telling aspect of any knowledge organization is the way in which it manages its information.

Knowledge management can equip intelligence organizations for the fast-paced, high- technology information age. By building upon the best aspects of total quality management, and observing the criteria for performance excellence adopted for the Baldrige National Quality Program, managers can theoretically lead professionals to work together effectively as a group in a situation in which their dealings with one another affect their common welfare.

STUDY DESIGN

The DIA is expected to ensure the satisfaction of the full range of foreign military and military-related intelligence requirements of defense organizations, UN Coalition Forces, and non-defense consumers, as appropriate. Specifically, DIA supports: (1) joint military operations in peacetime, crisis, contingency, and combat; (2) service weapons acquisition; and (3) defense policymaking. In the past fifteen years, several analysis and production units within DIA have received prestigious awards demonstrating the confidence and appreciation that senior U.S. leaders have in the agency’s performance. On the basis of these observations, a case could be made that DIA has become a world-class organization for meeting U.S. military intelligence requirements. Clearly, the DIA is a worthy organization to examine for evidence of how quality intelligence analysis can be carried out.

The three essential characteristics of operations research are (1) systems orientation, (2) use of interdisciplinary teams, and (3) adaptation of the scientific method. The Baldrige criteria provide a similar function for assessing information organizations. From a systems perspective, the Baldrige criteria focus on a leadership triad and a results triad.

A Typical Office

A typical analysis office consists of a manager, the senior intelligence officer (SIO), analysts, liaison officers, and administration/technical support staff. The manager handles a set of processes common to many organizations such as planning, organizing, staffing, directing, coordinating, reporting, and budgeting. The SIO is a senior analyst, a subject matter expert on a grand scale, and is responsible for the content of the analytical product. A typical SIO approves all finished intelligence products, coordinates budget to contract external studies, and is the unit’s chief training officer. Analysts tend to be subject matter experts and are accustomed to using technology to support analysis. Liaison officers connect the analysis office either to operational forces in the field or to other organizations in the Intelligence Community. The administration/technical support staff provide a variety of functions such as disseminating the finished intelligence product, providing graphic support, and arranging travel.

Consumer Groups

Analysts produce intelligence for a number of different consumer groups.2 These consumer groups include: (1) civilian and military policymakers; (2) military planners and executors; (3) acquisition personnel responsible for force modernization; (4) diplomatic operators; and (5) intelligence operators.

Participants

This intelligence analysis benchmarking study is the result of a class project for the Joint Military Intelligence College’s graduate-level course ANA 635, “Structured Techniques for Intelligence Analysts.” This course helps students understand and apply descriptive and quantitative methodologies to intelligence analysis.

Process

The team followed established benchmarking procedures in conducting this study, which included on-site visits and interviews with members of each unit. The study team adopted a “systems” perspective that isolated the concepts of leadership, strategic planning, consumer focus, information and analysis, people and resources, process management, and mission accomplishment. Taken together, these seven elements define an organization (or a unit within a larger organization), its operations, and its performance.

THE EVALUATION CRITERIA

The Malcolm Baldrige National Quality Improvement Act of 1987, Public Law 100- 107, provides for the establishment and conduct of a national quality improvement pro- gram under which awards are given to organizations that practice effective quality man- agement. The Act established the Malcolm Baldrige National Quality Award. In implementing this Act, the National Institute of Standards and Technology makes avail- able guidelines and criteria designed for use by business, education and health care orga- nizations for self-evaluation and improvement of their competitive positions.3 In this paper, the NIST criteria are applied to DIA, under the premise that a U.S. government organization, through its component offices, could benefit from a systematic evaluation of its “competitive” position; namely, its ability to develop and maintain agility in a change- intensive environment.

Task Assignment

Initiative. In many offices, analysis is self-generated. The office leader assigns each analyst to an area of the world and constantly pushes for finished products. The analyst is expected to act on his own initiative, do the analysis, and produce articles. In this scheme, there really is no clear top-down tasking process. One subject observed that leaders need to do a better job of prioritizing analysts’ work because, even as analysts assemble data to put in databases, create briefings and produce finished intelligence products, they often write on relatively insignificant items. Their workload is usually events-driven. Consum- ers call in to their office and request certain information. The analysts attempt to tell the consumer what they don’t already know. Their work may not always reflect the consumer’s interests. To counter this tendency, some subjects maintain, managers need to incorporate a perspective on vital national interests into their analysts’ tasks.

Another subject identified two ways in which analysis tasks are self-generated. In one, the analyst submits a formal proposal to do general research and, if approved, then does the work. In the second, the analyst submits a more structured proposal to study a region of the world by looking forward perhaps about 10 years. If approved, the analyst in this case identifies problems, examines them, and synthesizes them into most important issues. A problem and its attendant issues could become the basis for a National Intelligence Estimate (NIE).4

The self-generated analysis may have certain implications. First, the analysts may be doing analysis that serves the personal interest of the analyst, but does not meet a military leader’s or other policymaker’s requirements. Second, the analyst may be focusing attention on analysis projects that are the easiest or quickest to publish, satisfying a need of the analysis office, rather than producing analysis that is most beneficial to the Intelligence Community. Third, the analyst may focus on easier problems that reside in the short-term instead of tougher future-oriented problems in order to give the appearance of being more productive by producing more intelligence.

Leader and Analyst Interaction

Initiative. Most interview subjects agreed that leaders from inside or outside their office do not interfere with the analyst’s work. They provide oversight only when the ana- lyst asks for it because analysts do not like to be micro-managed. Each manager considers his or her analysts subject matter experts. They may ask the analyst, “are you finished yet?” or “do you need help on A/B/C?” For additional support, analysts often appeal to other experts as needed.

The SIO for an office gets involved only if an analyst brings an issue or question to him. Sometimes, the SIO walks around the office, talks to the analysts, and looks for ways to help obtain certain information. The SIO gets formally involved in the analytic process, however, when the analyst has created the final product. Usually, it has been coordinated through the analyst’s peers before it gets to the SIO. The SIO makes any substantive refor- matting corrections before the final product can leave the office.

Directive. Usually during a crisis situation, there is face-to-face interaction between an office leader and analysts from the moment the requirement appears until it is met. One subject agreed that there should be a constant exchange of information between the leader and the analyst even during routine analysis. Some leaders are better at it than others. The lower level managers in their office worked directly with the analysts.

Usually, they assign an analyst a project, discuss with them how to do it, and continually ask them how they are doing. Another subject described the production process this way: the analyst produces, the boss does a sanity-check review; the analyst coordinates his or her work with other staffs, and they suggest changes.

Feedback

Initiative. The interaction between leadership and the analyst during feedback sessions varies from office to office. It usually follows a similar pattern: the analyst produces his or her findings, managers review the work; analysts make the recommended changes, and analysts coordinate the work with appropriate offices. In some cases, the analyst decides who should coordinate on his or her analysis. Competition between offices impedes the coordination process. One subject regrets that neither top-down guidance nor tools exist to support the coordination process.

One subject said all input from leaders and peers comes after the analysis is done. As a result, the usual case is for the assessment in the final product to be conceptually close to the analyst’s own findings. Once the feedback process concludes, analysts are free to post their finished products on the electronic “broadcast” media.

An analyst may also send a copy directly to the consumer, or may inform specific consumers of its URL. As feasible, the product will be “downgraded” to a lower classification level by obscuring sources or methods to enable it to reach a wider audience on different electronic systems.

Directive. After completing the analysis, the analyst will normally present his or her information to a team chief or SIO for coordination. After considering the revisions, the SIO will give the analyst permission to send written findings forward. Because this information may appear unaltered in the U.S. President’s read book, an SIO does not hesitate to “wordsmith. This feedback may be frustrating and humiliating for the analyst. The SIO may direct the analyst to fix the logical development of an argument or to present more and better data. Some analysts will simply rework their product and resubmit it to the SIO. Others, however, will discuss the comments with the SIO. The SIO typically prefers the latter method because the analyst will usually understand the comments better and will learn more from the experience. Almost all of the products go to their consumers; only rarely will a product be killed within the analyst’s office.

Comments

Good leadership demands high quality products to inform warfighters and policymak- ers. Leaders need analysis to answer tough questions. At the same time, managers need to keep in mind important questions about the analysis process itself:

• Are military leaders able to do their job better because of their analyst’s product?
• Are the benefits of doing the analysis outweighing the costs?
• Are we doing risk analysis? That is, are we attempting to understand the balance between threat and vulnerabilities by examining local trade-offs?

Good leadership ensures a value-added coordination process. Analysts often find the coordination process tough. One subject said that before the first draft, the analyst’s job was the best job. After the first draft, however, the analyst’s job was difficult. Another subject called into question the coordination process because of the low quality of finished products posted in the database. Another subject said the coordination process is a hum- bling experience for the analyst because it is tough to get back your paper marked up with corrections. One subject said a co-worker tested the system. He worked really hard on one project and sent it forward. Next time, he slapped something together and sent it forward.

He noticed that it didn’t make any difference either way. He was devastated. There was no motivation to excel. From these testimonials, one could conclude that unsystematized (non-standard) coordination may hinder the work of analysts.

Good leadership theoretically removes such barriers to effective use of analytical resources by providing leading-edge administrative and computer support; quick-turn- around graphics support for urgent tasks; and hassle-free travel support. One subject com- pared DIA administrative support unfavorably with what he grew to expect when he worked for the Central Intelligence Agency. He could drop something off at 5PM for graphics support and pick it up when he got to work in the morning. When he traveled, his proactive travel office would remove some of the hassles associated with traveling. He would tell someone in the travel office where he needed to go. They would call him back a short time later with all the information he needed to know about the trip. Then, when he got back from his trip, he told them what he did and they reimbursed him. There were no required travel reports and no receipts. In fact, they told him to get rid of all his receipts. In case he was caught; it would be better for no one else to know where he had been!

Summary

Leadership is a critical aspect in determining the success of the intelligence analysis process. Leadership demands high-quality products, ensures a value-added coordination process, removes technical or administrative barriers, and intervenes when needed. Some leaders assign tasks directly to the analysts. Others rely on the Senior Intelligence Officer (SIO) to assign tasks or rely on the initiative of the analysts themselves to seek out the right problems and analyze them. Leadership tends to provide little or no interaction dur- ing the analysis process except if and when the analyst seeks it. Analysts do not like to be micro-managed. In a crisis situation however, the leader takes a more direct and hands-on approach. Although feedback varied from office to office, ideally the analyst should seek short, regular feedback on an informal basis with the office leader.

STRATEGIC PLANNING

Baldrige criteria compel an examination of how an analysis office prepares for future challenges, records its institutional history, and uses academic or professional sources. Within the category of future challenges, a distinction may be made between capability and issue challenges. Future capability challenges deal with the people and process issues. Future issue challenges, on the other hand, are country- or target-dependent. These challenges deal specifically with the emerging threat to U.S. interests. Although none of the offices has a written, formal strategic plan, in most, leaders claimed to have thought about strategic planning issues and were working toward a future vision. Their daily mis- sion though was to “get the right information to the right people at the right time.”

Future Challenges

Capability Challenges. Organizational capability centers on people and processes. For those subjects who indicated that they have thought about the future in strategic plan- ning terms, leaders recognized that the commitment of their people to an organizational ideal is a necessary ingredient for success. In practice, then, organizational capabilities must support strategic planning. Analysis organizations need to consider their staff capabilities and ensure that staffs have the necessary knowledge, skills, and tools for success. Our study subjects generally favored a team approach for intelligence analysis. Leaders wanted analysts to consult with experts and coordinate with stakeholders before sending the final product to consumers.

Since none of the subjects claimed to use a formal strategic planning process, our study team could not discuss the process with them in detail. A generic strategic planning process, however, should include (1) doing advanced planning, (2) performing an environmental scan based on external and internal information, (3) setting a strategic direction using the vision statement, (4) translating the strategic direction into action via an implementation plan, and (5) making a performance evaluation. A performance evaluation or assessment should include (1) defining goals, (2) describing success, (3) setting targets, (4) measuring performance, and (5) adjusting goals based on performance. The DIA has a formal process to do strategic planning, but it was not clear to the study team how the organizational strategic planning process might be linked to the lower levels of the organization.

Institutional History

Most subjects said they did not have a written institutional history. There exist no for- mal analyses of their office’s strengths or weaknesses. The office does rely, then, on retaining experienced individuals for its corporate memory.6 Some offices maintain briefings about their office’s operations as a surrogate for a written institutional history. Anyone assigned to an analytic unit is expected to set about building connections and learning operations and systems. Only one subject said they had formed a history office to record significant events for their office, beginning in the late 1990s. Even this admirable initiative, however, with its focus on events, rather that process itself, would fail to capture the nuances of truly useful and recoverable institutional history.7

Academic or Professional Sources

In the interviews, office leaders tended to support academic training, attendance at professional conferences and conventions as experiences useful to the development of professional expertise. However, there exist at the office level no guides or plans for such professionalization opportunities.

“Factions,” an analytical tool developed at CIA for systematic analysis of political instability, is a good example of research and development (R&D) work done in collaboration with members of the academic community.10 In the 1980s there was no particular pressure or encouragement across the Community to use structured analytic methods. Analysts gathered the facts, looked at them in their uniquely “logical” way, and came to some conclusions. Some DIA analysts use structured techniques to do intelligence analysis. The subjects we spoke to, however, described their analytical process as gathering facts, examining them logically and drawing some conclusions. More emphasis on structured techniques at DIA may be appropriate.

Summary

Although none of our subjects had a written, formal strategic plan, most of them had thought about strategic planning issues and were working toward a future common vision among the personnel in their office. Office leadership is often engaged with senior leader- ship about future issues. Managers know the direction they want to go in and recognize the need to support a strategic plan. Furthermore, a focus on important future issues such as which country is most problematic clearly makes an analysis shop proactive and saves time in crisis situations.

Theoretically, the strategic management process includes (1) advanced planning; (2) an environmental scan that includes a review of internal and external information; (3) setting a strategic direction; (4) translating the strategic direction into action; and (5) evaluating performance. Consumers, stakeholders, and subjects influence the process by stating their requirements, receiving products and services, and providing feedback. Information and analysis influence each step of the process by providing management with important information, data, and feedback.

Most of our subjects relied on the corporate memory of senior analysts instead of maintaining a recorded institutional history. As senior analysts approach retirement age and leave government service, analysis offices will lose a great deal of expertise. Some offices maintained a written office history or had on file significant briefings about their office’s operations. All of our subjects used academic or professional sources to develop their capabilities as needed. They also knew what commercial tools were available to help them to do more for their consumers.

CONSUMER FOCUS

The Baldrige criteria applicable to a consumer focus suit the DIA case very well. As an information organization, this agency is obligated, through its analysis units, to discern consumers’ requirements. A key question for this study is whether relationships with consumers tend to be ad hoc or the result of a deliberate strategy.

Consumer Requirements

Consumers make their requirements known through a formal production database, telephone calls, individual initiatives, by e-mail, and through personal visits. They make their initial requirements known primarily on an ad hoc basis. The most meaningful requirements are informal because the analyst gets to talk directly to the consumer. Subsequently, the analyst can help the consumer refine the formal request in the Community’s intelligence production database. Some consumers make telephone calls to analysts that require an answer in two hours or less. This can make the analyst’s job very stressful. Consumers use emails, on the other hand, as a secondary contact medium and mostly for follow-up activities. On occasion, a consumer will come to the analyst’s office to maximize the opportunity to gain a very specific pre-deployment, contextual understanding of target information. One subject noted that although most analytic tasks are management-driven, there are also many self-generated products. The latter could have a positive effect, especially in promoting analyst self-esteem, so long as the self-generated products are of interest to the entire Community.

Consumer and Analyst Interaction

The different frames of reference that one might expect to characterize any interaction between analysts and consumers are on vivid display in anecdotes related by most inter- view subjects. Much of the time, consumers appear to shy away from direct interaction with an analyst. On occasion, the consumer’s rank is too far above that of the analyst to allow an analyst to directly approach an individual consumer, especially in an organization like DIA, often still characterized by a culture that finds meaning in and derives behavioral clues from a hierarchy of authority. Occasionally, however, informal interaction does occur, to include face-to-face conversations or telephone calls to enhance clarity of requirements. One subject noted that his contact with consumers is mainly personal and direct. This analyst works with consumers to build training products that look as much as possible like the real world. Toward that end, an analyst might attend survival school as a student, rather than as a VIP, in order to absorb and then re-create more realistically some part of that training environment.

Production tasking usually passes through bureaucratic levels at both the producer and consumer ends, which reinforces pre-existing tendencies keeping analysts and consumers away from each other. Suddenly, an answer to a formal or infor- mal request simply appears. A general absence of feedback from consumers characterizes the intelligence production process in the offices included in this study. Any interaction with consumers that does occur happens on an ad hoc basis.

Because most products are published electronically on systems such as Intelink, it is possible for the agency to track in considerable detail the organizations and even particular analysts that access and explore particular documents. In practice, however, this is rarely done at the level of detail that allows an analyst to be certain who the consumer really may be. Even if one accepts the concept that within the Department of Defense, information becomes actionable intelligence mainly through the briefing process, it is wise to remember that cognitive processes do depend ultimately on purposeful information-seeking activity, which can hypothetically be tracked in such detail as to inform intelligence producers exactly who their true consumers really are.

Feedback

Analysts provide finished products to their consumers in a variety of ways: through the formal production requirements management database, in hard copy, posted to a website, or by email. Sometimes, depending on the size of the electronic file, analysts post a prod- uct on a “hidden” URL. Then, they call a known consumer to let them know where they have posted the product. Once the consumer has the product, the analyst can delete it from the system. Feedback appears in many forms. In one office, informal feedback pre- dominates through personal contact with the consumers. Another office obtains consumer feedback either via the office web site’s “feedback” button, or via a verbal response from the consumer that the product met or did not meet the need. Hard copies automatically include a publication office’s formal feedback form. All products do not get this form. One subject said a team in their division uses a questionnaire to find out if their consum- ers liked their product. One office maintains a sizable travel budget so analysts may travel to their consumer’s office regularly.

Marketing a product can attain great importance. An analyst typically does not want his or her finished product to sit on the table in the J-2’s office. The usual desire is for the information to reach much lower levels, where the material remains actionable information, which may only later become actionable intelligence. They do this by using different means of communication.

The analyst who writes the product can market it by mailing hard copies, by telling people where to find it on Intelink or by responding to specific requests by sending email attachments or making a personal phone call.

Summary

Building relationships with consumers tends to be more ad hoc than the result of a deliberate strategy. Although consumer feedback varies from office to office, most of the subjects said that “working-level” consumers support their intelligence analysis process. Some got feedback from an office web site.

INFORMATION AND ANALYSIS

Information and analysis criteria address the question of how well DIA offices are meeting challenges in this core functional area, and reveal whether a meaningful approach is being taken by the organization for measuring product quality. Representatives of most of the analysis offices indicated that they are meeting the requirements embodied in these criteria. Informal means for measuring analysis performance included counting products, meeting production schedules, regularly updating databases, answering key consumer-oriented questions, and gauging performance based on “perception.’’

Meeting Current Analytic Challenges

Specific indicators of success that were quickly asserted by interview subjects included the following:

1. Positive comments or lack of negative comments from consumers;
2. Evidence of consumer use of intelligence products;
3. Knowledge of the number of terrorist attacks averted;
4. A downward trend in the number of requests for information (RFI);
5. Evidence of improvement of analysts’ capabilities based on accumulated experience.

Not surprisingly, a tendency exists for a dissatisfied consumer to be more vocal. A lack of negative comments from consumers is therefore considered a positive indicator of success. Consumers never want DIA to stop sending them daily intelligence products. Even if routine DIA assessments are only placeholders, the consumer still expresses a demand for the products because one article could change everything.

Measurement of actionability is never easy because often no recordable action is required, and the producer faces the difficult task of determining whether the product has simply influenced the way a consumer views an important issue. No one from among the offices under study here claims to have an answer to this difficult issue.

A different type of indicator of success is the level of improvement in analyst savvy, given the accumulation of experience. An office’s SIO is in a good position to determine whether the analyst is making improvement. Comments from the SIOs contacted in the course of this study indicate that they examine the analyst’s work with a critical eye to determine whether the analyst is growing in the job. If it is determined that the analyst is in fact getting better at doing analysis, then the SIO rewards the analyst with more work and more challenging jobs. The SIO also rewards the analysts with an appropriate performance review. If however, the analyst does not show improvement in the current task, then the SIO working with the analyst and manager will attempt to diagnose the problem and provide appropriate training to improve proficiency in doing analysis.

A Standard Approach to Measuring Quality

No subject claimed to have plans to implement any kind of standard approach to mea- sure analytic quality. One subject said he was not opposed to some standard for measuring quality. He has not, however, seen a useful measure for it. In another office, analysts “just publish in order to keep their database filled.” Little or no attention is given to quality. Either the product meets the requirement or it doesn’t.

Comments

All of the subjects said they had no formal performance measurements, but neither did they employ informal means. To justify this situation, one subject said intelligence analysis is purely subjective. There are no numbers to count. If analysts meet their production schedule, then they are considered to be successful. If analysts get all of their work done, then they have met the minimum success requirement.

Managers of intelligence analysis are ambivalent about managing for results. Though interested in demonstrating success, they are uneasy about being held accountable for poor organizational performance. There are severe problems in measuring performance in practice: (1) problems in getting agreement on what good performance means for intelligence analysis; (2) intelligence analysis goals are unreasonable compared with available resources; (3) key performance indicators are unavailable or too costly to collect; and (4) management is unwilling to make necessary changes to improve intelligence analysis.

Office leaders appear to be most interested in how many products the analyst finished, even though this crude measure is not supplemented with any method for independently weighting the relative value of those products.

One subject noted that his office does track certain indicators of quality, such as whether the consumer liked the product. A subsidiary indicator lies in evidence of whether the consumer responded to the product. Did anyone else want a copy of the product?

One subject prefers using the term “gauge” instead of “measure” because “measure” assumes some sort of reasoned technique. Often, there is a bit of a “gut check” in any office with respect to how well managers perceive their office to be meeting its mission. Most of the measurement comes in the form of ad hoc consumer feedback. A well-executed mission, like a successful noncombatant evacuation operation (NEO) for instance, tells the office that they are doing a good job.

Summary

Informal means for measuring analysis performance included counting products, meeting production schedule, updating databases, answering key consumer-oriented questions, and gauging performance based on perception. Two common ways for analysts to know if they are meeting their consumer’s needs are by a lack of negative feedback and by an increase in product use. Our subjects do not keep track of forecast accuracy because it is too difficult to track. Instead analysts individually keep a mental checklist on any forecasts they make and how the situation developed over time. Although none of our subjects planned to implement any standard approach to measure analytic quality in the near future, they continue to examine ways to do analysis more effectively without destroying their organization’s “can do” ethos.

PEOPLE AND RESOURCES

The people and resources category offers criteria to examine the knowledge, skills, and experiences of the analysts and the availability of analysts’ tools. Leaders can improve the quality of a product by promoting greater expertise among the intelligence analysts and by inducing analysts to take advantage of information found in open sources. Study subjects said they need analysts who think across interdisciplinary boundaries and they prefer analysts who have military operations experience.

Knowledge, Skills, and Experience

According to representatives of the five offices in DIA, analysts bring with them or develop certain skills to help them perform their analysis and at the same time they must develop an awareness of how certain analytic pitfalls may influence their work. The ana- lytic skills include: (1) technical or topical expertise; (2) knowledge of the target, sources, and analytic techniques; (3) ability to search and organize data; (4) ability to synthesize data or use inductive reasoning; and (5) ability to express ideas. The analytic pitfalls include: (1) personal biases or inexperience; (2) lack of data or access to data; (3) asking the wrong question;14 (4) misunderstanding data; (5) flaws in logic; (6) no review or evaluation process; (7) denial and deception by the target; (8) politicization or non-objectivity; (9) groupthink; and (10) mirror imaging.

One problem frequently voiced is that new hires top out in four years at the GS-13 level, then they have to move out of the office to continue advancing in their career. A consensus exists among office managers that any office struggles to maintain their analysis capability as people move to better jobs for promotion. Having people who are career intelligence officers, especially if they also possess a military operations background, is commonly cited as a key to a quality staff.

PROCESS MANAGEMENT

Process management criteria apply to information handling, intelligence analysis, and collaboration. Processes associated with the dissemination of information and intelligence are beyond the scope of this study, but in this area, the advent of online publication and the loosening of “need to know” rules governing access to sensitive information have relieved some classic dissemination bottlenecks.15 Good analysts are commonly charac- terized by DIA officials as “focused freethinkers.” Other commonly acknowledged max- ims are that the analysis process is a highly competitive process, and that analysts rush to be first to publish vital intelligence findings. At the same time, analysts must work together by collaborating with other analysts to ensure accuracy.

None of our subjects said they observed a formal or structured process in doing intelligence analysis. One subject summed up this approach as simply “sitting down and going to work.” A second subject claimed to use an informal and semi-structured, term research process. In this mode, the analyst first has an idea related to their “account.” They discuss with their team chief the problem they are interested in solving. If they get permission, then they do the research. Another subject added that, likewise, he does not have a “hard- wired” approach, yet applies a routine to the process. One subject said his office has a process that only remains in the mind of each of the office’s analysts. It is, in the estimation of this subject, an application of the scientific method or at least a disciplined thought process. In this scheme, managers give the analyst a problem, and the analyst does research to determine what they have and what they need. If there is time, he or she will request collection of certain data. The analyst reviews the data and answers the question.

Information Handling

Analysts do their own fieldwork to gather any information beyond that from routinely available, written sources. One SIO emphasized that his analysts “gather” information, they do not “collect” it. For example, they do not have diplomatic immunity when they are visiting other countries; they must get specific permission from the country to be there.

Analysts report that they try to lay down the steps of their analysis in a logical way. For example, they use several paths to move from general information to more specific infor- mation. Some analysts write their papers as if they were doing research in a graduate school. Others write products that look like newspaper articles. Analysts tend to protect their sources in a manner similar to the way newspaper reporters protect the identity of their sources.

Intelligence Analysis Process

The analysis process includes (1) problem identification; (2) data gathering; (3) analy- sis, synthesis, collaboration, and examination of alternative propositions using all sources of data; and (4) direct support to consumers with briefings and publications. In the end, analysts make an assessment of what the data probably mean or what the data could mean. General Colin Powell (USA, Ret.) sums up his expectations as a consumer of intelligence in this way: “Tell me what you know, what you don’t know, and what you think— in that order.” In this oft-quoted dictum, he suggests that analysts can best convey what is known about the problem, evaluate the completeness of their knowledge, and interpret its meaning. It remains unclear whether the same guidance is as suitable for written products as for oral briefings. In the course of analysis, whether presented in written or oral format, it is important for the analyst to answer for himself certain key questions:

1. What do you know about the issue?
   2. What is a logical, sequential way to present the facts?
   3. What conclusions or implications can be drawn from these facts?

Analysts generally find it distasteful to coordinate their products with members outside of their team. Nonetheless, when the “corporate” analysis process works as it can, all of the teams contribute to the analyst’s final product and, as a result, the analyst produces a superior product. Managers do want analysts to exchange information with each other. They have a distaste for competition among analysts. The analysts’ rationale for resisting widespread coordination is that they consider themselves subject matter experts.

Collaboration and Competition

Collaboration. If analysts choose not to collaborate with other Community experts, then the SIO may force them to do so during the coordination and review process. Manag- ers want analysts to verify the validity of all of their information in a manner similar to the way that they feel compelled to check and verify human-resource information.

Competition. Collaboration among intelligence analysts is always a problem because analysts, like many school children, tend to be competitive with each other. The acknowl- edged lack of coordination within DIA is not a new or exceptional phenomenon. New analysts may not know who to talk to on a particular subject or where to go to get useful information. A more senior mentor or manager can usually provide the analysts with that kind of information, but the exchange of information between analysts doesn’t normally happen in the unsupervised work environment. Analysts are very competitive within the office and especially with other agencies in the IC. There is a strong desire to publish first, a tendency that inhibits full disclosure to competing analysts.

Summary

Good analysts are “focused freethinkers.” Nevertheless, “out-of-the-box”-type thinking needs to be tempered by using an informal, disciplined thought process and collaborating with other analysts The scientific method is a streamlined process that ensures the analyst doesn’t wander too far

Definition of Success

All of our subjects agreed that success is a difficult concept to define. Our subjects did, however, agree that it might be possible. First, success is positive feedback from consum- ers. If they do not provide explicit positive feedback, then the analysis shop can look for indirect indicators.

Second, an intelligence analysis success occurs when a unit that is known to depend on DIA products completes its military operation successfully.

Third, success is obtaining an expanding budget and a growing workforce. These developments indicate that the topic focus is perceived as interesting by agency leader- ship. One subject labeled this type of success as “bureaucratic success.”

Fourth, success to some analysts means that evidence indicates that they are able to look at things in the open sources and link them in ways others could not envision. The analyst would observe a situation, notice a trend, and take a position on a tough issue.

One possible way to measure success is to count the number of electronic hits on a particular document on a database. If consumers are asking questions, then analysts can feel they are being helpful. An informal summary metric, then, is the measure of information flow out of the office.

Summary

Intelligence analysis supports DoD organizations and warfighter operations. Mission success is positive feedback from consumers or a successful completion of military operations. Bureaucratic success, on the other hand, results in more money to spend and more analysts to do the job. Analysts consider themselves successful if their consumers changed their behavior based on the information they provided to them.

STRATEGIES FOR SUCCESSFUL INTELLIGENCE ANALYSIS

“What is good intelligence analysis?”

Good or useful intelligence analysis may be best defined by its opposite. Bad intelligence does not specify what the threat is or how it will manifest itself. For example, the analyst may conclude there will be a 50 percent chance of a chemical weapons attack against the U.S. in the next 10 years. That conclusion is not meaningful or helpful to planners and users of intelligence.17 In another example, the analyst may conclude that all ports are dangerous to U.S. ships. This analysis is impractical because ships have to use ports occasionally.

In government work, the applied nature of intelligence production would seem to offer the opportunity to develop and apply consistent measures of productivity and quality, just because the products are indeed used by specific consumers. If this is true, then there is room to bring “operations research” into play as well as the Baldrige criteria, to encour- age the development and collection of carefully codified, surrogate measures of productivity and quality. Successful use of these tools does not remove from the manager the task of decisionmaking but rather it requires of him or her rather different kinds of decisions. In other words, using operations research tools provides managers with extra insight into their particular subject and hence leads them into much more difficult but fruitful areas. Consultants with scholarly backgrounds could come into the intelligence analysis work environment and make vast improvement to the intelligence analysis capability using such tools to assist them, in the author’s opinion.

CORE COMPETENCIES FOR INTELLIGENCE ANALYSIS AT THE NATIONAL SECURITY AGENCY David T. Moore and Lisa Krizan

Seekers of Wisdom first need sound intelligence.

— Heraclitus1

What makes an intelligence analyst successful in the profession? This question strikes at the heart of the National Foreign Intelligence Community’s mission to provide actionable information to national leaders and decisionmakers.

What is a qualified intelligence analyst?

In this paper the authors propose a set of functional core competencies for intelligence analysis, shown in the figure below, which provides a starting point for answering fundamental questions about the nature of ideal intelligence professionals, and how analysts who share these ideals can go about doing their work. Keeping in mind the complex nature of the threats to U.S. national security, we argue that the strategy for deploying intelligence analysts and for carrying out intelligence production must become more rigorous to keep pace with 21st Century foes, and to defeat them.

Functional core competencies for intelligence analysis

The authors began exploring the art and science of intelligence analysis at their agency as part of a corporate initiative to add rigor to its analytic practice.

Sherman Kent, who helped shape the national peacetime intelligence community, argues that intelligence requires its own literature. According to Kent, a key purpose of this literature is to advance the discipline of intelligence. Kent believed “[as] long as this discipline lacks a literature, its method, its vocabulary, its body of doctrine, and even its fundamental theory run the risk of never reaching full maturity.”6 Through the publication of articles on analysis and subsequent discussion, “original synthesis of all that has gone before” occurs.7 In keeping with Kent’s mandate to develop an intelligence literature that provokes discussion and further methodological development, we seek comment and fur- ther discussion among scholars of intelligence studies.

DEFINITIONS AND CONTEXT

Intelligence refers to information that meets the stated or understood needs of policymakers…. All intelligence is information; not all information is intelligence.

— Mark Lowenthal8

Intelligence is timely, actionable information that helps policymakers, decisionmakers, and military leaders perform their national security functions. The intelligence business itself depends on professional competencies, what John Gannon, former Chairman of the National Intelligence Council, refers to as “skills and expertise.” He notes that “this means people—people in whom we will need to invest more to deal with the array of complex challenges we face over the next generation.”

Ultimately, analysis leads to synthesis and effective persuasion, or, less pointedly, estimation.10 It does so by breaking down large problems into a number of smaller ones, involving “close examination of related items of information to determine the extent to which they confirm, supplement, or contradict each other and thus to establish probabilities and relationships.”11

Since the advent of the Information Age, “[collecting] information is less of a problem and verifying is more of one.”12 Thus the role of analysis becomes more vital as the supply of information available to consumers from every type of source, proven and unproven, multiplies exponentially. Intelligence analysts are more than merely another information source, more than collectors and couriers of information to consumers. Further,

[the] images that are sometimes evoked of policymakers surfing the Net themselves, in direct touch with their own information sources, are very misleading. Most of the time, as [policymakers’] access to information multiplies, their need for processing, if not analysis, will go up. If collection is easier, selection will be harder.13

At its best, the results of intelligence analysis provide just the right information permit- ting national leaders “to make wise decisions—all presented with accuracy, timeliness, and clarity.”14 The intelligence provided must “contain hard-hitting, focused analysis relevant to current policy issues….Therefore, analysis of raw information has the most impact on the decisionmaker and [therefore] producing high-quality analytical product should be the highest priority for intelligence agencies.”

Treverton adds that intelligence must anticipate the needs of policy. “By the time policy knows what it needs to know, it is usually too late for intelligence to respond by developing new sources or cranking up its analytic capacity.”

A former policymaker himself, he asserts that intelligence is useful to policy at three stages during the life of an issue:

• If the policymakers are prescient, when the issue is just beginning; however there is likely to be little intelligence on the issue at that point.
• When the issue is “ripe for decision.” Here policymakers want intelligence that permits alternatives to be considered; however, intelligence often is only able to provide back- ground information necessary for understanding the issue.
• When the policymakers have made up their minds on the issue, but only if intelligence supports their view. They will be uninterested or even hostile when it does not support their view.21

These limitations notwithstanding, Treverton suggests that policymakers can and should establish a symbiotic relationship with the intelligence analysts who advise them:

[If] you call them in, face to face, they will understand how much you know, and you’ll have a chance to calibrate them. You’ll learn more in fifteen minutes than you’d have imagined. And you’ll also begin to target those analysts to your concerns and your sense of the issue.22

Similarly, the analyst has responsibilities to the policymaker. In commenting on this rela- tionship, Sherman Kent asserts

[intelligence] performs a service function. Its job is to see that the doers are generally well informed; its job is to stand behind them with a book opened at the right page to call their attention to the stubborn fact they may be neglecting, and—at their request—to analyze alternative courses without indicating choice.23

In Kent’s view, the intelligence analyst is required to ensure, tenaciously, that policy- makers view those “right” pages, even when they may not wish to do so.

MEASURING SUCCESS IN INTELLIGENCE ANALYSIS

Intelligence must be measured to be valued, so let us take the initiative and ask our management, [and] the users, to evaluate us and our products.

— Jan P. Herring24

Any observer can expect that a successful intelligence analyst will have certain personal characteristics that tend to foster dedication to the work and quality of results.

Intelligence Process

Successful intelligence analysis is a holistic process involving both “art” and “science.” Intuitive abilities, inherent aptitudes, rigorously applied skills, and acquired knowledge together enable analysts to work problems in a multidimensional manner, thereby avoiding the pitfalls of both scientism and adventurism. The former occurs when scientific methodology is excessively relied upon to reveal the “truth”; the latter occurs when “inspiration [is] unsupported by rigorous analysis.”26

A vital contributor to the analytic process is a spirit of competition, both within an intelligence-producing agency and especially between intelligence agencies. There is a tendency for analysts working together to develop a common mindset. This trap occurs typically when analysts fail to question their assumptions about their role in the intelligence process and about the target. The Council on Foreign Relations’ independent task force on the future of U.S. intelligence recommends that “competitive or redundant analy- sis be encouraged” precisely for these reasons.27

Successful analysis adds value—to the information itself, to institutional knowledge, to fellow intelligence professionals, to the process, and to the institution or unit itself—in terms of reputation and the degree to which good analytic practices endure despite changes in target, consumer, and personnel. Successful analysts are those whose work, whenever possible goes to the level of making judgments or estimating.

What role does management play in ensuring analytic success? First and foremost, management effectively uses financial and political capital to ensure that analysts have access to consumers, and to the resources they require to answer those consumers’ intelli- gence needs. This includes the organization of the work itself, allocation of materiel and personnel, and coordination with consumers and other producers. When management is successful, the analyst has the necessary tools and the correct information for successful intelligence analysis. Good morale among analytic personnel becomes an indicator of effective management. A good understanding of the unit’s mission and the analysts’ own satisfaction with his or her performance naturally produces a feeling of empowerment and a belief that the organization places great value on analytic talent.

Intelligence Product

The products of successful analysis convey intelligence that meets or anticipates the consumer’s needs; these products reveal analytic conclusions, not the methods used to derive them. Intelligence products are successful if they arm the decisionmaker, policy- maker or military leader with the information and context—the answers—needed to win on his or her playing field.

Readiness: Intelligence systems must be responsive to existing and contingent intelligence requirements of consumers at all levels.
Timeliness: Intelligence must be delivered while the content is still actionable under the con- sumer’s circumstances.

Accuracy: All sources and data must be evaluated for the possibility of technical error, misper- ception, and hostile efforts to mislead.

Objectivity: All judgments must be evaluated for the possibility of deliberate distortions and manipulations due to self-interest.

Usability: All intelligence output must be in a form that facilitates ready comprehension and immediate application. Intelligence products must be compatible with the consumer’s capabilities for receiving, manipulating, protecting, and storing the product.

Relevance: Information must be selected and organized for its applicability to a consumer’s requirements, with potential consequences and significance of the information made explicit to the consumer’s circumstances.

Measures of success for intelligence products28

Six “underlying ideas or core values” for intelligence analysis, identified by William Brei for operational-level intelligence, and shown in the figure above, establish the analyst’s “essential work processes.”29 Since they are defined in terms of the consumer, they also can be used as a checklist to rate the quality of products provided to the consumer.

William S. Brei, Captain, USAF, Getting Intelligence Right: The Power of Logical Procedure, Occasional Paper Number Two (Washington DC: Joint Military Intelligence College, 1996), 6.

• Was the reported intelligence accurate? (Accuracy)
• Are there any distortions in the reported judgments? (Objectivity)
• Is the reported intelligence actionable? Does it facilitate ready comprehension? (Usability)
• Does it support the consumer’s mission? Is it applicable to the consumer’s requirements? Has its significance been made explicit? (Relevance)

Brei asserts that accurate data provide the foundation for subsequent objective judgments, and the expression of objective judgments in a usable form provides much of the basis of a relevant product. Thus, unverified data cannot only cost an intelligence product its Accuracy, but also damage its Relevance to the customer.32

Brei’s principles provide a means for evaluating a given intelligence product based on the meaning it conveys and the value of that intelligence to the consumer. His approach, when combined with an “insider’s” view of the intelligence production process, analytic methods and personnel management practices, makes a comprehensive evaluation of intelligence analysis appear possible

CHARACTERISTICS OF SUCCESSFUL INTELLIGENCE ANALYSTS

A sophisticated intelligence analyst is one who is steeped in the history and culture of a region, has lifelong interest in the area, and approaches the study of the region as a professional responsibility, and probably as an avocation as well.

— Ronald D. Garst and Max L. Gross34

Who are the most successful intelligence analysts? What makes them successful? In setting forth the functional core competencies for successful intelligence analysis we observe there are characteristics which, while not necessary for successful intelligence analysis per se, do seem to be associated with analysts considered to be the most success- ful at their trade.35

Probably the most indispensable characteristics of successful intelligence analysts are high self-motivation and insatiable curiosity. Analysts want to know everything they can about the objects under their scrutiny. Reading and observing voraciously, they ferret out every possible piece of information and demonstrate a sense of wonder about what they discover. As new fragments appear, novel connections are discovered between the new and older information as a result of intense concentration leading to epiphanous moments of “aha” thinking. The most successful analysts tend to enjoy their work—“It’s play, not work.” Indeed, they often will stay late at the office to pursue a thorny problem or an engaging line of reasoning.

Employee orientation programs that acknowledge these characteristics may be most successful in initiating new employees into the analytic culture. When personal characteristics are embodied in compelling “war stories” told by mentors and peers, they can reinforce the cultural values of the agency, building corporate loyalty by reinforcing the sense of membership.

ABILITIES REQUIRED FOR INTELLIGENCE ANALYSIS

The competent intelligence analyst must have a unique combination of talents.

— Ronald D. Garst and Max L. Gross37

Abilities arise from aptitudes that can develop from an individual’s innate, natural characteristics or talents. Although aptitudes may largely be determined by a person’s genetic background, they may also be enhanced through training.38

Communicating

Teaming and Collaboration

Teaming and collaboration abilities enhance intelligence analysis, since the analyst’s relationship with consumers, peers, subordinates, and supervisors shapes the intelligence production process. Formalized means of enhancing all these abilities can lead intelligence professionals to considerably greater effectiveness as analysts and leaders of analysts. This is why the Director of Central Intelligence has indicated that collaboration is a cornerstone of strategic intelligence.41 A collaborative environment also minimizes the likelihood of intelligence failures.

We identify four distinct teaming abilities, to show the complexity of the concept. Typically, formal training programs address leadership abilities only in the context of the management function; here, we focus on the analysis process itself.

Influencing: Those with this ability effectively and positively influence superiors, peers, and subordinates in intelligence work. Analysts often need to persuade others that their methods and conclusions are valid, and they often need to leverage additional resources. The ability to influence determines the level of success they will have in these areas.

Leading: Those who are more senior, more skilled, and more successful in intelligence analysis have an obligation to lead, that is, to direct others and serve as role models. The ability to lead involves working with and through others to produce desired business outcomes. Thus, developing leadership abilities enhances the field of intelligence analysis.

Following: Almost every grouping of humans has a leader. Everyone else is a follower. Analysts must enhance their abilities to work within a team, taking direction, and act-ing on it.

Synergizing: Drawing on the other three teaming abilities, players in the intelligence process cooperate to achieve a common goal, the value of which is greater than they could achieve when working alone.

Thinking

As our species designation—sapiens—suggests, the defining attribute of human beings is an unparalleled cognitive ability. We think differently from all other creatures on earth, and we can share those thoughts with one another in ways that no other species even approaches.

— Terence W. Deacon, The Symbolic Species.43

Intelligence analysis is primarily a thinking process; it depends upon cognitive func- tions that evolved in humans long before the appearance of language.44 The personal characteristics of intelligence analysts are manifested in behaviors that reflect thinking and/or the inherent drive to think. Our national survival may depend on having better developed thinking abilities than our opponents.

Information Ordering: This ability involves following previously defined rules or sets of rules to arrange data in a meaningful order. In the context of intelligence analy- sis, this ability allows people, often with the assistance of technology, to arrange infor- mation in ways that permit analysis, synthesis, and extraction of meaning. The arrangement of information according to certain learned rules leads the analyst to make conclusions and disseminate them as intelligence. A danger arises, however, in that such ordering is inherently limiting—the analyst may not look for alternative explanations because the known rules lead to a ready conclusion.

Pattern Recognition: Humans detect patterns and impose patterns on apparently ran- dom entities and events in order to understand them, often doing this without being aware of it. Stellar constellations are examples of imposed patterns, while criminal behavior analysis is an example of pattern detection. Intelligence analysts impose or detect patterns to identify what targets are doing, and thereby to extrapolate what they will do in the future. Pattern recognition lets analysts separate “the important from the less important, even the trivial, and to conceptualize a degree of order out of apparent chaos.”45 However, imposing or seeking patterns can introduce bias. Analysts may impose culturally defined patterns on random aggregates rather than recognize inher- ent patterns, thereby misinterpreting the phenomena in question.

Reasoning: The ability to reason is what permits humans to process information and formulate explanations, to assign meaning to observed phenomena. It is by reasoning that analysts transform information into intelligence, in these three ways:

1. Induction: Inductive reasoning combines separate fragments of information, or specific answers to problems, to form general rules or conclusions. For example, using induction, a child learns to associate the color red with heat and heat with pain, and then to generalize these associations to new situations.46 Rigorous induction depends upon demonstrating the validity of causal relationships between observed phenomena, not merely associating them with each other.
2. Deduction: Deductive reasoning applies general rules to specific problems to arrive at conclusions. Analysts begin with a set of rules and use them as a basis for interpreting information. For example, an analyst who follows the nuclear weapons pro- gram of a country might notice that a characteristic series of events preceded the last nuclear weapons test. Upon seeing evidence that those same events are occur- ring again, the analyst might deduce that a second nuclear test is imminent.47 How- ever, this conclusion would be made cautiously, since deduction works best in closed systems such as mathematics, making it of limited use in forecasting human behavior.
3. Abduction: Abductive reasoning describes the thought process that accompanies “insight” or intuition. When the information does not match that expected, the analyst asks “why?,” thereby generating novel hypotheses to explain given evidence that does not readily suggest a familiar explanation. For example, given two ship- ping manifests, one showing oranges and lemons being shipped from Venezuela to Florida, and the other showing carnations being shipped from Delaware to Colombia, abductive reasoning is what enables the analyst to take an analytic leap and ask, “Why is citrus fruit being sent to the worldwide capital of citrus farming, while carnations are being sent to the world’s primary exporter of that product? What is really going on here?” Thus, abduction relies on the analyst’s preparation and experience to suggest possible explanations that must then be tested. Abduction generates new research questions rather than solutions.48

SKILLS REQUIRED FOR INTELLIGENCE ANALYSIS

Any institution that relies on professionals for success and seeks to maintain an authentic learning climate for individual growth must require its members to read (to gain knowledge and insight), research (to learn how to ask good questions and find defensible answers), discuss (to appreciate opposing views and subject their own to rigorous debate), and write (to structure arguments and articulate them clearly and coherently).

Critical Thinking

It is by thinking that analysts transform information into intelligence. Critical think- ing is the cognitive skill applied to make that transformation. Critical thinking can be defined as

[An] intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action….Thinking about [our] thinking while [we’re] thinking in order to make [our] thinking better.50

There is a clear need to educate and train intelligence analysts to use their minds…[Only] by raising their awareness can the intelligence unit be assured that the analysts will avoid the traps in being slave to conformist thought, precedent and imposed cultural values—all enemies of objective analysis.51

An ordered thinking process requires careful judgments or judicious evaluations leading to defensible conclusions that provide an audit trail. When the results of analysis are controversial, subject to alternate interpretations, or possibly wrong, this audit trail can prove essential in defending the process used to reach the conclusions.

Foreign Language Proficiency

Furthermore, foreign language proficiency provides more than just a translation of non-English materials. The structure of a target’s language and that target’s culture are closely related. One well-known theory of this relationship, by Edward Sapir and Benjamin Whorf, posits that “language is a force in its own right and it affects how individuals in a society conceive and perceive reality.”64 Thus concepts essential to understanding the target are communicated in a context that goes beyond simplistic translation.

Context must determine the translation, and an analyst lacking foreign language skills must trust the linguist to correctly understand that context. The expertise required for that understanding might render the linguist a better intelligence analyst than the original analyst. This begs the question: “Is such duplication of personnel affordable?”

Research

Research skills provide discipline and consistency for the creation of value-added intelligence. By providing methodologies for defining the requirement to be answered, as well as methodologies for answering that query, research skills ensure analytic consistency and enable thorough exploration of the issues. Necessary research skills include methods of problem definition that ensure that, in collaboration with the consumer, analysts correctly define or redefine the problem in terms of a “research question,” so as to understand the consumer’s and the analyst’s own objectives.

Information Gathering and Manipulation

Information is the grist for intelligence analysis, and to be successful, analysts must aggressively seek it out. Different information/data manipulation skills are required for the various stages of the intelligence process.

• Collection: This stage involves gathering information from all sources. The intelligence analyst directs the collection process, causing specific resources to be tasked. Related information manipulation skills include selecting and filtering in order to assess whether the information and its sources are of value.
• Monitoring: Reliability of sources and the validity of the information are always in question. Monitoring skills focus on information review, and often may involve analy- sis of descriptors and summaries of that data.
• Organizing: Skillful arrangement, formatting, and maintenance of data for analysis and technical report generation ensure access to materials in a usable format.
• Analysis/Synthesis: Information manipulation skills can point to patterns, relation- ships, anomalies and trends.
• Interpretation: This is the stage in the process where information is transformed into intelligence by cognitive manipulation, that is, assigning meaning to analyzed and syn- thesized information using critical thinking. Computers aid in this step, however, a study of 12 major “analytic” software tools concludes “true analysis will remain a peo- ple function, assisted by computer technology.”68
• Dissemination: Dissemination, except for some graphic products, is now of course mostly electronic. Information preparation and presentation skills allow its transforma- tion and publication, so that the results of analysis appear in usable formats, which may be further tailored by users.
• Coordination: Coordination requires analysts as well as their managers to employ “collegial” skills in the bureaucratic environment; these skills are also needed to avoid diluting the intelligence message down to the “lowest common level of agreement.”
• Evaluation: Internal and intra-community evaluation allows the intelligence to be dis- cussed and placed in larger contexts than that viewed by a single agency. Such collab- oration may also identify the additional intelligence required to clarify issues. Evaluation can become a continuous part of the production process.69

Project/Process Management

Few analysts enjoy the luxury of working full time on only one problem or on one aspect of a particular problem. We distinguish between projects and processes. The former tend to have finite scope and goals whereas the latter are open-ended. Both require planning, imple-

mentation, monitoring, and negotiating skills.70 A project/process plan defines and clarifies what needs to be accomplished; identifies necessary resources; creates a timeline, including milestones; and makes the analyst accountable for successful completion.

KNOWLEDGE REQUIRED FOR INTELLIGENCE ANALYSIS

Without a solid knowledge base concerning the region or issue to which the analyst is assigned, . . . the individual will not even know what questions to ask. That is, the person will not really be qualified to be called an “analyst.”

— Ronald D. Garst and Max L. Gross71

Knowledge consists of familiarities, awareness, or understanding gained through experience or study; it includes both empirical material and that derived by inference or interpretation.72 Depending on the specific target, the knowledge required can vary widely.

Target Knowledge

Doing intelligence analysis in the information age is often like “being driven by some- one with tunnel vision.”73 In the quest to answer a consumer’s questions, the analyst often pushes aside “all the fuzzy stuff that lies around the edge—context, background, history, common knowledge, social resources.”74 Yet, to do so is perilous, for these provide bal- ance and perspective. They offer breadth of vision and ultimately allow analysts to make sense of the information under study. By providing the context into which analysts place their work, fields of study such as anthropology, comparative religion, economics, geography, history, international relations, psychology, and sociology all interact to contribute vital knowledge about the target, which both analysts and consumers need to understand. Changes in the culture, religion, geography, or economic systems (among others) of a tar- get may themselves be subjects of an intelligence requirement.

The following selection of topics exemplifies some non-traditional but essential target knowledge areas required for thorough intelligence analysis.

Culture: Culture can be defined as a group’s values, standards, and beliefs. In turn, culture defines that group. The study of culture reveals the roles of individuals in the community, and how they relate to non-members of the culture. This provides insights into behaviors that are of value in predicting future behavior. This is true when the tar- get is a people or a nation as well as when the target is a specific subgroup or individual member within a culture. Adda Bozeman points out that because political systems are grounded in cultures, “present day international relations are therefore by definition also intercultural relations … [A]nalysts and policymakers in the West would be more successful in their respective callings if they would examine the cultural infra- structures of the nations and political systems they are dealing with.”

Message of Language: The message of language is a part of culture, and while isolating it makes an artificial distinction, we do so to reiterate its importance for intelligence analysis. What languages are utilized, by whom, and in what context, is essential in understanding the target’s culture. For example, much is revealed if members of an insurgent group primarily communicate using the language of the elite members of their culture. Additionally, what the language indicates about class and personal relationships may provide clues to behaviors.

■ Technology: Technology itself can be the subject of study by the intelligence analyst. Someone developing a target may analyze specific technologies and their infrastructure as they pertain to that target. Further, the role of technology within a region, nation, or people is an indicator of behavior. The domains of communications, utilities, transportation, manufacturing, and others, as well as the attitudes of the people to them, are rich sources of study. Technology also can provide insights into sources of information that will be available to the intelligence analyst.

Professional Knowledge

In addition to understanding their targets, intelligence analysts need to know a great deal about the context and nature of the intelligence profession, and the resources available to help them do their job well. Understanding the plans and policies of their own government enables analysts to frame their work in terms of the nation’s strategic and tactical objectives. Intelligence consumers are government officials. Their needs drive analytical process and priorities. Analysts base collection tasking on the imperative to match information sources to consumer needs. These information sources, such as human-source reporting, signals intercepts and documentary research, provide the analyst with raw materials for the creation of intelligence through analysis, synthesis and interpretation.

In addition, analysts need to know what specific sources of information relevant to a particular inquiry are available for exploitation. Knowing which expert sources and subject matter experts can guide the analytic process, or can offer different or additional perspectives, enhances intelligence work. The reliability of these sources is also critical. When different sources provide contradictory information, the reliability of one source versus another may provide insights into which information is accurate; the sources may be open or secret, technical or human.

Finally, others, known and unknown, may be examining similar information for the same or different consumers. Awareness that sources of information, possibly vital information, exist, even though they remain undiscovered or untapped, keeps the analyst constantly seeking out new connections.

IMPLICATIONS FOR THE INTELLIGENCE ANALYSIS WORKFORCE

Returning to our thesis, what makes an intelligence analyst successful? Given that the analyst’s purpose is to create intelligence, success means following an effective process (rigorous analysis, sound management) and creating a quality product (that conveys intelligence and meets the consumer’s needs). To do this requires appropriate abilities, knowledge and personal characteristics for rigorous intelligence analysis and production. Well- honed capabilities to communicate, cooperate and think, coupled with the skills that ensure technical competency, provide the means for intelligence work. Informed, deep knowledge of the issues and their background provides both content and context for analysis. Analysts who are motivated to succeed, to know targets, and to share that knowledge ensure that consumers receive intelligence of the highest caliber.

IMPLICATIONS FOR THE INTELLIGENCE ANALYSIS WORKFORCE

[Much] of the work of the intelligence community is highly specialized and requires exceptional creativity…. It is also safe to say that some of the most pressing analytic skills the community will require are precisely those we cannot even foresee at this time.

— Bruce D. Berkowitz and Allan E. Goodman79

a possible uptick in hiring and rising rates of eligibility for retirement mean that, at the least, the savvy of the analytic population will continue to dwindle at the lower end and retire from the upper end.80 Even an adequately sized analytic workforce, lacking adequate mentoring and training from senior, expert analysts, will leave the Intelligence Community unable to meet security challenges.

According to the Council on Foreign Relations’ independent report on the future of intelligence, “less than a tenth of what the United States spends on intelligence is devoted to analysis; it is the least expensive dimension of intelligence… This country could surely afford to spend more in those areas of analysis where being wrong can have major adverse consequences.”82 Winning the talent war requires smart investment in the hiring, training, and deployment of analysts.

training is of little value unless it can be immediately applied. Thus organizational structures, culture, and processes must be aligned to permit and to reward rigorous analysis. Unless analysts are recognized and appreciated for performing sophisticated analysis, they will not embrace change. Significant recognition for high-level analysis will inspire others to follow, creating a culture that fosters and sustains excellence in tailored intelligence production.

employees transferring into the analytic disciplines from other fields must have the prerequisite abilities and skills for analysis before joining this discipline. The field of intelligence analysis cannot safely be a catchall for employees transferring from downsized career fields.

The Aspin-Brown Commission on the Roles and Capabilities of the United States Intelligence Community identified several additional actions to improve the quality of analysis. These include a minimal prerequisite to visit target countries as part of analytic orientation, rewards for acquiring and maintaining foreign language proficiency, encour- agement to remain within substantive areas of expertise, and periodic rotational assign- ments to consumer agencies.89 Enacted as part of employee training and orientation, these measures can substantially enhance analysts’ target knowledge and skills.

Sensemaking: A Structure for an Intelligence Revolution

By David T. Moore – NATIONAL DEFENSE INTELLIGENCE COLLEGE WASHINGTON, DC

March 2011

FOREWORD

Gregory F. Treverton
Director
RAND Corporation Center for Global Risk and Security

We at NGA used to look for things and know what we were looking for. If we saw a Soviet T-72 tank, we knew we’d find a number of its brethren nearby. Now, though, we’re not looking for things. Instead, we’re looking for activities or transactions. And we don’t know what we’re looking for.

In fancier language, the paradigm of intelligence and intelligence analysis has changed, driven primarily by the shift in targets from the primacy of nation-states to trans-national groups or irregular forces. In the world of the national-state, I and others divided intelligence problems into puzzles and mysteries (or variants of those words).1 Puzzles are those questions that have a definitive answer in principle. How many nuclear missiles the Soviet Union had was a puzzle. So is whether Al Qaeda possesses fissile material. By contrast, mysteries are questions that cannot be answered with certainty. They are future and contingent.

For puzzles, intelligence tried to produce the answer.

For mysteries there was no answer. Instead, analysts sought to frame the mystery by providing a best estimate, along, perhaps, with excursions or scenarios to test the sensitivity of critical factors. If intelligence failed to understand the full picture of Soviet missiles, and puzzle became mystery, it at least knew something about where to look: there was experience and theory about missile building, plus historical experience of Soviet programs. The mystery came with some shape.

However, today’s transnational threats confront us with something more than mysteries. I call these shapeless mysteries-plus “complexities,” borrowing Dave Snowden’s term. They are sometimes called, as Moore notes, “wicked problems” or simply “messes.” The come without history or shape. Large numbers of relatively small actors respond to a shifting set of situational factors. Thus, they do not necessarily repeat in any established pattern and are not amenable to predictive analysis in the same way as mysteries. Those characteristics describe many transnational targets, like terrorists—small groups forming and reforming, seeking to find vulnerabilities, thus adapting constantly, and interacting in ways that may be new.

For complexities, especially, the challenge is to employ sensemaking— the term is from Michigan psychologist, Karl Weick. Exactly how to accomplish sensemaking is a task that still mostly lies before us, which makes this book such an important contribution. Sensemaking departs, as Moore notes, from the postwar tradition of Sherman Kent, in which analysis meant, in the dictionary’s language, “the process of separating something into its constituent elements.” Sensemaking also blurs America’s bright white line between intelligence and policy, for, ideally, the two would try to make sense together, some- times disaggregating events, sometimes aggregating multiple perspectives, always entertaining new hypotheses, all against the recognition that dramatic failure (or success) might occur at any moment.

he [David More] is very careful about classification. That means the visible trails of his practice in his scholarship are sparse, and his cases are mostly familiar ones, albeit ones often spun in new directions.

The new paradigm makes the use of machines and method imperative, letting machines do what they do best—searching large amounts of data, remembering old patterns, and the like—while letting humans use the judgment they alone can apply. Yet the tests by Moore and his colleagues remind us that methods are critical but only if they have been tested. It turns out, for instance, that ACH, analysis of competing hypotheses, a method more frequently used now and one that has been tested, isn’t all that valuable, at least not for analysts beyond the novice level.

COMMENTARY

Anthony Olcott, PhD
Associate, Institute for the Study of Diplomacy Georgetown University

David Moore is right to talk of the need for an intelligence revolution. However, as Lenin learned in the 18 years that passed between publication of The Development of Capitalism in Russia and taking over the Winter Palace, it takes more than a diagnosis and a prescription to make a revolution. Although his is among the best, Moore’s book is also but the latest addition to a groaning shelf of books devoted to intelligence and analytic reform while the companion shelf, for books on how to improve the policy process, sits dusty and all but empty. In that regard, even though Moore’s discussion of the processes of analysis and how the ways we answer questions might be improved is one of the strongest in recent memory, the most valuable part of the book could well be the somewhat smaller amount of attention it devotes to the problem of how we formulate our questions in the first place.

Kendall did not share Kent’s conviction that the job of the analyst was “to stand behind [the policymakers] with the book opened at the right page, to call their attention to the stubborn fact they may be neglecting.”4

Moore has done a deep and convincing job of diagnosing the ills of the IC, and has proposed a rich and promising cure. This, as Hilton points out, is an extended act of cognition. What lies between this book and Moore’s revolution, however, is the need to have others come to the same conclusion— which, as Hilton points out, requires communication, not cognition.

Sixty years ago a small group of analysts—dubbed “Talmudists” for their pains—worked out a complex, sophisticated method of deriving action- able intelligence from the tightly controlled propaganda outlets of the USSR and Mao’s China. This let IC sinologists spot the first signs of the Sino-Soviet split as early as April 1952, and by 1955 Khrushchev had been tagged as the likely winner in the struggle to consolidate power in the Kremlin after Stalin’s death. Those early indicators, however, remained scoffed at and un-acted upon precisely because the methodology—which a colleague in the CIA compared to studying “invisible writing on slugs”6—was too complex and too weird to be easily explained to policymakers—who, in any case, already believed other hypotheses, and had their own “facts.” 7

(6) Richard Shryock, “For An Eclectic Sovietology,” Studies in Intelligence, vol. 8, no. 1 (Win- ter 1964).

COMMENTARY

Emily S. Patterson, PhD Assistant Professor College of Medicine
The Ohio State University

it is an achievable goal for the vast majority of United States policy to be directly informed by evidence that is systematically validated, collated, and synthesized by teams of professional intelligence analysts.

This book is a critical milestone in attaining the goal of analysis directly supporting evidence-based policymaking. This book’s primary contribution is to conduct sensemaking on the label sensemaking. Decades of relevant academic literatures have been synthesized into one framework that illustrates how disparate research streams relate to each other and to the framework. Until now, there has not been such an extensive effort to pull together related research on sensemaking from such diverse disciplines as psychology, political science, philosophy, organizational science, business, education, economics, design, human-computer interaction, naturalistic decisionmaking, and macrocognition.

The contributions of this book go beyond a literature review, how- ever, in that an action-oriented stance is taken toward capturing nuggets of insight on how to improve aspects of analysis. The categories themselves are useful in putting some shape and structure to the amorphous value that expertise brings to creating a solid analytic product in an uncertain world: planning, foraging, marshaling, understanding, and communicating. Of par- ticular value is describing different aspects of validation that are relevant to intelligence sensemaking, and distinguishing processes for predicting future events (foresight) from processes for describing past events and assessing their impacts (hindsight).

look at what is measured [to] operationally (to) determine how people truly define a concept.

Even if high rigor is not possible under extreme time pressure, data overload, and workload conditions, the measure has potential value in supporting negotiations for what aspects are most important to do well for a given task, as well as communicating the strengths and weaknesses of the process behind an analytic conclusion.

COMMENTARY

Christian P. Westermann
Senior Analyst
Bureau of Intelligence and Research U.S. Department of State

History will tell us if current intelligence reforms are evolutionary or revolutionary, but the Intelligence Community is responding to mandated change brought about by the 2004 Intelligence Reform and Terrorism Prevention Act (IRTPA).8 In particular, the analytic and collector communities are adjusting to one of IRTPA’s pillars—improved information sharing. As reforms unfold, the collector and analyst must adapt to new rules and new analytic standards, and incorporate more methodologies, techniques, and alternatives in their analysis, in collaboration with managers and tradecraft cells in the national intelligence organizations. These new structures and guidelines present an intellectual challenge as well as a bureaucratic maze for the collector and analyst struggling not only to “produce” intelligence in a timely fashion but also to improve their product. This is not easy for the intelligence professional because time is not on their side. This is why improving the way in which all analysts think is so important and why an understanding of sensemaking will help advance the profession beyond the “established analytic paradigm” for complex problems and create greater possibilities for the application of imagination in the IC. The failure to properly assess Saddam Hussein’s WMD programs during the lead-up to Operation Iraqi Freedom is the preferred example of this failure to imagine alternatives. The corporate solution to this problem is increased collaboration and information sharing; David Moore is not in disagreement but has suggested that it must go beyond new methodologies or techniques—it must be done with a strong sense of rigor and individualism in one’s thinking.

David Moore has written for the Intelligence Community a revolutionary epistemology. His novel construct for intelligence professionals is the foundation for a philosophy of intelligence.

Moore’s prescription is to take the disaggregation of data, commonly referred to as analysis, synthesize it, and then apply to it one’s interpretation and communication skills to make sense of the information. Sensemaking therefore is a theory of knowledge for the intelligence professional and also a practice to aid the difficult art of intelligence reasoning.

His attention to revolutionary change in the art of intelligence thinking grows from his recognition that organizational reform has been ongoing for decades and despite those changes attendant failures have occurred and continue to occur. Therefore the only hope for achieving positive reform rests with changing the practice of intelligence whereby the individual collector and analyst, working together, and accepting the responsibility to think critically but also independently and across the Community, make sense of the 21st century national security environment.

COMMENTARY

Phil Williams, PhD
Director, Matthew B. Ridgway Center for International Security Studies Wesley W. Posvar Chair of International Security
Graduate School of Public and International Affairs
University of Pittsburgh

Moore’s Law for Intelligence

Any book that discusses amongst other things, red brains and blue brains, kayaking, information foraging, flashlights as blindfolds, space-time envelopes, and intellectual audit trails, is out of the ordinary. When you throw in the contention by the author that intelligence as currently practiced is akin to medicine in the 14th Century you have a book that will raise hackles, blood pressure, and voices.

This is not an easy read. But the overall thesis is straightforward and compelling: the environment within which the U.S. intelligence community now finds itself is not only highly complex but also full of wicked problems. To provide the kind of intelligence that is useful, relevant, and helpful to policy makers who have to anticipate and respond to these problems and challenges, Moore argues that the traditional paradigm developed largely by Sherman Kent has to be superseded by a new paradigm based largely on ideas initially outlined by Willmoore Kendall, a contemporary critic of Kent. The original Moore’s Law11 was narrowly technical; David Moore in contrast argues that a complex environment full of mysteries, not puzzles, requires holistic thinking (as opposed to simply disaggregation of problems), mindfulness (as opposed to mindlessness which he also elucidates), and a dynamic willing- ness to change paradigms, shift perspectives, and abandon strongly held perceptions. The book also develops the notion of sensemaking rigor and shows how metrics of rigor can be applied to several studies examining the rise and impact of non-state actors.

David Moore’s analysis is important and deserves to be widely read in the intelligence community and in the academic world.

it would have been helpful, for example, if David Moore had considered more explicitly the argument by David Snowden that making sense of a complex environment requires probing the environment. Further thought about this suggests that law enforcement is particularly good at this form of knowledge elicitation and sensemaking: sting operations, controlled deliveries, infiltration of criminal organizations, are all probing mechanisms that can contribute significantly to an increased level of understanding and, concomitantly, to an enhanced capacity for effective action. For many intelligence professionals, especially those who have had a dismissive view of law enforcement, the idea that law enforcement approaches to sensemaking might be ahead of those in the intelligence community, is likely to be as uncomfortable as most of the arguments in David Moore’s book. Certainly Moore’s volume is designed to shake and to stir. It is a manifesto for an intellectual revolution in the approach to intelligence and, as such, is likely to be both acclaimed and reviled.

PREFACE

On Being Mindful

What Is Mindlessness?

We are surrounded by errors and they are ours. Intelligence officials at the national level repeatedly use the same excuses for professional errors and for the systemic failures that follow. Despite directives to “fix” the structures, and most recently the means, by which intelligence is created, we insistently fail at our obligation to make early sense of vital threats and opportunities.

Ellen Langer, summarizing her pioneering social psychology research, finds mindlessness to arise from an over-reliance on “categories and distinctions created in the past.”12 She holds that such categories “take on a life of their own.”13

Langer also sees mindlessness arising from “automatic behavior.” Here, people rely on automatic responses as the basis for their behavior, as when one writes “a check in January with the previous year’s date.”14 By extension, intelligence professionals, in assessing sources, may develop a habit of discounting human intelligence sources because some are untrustworthy. As a result, they may miss novel insights because they use certain sources to the exclusion of others.

Finally, mindlessness can result from a failure to take into account alternative information that transcends our comfortable worldview. Langer observes that “[highly] specific instructions…encourage mindlessness” because they define what is acceptable and limit the viability of alternative signals that could lead to more accurate understanding of a phenomenon.

There were in fact 100 nuclear- tipped tactical missiles deployed on the island months before the arrival of the more infamous strategic missiles.17 A rigid notion of what constituted a nuclear missile, usually conceived as an offensive weapon, appears to have contributed to the case officers’ mindless disregard of the witnesses.

With respect to intelligence consumers, two faculty members at the International Institute of Management Development (IMD), corporate strategy expert Cyril Bouquet and corporate leadership and organization expert Ben Bryant, suggest that “decision makers often suffer from poor attention management, being obsessed with the wrong types of signals and ignoring possibilities that could significantly improve the fate of their undertakings.”18 They characterize these behaviors as fixation and relaxation. People who fixate “become so preoccupied with a few central signals that they largely ignore things at the periphery.”1

Bouquet and Bryant identify relaxation as when, after a “sustained period of high concentration,” people become unfocused on the task at hand and look to the ultimate goal.23

Sometimes ascribed to intelligence professionals’ and national consumers’ falling prey to “creeping normalcy,” relaxation was also a contributor to Israel’s failure to anticipate the attacks by Egypt and Syria in 1973.

In sum, mindlessness too often guides the assessment of affairs in too many domains, leading to errors, failures, and catastrophes. Mindless- ness is deemed unacceptable within the larger American society only when the resulting errors do lead to accidents and disasters. However, mindlessness is completely unacceptable within the domain of intelligence. One can never be certain in foresight whether errors will occur, so intelligence professionals must seek to anticipate, recognize and avoid them at all costs.

Attaining Mindfulness

The antithesis of mindlessness is mindfulness.

For Langer, a mind- ful state corresponds with: “(1) [aptitude for the] creation of new categories; (2) openness to new information; and (3) awareness of more than one perspective.”26 For example, as an intelligence professional considers who might be a member of Al Qaeda, a mindful attitude would involve constant reassessment and categorization of who might hold such membership— leaving the path open to new information for making sense of the organization and its membership. Thus, as we apply the idea that “[a] steer is a steak to a rancher, a sacred object to a Hindu, and a collection of genes and proteins to a molecular biologist,”27 the notion of a Nigerian male or even a blonde woman from Pennsylvania as a possible Al Qaeda affiliate would emerge from a mindful perspective.

Leadership scholar Deepak Sethi sees mindfulness as a “form of meditation” teaching “three simple-on-the-surface yet revolutionary skills: Focus, Awareness, and Living in the Moment.”28 This definition descends from millennia of Buddhist tradition. He argues that rather than an esoteric method it is “very practical, action oriented, and transformational.” Sethi believes that one practical way to bring about mindfulness is through the use of daily meditation, first using one’s breathing as a focus, and then using “specific daily activities such as meetings with another colleague.” However, the “real challenge [of employing mindfulness] is to take it from the meditation chair to the office chair and the real world.”29 Intelligence journeymen face this dilemma from a different perspective. They confront the real world and are challenged to contemplate their own thought processes as they engage it.

Ben Bryant and IMD research associate Jeanny Wildi write that mindfulness “involves the ability to accurately recognize where one is in one’s emotional landscape and allows…understanding, empathy, and capacity for accurate analysis and problem-solving.”34 They identify a process of detaching, noticing, and developing “here and now awareness.”35 Detachment, for example, allows a viewer to remember that a movie is really merely a “beam of light passing through a piece of moving celluloid projecting onto a screen with some sound and music that are designed to generate particular emotions.”36

In intelligence work, detachment involves stepping back from the full sensual experience of an issue to consider the actors involved, their motives, the larger context. Critical thinking as it is taught in the Intelligence Community attempts to make sense of the overall purpose or goal of a phenomenon, the points of view and assumptions of the actors involved, the implications of their acting in certain fashions, and other aspects of the larger context sur- rounding the issue.37 Questioning the available evidence and the inferences arising from it brings further detachment from the issue.

Noticing involves remaining open to both internal and external stimuli. Ultimately, situational information is conveyed from external sources through sight, sound, touch, smell, and taste. People can think consciously about these but they tend to process them using more autonomic brain structures, often without noticing they are doing so. The unease one feels about getting into a taxi or onto an elevator in an unfamiliar setting are examples of such input. In intelligence work this might be represented as a hunch about what an adversary will do. As Daniel Kahneman and Gary Klein note, in certain environments—where one can learn the cues—these intuitions may be quite accurate.38 However, in domains where one has not developed expertise, such intuitions can be inaccurate.39 The challenge is determining which of these situations one is in. This brings us back to the imperative of applying mindful detachment from the situation.

34 Ben Bryant and Jeanny Wildi, “Mindfulness,” Perspectives for Managers, no. 162 (September 2008), 1, URL: <http://www.imd.ch/research/publications/upload/PFM162_ LR_Bryant_Wildi.pdf>, accessed 14 January 2010. Cited hereafter as Bryant and Wildi, “Mindfulness.”

as Warren Fishbein and Gregory Treverton note, as “[mindful- ness] is the result of a never-ending effort to challenge expectations and to consider alternative possibilities.”43

“executives need to meditate in their own way, find ways to step back and reflect on their thoughts, actions, and motivations, and decide which ones are really supportive of their strategic agendas.”

Definitions for Making Sense of Sensemaking

Intelligence is a “specialized form of knowledge…[that] informs leaders, uniquely aiding their judgment and decision-making.” It is a type of knowledge created through organized activity that adds unique value to the policy- or decisionmaker’s deliberations. In the U.S. context, it makes sense of phenomena of interest to national leaders, warfighters, and those that directly and indirectly support them. Intelligence makes sense of phenomena related to the social behaviors of others. It reflects interest in what anyone will do to, and with, others that could affect the national interests of the United States as well as the prosperity and security of its citizens. Intelligence maintains an interest in external phenomena, such as epidemic or pandemic diseases, that impact U.S. national interests. In contrast to some popular portrayals, it really is not voyeuristic: what others do privately and alone is generally of little interest or value except as it affects how they relate to, and behave toward others. In other words, when private behaviors reveal either vulnerabilities or preferences, they may become of value to intelligence practitioners.

Sensemaking as it is used here refers to “a set of philosophical assumptions, substantive propositions, methodological framings, and methods.” As Mark Stefik notes (referring to work done with col- leagues Stuart Card and Peter Pirolli), it “is how we gain a necessary understanding of relevant parts of our world. Everyone does it.” Sensemaking goes beyond analysis, a disaggregative process, and also beyond synthesis, which meaningfully integrates factors relevant to an issue. It includes an interpretation of the results of that analysis and synthesis. It is sometimes referred to as an approach to creating situational awareness “in situations of uncertainty.” Gary Klein, Brian Moon, and Robert Hoffman consider the elements of sense- making and conclude that it “is a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively.”

These authors conclude that “the phenomena of sensemaking remain ripe for further empirical investigation and [warn] that the common view of sensemaking might suffer from the tendency toward reductive explanation.” By reductive explanation Klein, Moon, and Hoffman refer to a tendency to overly simplify explanations—to “reduce” complex phenomena to simplistic models facilitating an apparently needed but shallow understanding.

Intelligence sensemaking encompasses the processes by which specialized knowledge about ambiguous, complex, and uncertain issues is created. This knowledge is generated by professionals who in this context become known as Intelligence Sensemakers.

These terms are used as defined here throughout this book.

Sensemaking: A Structure for an Intelligence Revolution

CHAPTER 1 Introduction

Where We Are

How people notice and make sense of phenomena are core issues in assessing intelligence successes and failures. Members of the U.S. Intelligence Community (IC) became adept at responding to certain sets of phenomena and “analyzing” their significance (not always correctly) during the Cold War. The paradigm was one of “hard, formalized and centralized processes, involving planned searches, scrupulously sticking with a cycle of gathering, analyzing, estimating and disseminating supposed enriched information.”

A growing professional literature by intelligence practitioners discusses these trends and their implications for advising and warning policymakers.58

The literature by practitioners embodies a trust that national intelligence producers can overcome the “inherent” enemies of intelligence to prevent strategic intelligence failure.59 The disparity between this approach and accepting the inevitability of intelligence failure has grown sharp enough to warrant the identification of separate camps or schools of “skeptics” and “meliorists.”60 As a leading skeptic, Richard Betts charitably plants the hopeful note that in ambiguous situations, “the intelligence officer may perform most usefully by not offering the answer sought by authorities but by forcing questions on them, acting as a Socratic agnostic.”61 However, he completes this thought by declaring, fatalistically, that most leaders will neither appreciate nor accept this approach.

Jervis asserts that policymakers and decisionmakers “need confidence and political support, and honest intelligence unfortunately often diminishes rather than increases these goods by pointing to ambiguities, uncertainties, and the costs and risks of policies.”63 The antagonism is exacerbated when policy is revealed to be flawed and to have ignored intelligence knowledge.

Jervis’ article on intelligence and policy relations, while it correctly notes the tensions arising from the differing roles of intelligence and policy, over-generalizes the homogeneity of the policy community. It is the author’s experience that outside of the highest levels, there are many levels of policymaking that both encourage and welcome the contributions of intelligence. Indeed, some parts of the policy community, beyond the Department of Defense (DoD) where it is the norm to do so, rely strongly on intelligence. Further, disagreements (which Jervis consistently labels conflict) are inherent and typically welcome in the process. Hard questions about the accuracy of judgments must be asked. If we are doomed to such “disagreements,” then it is a doom we should be eager to embrace.

The other perspective is that of the meliorists—those who feel intelligence processes can be improved. The present authors reside in this camp, preferring to believe that the application of well-informed, mindful exper- tise, as developed in the present work, can bring positive and substantive value to the fulfillment of the IC’s obligations.

intense attention within and outside the IC has focused on the means by which pertinent phenomena are to be under- stood. So-called intelligence “analytic” methods are being unshelved or developed and taught to novice and experienced intelligence professionals alike. However, less fully considered are the appropriateness and validity of these methods as well as the underlying assumptions they enshrine. Even less well understood is what happens when specific methods are combined and how those combinations may be made. Several ways exist to characterize these methods in terms of their purpose. However, to date, there is no readily avail- able way to characterize methodological appropriateness or effectiveness, nor the limitations of individual methods. We also lack sound guidance on the use of combined methodologies, despite some recent, promising literature.

Before these deficiencies can be remedied, however, we need to reframe the way in which intelligence is created. Such a re-conceptualization involves critically examining what intelligence practitioners actually do, and why. The examination demands methodological rigor with particular attention to how we might ensure the validity of our approach to the work of intelligence. If the examination indicates that the existing paradigm for intelligence creation is inadequate, then a revolutionary shift in IC habits will be justified.

the intelligence-creation process remains largely a product of Cold War-era institutions and thinking, using the same cognitive frameworks that have been employed for decades. Some argue that what worked in the past is still appropriate. However, as numerous executive and legislative reports confirm, intelligence targets have in fact evolved: adversaries’ goals have changed, and their methods have evolved, even if the threats they pose seem very familiar. In sum, the old national intelligence paradigm is woefully out of date.

Intelligence issues are not the same as the issues framed separately by policymakers. To partner successfully with policymakers, intelligence professionals must consider issues from multiple perspectives. This is the role of sensemaking. Yes, the sensemaking process includes “analysis” or attacking issues by “taking them apart.” The process also includes synthesis—putting the pieces back together; interpretation—making sense of what the evidence means; and communication—sharing the findings with interested consumers. Essential to these processes is another, that of sound planning or “design.”69 While it could be said that this is what intelligence analysts do, such a statement is epistemologically false. Strictly speaking, intelligence analysts only take issues apart.

Why should we be concerned with a matter of semantics? In short, because the terms we use within the Intelligence Community shape and reflect our practice. If we are to change the culture of intelligence, and be changed by it, our practice of intelligence must also change. New language encourages a new paradigm, and paradigm shifts are revolutionary, not evolutionary.

Kent’s Imperative70

When much of the tradecraft of intelligence was put in place sixty or more years ago, the dominant framework was that of the historian as scientist. The primary intellectual framework for Cold War intelligence at the national level grew from Sherman Kent’s seminal work, Strategic Intelligence for Ameri- can World Policy.71 Kent’s legacy remains active in the National Intelligence Council and the Community at large.72 Although decision theory and other social science thinking began to influence the creation of intelligence in the 1960s and 1970s, these inputs languished until the reform efforts of recent years. More recently, advances in cognitive science, anthropology, decision theory, knowledge theory, and methods and operations research have brought us to the brink of informed, mindful intelligence sensemaking.

Sherman Kent argues that in creating predictive intelligence about its adversaries “the United States should know two things. These are: (1)…strategic stature, (2)…specific vulnerabilities.”73 These objectives focus on capabilities and draw heavily from the “descriptive and reportorial elements” of intelligence for basic data.74

The Failure of an Analytic Paradigm…

Kent’s preference for gathering and disaggregating more and more data to find answers fails today in the face of information volume, velocity, and volatility. Marshaling and disaggregating ever more data does not equate to contextual understanding. Further, the assumption that larger pipes to collect data and larger arrays to store it will then allow us to uncover the hid- den, clarifying nuggets, is misleading.

Consider what actually happens when intelligence professionals look for an answer to a problem or question. They do not just disaggregate data. Instead, people inquisitively (and selectively) interpret patterns by comparing observed, newly emergent phenomena to what they already “understand.” They make sense of phenomena by asking questions; foraging for information; marshaling it into evidence; analyzing, synthesizing, and interpreting that evidence, and communicating their evidence-based understanding of issues to others. Something makes sense because, based on their experience, its pattern is similar to something they previously have seen and that made sense to them. They may even employ a new, self-generated pattern based on previously learned and remembered patterns if they do not get a good match to an ostensible pattern.78

In other words, one must be able to convincingly correlate ostensible patterns to the data or information for which one is attempting to “make sense.” This is not always possible, especially if the phenomenon or issue is broad, novel, or poorly understood; that is, not easily subject to confirmation by universal human sensory apparatii.

For practitioners to create intelligence knowledge—even with an acknowledged degree of uncertainty—therefore requires much more than mere “analysis.” One alternative framework is embodied in the concept of sensemaking. Sensemaking begins with a mindful planning and questioning that leads to foraging for answers. It is true that along the way the resulting relevant assemblage of information—or evidence—is disaggregated into its constituent elements. However, it is also synthesized or combined to form a theory or systematic interpretation of the issue that subsequently must be explained, and convincingly. Throughout sensemaking, a continuous assessment is demanded of both the processes by which the intelligence is created and of the intelligence knowledge itself.81 Mindfulness—as discussed above in the Preface—coupled with a critical thinking-based approach, pro- vide the vigilance, awareness, and self-reflection needed to assess an issue rigorously. This is a central point: Intelligence does not exist in a vacuum. It must contribute to the understanding of an issue by informing the concerned parties of a perspective or information they did not already know. Ultimately, if no one is concerned about the knowledge sensemakers create, it is not intelligence.

IARPA, the Intelligence Advanced Research Projects Activity, employs a definition of sensemaking that is complementary to that developed here.86They propose that sensemaking is “a core human cognitive ability [that] underlies intelligence analysts’ ability to recognize and explain relationships among sparse and ambiguous data.”87 This book accepts that perspective and develops the psychological, behavioral, and social levels of sensemaking as they apply to intelligence creation. By contrast, IARPA’s own program on sensemaking seeks to build upon advances in computational cognitive neuroscience that reveal “the underlying neuro-cognitive mechanisms of sensemaking.”88

IARPA, BAA-10-04, 4. On the emerging discipline of cognitive neuroscience, see The 4th Computational Cognitive Neuroscience Conference, URL: <http://ccnconference.org/>, accessed 7 June 2010.

As characterized by Peter Pirolli, the process of sensemaking is highly iterative, involving a foraging loop and a sensemaking loop.89 In the former the sensemaker seeks information, “searching and filtering it,” while in the latter an iteratively developed mental model or schema is developed “that best fits the evidence.”90 While the overall flow is “from raw information to reportable results,” top-down and bottom-up processes act in concert to reframe issues: information either does or does not fit the hypotheses being considered; hypotheses are refuted or refined, and the larger issue and its context are also reframed, as it comes to be more thoroughly understood.91 How this can occur within the context of intelligence creation is developed in the following chapters.

To sum up, this book argues that intelligence built around a model of disaggregation as it originated with and developed under Kent, and is still largely practiced today, is at best insufficient. A paradigm based on the concept of sensemaking and employing insights from other knowledge-creation disciplines provides a more appropriate means of skillfully creating intelligence. This book draws a general picture of 21st Century intelligence under a revolutionary paradigm, although it does not explain how all its contours can be fleshed out. We believe that intelligence could be a true profession and moving toward that goal is our desire.

CHAPTER 2
The Failure of “Normal Intelligence”

Intelligence Challenges

Our understanding of everyday phenomena is confounded by every- day strategies employed to mitigate cognitive dissonance, a stressful condition arising when reality clashes with one’s perceptions. Two broad strategies, selective exposure and selective perception, can prevent dissonance, but at the expense of sound, mindful reasoning. Through the former, we limit the evidence to that which agrees with or otherwise supports our positions; in the latter, we interpret what we experience in terms of our pre-existing world-view.

the differences between “intelligence error” and “intelligence failure.” Anthropologist Rob Johnston defines intelligence error in terms of “factual inaccuracies in analysis resulting from poor or missing data.”98 Conversely, intelligence failures are “systemic organizational surprise resulting from incorrect, missing, discarded, or inadequate hypotheses.”99 Thus, the term “failure of imagination” makes sense as a synonym for intelligence failure, where members of an intelligence creating organization fail to imagine in advance the essential outlines of an incident that subsequently occurs.

The etymology of “imagination”—generating images—reminds us of the contemporary critic of Kent, Willmoore Kendall, who suggested that the job of national intelligence is to communicate with decisionmakers in a “holistic” way so as to generate the “pictures [mental models] that they have in their heads of the world to which their decisions relate.”112

Considering Standard Models

Intelligence failures occur as practitioners employ a “standard model”113 of intelligence: In it, analysts “separate something into its constituent elements114 so as to find out their nature, proportion, function, relationship, etc.115 and “produce reports” based on “collected” information and data. There is a definitional presumption that disaggregation will lead to answers. However, this model incompletely describes what the intelligence professional does and its underlying presumption about finding answers may be false.

One problem is that in Kent’s data-based analytic framework, analysts need to have all the data available so they can be marshaled into a coherent account. “Dots”—if they exist at all—can be connected in more than one way.116 In foresight it is difficult at best to determine which combination and order is valid. Such determinations can be further complicated by the fact that adversaries may change their actions if they suspect we have arrived at a certain conclusion.

An additional problem is that with an increased number of signals there is also an increased level of noise. Which signals, which facts, or which inferences the intelligence professional should consider valid becomes a very important consideration. At best, warning of a pending incident is a problem of assembling and making sense of the details of a specific incident in advance. However, many intelligence problems inherently defy such linear characterization. They are in fact “wicked” problems—a formal designation of a complex issue with myriad linkages. We turn next to an exploration of problem types to see how their nature directs our making sense of them.

Types of Problems

In order to understand “wicked problems,” one must first understand the nature of “tame problems.”

Tame Problems

In a tame problem there is general agreement as to what or who an adversary is, what the “battlefield area” is, and what an attack is. Such problems, while difficult, exhibit specific characteristics: They are clearly defined and it is obvious when they are solved. Solutions to these problems arise from a limited set of alternatives that can be tested; the correct solution can be objectively assessed. Finally, solving one tame problem can facilitate creating valid solutions to other, similar tame problems.

It is important to note that analysis protocols for tame problems con- tain little or no room for “emergent” properties. One may not know that the analytic protocol is insufficient until the puzzle has been incorrectly defined, characterized, and solved, if it is in fact solvable. One arrives at one solution that at first appears to have resolved the issue, but in fact, the issue reemerges elsewhere. For example, the implementation of a linear, intelligence-driven solution to crack down on insurgents and their improvised explosive devices (IEDs) in one area may lead to an emergence of IED-caused explosions some- where else. In such a case, the application of “tame problem protocols” may in fact have been inappropriate—the problem is in fact not tame.

Admittedly, many 21st Century intelligence issues remain puzzles or tame problems. This occurs when the events surrounding the issues have already occurred, appropriate questions are readily identifiable, and answers exist, even if they are difficult to find.

Wicked Problems

However, seen in a larger context, are such puzzles truly tame? Or are they components—as Russell Ackoff suggests—of something larger: a mystery in Treverton’s terms, or a “mess” according to Ackoff.119 Treverton’s intelligence mysteries defy easy definition. They belong to a class of problems defined by social researchers Horst Rittel and Melvin Webber as “Wicked Problems.”

The adaptive nature of adversaries makes seemingly tame puzzles wicked, moving them into the realm of “unknown unknowables.”

By definition, wicked problems are “incomplete, contradictory, and changing.”121 They do not have single answers and in fact, are never truly answered. In the context of intelligence, the sensemaker may never realize a problem has been resolved. This is because “the solution of one of its aspects may reveal or create another, even more complex problem.”122 The emergent complexity of the problem itself, its adaptive nature, efforts at denial and deception by adversarial actors, as well as cognitive frailties on the part of sensemakers, compound the problem, confounding sensemaking, leading in some cases to disastrous courses of action or consequences.

A Wicked Look at Wicked Problems in Intelligence

Characterizing intelligence issues in terms of their problem type— admittedly somewhat vaguely (in keeping with their nature)—reveals just how prevalent wicked problems are within the domains of intelligence.

Wicked problems have no definite formulation. To Rittel and Webber, “the process of solving the problem is identical with the process of understanding its nature, because there are no criteria for sufficient understanding.”124 In other words, making sense of problems deemed sufficiently complex so as to be considered wicked is equivalent to characterizing them in the first place; the description encompasses all possible solutions.

one wicked problem example could be “how best to stem the growth of terrorism in the Middle East.” An assumption in considering this problem is that if intelligence professionals can understand what motivates people to become terrorists in the first place, intervention might be possible. Mitigating the creation of new terrorists could aid in reducing both their numbers and by extension, their attacks. Do people become terrorists because they are dissatisfied with what they see as contradictions and hypocrisies in their lives? If so, what then are the specific roots of dissatisfaction and contradiction? One commonly cited is a lack of economic opportunity for males within societies. In that light, Rittel and Webber ask, “where within the…system does the real problem lie? Is it deficiency of the national and regional economies, or is it deficiencies of cognitive and occupational skills within the labor force?”125 The possible solutions to this problem extend the domain of questions, spreading ever outward.126

as Nicholas Taleb notes, [Our] track record in predicting those events is dismal; yet by some mechanism called the hindsight bias we think that we understand them. We have a bad habit of finding “laws” in history (by fitting stories to events and detecting false patterns); we are drivers looking through the rear view mirror while convinced we are looking ahead.

Even with the addressal of major assumptions, there remain additional underlying factors that do not get questioned—almost an endless succession of assumptions that must be peeled off the problem much as one peels layers off an onion. There is an added complication that individual layers are not sequential and in fact may lead (to continue the analogy) to other onions or other vegetables, or even fruit. In intelligence, such assumptions are themselves a mess: a complex system of interrelated experience, knowledge, and even ignorance that affects reasoning at multiple levels sequentially and simultaneously.

Wicked problems have no clear end-point.

With tame and well-structured problems one knows when the solution is reached. In wicked problems this is not so, as Rittel and Webber make clear:

There are no criteria for sufficient understanding and because there are no ends to the causal chains that link interacting open systems, the would-be planner can always try to do better. Some additional investment of effort might increase the chances of finding a better solution.

We have a bad habit of finding “laws” in history (by fitting stories to events and detecting false patterns); we are drivers looking through the rear view mirror while convinced we are looking ahead.127

This is not a new consideration. Writing in the 1930s, John Dewey observed that “the ‘settlement’ of a particular situation by a particular inquiry is no guarantee that that settled conclusion will always remain settled. The attainment of settled beliefs is a progressive matter; there is no belief so set- tled as not to be exposed to further inquiry.”131 Intelligence sensemakers routinely confront this challenge. Reports and assessments often update or revise previous conclusions. Often the previous reporting is consulted before the new report is written so that the author can determine the preexisting point of view on the issue. Such consultations at best determine whether the current situation deviates from the norm. Unfortunately, sometimes such consultations lead to the rejection of the new evidence, opening the way to intelligence errors and failures. One goal of an adversary’s denial and decep- tion activities is to facilitate rejection of the novel deviation. It was in this way that the possibility of nuclear missiles deployed to Cuba was rejected amid outlandish noise during the summer of 1962, and military exercises along the Suez Canal lulled Israel into a sense of creeping normalcy prior to October 1973.

Solutions to problems may be implemented for “considerations that are external to the problem” itself: problem solvers “run out of time, or money, or patience.”132 In intelligence, sensemakers may only be able to work for a given time on a problem before they have to issue their report. Changes in funding may mean that an effort to understand a phenomenon has to be dis- continued. The practicalities of resource limitations force changes in sense- makers’ foci. However, this does not mean that the problem does not continue to exist and perhaps, threaten. Rather, an answer has been developed to a dis- tilled problem, communicated, and now other things must be done.

Solutions to wicked problems are at best good or bad.

Some problems have true or false, yes or no answers. These are not wicked problems. Wicked problems have no such answers. Differing perspectives applied by different problem solvers, differing sets of assumptions, and differing sources of evidence are several of the factors that lead separate groups to come to different judgments about wicked problems. The impossibility of exhaustively considering all the factors and solutions of the problem also contributes to a multiplicity of solutions.

Focusing on the economics sur- rounding the growth of terrorism leads to different proposed solutions than does focusing on the demographics involved in the issue. Religious consider- ations or broader cultural considerations also create different solutions. Each of these perspectives in turn optimizes multiple points of view with differing, good and bad solutions. Overlap is possible and even desired. Good solutions encompass multiple domains.

Tests of solutions to wicked problems may not demonstrate their validity and may provoke undesired consequences. Implemented solutions to wicked problems “generate waves of consequences over an extended— virtually an unbounded—period of time.”

Further, these consequences may themselves prove so undesirable as to negate any and all benefits of the original decision—and this cannot be determined in advance. Thus an intelligence-based decision to invade a country’s possessions may create circum- stances that offset any gains initially won, as the Argentineans discovered in 1982 when they—unwisely in retrospect—seized the British-owned Falkland Islands.

Implementing solutions to wicked problems can change the problem. In intelligence problems, real solutions cannot be practiced; there are no “dry runs.” True, sensemakers and their policy-making customers can (and should) consider what might happen or the “implications” of the decisions or solutions of the problem at hand. Doing so might increase the likelihood that the decision selected is the best or the less bad of a set of bad alternatives.

Modeling the situation is one common means of assessing the implications of a potential action. However, models must by their very nature limit the factors considered. This raises the question of how one might know in advance if the eliminated factors are in fact significant.

As Rittel and Webber note, “every attempt to reverse a decision or to correct for…undesired consequences poses another set of wicked problems,”135 as sensemakers and planners involved in the U.S.- led “war on terror” have discovered. Actions, once taken, may mitigate the threat, or may not, which leads to the next facet of wicked problems.

Sensemakers can never know if they have determined all the solutions to wicked problems. They can expect, however, that they almost certainly have not determined all the solutions. In developing the range of alternatives within scenarios, two goals predominate: mutual exclusivity and collective exhaustion.

In other words, each alternative must preclude the simultaneous possibility of the others, and the entire set of known alter- natives must be considered. In practical terms, this is much more difficult to achieve than it sounds. Intellectual frameworks and so-called “biases” such as vividness, anchoring, confirmation, and others combine to prevent people from being able to consider all the alternatives. Adding to this is the fact that issues evolve in unpredictable ways. All the solutions simply are not knowable because they lie in the future

Each wicked problem is unique. While it is true that common elements can be found between problems, there remain additional and unique properties of “overriding importance.”136 In other words, wicked problems cannot be characterized into “classes…in the sense that principles of solu- tion can be developed to fit all members of a class.”

Every wicked problem is embodied in another one. Rittel and Webber describe problems as

[discrepancies] between the state of affairs as it is and the state as it ought to be. The process of resolving the problem starts with the search for causal explanation of the discrepancy. Removal of that cause poses another problem of which the original problem is a “symptom.” In turn, it can be considered the symptom of still another, “higher level” problem.139

What policies and actions, for example, are necessary to “fix intelligence?” Answering this involves asking what is causing intelligence to fail. One place to start is to consider why analysts are wrong and how intelligence errors lead to intelligence failure.140 Yet such considerations lead one to consider how consumers may ignore intelligence, and how adversaries may in fact be “more capable” than expected. These in turn lead to what Jeffrey Cooper considers “analytic pathologies” that decrement both individual and corporate efforts to make sense of issues (table 2). Each of Cooper’s specific pathologies is furthermore at least partially embodied in the others, giving rise to error-producing systems.141 For example, Cooper argues that intelligence professionals’ pathological focus on both “the ‘dots’ analogy and the model of ‘evidence-based’ analysis…understate significantly the need for imagination and curiosity.”142 Related to this is what he calls the myth of “Scientific Methodology.”

Analysis is not [hard] science and is not about proof. Rather it is about discovery.143 These are embodied in the protocols he refers to as the flawed “Tradecraft Culture,”—a guild system of potential sensemakers and their historically unchanging ways of working.144

How wicked problems are resolved is determined by the means and methods used to make sense of them. In other words, how problems are perceived determines the kinds of solutions that are proposed. Point of view becomes essential in defining what a problem is and how it is to be resolved. Complex, wicked problems (as well as many “tame” ones) cannot be defined from one point of view.

Sensemakers have no right to be wrong.

[T]he primary function of the Central Intelligence Agency is to seek the truth regarding what is going on abroad and be able to report that truth without fear or favor. In other words, the CIA at its best is the one place in Washington that a President can turn to for an unvarnished truthful answer to a delicate policy problem.148

Will Pitt, “Interview: 27-Year CIA Veteran,” Truthout, 26 June 2003,

This aphorism may have validity in the domain of tame problems where the truth is known or knowable. However, it has much less (if any) validity in the world of wicked problems where many truths can coexist, depending on the point of view expressed, the context can be simultaneously true and contradictory, and may in fact be unknowable.

The goal of assessing wicked problems may be to “improve some characteristics of the world where people live. Planners are liable for the con- sequences of the actions they generate; the effects can matter a great deal to those people that are touched by those actions.”

An Intelligence Example: Pandemics as Wicked Problems

One of the threats faced by intelligence organizations and their professionals is that of an emergent global pandemic. What kind of a threat is a pandemic? Is it a tame or wicked problem, or something in between? Such considerations matter because they define what approaches are suitable for alleviating or mitigating the threats to national security that pandemics pose.

When the stakes are the lives of many people, sensemakers and policymakers who miscalculate or underestimate or are otherwise wrong about a pandemic and its impact on their countries or region can expect vilification at best. A fear of such vilification from the public and the media might contribute to the situation whereby pandemic-tracking organizations such as the U.N. World Health Organization or the CDC overestimate the severity and threat posed by a pandemic such as the 2009-2010 Swine Flu pandemic.158 For intelligence professionals this phenomenon is not unknown. Common wisdom among intelligence sensemakers is that it is far better to warn and be mistaken (and nothing hap- pens) than to not warn and be mistaken (something happens).

Complexity

Rittel and Webber’s notions of wicked problems can also be characterized through the lens of complexity theory. As developed by Jonathan Rosenhead, “systems of interest to complexity theory, under certain conditions, perform in regular, predictable ways; under other conditions, they exhibit behaviour [sic] in which regularity and predictability is lost.”159 This is exceptionally true of intelligence. Certain kinds of issues, including the interpretable indications of a build-up to armed conflict, can be extremely predictable.

However, in other situations, there may be a number of unknowable, unpredictable, and unanticipatable outcomes. Thus, reliable prognostication is simply not possible.160 For instance, if a coalition of nations removes an oligarch in another nation from power, the specific outcomes of that action cannot be known in foresight. While alternative outcomes can be modeled and simulated, they remain valuable only as discussion points: there is no guarantee in advance that they have captured the reality that will occur. Modeling and simulation are feasible because complexity science shows that the “indeterminate meanderings of these systems, plotted over time, show there is pattern in the movements…the pattern stays within a pattern, a family of trajectories.”161 Unfortunately, because intelligence must address the “real” world, rather than its modeled or simulated semblance, events often are unique and therefore their patterns also are unique.

Thus, there exists an inability to guarantee a future reality; even probabilities may be suspect.

Analysis as here defined is insufficient to address complexity. Disaggregation simply does not reveal future alternatives. That this is so becomes obvious if one finds that it is the emergence of unique and novel behaviors arising from different and minutely differing initial conditions that characterize many 21st Century intelligence issues. In these circumstances, the whole of an issue is greater than its parts. But, in analysis, the issue is by definition and practice the sum of its parts.

Given these complex issues, the concept of “analysis” is simply insufficient for sensemaking. Instead, greater conceptual accuracy and precision of terminology is required.

To achieve the needed accuracy and precision requires more than semantic invention. It also demands that underlying concepts, known as assumptions or premises, be identified and accounted for. Therefore, in developing the case for considering new paradigms for intelligence, certain terms require explicit (re)definition.

Implications of Complexity

Viewed from a larger context, complexity stymies the entire “standard model” of intelligence creation. With regard to Kent’s concept of knowledge, or how intelligence is created, complexity—as viewed from the framework of wicked problems—confounds the consideration and mitigation of such problems. Kent’s model of predictive and specific warning seems more miss than hit. Complexity further confounds the collaborative processes contained within Kent’s notions of Activity and Organization, by which intelligence pro- fessionals are tasked to interact. How does the intelligence professional know in advance whose imagination will be most helpful in making sense of the problem at hand in time to prevent a catastrophe or even imagine one?

Given the challenges of both tame and wicked 21st Century intelligence problems and their inherent complexity, what are intelligence professionals to do? One avenue open to them, and presented below, is the development and validation of methods of reasoning about key evidence.

CHAPTER 3
From Normal to Revolutionary Intelligence

Evidence-Based Intelligence Creation

Intelligence sensemakers use more than context-less data and information. They employ assemblages of evidence—at a minimum, collections of data and information determined through marshaling to be relevant to the issue under consideration—in other words, contextualized to specific issues. Evidence reveals alternative explanations through pattern-primed, induced inferences about what is going to happen or what has happened already in the past.

While the inferences are typically uncertain, they do justify beliefs about phenomena. Justifying beliefs (or theories or hypotheses) presents a case for their accuracy but does not guarantee ground (or any other) “truth.” Rather, as Peter Kosso notes, justifying beliefs is “about meeting the standards of evidence and reason [to] indicate [the] likelihood of accuracy.”168 Sensemakers go further and seek to demonstrate that the knowledge of tendencies they establish provides for “a correlation between being more justified and being true.”

It is arguable whether greater evidentiary justification demonstrates the likelihood of a strongly accurate correlation with truth. As Kosso makes clear, even with abundant justification, there is no certainty of truth.

As figure 1 illustrates, intelligence sensemaking is conducted in ser- vice of a number of goals, including describing states of affairs, explaining phenomena, interpreting events and actions, and estimating the likelihood and impact of a foe’s future actions. As intelligence professionals move from describing events, explaining patterns of behavior, and grasping underlying factors and intentions, ever more justification of beliefs about the phenomena under scrutiny is required. Yet, as intelligence professionals attempt to apply greater scrutiny in this sequence, their capability to do so decreases as they face greater ambiguity.

intelligence evidence, while it may appear to remain “haphazard,” is the result of systematic foraging, gathering and interpretation. The past tells intelligence practitioners what to look for in the future. This poses dangers when those indicators are no longer (if indeed they ever were) valid.

If using the past to gain wisdom about what the future holds is not feasible, what about studying the past to avoid folly? Tversky and Kahneman’s work on availability leads one to suspect (as Fischoff also notes) that focusing on misfortunes “disproportionately enhance[s] their perceived frequency.”181 Another challenge to considering the past as a teacher of what not to do is that one may not properly understand the problem.

With the intention to improve evidence-based intelligence creation, recent legislation “reforming” intelligence goes so far as to require that “alternative analysis” be conducted.182 The IC, at least through its schools, interprets this to mean that multiple hypotheses be considered. The relevant act mentions “red teaming: a means by which another group of intelligence professionals consider alternative explanations for an issue being scrutinized.183 The legislation leaves unexamined the question of whether the criteria for sensemaking will be met in examining tame problems and especially wicked problems arising from consideration of adversarial intentions.

If, for example, one estimates that a particular country whose policies one’s own government generally opposes will develop both a long-range missile capability and a nuclear weapons capability and then marry the two together, one has to have already imagined, in the context of the target country’s political and technological environment, what a long-range missile capability is, what a nuclear weapon is, what a weapon of mass destruction is, and a strong sense of the will to combine these threat elements. Policymakers may challenge the target country’s actions, making their leaders more adversarial. Thus, at a minimum, intelligence and policy create the future—or a version of it. Done poorly, this can lead to unintended and dangerous implications.

In a tense bilateral or even multilateral environment, rhetoric and actions can precipitate events so as to create a future consistent with those pattern-derived conclusions, driving the target country to produce the weapons. Each side then blames the other nation’s government for having “caused” the crisis.

when interpretations of the evidence lead to coherent alternative inferential conclusions, then the existing or accepted theories require changing. What must not happen is to reinterpret the evidence to support the prevailing pre- existing theory,

People are often unwilling to abandon their cherished positions. This occurs in part because they are not dispassionate as they reason about evidence. In other words, positions are influenced by various worldviews or cognitive approaches, particularly selective perception and selective exposure. These combine to steer how people recognize issues, the phenomena that comprise them, and how they go about making sense of them.184 These influences or theoretical frameworks shape the patterns people use to interpret new phenomena. The benefit is that these frameworks make people smart and do so quickly.

In an information-rich environment brought about by technical collection, intelligence professionals can select inappropriate patterns to use in making sense of new phenomena. In intelligence work, if such patterns conspire to affect the search for and the selection of the evidence sensemakers use, and that they and their consumers then accept, selective perception and selective exposure set the stage for intelligence error and failure.

Evidence always requires a context, and as the missile example illustrates, there may be more than one explanatory context that makes sense. In intelligence, “evidence is [particularly] rarely self-sufficient in information or credibility.”

unless the correct context is known, evidence—if its constituent information can even meet that threshold— is subject to many different interpretations. Without context the person assessing the evidence has no way of knowing which interpretation is correct. Multiple contexts further confound the situation, for different contexts often lead to alternative conclusions as was illustrated in the missile development scenario just described. Finally, as Hampson’s essay reveals, the political context of the policymaker may skew the actual context conveyed by intelligence.

Considering the Normal

The process described in the preceding section can be thought of as “normal intelligence.” As conceived by Thomas Kuhn, “normal” refers to “the relatively routine work…within a paradigm, slowly accumulating detail in accord with established broad theory, not actually challenging or attempting to test the underlying assumptions of that theory.”195 We can thus see that “normal intelligence” is an activity of expanding knowledge in which most intelligence professionals engage and which incrementally increases knowledge about targeted phenomena.

The perceived and recalled successes of the past contribute to the repeat use of unvalidated tradecraft. The paradigm presumes state-level adversaries—eventually with mutually destructive capabilities.

As used in this context, “normal intelligence” is to “intelligence” as Thomas Kuhn’s “normal science” is to “science.” In both domains newly created knowledge incrementally adds to an increasingly established paradigm; new knowledge does not easily redefine the paradigm.

Normal science or normal intelligence does not seek to revise significantly the paradigm by which new phenomena are known and understood. This may be seen in the way new intelligence personnel adopt existing job accounts. A common practice involves their reviewing previous reporting on the account, with a tendency for new reporting to stay within the conceptual boundaries of what has gone before. Knowledge increases only incrementally.

Normal paradigms prevail until previously unnoticed and unnoticeable discrepancies create sufficient inconsistencies in explaining and understanding phenomena so as to cause errors that cannot be ignored.

In the cultural environment of human interaction, the new perceptions of reality can be enough to force a reconsideration of the old. In social scientific terms, a new paradigm not only explains the new, it does better at explaining the old.

The existence of particular intelligence errors does not necessarily indicate a paradigm has changed. However, repeated intelligence errors do. As is the case with science, small errors in adequately characterizing phenomena lead to the emergence of “corrective constants.”

The state-as-adversary paradigm for intelligence creation is obsolete. Two decades now separate the interpretable intelligence context from that of the Cold War: the adversaries and issues are now strikingly different.202 The power of the Soviet Union waned dramatically after 1990 as that of China increased. But even more central to the intelligence context, novel phenom- ena also appeared that were non-state based: emerging non-state actors posed new challenges by threatening traditional state structures.

The anomalies these new phenomena have created illustrate how and why normal intelligence is no longer adequate: it could no longer characterize these phenomena within the threat and opportunity framework of strategic intelligence. The “normal” means by which error is explained remain inadequate. As documented in the various Congressional and independent commission reports, intelligence no longer adequately describes, explains, or predicts with respect to the phenomena its consumers need to understand. Thus, intelligence change is necessary—revolutionary change.

Paradigm Shift

Revolutions in science, politics, and military affairs occur because crises reveal the insufficiency of the reigning paradigm.

periodic revolutions change how phenomena are perceived and understood.204 Crises are a precursor of such paradigm shifts.

here are repeated attempts to impose methods of “[social] scientific study…to analysis of complex ongoing situations and estimates of likely future events.”206 What is lacking is any sort of a systematic approach across the Intelligence Community. As long-time practitioner and observer Jack Davis noted a decade ago, no corporate standards for how intelligence is created, including the methods employed, exist.207 Although sound practice does not ensure that intelligence assessments will be correct, its absence, by definition, contributes to flawed conclusions.

In short, U.S. intelligence professionals operate in an environment similar to an unfolding Kuhnian revolution: the epistemology of normal intelligence is insufficient and new knowledge is needed. The recent failures highlight the necessity for change, as does the graying of the intelligence sensemaking workforce—new people faced with new and emerging issues should be comfortable with finding new ways to systematize their work.

Not all “old school” intelligence practices are without continuing value. Several significant state-level adversaries remain as threats to the security of the American nation although they too are challenged by the new non-state actors and issues that populate the paradigm of the new intelligence—something that compounds any estimate of how they are likely to engage the United States. Further, in many circumstances and in dealing with certain issues, the tacit expertise of highly experienced intelligence professionals is appropriately tapped for “recognition-primed” sensemaking.209 These “old hands” possess both current knowledge and a highly evolved skill set. Years of innovative and critical thinking mean they are skilled in looking at issues from a variety of perspectives and have the wisdom of deep context.

The challenges involve knowing when such expertise is valuable and needed in the first place, and encouraging the intelligence enterprise to develop and retain the cognitive and organizational flexibility that such thinking requires.

Indeed, a part of successful and revolutionized intelligence work involves gleaning new meanings from old patterns that have remained hidden to those who have stopped short of sensemaking. One challenge is that the “fresh” eyes lack the knowledge of potentially relevant patterns while the “old” eyes cannot see things as new. Each lacks the other’s strength. Experience acquired by newer professionals who engage in the practice of traditional “analysis” jaundices their once-fresh viewpoints even as they start to acquire the relevant and necessary experience.

One solution may be to adopt a model of core competencies broken out according to task analyses of existing intelligence missions and functions. Such a model identifies what is needed and has been at least partially imple- mented in the IC’s Analytic Resource Catalog developed during the tenure of former Director of Central Intelligence George Tenet.

Mark M. Lowenthal, “Foreword,” in Moore, Critical Thinking, xi.

Rewarding the successful use of some of the most important competencies may also encourage their retention in the catalog. Among these are curiosity, perseverance, and pattern recognition.

Simply put, intelligence practitioners create knowledge to support their customers. As used here, intelligence practitioners are presumed to be contributors to government plans and policies at a variety of levels where they have the opportunity to share broad strategic perspectives with national leaders as well as ensure that deployed warfighters have at hand the fruits of technical collection and marshaling of tactical data.

Finally, it should be noted that intelligence Knowledge is only one component of a strategic, operational, and tactical intelligence triumvirate.213 Activity and Organization are the other two. It is the author’s belief that Activity and Organization also are in need of new paradigms. However, such a discussion hinges on what intelligence Knowledge is and how it is created—in short, the sensemaking involved. Insofar as Activity describes how the precursors of intelligence are hunted, gathered, made sense of, and transformed into knowledge, it is considered here. However, the uses of intelligence (also an activity) and how intelligence professionals are grouped, led, and managed to act and create knowledge—the realm of Organization—lie beyond the scope of this book.

CHAPTER 4:
The Shape of Intelligence Sensemaking

Intelligence Sensemaking involves a number of overlapping high- level activities. First, intelligence professionals engage in planning or design and then hunt for and gather the materials they require in order to under- stand issues, answer questions, or explore new ideas. They can be externally motivated by the needs of a customer or they can be self-motivated as a result of an observation, or both. Second, these professionals disaggregate and then reassemble relevant information, trying to determine what it means.

At every stage in their work they assess critically their processes and results, seeking to validate both how they are engaged and the outcomes of their engagements. These overlapping activities can be characterized as Planning, Foraging, Marshaling, Understanding, and Communication. They are supported by Questioning and Assessing.

Planning for Tame and Wicked Intelligence Problems

Making sense of either tame or wicked problems is predicated upon planning. Plans, according to Gary Klein, are “prescriptions or roadmaps for procedures that can be followed to reach some goal, with perhaps some modification based on monitoring outcomes.”214 Creating plans requires “choosing and organizing courses of action on the basis of assumptions about what will happen in the future.”215 Known as planning, this process characterizes the “contingencies and interdependencies such as actions that must occur first as a precondition for later actions.”216 When we add the concept that practitioners—through critical thinking—also engage in reflective thinking and learning, both singularly and collaboratively, we may similarly label this process “the art of intelligence design.”

With tame problems, where answers and solutions can be anticipated, algorithms can calculate actionable probabilities and repeatedly make sense of the problem.217 The design of useful algorithms may be complex and they operate well only in finite and specific environments. How, on the other hand does one plan or design for wicked problems? One answer is to re-imagine the wicked problem as a tame one. However, the repeated occurrence of “unintended consequences” in past scenarios suggests that this is not a good option. Disaggregating what are assumed to be tame problems into their component parts, regardless of the actual problem type, often proves inadequate, as unintended and unforeseen consequences make clear.

Yet planning must occur regardless of problem type.

Klein considers—along with Rittel and Webber—that planning is an emer- gent process: Goals are clarified and revised as understanding of the problem grows.218 He notes,

Goals can be dynamic and can change completely as a function of changing circumstances. Goals can conflict with other goals in ways we can’t anticipate or resolve in advance. Goals can carry implications we can’t perceive or anticipate until events transpire.219

In terms of intelligence creation, this means that larger, strategic goals can—and perhaps must—emerge as sense is made of the problems under scrutiny. Thus, a tasking from an intelligence consumer changes as mindful sense is made of the tasking itself, of the resources that are available for understanding it, and the mix of actors involved.

Klein refers to this reflective problem planning as “flexible execution” or “Flexecution.”220 Within the framework of intelligence sensemaking, it provides a self-reflective process—at the individual and organizational levels—that monitors the goals and whether what is understood or being done is consistent with those goals, modifying those goals as understanding emerges.

Foraging
Hunting and Gathering

If, as Baumard asserts, “Intelligence, a continuous human activity, gives sense to the stimuli received from the environment [then] these stimuli [must] be passively or actively sought.”223 This requires hunting and gathering. They comprise foraging, which in turn refers to “a wide search over an area in order to obtain something.”

An apt analogy for the foraging activities of intelligence professionals can be drawn from anthropology, where the activities of the hunter-gatherer have been immortalized. In nonagricultural societies people both hunt for specific game and take advantage of what the local environment provides.225 Similarly, intelligence professionals may seek specific information, often tasking collection systems as part of the search. They also take advantage of existing repositories of information. Neither approach is wholly satisfying nor provides for all of the sensemaker’s needs all of the time. However, without the basic act of foraging there can be no sensemaking as there is nothing from which to make sense.

Information foraging is a rich subject about which Peter Pirolli has done extensive work, some of it sponsored by the Intelligence Community’s Novel Intelligence from Massive Data research project funded by IARPA’s predecessor, ARDA (Advanced Research and Development Activity).

Underlying information foraging theory is the idea that “humans actively seek, gather, share, and consume information to a degree unapproached by other organisms” and therefore, “when feasible, natural information systems evolve toward stable states that maximize gains of valuable information per unit cost.”

Efficiency derives from optimizing the time necessary to achieve a goal, the quality of the achievement, and the satisfaction obtained in doing so.227In application to the human information foraging scene, the theory becomes “a rational analysis of the task and information environment that draws on optimal foraging theory from biology and…a production system model of the cognitive structure of [the] task.”

Peter Pirolli and Stuart Card, “Information Foraging,” Psychological Review, vol. 106, no. 4

Toward a Practice of Intelligence Foraging

Developing an optimal foraging model for information acquisition requires the subject to consider whether to remain at a source that provides a superabundance of information of questionable value, or to seek another, more valuable, source.229 Pirolli and colleague Stuart Card observe that for human foragers, this involves “a tradeoff among three kinds of processes”: exploring, enriching, and exploiting.230

These three foraging steps will not seem foreign to traditional intelligence practitioners. “Exploring” is a breadth activity whereby a sensemaker broadly examines a wide variety of information that may or may not be relevant to the issue. The premise is that when one considers a broad variety and volume of data, there is less opportunity to miss “something novel in the data.”231 Speaking in traditional intelligence terms, exploring is like reconnaissance. By contrast, “enriching” is a depth activity. Here the sensemaker identifies areas of interest and focuses attention on those areas. As Pirolli and Card note, this is “a process in which smaller, higher-precision sets of documents are created.”232 Reconnaissance has become more narrowly focused, but highly targeted “surveillance” is not yet in play. Finally, the practitioner “exploits” the results of foraging by thoroughly examining what is found and extracting information as needed. This activity extrapolates from tacit sensemaker behaviors and information-based patterns to create hypotheses about what the information means. At this point, foraging evolves to sensemaking.

The appropriate amount of exploration depends on the context. However, there appears to be a limit after which more information does not increase accuracy although it does increase the sensemaker’s overall confidence.

The discussion of how much exploration is needed is germane because in the Pirolli-Card framework the sensemaker may believe she controls the amount of foraging. In a sense this is true. The sensemaker will stop foraging once she believes she has what she needs. But how much information is sufficient? Slovic’s results and Heuer’s subsequent discussion suggest that practical sufficiency is achieved at lower levels of exploration than expected.235

Further, contrary to the beliefs of the sensemaker, it is often information itself that controls the processes. Intelligence professionals are over- whelmed with more and more information that arrives faster and faster and may be valuable for shorter and shorter periods of time. This information flood challenges the sensemaker to efficiently find the information needed in order to make sense of the phenomena or issue under scrutiny in a timely manner. Does one need to peruse all the information received? Would a different source be more productive by providing more focused information? These are examples of the difficult questions the sensemaker must consider. Compounding her deliberations further is the fact that she must answer this question in foresight; hindsight is too late.

Foraging practice begins with an understanding of what it is the sensemaker seeks to know, the foraging resources available, and the urgency of the issue. But how can an intelligence sensemaker know what to look for? To what degree does she need to explore, enrich, or exploit the information? Further, how does she know if she is getting what she needs?

A first step is to think critically about the issue itself and the resources needed. Using a metacognitive, process-focused critical-thinking model such as that adapted from Richard Paul, Linda Elder, and Gerald Nosich, the practitioner can dissect the issue and her thinking on the issue.

She makes assumptions explicit, explores relevant points of view, starts to con- sider the ramifications of the issue, and she asks important questions about what resources will best inform her about the issue; she considers the con- text in which she is working, and ponders the alternatives to her reasoning about where and how to forage. This critical thinking defines her foraging activities. She may engage in all three strategies at once or at different times as she engages in the analyses that produce her syntheses and the interpretations necessary to generate knowledge. A well-developed understanding of exploited information may direct her back to do additional exploring or enriching (or both).

In intelligence foraging as it traditionally has been practiced, there is a tendency to linger at a fruitful source rather than to explore elsewhere for the required information.

The danger, of course, is that the practitioner may limit the information she acquires and the relevant perspectives it informs. If, for example, the practitioner has a belief that two parties in whom she has an interest communicate via one means and she can acquire technical collection that captures the communications via those means, she may ignore the fact that they also use other methods to communicate. She may subsequently fail to task systems that collect those other communications in the belief that what she is getting suffices. Should the parties suspect that their communications are being targeted, they may engage in deceptive practices over that collected means and use the other non-collected methods for their real exchanges.

Foraging Challenges

Ongoing research at University College London offers a view of how younger sensemakers likely search for information. The researchers report that people foraging for information spend four to eight minutes viewing each resource.237 Thus, a great many resources may be consulted but none of them very deeply. Within the context of intelligence, such foraging strategies may facilitate broad searches but leave open the question of whether deeper searches are also accomplished.

An additional challenge with information foraging is that if the practitioner misses the opportunity to acquire something, it may never again be obtainable. Like the elements of a fleeting interpersonal conversation, the original foraging behavior, if left un-captured, can never be recaptured. Indeed, there may be no indications that such a conversation even occurred.

A further consideration for the practitioner is the “self-marketing” of the information. Vivid stories market themselves much better than do flat ones. Exploited information that supports a favored hypothesis may be preferred over information that does not; an unfortunate reality is that little motivation remains for further exploration. The practitioner is human—she will not likely have a truly agnostic attitude about what she seeks and why.

Compounding this is the idea that sources and means for foraging are self-protective. For example, there is a presumption that sources will continue to communicate via specific means. The methods that capture those communications and the people that support them tend to seek justification. Assets may be kept active after their usefulness expires. The practitioner returns to the same sources over and over because they have been useful in the past and such attention helps keep those sources actively collecting.

Critically assessing what she is doing is one way the sensemaking practitioner may be able to overcome these limiting tendencies. By constantly asking herself how she is thinking about the issue, what she seeks, other perspectives, her assumptions, as well as relevant concepts such as self-deception or adversarial deception, the practitioner may diminish the impact that her preferences play on her foraging decisions.

Another part of this critical assessment is the consideration of the costs of foraging. As Herbert Simon notes,

What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.

The superabundance of information available on the Internet (and elsewhere) creates a “poverty of attention” to any one source. Rather, people skim across a great many sources.

A predator may go out seeking one type of prey and find none of it but there may be an abundance of some other kind of game. Within the context of intelligence such opportunism may or may not be appropriate (or even legal) when a technical system or an asset is tasked to provide information for intelligence. Lacking the desired information, a human source might opportunistically substitute what might be perceived as desired or desirable, even if it is not closely related to the issue at hand—or for that matter, even “true.” It is used because it satisfices for the immediate term.

Harvesting

A special case of foraging involves “harvested” information. Technical agencies that field systems to gather information also can be characterized by a different model, that of harvesting. The systems employed simply harvest that which lies within their purview, then process it and store it in silos— data repositories from which sensemakers subsequently must forage.

Such systems are efficient at creating broad collections; they tend to be inefficient and unreliable when very narrowly focused. Thus, directed rather than broad collection of specific phenomena is needed.

Technical collection systems tend to provide—even in the negative—what sensemakers want to find. This can create a potentially dangerous confirmation of an idea that may be invalid.

Automated retrievals from information repositories typically pro- vide sensemakers with what they believe to be the needed and relevant information—in short, their evidence. The evidence pertaining to specific issues arrives at the sensemaker’s desk and reports are issued. At first, the evidence is carefully scrutinized and the system that provides it assessed. As the process repeats, however, as it certainly did in the Cold War era, complacency may set in. Critical assessment of quality and quantity may cease.

Finally, no amount of foraging can discover valuable information if it has not been collected by some system—human or technical—in the first place.

Marshaling

What can be done to revolutionize the way information foraging is accomplished so as to overcome or at least mitigate these problems? Some answers lie in an understanding of marshaling. Part of the sensemaker’s practice is to turn foraged and gleaned information into evidence.

This is a broad activity, for if the issue has multiple explanations or future possibilities, then evidence will be information relevant to any, many, or even all of those explanations or possible outcomes. In order to make that determination, the sensemaker will need to have identified what those alternatives are and to have collected information (both dis-confirming and confirming) about them.243 This may require foraging from additional resources with all the attendant challenges discussed above.

Understanding

If we presume that foraging has yielded relevant and valuable information—evidence—on the issue under study, the next step is to determine what it means. This is the heart of sensemaking: evidence is dissected, reassembled

is the sensemaker focusing on individual actors, the actions of a collection of actors, the beliefs that guide the activity, or the processes that determine the actions of the collective?

The disaggregation of each of these perspectives and their associated stories provides a rich brew for sensemaking.

Synthesizing

Synthesizing is “the combination of ideas to form a theory or system.”245 Even as the intelligence professional analyzes the individual pieces of information, they are synthesized into a mental picture of the larger issue. Pieces of information are implicitly combined even when the sense- maker works within the yield of a particular foraging discipline or within a frame or reference. Such synthesis drives further foraging and analysis.

Synthesis needs to be explicit. In the example developed above, the intelligence professional is required to synthesize the differing trajectories

of the three principal actors, considering how their beliefs harden or soften their positions and how they are vulnerable to the actions, influences and processes of the groups. Doing so in a systematic fashion leads the intelligence professional to new insights about the situation: what is going on… and (from the U.S. perspective) what to do about it.

 Interpreting

Issues can be dissected and reconstructed in a variety of ways, creating different meanings. Sense must be made of these different meanings. Interpreting, or “the action of explaining the meaning of something,” is another component of sensemaking.246 We may say that whereas analysis and synthesis establish the what, interpretation establishes the so what.

A revolutionary approach to sensemaking now being undertaken by analysts from DIA, State, and CIA, is to engage in “adversarial briefing” of principals, where briefers adopt opposing perspectives for a thorough airing of the issue, complete with the participation of the principals themselves.

Communicating

New models of knowledge transfer recognize change in both message and medium. Social networking, peer-reviewed shared multimedia, and interactively blogged communications are examples of these new mediums. The message is short and subject to change by different contributors. Authority is based on consensus. The distinction—if it exists at all—between formal and informal communication is blurred. There are dangers here as authority and truth are no longer necessarily linked. One risk is that the “wisdom of a crowd” can in fact be the “madness of a mob”—a phenomenon occur- ring in both the public arena and within the IC’s blogosphere. In both arenas the loudest voices strive to bludgeon into silence those who would disagree, all the while advancing their egocentric or sociocentric positions. Scientific knowledge and empirical facts matter little in such cases.

In summarizing the “introspective works responding to…intelligence failures,” Charles Weiss agrees that intelligence practitioners’ failures include a lack of proper attention to hypotheses and data collection efforts that are contrary to what they regard as the most likely interpreta- tion of available information.”248 One danger is that the very judgment about which the sensemaker is least confident might be the one that turns out to be correct. The fallacy of depending on the communication of confidence levels relates to the fact that each assessment or report only fills in some unknown portion of the gaps in the sensemaker’s and policymaker’s knowledge.

a carefully considered, standardized metric of uncertainty could pro- vide one means of assessing and communicating confidence independently from the sensemaker. Weiss suggests that either Kent’s scale250 or its more recent instantiation by the Office of the Director of National Intelligence offers an appropriate means by which the uncertainty could be systematically captured.251 The challenges inherent in such metrics are twofold. First, the evidentiary statistics necessary for their use are “typically unavailable to intelligence analysts—or it they are available, must be based on small samples of past events.”252 Additionally, scoring the conclusions from such small samples across production lines and even from day to day by a single intelligence professional can be observed to be inconsistent. Steve Rieber discusses calibrating sensemakers as a solution.253 To date no such strategy has been implemented.

CHAPTER 5
A Practice of Understanding

Judgment in intelligence sensemaking, as in a number of other domains, likely improves as one progresses to higher levels of proficiency and expertise. However, prediction is both difficult and inherently unreliable because events of different kinds vary considerably in their inherent predictability.

Intuition

It is difficult to wrap appropriate words around the concepts that are at hand, thus care should be taken to make certain distinctions. Sometimes, judgments can be rapid, non-conscious, non-deliberative, and almost seem as if they are immediate perceptions and feelings rather than judgments.

Intuitive, or, as it is sometimes called, automatic thinking forms the basis for much of our personal sensemaking.260 It allows us to process complex inputs that far exceed the “span of immediate apprehension” of approximately seven “chunks” or elements that individuals can consciously process in working (or short-term) memory.261 Such thinking, for example, explains how people (mostly) drive successfully. Recent research revealing a correlation between cell phone use (especially texting) and accidents leads one to extrapolate that attention-requiring activities such as engaging in professional-level office conversation or dialing and conversing while mobile respectively impede otherwise effective automatic activities such as ten-finger typing or absent-mindedly but safely driving an automobile.

There is another side of intuitive reasoning that sometimes works in opposition to one’s survival. Intuitively reasoned responses to stress are often highly focused and narrow. Laurence Gonzales notes that in such reactions “the amygdala…in concert with numerous other structures in the brain and body, help to trigger a staggeringly complex sequence of events, all aimed at producing a behavior to promote survival.”266 But, in many cases, behavior is also locked down.

Intuitive or automatic thinking is a survival mechanism.268 Gonzales notes that such mechanisms “work across a large number of trials to keep the species alive. The individual may live or die.”269 But over time—and in reference to humans—generally more live than die, leading to evolution and the genetic transmission of the “reflex.” If a particular set of behaviors confers a survival value, that set can become more widespread in the population. Seen in this light, unease at entering an elevator at night could be a modern instance of sensing shadows in the tall grass.

On a shorter time horizon, people use experience-based intuitive patterns or mental models. These patterns or models direct how situations are perceived and how they are responded to.

We turn next to an examination of types of judgment, distinguishing between those that are skill-based and those that rely on “heuristics,” or learning through personal discovery whether rules of thumb are valid shortcuts to understanding an issue. We then link this dissection of judgment to the work of intelligence professionals.

Types of Judgment

First we must clarify the meaning of “judgment.” A judgment can be an observer’s belief, evaluation or conclusion about anything—one can form a judgment about anything of interest, including one’s own reasoning.275 Judgment also describes a process, surely more than one kind of mental process, by which one reaches a decision.

Judgment can be expressed as affective evaluation (example: That is a good thing), objective evaluation (It looks like a cat, but it is just a stuffed cat), or categorical assignment (My judgment is that this is a case of highway robbery). Judgment as process can also be described as apodictic, modal, or oral, among others.276

In terms of intelligence sensemaking, successful intuitive judgment arises from the tacit knowledge of experts who assess “normal” (in Kuhnian terms) situations, or as has been discussed above, the tame, familiar or regularly occurring kinds of problems (although they may be quite complex).

However, it should be noted that the combination of skill-based and heuristic-based intuition confers a benefit to mindful experts: a sense of when a case seems typical at first glance, yet there is something not quite right about it. While the less experienced person may be lulled into believing the case fits a certain type, expert decision makers react differently. They note some worrisome clue that raises questions. “Maybe this is not a typical case,” they venture. Eventually they may come to see that the case is in fact atypical. So informed, they make a different judgment, sometimes in disagreement with other experts. As we will now see, intuition certainly becomes a part of the intelligence process when practitioners make, or fail to make, useful and accurate predictions.

The Question of Predictability

Some in the IC argue that the pertinent aspects of all intelligence problems can be adduced, “if we only knew more” or “had the right algorithm or method.” However, we mislead ourselves if we believe that any messy problem can be resolved with a probability-juggling program. The authors are reminded of the observation, “There are the hard sciences and then there are the difficult sciences.” It is both impossible and inappropriate to attempt to remake social and cognitive sciences into emulations of calculational physical sciences. If the reduction were possible, someone would have achieved it, or would have at least made demonstrable progress toward its realization, in the 200-plus years during which psychology and the other “social sciences” have called themselves “sciences.” If reduction was appropriate, and we could get “the right information to the right person at the right time,” we would not need that right person—“truth” would be self-evident.

Some examples of mindful, heuristic-based decision making, especially pertinent because they involve the thinking habits of senior U.S. civilian and military officials as well as of their strategic advisors, are discussed in Neustadt and May’s Thinking in Time.292 The authors point out that a sensemaker’s awareness of historical decision making in even loosely analogous situations helps to keep at bay the further unsettling idea that the present circumstances constitute a “crisis.”

Anchoring,” which is the biasing of a judgment because of the framing of the initial question, and “attribute substitution,” arising from replacing a difficult question with an easier one, are two contributors to such flawed intuitive judgments.

Thinking About Anticipating

Jurisprudence, clinical psychology, and economic forecasting are all examples of domains where accurate prediction is difficult or impossible, and it is not terribly clear what it means for a person to be an expert in any of those fields. In the realm of jurisprudence, studies of the low rate of successful intuitive predictions about future recidivism among paroled offenders serves as one of many pieces of evidence showing that even highly experienced professionals in certain domains may be no better than laypersons at making intuitive judgments.

Anticipating Deception: Applying the Space-Time Envelope

A recurring worry within alert intelligence services is whether they are being deceived by their adversaries. From Troy, in the second millennium BCE, through to mis-direction schemes in World War II, and on to the lead-up to the Iraq invasion of 2003, adversarial deception has played a strong or decisive role in final outcomes.307

Implications of Visualizing Anticipation

Diagramming using Concept Maps (and related kinds of diagrams called causal maps and cognitive maps) has been used as a de-biasing technique for analysis under uncertainty. This use is well known in the field of business and strategic management:

in the pages of The Journal of Strategic Management, Gerard Hodgkinson and his colleagues added:

In addition to providing a useful means for gaining insights into the nature and significance of cognitive processes underpinning strategic decision making, this dynamic emphasis on antecedents, behaviors and consequences, renders causal cognitive mapping techniques particularly attractive as a potential means for overcoming the effects of framing (and possibly other cognitive biases) in situations involving relatively complex decision scenarios.313

Hodgkinson et alia investigated “the extent to which judgmental biases arising from the framing of risky decision problems [could] indeed be eliminated through the use of this particular cognitive mapping technique” and found cognitive mapping to be “an effective means of limiting the damage accruing from this bias.”314

The Roles of Intuitive Thinking in Intelligence Sensemaking

Given these considerations, what are (or should be) the roles of skills- based intuitive and heuristic-based intuitive thinking in intelligence sensemaking? Many, if not most, intelligence professionals have had a “feeling” about an issue and what is going to happen. Sometimes those intuitions are correct, particularly if the requirement entails real-time observation and situational awareness. When it comes to anticipatory sensemaking, however, the authors suspect that intelligence professionals may fare no better than does the average citizen in predictive situations.315

There are a number of reasons for this, not the least of which has to do with the availability of evidence, or relevant information. A somewhat persistent myth about intelligence is that its professionals have access to all the information they need and that they can get any and all other necessary information. This not only simply is not true but is likely highly undesirable. While it is true that intelligence professionals must make their assessments based on incomplete, often faulty, and sometimes deceptive information, at least they can do so. Forcing them to try to make sense of all the relevant information relating to an issue would likely burden them sufficiently so as to preclude anything but the most general findings being issued—if anything can be issued at all. Finally, as was discussed above in relation to figure 1 (see Chapter 3), complexity, ambiguity, and uncertainty increase as one moves from Descriptive to Estimative (or Anticipatory) Intelligence.316

intelligence professionals compete against other foreign intelligence organizations whose professionals may be as skilled at obfuscating what their factions, groups, or nations are doing as we are at making sense of what they are doing. Sometimes that other side is, in fact, better. And, like a closely matched sports event, the difference between valid and true sense- making versus invalid or untrue sensemaking—or even no sensemaking at all—might be the result of luck.

Since such Type 2 domains are ones in which the primary task goals involve the understanding and prediction of the activities of individuals or groups, accuracy and precision are elusive. Consequently, as has been noted, Type 2 domains are also characterized by tasks involving a lack of timely feedback and a paucity of robust decision aids. Further, if the object of study does not know what she or they will do, how can someone else predict it reliably? Therefore, is it any surprise that in such domains, intuition is limited in its usefulness? What are intelligence professionals to do?

It should be reiterated that although over-estimative errors in intelligence sensemaking, as has been noted, are unacceptable, under-estimative errors are even less tolerated. It is better to have warned and been wrong than not to have warned and been wrong. False alarms are better than misses. A warning about an attempt by another individual to set off a bomb on a subway system—which subsequently does not occur—generates far less uproar (if any at all) than does a failure to warn of an individual who in fact plans to blow up an airliner, and through anticipatory sensemaking, being able to catch him preemptively.

It is by no means obvious that simply throwing more information at a problem will make solving it any easier.

Future Vision: Red Brains, Blue Brains?

Darren Schreiber et alia used functional MRIs (magnetic resonance imaging tests) to assess how people associated with the U.S. Republican and Democratic political parties deal with risk. The researchers discovered that members of the two groups used distinctly different portions of their brains when making “winning risky versus winning safe decisions.”324 The authors note that the different portions of the brain play different roles in human cognition and conclude that

it appears in our experiment that Republican participants, when making a risky choice, are predominantly externally oriented, reacting to the fear-related processes with a tangible potential external consequence. In comparison, risky decisions made by Democratic participants appear to be associated with monitoring how the selection of a risky response might feel internally.325

While neurocognitive paradigms for intelligence sensemaking have not yet formally been identified or established, implications of this work—to the degree that intelligence professionals can speak to the concerns of decisionmakers who are, after all, particular political partisans—may be significant. The research to date shows that the cognitive mechanisms and especially the emotion-based attitudes of partisan sensemakers shape their reasoning as they assess uncertain and risky phenomena.

Looking Ahead

Intuitive reasoning is something that we do naturally, all the time. It cannot be prevented, is not easily neutralized, and it is sometimes useful and necessary in sensemaking. While it can be reliable when employed as the sole basis for actionably valid, predictive intelligence creation in Type 1 domains, it is highly fallible when used for intelligence creation in Type 2 domains.

What can be done to challenge and validate the surety one has about an intuitive intelligence judgment? Employing approaches to reasoning such as those found in critical thinking seminars and courses, especially as currently offered across the IC’s educational institutions, and developing skills that aid mindfulness (as discussed in the Preface), offer possible means of accomplishing calibrated reasoning.

CHAPTER 6 Considering Validation

How does one know if the knowledge that intelligence sensemakers create is itself valid? Does accuracy alone ensure validity? What was accurate when findings were communicated may not be accurate subsequently. This flux suggests a strong procedural basis for validation. For example, were steps followed to avoid perceptual errors and cognitive traps? Was the process documented? Were alternatives adequately explored? Given the inherent uncertainty in intelligence judgments, it remains possible that all the appropriate processes may be sufficiently applied and yet the judgment is wrong

Analogies from Other Fields

Medicine

Medical practice is at times presented as having notable similarities to intelligence practice. For example, with respect to validation, an ultimate metric for failure in medicine is that the patient dies. But is medicine successful if the patient lives? At what quality of life and for how long are two additional questions. Perhaps death with a minimum of suffering is the most favorable medical outcome—is this a success? Depending on the specifics of the case, maybe it is.

Jurisprudence

Jurisprudence is an adversarial system in which the ultimate confrontation is a trial wherein two advocates make inferences about evidence to argue opposite sides of a case before an impartial third entity or body (often of non-experts).

So, how does one measure success? There are at least five points of view involved in jurisprudence: That of each of the advocates, the judging entity (a jury or judge) the accused person, and the community or government. Each party, depending on the verdict, has a different metric for success. In certain types of cases such as those involving child molestation or alleged terrorism, the accused person tends to be deemed guilty by the community, prosecuting advocate, and government even if exonerated. In all cases where an opinion—particularly in the media—runs counter to the majority’s views, the conclusion may be made that the court failed to render the “right” verdict

Science

Science involves a number of metrics that include a sound method of documenting both process and results, as well as replication. Work is considered preliminary and non-definitive if it has not been replicated.

despite the intellectual appeal of theories about genetic links to specific behaviors, “there are few replicated studies to give them heft.”331 In other words, the underlying theories may not be sound.

Science depends on refutation of alternative hypotheses, and replication studies attempt to refute that which has been shown. It is quite acceptable to be wrong so long as one admits it when the fact becomes apparent. Indeed, one model for science is that of competitive cooperation. Scientists attempt to tear down the new work of colleagues—without resorting to personal attacks. This dialectic approach may last generations or longer. In the process new knowledge is discovered and—if it cannot be refuted—validated.

Replication in Intelligence

The inability to replicate much of the process of sensemaking in intelligence limits the application of this indispensable practice of science. The pressures of real-time production inhibit the re-visitation of past judgments, although with at least one recent National Intelligence Estimate, the repetition of “alternative analysis” led to the questioning and revision of the original conclusions. Essentially, any meaningful replication of intelligence phenomena can only be accurately made in foresight, as in a National Intelligence Estimate.

when intelligence students faced with a scenario involving three fictitious nations at odds with each other develop a common set of hypotheses regarding who will initiate a war and with whom, and are then given a finite set of evidence and a common method such as the Analysis of Competing Hypotheses, they come to similar conclusions as to which hypotheses are the least likely and therefore which eventualities can be expected.334 Unfortunately there does not yet exist a similar body of results for real intelligence problems interpreted through the lenses of different intelligence disciplines and sources.

Yet, replication remains an important metric of the intelligence sensemaking process. As Caroline Park notes, “[the] basic reason research must be replicated is because the findings of a lone researcher might not be correct.”

Lynn Hasher, David Goldstein, and Thomas Toppino concluded that confidence in assertions increases through repetition of the assertions in situations when it is impossible to independently determine their truth or falsity.338

Validation in Foresight and Hindsight

People—and intelligence practitioners and their customers are merely people—evaluate judgments they have made in hindsight. They believe, according to Mark and Stephanie Pezzo, “that one could have more accurately predicted past events than is actually the case.”340 Thus, hindsight occurs at least in part because as people make sense of “surprising or negative” events, “the reasons in favor of the outcome [are] strengthened, and reasons for alternative outcomes [are] weakened.” Further, in hindsight all the relevant facts may be known whereas in foresight this is not the case. But evaluating “mistakes” in hindsight obscures an important point made clear by Taleb: Mistakes can only be determined as such by what was known at the time they were made and then only by the person making the mistake.341 In other words, mistakes need to be evaluated from the points of view held in fore- sight. And seen from that perspective they may not be mistakes at all.

Applied to intelligence sensemaking, this means that many so-called intelligence errors and failures may, in fact, be well-reasoned and reasonable judgments based on what is known prior to the decision. Certainly, when viewed in hindsight they were wrong. But in foresight they were accurate and valid to the best of the sensemaker’s abilities. How can this enduring problem be mitigated?

Validating the Practice of Intelligence Sensemaking

What else contributes to bringing about validated sensemaking? If a method does not do what it is commonly purported to do, is it invalid? This is one question that has been raised with regard to the Analysis of Competing Hypotheses, or as it is commonly known, ACH. Richards Heuer, Jr. initially developed the method for the detection and mitigation of attempts at adversarial denial and deception.343 ACH forces consideration of alternative explanations for, or predictions about, phenomena.344 It forces consideration of the entire set of evidence, not “cherry-picked” trifles that support a favored hypothesis.

a study by MITRE failed to show that it does eliminate the confirmation bias.345 Both the MITRE study and an earlier one by NDIC student Robert Folker do suggest that ACH is of value when used by novice intelligence professionals. However, Folker tentatively concluded that experts seem not to be aided by the method.346 Is it still a valid method for intelligence sensemaking?

Perhaps it is. The method provokes detailed consideration of the issue and the associated evidence through the generation of alternative explanations or predictions and the marshaling of the evidence. It asks the sensemaker to establish the diagnosticity of each piece of evidence.

ACH further makes explicit the fact that evidence may be consistent with more than one hypothesis. Since the most likely hypothesis is deemed to be the one with the least evidence against it, honest consideration may reveal that an alternative explanation is as likely or even more likely than that which is favored. The synthesis of the evidence and the subsequent interpretations in light of the multiple hypotheses is also more thorough than when no such formalized method is employed.

Indeed, Robert Folker’s “modest” experiment in applying qualitative structured methods—specifically ACH—to intelligence issues showed that “analysts who apply a structured method—hypothesis testing, in this case— to an intelligence problem, outperform those who rely on “analysis-as-art,” or the intuitive approach.”347 Simply put, Folker experimentally showed that method improves the quality of practitioners’ findings. Folker’s study offers evidence that “intelligence value may be added to information by investing some pointed time and effort in analysis, rather than expecting such value to arise as a by-product of ‘normal’ office activity.”348

One variation on ACH implementation provides a structured means of developing issues. As applied by the faculty and students of the Institute for Intelligence Studies at Mercyhurst College, practitioners begin with a high-level question and use sequential iterations of ACH to eliminate alternative explanations.349 The next phase takes the “non-losers” and develops them further. Another round is conducted and again the “non-losers” are selected and further developed. While this could generate a plethora of branching explanations, in reality it is the author’s experience that it tends to disambiguate the issue fairly efficiently. At worst the structuring inherent in the method leaves the sensemaker with an in-depth understanding of the issue; at best, a couple of eventualities and their likely indicators are determined. Assets can then be tasked, foraging conducted, and more exact determinations made as the issue develops.

Johnston observed that the IC has at its disposal “at least 160” [ana- lytic methods]…but it lacks “a standardized analytic doctrine. That is, there is no body of research across the Intelligence Community asserting that method X is the most effective method for solving case one and that method Y is the most effective method for solving case two.”353 As referenced here, such a doctrine arises out of knowledge that the specific methods are valid, in other words it has been demonstrated empirically that they actually do what they claim to do. Such a doctrine proffers a menu of sensemaking options dependent on the goals of the sensemaker. The current model, where validity is presumed by intelligence professionals because they are taught the method(s) in the community’s training schools, is insufficient because—at the most basic level such a metric is insufficient for determining validity. Further, there is little sense of what methods are appropriate in what situations. A claim of “we always do it that way,” is known to be insufficient but remains part of the “corporate analytic tradecraft.”

Heuer noted that “intelligence [error] and failures must be expected.”354 One implication of this assertion is that intelligence leadership cannot fall back on a “lack of skills” excuse when the next major intelligence failure occurs. However, without validating a canon of method and a taxonomy to characterize its use, intelligence professionals will remain hamstrung in their efforts to make fuller sense of threatening phenomena, increasing the likelihood of error and failure.

Seeking Validation: Toward Multiple Methods

Within the canon of social science method lies an approach to sense- making that may offer intelligence practitioners a means of disambiguating the wicked mysteries as well as the hard puzzles they face daily. Even in current practice, intelligence practitioners employ this approach when they do not rely on merely one method for sensemaking. Multi-method intelligence sensemaking explores complex issues from multiple perspectives. Each method used—such as ACH—provides an incomplete understanding of the issue, leaving the intelligence professional the task of making sense of the differing sensemaking conclusions.

intelligence professionals who engage in a “multiframe” sensemaking approach consider issues from multiple points of view created from the intersections of action- and process-focused vantage points and the perspectives of the individual and the collective. As developed by Monitor 360 for the National Security Agency’s Institute for Analysis, it facilitates sensemakers’ developing different answers to the intelligence question at hand.356 They must combine the differing results, in other words, synthesize and interpret partial answers, in order to better understand the issue underlying the question and to determine a best (at the time) understanding of the issue.

The lexicon of multi-methodology provides a term for this combinatorial activity: triangulation, or pinpointing “the values of a phenome- non more accurately by sighting in on it from [the] different methodological viewpoints employed.357 This is a process of measurement—which to be useful (accurate) “must give both consistent results and measure the phenomenon it purports to measure.”358 In other words, triangulation requires that the methods employed are repeatable and valid. Intelligence creation requires that those methods be applied with rigor.

CHAPTER 7
Making Sense of Non-State Actors: A Multimethod Case Study of a Wicked Problem

we find that a diversely practiced, multimethod approach that does incorporate a specific process, organizing principles, and an operational structure can fulfill the need for 21st century intelligence sensemaking. Such an approach reflects a Kendallian approach to intelligence sensemaking: It collaboratively paints a picture for a decision-maker rather than presenting a “scientific fact.”

Introducing the Wicked Problem of Non-State Actors

It is commonly thought that non-state actors are emerging as a dominant global force in the realm of national and international security, yet conclusive evidence confirming this belief is lacking. Part of the challenge is that non-state actors fit the profile of a wicked problem. While it is true they can be identified (the name accomplishes this—they are non-state versus state actors), there is no commonly accepted definition.361 In other words, non- state actors are defined by what they are not, leaving room for disagreement as to what they are. Further, some non-state actors can be characterized as “good” and others as “bad.” Differing points of view about whether a particular non-state actor is “good” or “bad” leads to varying characterizations of their activities.

National Intelligence Council, “Nonstate Actors: Impact on International Relations and Implications for the United States,” Conference Report, August 2007, URL: <http://www.dni. gov/nic/confreports_nonstate_actors.html>, accessed 10 May 2010, The Conference Report suggested that “Nonstate actors are non-sovereign entities that exercise significant economic, political, or social power and influence at a national, and in some cases international, level. There is no consensus on the members of this category, and some definitions include trade unions, community organizations, religious institutions, ethnic groupings, and universities in addition to the players outlined above”

Issues involving non-state actors lack clear definitions and are resistant to traditional intelligence approaches due to their open-ended nature; potential solutions to problems are neither clearly right or wrong; and difficult-to-discern and complex inter-linkages exist, although drivers for issues involving non-state actors can be identified

Three Approaches to Making Sense of Non-State Actors

The starting point for this case study was a 2007 National Intelligence Council (NIC) Desktop Memorandum that analyzed key findings from a series of seminars co-hosted with the Eurasia Group, a global political risk research and consulting firm.

The Memorandum observes that non-state actors are of interest “because they have international clout, but are often overlooked in geopolitical analysis.” The implicit but demanding questions of why and how much non-state actor “power” and “influence” have increased worldwide was not answered.

363 See the Eurasia Group web site, URL: <http://www.eurasiagroup.net/about-eurasia- group>, accessed 14 May 2010.

364 National Intelligence Council, “Nonstate Actors: Impact on International Relations and Implications for the United States,” Conference Report, August 2007, URL: <http://www.dni. gov/nic/confreports_nonstate_actors.html>, accessed 27 April 2010, 2. Cited hereafter as NIC, “Nonstate Actors.”

Key Findings of the Mercyhurst Study on Non-State Actors

Students in the Mercyhurst College Institute of Intelligence Studies (MCIIS) focused on the roles non-state actors play and their expected impact in Sub-Saharan Africa over the next five years (results), and on building a multi-methodological paradigm for considering the issue (process).369 Within this context, three additional questions were raised:

• What is the likely importance of [Non-State Actors] vs. State Actors, Supra-State Actors and other relevant categories of actors in Sub- Saharan Africa?
• What are the roles of these actors in key countries, such as Niger?
• Are there geographic, cultural, economic or other patterns of activity along which the roles of these actors are either very different or strikingly similar?370

The students developed a scoring system for both lawful and unlawful non-state actors, in terms of the socio-political environment, and applied this index to all 42 Sub-Saharan African countries (figure 7). The scoring characterized the roles of non-state actors vis-à-vis government and non-government interactions based on four drivers: An “ease of doing business” variable and a contrasting “corruption perception” variable; a democracy variable and a contrasting failed states variable.373 Stable and failing states were revealed to have differing interactions with non-state actors. In the former, non-state actors were lawful actors who tended to have government-sanctioned role potentials, whereas in the latter they were typically unlawful actors engaged in anti-government roles.

Mapping significant multinational corporations, NGOs, and terrorist organizations to specific countries as representative of non-state actor activity revealed correlations between role potential spectra and geospatial data, whereby each generally supported the other.375 Thus, geospatial sense- making tended to confirm the conclusions derived from the non-state actor role spectra.

Key Findings of the Least Squares Study on Non-State Actors

The Least Squares study of non-state actors began with the hypothesis that “non-state actors emerge in vacuums and voids.”376 Their study focused on the issue of violent and non-violent non-state actors but also explored a set of contingent methodological approaches. The inquiry sought to contribute novel understanding of non-state actors by

synthesizing available data and disparate taxonomies,…by generating and testing hypotheses concerning the key dynamics driving the transfer of power from states to [non-state actors] and favoring the emergence of novel [non-state actors] under globalization; and…by investigating the development of methodologies that might be most useful for future research.377

Two key findings revealed the critical role of environmental knowledge and of public expectations in motivating non-state actors, both as individuals and as members of the collective. Such findings were found to be significant to efforts aimed at mitigating the recruitment of specific Al Qaeda- associated individuals to assail the United States. Additionally, the team found that an approach based on critical thinking led to reasoning pathways that likely would not have been noticed or explored had a more intuitive and less rigorous approach been employed.378

Approaches and Methodologies Thinking Critically about the Issue

In order to impose structured thinking on a highly unstructured problem, the NIC advisor to, and the members of LSS first inventoried their own understanding of the non-state actor issue using the “eight elements of reasoning” developed and espoused by the Foundation for Critical Thinking and used throughout much of the IC.379 These elements include:

• Question at issue (What is the issue at hand?)
• Purpose of thinking (why examine the issue?)
• Points of view (What other perspectives need consideration?)
• Assumptions (What presuppositions are being taken for granted?)
• Implications and consequences (What might happen? What does happen?)
• Evidence (What relevant data, information, or experiences are needed for assessment?)
• Inferences and interpretations (What can be inferred from the evidence?)
• Concepts (What theories, definitions, axioms, laws, principles, or models underlie the issue?

Literature Consultation

Concurrent with their critical thinking, MCIIS, LSS, and others examined key academic and applied-academic works related to the assessment of non-state actors. Notable among them, work by Bas Arts and Piet Verschuren describes a qualitative method for assessing the influence of stakeholders in political decision-making.380 The “triangulation” referred to in their title encompasses “(1) political players’ own perception of their influence; (2) other players’ perceptions of the influence brought to bear; and (3) a process analysis by the researcher.”

Another contribution in the applied realm came from recent work by a new generation of military (and ex-military) authors who see the rise of non-state actors as a seminal event that will drive U.S. national security strategy. Among these sources is Warlords Rising: Confronting Violent Non-State Actors, whose authors anchor their work in open systems theory (the concept that actors and organizations are strongly influenced by their environment).382 In particular, they ask what environments give rise to violent383 non-state actors, what sustains them, and how changes to those environments might disrupt them.

Application: Indicators of Non-State Actor Power in Africa

The students were able to validate their findings employing three different methods as well as different evidence sets and also assess their methodological validity. This kind of meta- sensemaking could constitute a bridge between now-traditional IC efforts and a revolutionary approach to building a sensemaking argument in official circles.

Of note is a remark by project supervisor Professor Wheaton: “The big advantage [of the multimethodological approach] was the ability to see similar patterns crop up again and again by looking at the data in different ways. This increased their [the students’] confidence enormously.”384 Additionally, given the temporal context (short) and the scale of the project (large) a multimethodological approach was perhaps the only means of tackling the problem.

Application: A Multi-Disciplinary Workshop on Non-State Actors

Participants used three frameworks to consider the environment within which the three groups exist and operate.

• Points of segmentation are the boundaries or borders between and among groups of people, where the degree of disagreement on issues is indicated numerically.386 Points of segmentation can track inherent characteristics such as gender or ascribed cultural differentiators such as Sunni or Shiite. The set of points distinguishes one individual or group from another and identifies possible points of cooperation and conflict that can be exploited. Specific values for points of segmentation are derived from an expert assessment of the strength of the actors’ expressed attitudes, reinforced by observable behavior. They distinguish one individual or group from another, and identify the points most suitable for exploitation by the protagonist.
• Prospect theory, originally developed by Kahneman and Tversky, posits that “people tend to be risk-preferring when facing long shot risks involving significant gains, such as betting on race horses, and are risk averse when facing significant losses: [in other words, when] buying a home or car insurance” respectively.”388
• Institutional interactions is the name associated with a systematic model that allowed workshop participants to explore the complex roles non-state actors play as they influence (and are influenced by) overlapping institutional capabilities and needs.389 The participants concluded that even a simple model of institutional networks has enormous complexity—or high entropy—making it a good candidate for a subsequent in-depth modeling project. Due to imposed time constraints, development and application of the modeling was not completed.
• Morphological Analysis was identified as an additional approach through the institutional interactions method. Morphological analysis considers an entire space of possible implications opening the way for follow-on disambiguation (perhaps using additional multimethodological approaches) in order to abductively and soundly derive the kind of judgments that become useful knowledge.390

Their discussions and modeling, however, supported the Warlords Rising thesis: that environment is a critical factor in understanding the emergence and roles of non-state actors.

Critical Assessment: Lessons Learned from the Study of Non-State Actors

No matter what the methodological approach, project participants emphasized that close attention to environmental factors remains a key to understanding non-state actors. Nonetheless, even those approaches that emphasized environmental factors fell prey to certain inadequacies.

Changes in the Roles of Non-State Actors: An Alternative View

A systematic review of what was done and not done in the three non-state actor studies provides insights into how critical thinking can com- bine with multimethodological, mindful sensemaking, to provide a paradigm for 21st Century intelligence creation and its active communication to policymakers in a fashion that transcends the Sherman Kent tradition. This review is facilitated by employing the ten elements of reasoning noted above.

• Question: the beginning question of the NIC-Eurasia Group seminars was, “If non-state actors are emerging as a dominant global force, where is the evidence?” In other words, while there appears to be a consensus that they are a dominant global force, where is the formal evidence?
• Purpose: Determine whether or not there is evidence that non-state actors are emerging as a dominant global force. This problem is one of basic research to determine if the evidence in fact exists. However, the underlying issue of how we might measure relative power must first be conceptualized and addressed.
• Points of View: As we consider the original and complementary studies, there are two predominant points of view at issue: first, that of the NIC and its customers—who may believe that non-state actors are an emerging global force and want to quantify this shift in influence and power. The other, unavoidable point of view is that of non-state actors—some of whom would believe they are an emerging force and some who would not believe they are.
• Assumptions: The use of the term “non-state actor” as an apparent all-encompassing term in the initial problem question and statement presumes an initial understanding and consensus about what is or is not a non-state actor. This is actually inaccurate as the differing foci of the three groups make clear. However, the differences in this case become evident in hindsight although measures could be taken in foresight to at least check the understanding of different groups engaged in collaborative assessments.

Greater precision of the term non-state actors is needed. Differentiating between benign and non-benign non-state actors is a first step. Subsequent refinements of “benign non-state actors” into non- governmental organizations, multinational corporations, and super- empowered individuals is also useful. A similar set of distinctions within the set of violent non-state actors is also necessary. Then, a crosscheck among the teams must be accomplished so that consensus on the meaning and use of these terms is achieved.

Another assumption involves what is meant by the term “dominant global force.” Again, both greater precision and clarity is needed in coping with this assumption. One key question is, “Exactly what does dominant global force mean?” One answer to this could be that everywhere on the planet non-state actors are the force affecting politics and life. Such a simplified and simplistic view is likely inaccurate, and a range of political process models—among them those of the “rational actor,” of “bureaucratic politics,” and of “organizational process,” need to be parsed.

• Implications and Consequences: The consideration of implications and consequences means to anticipate and explore the events that follow a decision, and to put in play especially the interpretive aspect of sensemaking. In the context of non-state actors, it means to explore what happens if non-state actors are (or are not) emerging as dominant global forces and we are right or wrong about their power. Regardless of whether non-state actors are a dominant global force, if their influence is underestimated then surprises can be expected: Some non-state actor is likely to act in a fashion that is completely unexpected and with unanticipated results. On the other hand, overestimating the influence of non-state actors might create self-fulfilling prophecies. If, though, the influence of non-state actors is accurately measured it may be possible to mitigate that influence (where the non-state actors are acting on interests at odds with those of the United States). Alternately, where non-state actors are acting in consonance with the interests of the United States or are able to exploit opportunities put in place to get them to be helpful, the United States fulfills its goals.

Finally, the sensemakers’ interpretation of likely actions or events allows the implications and consequences of those actions to be considered, even if absolute prediction is elusive. Here, in the con- text of collaborative sensemaking through the communication of intelligence to a policymaker, we understand the admonition of Sherman’s Kent’s contemporary critic, Willmoore Kendall, that intelligence most critically “concerns the communication to the politically responsible laymen of the knowledge which…deter- mines the ‘pictures’ they have in their heads of the world to which their decisions relate.”392 This vision suggests communication of intelligence as an “insider” rather than offering “intelligence input” at arms length in the Kent paradigm.

• Evidence: What evidence is needed to determine that non-state actors are, and as importantly, are not an emerging dominant global force? As we have seen, each group gathered and sifted consider- able information on non-state actors, some of it highly relevant to the central question and some not. To the best knowledge of the authors, each of the three groups chose and evaluated evidence only with inductive logic. They did not take advantage of a means, avail- able in particular to a community with robust intelligence capabilities, to deductively eliminate one of the two possibilities.
• Inferences and Conclusions: With three different and independent efforts, the challenge lies in ensuring a useful triangulation of the results of those potentially disparate efforts. The Mercyhurst approach (internally triangulated) found that non-state actors, both legal and extralegal, are least effective in authoritarian states.

the Least Squares workshop demonstrated that within the context of either failed or failing states, expectations and perceptions of the public, or the political environment, are key drivers in anticipating the likelihood of actions by (violent) non-state actors. Strident or acrimonious expression of dissent that arises when domestic and international political/economic issues reinforce each other within the United States and Europe suggest a possible correlation in post-industrial states. This leads to a general conclusion that when expectations are at odds with situational reality, non-state actor activity increases.

• Concepts: Not only the assumptions, but other concepts as well were in play at multiple levels in the non-state actor case studies. The very notions of “non-state actor” and the ideas of democracy, authoritarianism, and anarchy needed clarification, ideally through well-grounded, empirical as well as theoretical research, to ensure common understanding.
• Alternatives: If non-state actors are not emerging as a dominant global force, then what can we say about their global role? Is their influence staying the same? Is it diminishing? Given a credible means of measuring change in the influence and power of non-state actors, the next step in this study of non-state actors would be to examine hypotheses generated from these alternative questions.
• Context: As has been repeatedly noted, non-state actors present both a challenge to U.S. interests and an opportunity for advancing those interests. The U.S. would like to mitigate the challenges and take advantage of the opportunities. How to make that happen in domains and regions of little existing U.S. influence or of waning U.S. and Western influence becomes a key concern as the United States strives to carry out a meaningful global role. Future attempts to make sense of the role of non-state actors may benefit from tap- ping into the larger context of recent policy-relevant literature on the problem of fragile states in applied academic journals.

Moving Beyond a Proto-Revolution

Microcognition and Macrocognition in the Study of Non-State Actors

There emerge two very general domains of which intelligence professionals must make sense: that of the relatively static, state-based system and that of the much more dynamic non-state actor. Of course, these do not exist in isolation from one another. There are boundaries, interstices, and points of segmentation; there is considerable overlap when one usurps or adopts the actions of the other. Further, the separate domains of domestic and foreign areas of interest and action, embraced by the Kent model of intelligence creation and communication, have been superseded by an indivisible, world- wide web of personal and organizational relationships. Broadly speaking, the “classic” model of intelligence sensemaking largely sufficed and perhaps continues to suffice when issues remain clearly tied to the political entities associated with the Westphalian system of state-based power.

However when dealing with non-state actors, a new, revolutionary paradigm becomes essential for making sense of issues as well as their interactions with the states of the other paradigm. In the former, a traditional, intuitive and expert-supported approach was largely adequate. In the case of the latter, as is glimpsed in this case study, a more rigorous approach is required.

In national intelligence terms, practitioners and their customers work in a macrocognitive envi- ronment as they manage the uncertainty they face in dealing with wicked problems.

Macrocognition, then, includes a focus on process as well as results—what we have labeled mindful, self-reflective sensemaking.

Klein et alia observe that intelligence professionals and decision- makers traditionally are “microcognitively” focused. That is, like those who follow in the Sherman Kent tradition, they are concerned with solving puzzles, searching, and “estimating probabilities or uncertainty values” for different phenomena of interest.403 As has been discussed, such an approach still may be suitable for solving tame problems or those of the Type 1 domain. Thus, microcognition describes the reductionist foci of the current intelligence “analysis” paradigm. However, this is not sensemaking, which requires another approach.

The transition or shift to macrocognition requires a focus on “planning and problem detection, using leverage points to construct options and attention management.”

Elements of the foregoing case study exemplify this strategy. Both the Mercyhurst Role Spectrum Analysis and the Least Squares Points of Segmentation identified potential leverage points that revealed truths about non-state actors, leading to more robust problem detection. A next step would have been to take the triangulated results from all the deployed sensemaking methods and use the results to construct options for dealing with nonstate actors in multiple environments. Such a macrocognitive approach would allow more persistent attention to the anti ipation of the broad course of events (in this case involving non-state actors), in contrast to a microcognitive focus on predicting more isolated and specific future incidents.

Next Steps in Revolutionary Sensemaking about Non-State Actors

The foregoing elaboration of non-coordinated sensemaking activities, even with its limitations, moved beyond the traditional model of intelligence creation. It specifically identified the multiple approaches taken by independent teams who used alternative schema and methods that, perhaps unsurprisingly, resulted in a broader understanding of the problem.

Triangulation was largely informal both within and between the groups. Thus, the work met the criteria for a transitional intelligence sensemaking project. The participants in all three efforts engaged in critical thinking to one degree or another. All were also mindful of the wicked issue of non-state actors and its significance.

the adoption of the new paradigm for sensemaking depends on bringing into play a cooperative spirit of science and scientific inquiry to the process of intelligence creation and communication. Mindful, critical thinking-based, multimethodological approaches to analysis, synthesis and interpretation are one means of doing this. Additionally, a means needs to be found to ensure that this approach to sensemaking remains rigorous. This becomes the subject of the next chapter.

CHAPTER 8
Establishing Metrics of Rigor

Defining Intelligence Rigor

I know the distinction between inductive and deductive reasoning. An intelligence officer is inherently inductive. We begin with the particular and we draw generalized conclusions. Policymakers are generally deductive. They start with the vision or general principle and then apply it to specific situations. That creates a fascinating dynamic, when the intelligence guy, who I call the fact guy, has to have a conversation with the policymaker, who I tend to call the vision guy. You get into the same room, but you clearly come into the room from different doors. The task of the intelligence officer is to be true to his base, which is true to the facts, and yet at the same time be relevant to the policymaker and his vision. That’s a fairly narrow sweet spot, but the task of the intelligence officer is to operate in that spot.

— GEN Michael V. Hayden (Ret.), former Director of the Central Intelligence Agency and the National Security Agency

Michael Hayden’s view that the intelligence officer needs to operate in the “sweet spot” linking intelligence and policymaker cognitive worlds coincides with the aim of the sensemaking paradigm. To bring these two worlds together, intelligence professionals can take advantage of the opportunity to meld their fact-based inductive tendencies with the visionary, deductive model of policymakers through the application of collective rigor to well-conceived questions. This approach allows intelligence professionals to embrace a triangulation on wicked problems from their professional perspective, and to improve their chance to communicate with policymakers whose circumscribed comfort zone may accept or even welcome wicked problems as opportunities to apply their vision to bring about politically rewarding solutions.

At present most tradecraft for sensemaking triangulation remains intuitive, operating in the realm of tacit knowledge. Thus, part of a revolution in intelligence requires that more formal and explicit means of triangulation be developed. It may be that some existing analytic tradecraft, when conscientiously applied, will improve synthesis and interpretation. Another option is to explore and experiment with new tools for conceptualizing rigor in information analysis, synthesis and interpretation.

Rigor in sensemaking can refer to inflexible adherence to a process or, alternatively, to flexibility and adaptation “to highly dynamic environments.”408 As proponents of the latter approach, Daniel Zelik, Emily Patterson, and David Woods recently reframed the idea of rigor into a more manageable concept of “sufficiency.”409 In the applied world of sensemakers, then, an apt question is: “Were sufficient considerations made or precautions taken in the process of making sense of the issue?” Zelik et alia observe that this requires a “deliberate process of collecting data, reflecting upon it, and aggregating those findings into knowledge, understanding, and the potential for action.”410 In order to achieve answers to this question Zelik et alia developed an eight-element taxonomy of sufficiency and a trinomial measurement of rigor: Each element was calibrated in terms of high, medium or low rigor. In their examination of information products, an overall score could be computed that, in intelligence terms, would communicate to both the practitioner’s management and to consumers the rigor of the crafted intelligence product.

Attributes of the Rigor Metric

Hypothesis Exploration describes the extent to which multiple hypotheses were considered in explaining data. In a low-rigor process there is minimal weighing of alternatives. A high-rigor process, in contrast, involves broadening of the hypothesis set beyond an initial framing and incorporating multiple perspectives to identify the best, most probable explanations.

Information Search relates to the depth and breadth of the search process used in collecting data. A low rigor analysis process does not go beyond routine and readily available data sources, whereas a high rigor process attempts to exhaustively explore all data potentially available in the relevant sample space.

Information Validation details the levels at which information sources are corroborated and cross-validated. In a low-rigor process little effort is made to use converging evidence to verify source accuracy, while a high-rigor process includes a systematic approach for verifying information and, when possible, ensures the use of sources closest to the areas of interest.

Stance Analysis is the evaluation of data with the goal of identifying the stance or perspective of the source and placing it into a broader context of understanding. At the low-rigor level an analyst may notice a clear bias in a source, while a high-rigor process involves research into source backgrounds with the intent of gaining a more subtle understanding of how their perspective might influence their stance toward analysis- relevant issues.

Sensitivity Analysis considers the extent to which the analyst considers and understands the assumptions and limitations of their analysis. In a low-rigor process, explanations seem appropriate and valid on a surface level. In a high-rigor process the analyst employs a strategy to consider the strength of explanations if individual supporting sources were to prove invalid.

Specialist Collaboration describes the degree to which an analyst incorporates the perspectives of domain experts into their assessments. In a low-rigor process little effort is made to seek out such expertise, while in a high-rigor process the analyst has talked to, or may be, a leading expert in the key content areas of the analysis.

Information Synthesis refers to how far beyond simply collecting and listing data an analyst went in their process. In the low rigor process an analyst simply compiles the relevant information in a unified form, whereas a high- rigor process has extracted and integrated information with a thorough consideration of diverse interpretations of relevant data.

Explanation Critique is a different form of collaboration that captures how many different perspectives were incorporated in examining the primary hypotheses. In a low-rigor process, there is little use of other analysts to give input on explanation quality. In a high-rigor process peers and experts have examined the chain of reasoning and explicitly identified which inferences are stronger and which are weaker.

The tradecraft underlying these attributes assesses the domains of intelligence foraging and intelligence sensemaking (analyzing, synthesizing, and interpreting) as described here. The study by Zelik et alia reveals that what might at first glance be considered the application of a low level of rigor really is merely a function of a varying distribution of rigor applied among the attributes.

This distinction suggests a profound insight, namely that information search is perceived as more highly valued than information synthesis. In the case above, a definitive judgment about this insight cannot be made, as the other attributes of the two assessments were not identical. Still, it seems clear that at least the participants in the study were still wrestling with a consideration formally discussed by Richards Heuer, Jr. (and many others before and since—including Moore and Hoffman above): How much information is necessary for effective sensemaking?

Assessing Sensemaking Rigor in Studies of Non-State Actors

Rigor and the NIC-Eurasia Group Effort

The NIC-Eurasia Group effort (summarized in figure 12 and column one of table 7) garnered the fewest points of the three groups. Hypothesis Exploration — Low: The NIC-Eurasia Group memorandum noted that non-state actors are of interest “because they have international clout, but are often overlooked in geopolitical analysis.” The implicit but demanding questions of why and how much non-state actor “power” and “influence” have increased worldwide were not answered, nor was a time frame established. This failure to broaden the hypotheses beyond the initial framing of the issue led to a lack of incorporation of multiple perspectives to identify at least “best,” and perhaps most probable answers to these questions.

Rigor and the Mercyhurst Effort

The Mercyhurst students relied on published data. No evidence of consultation with external experts was evident. While this was to be expected given the demographics of a student team, it nevertheless led to a score of Low for the Specialist Collaboration metric.

Rigor and the LSS Effort

The LSS social science study of non-state actors scored the highest of all three groups, earning a high in each metric save one, Information Validation, where they scored a Medium. In this case, while converging information was employed to cross-validate source accuracy for the evidence closest to the areas of interest, a systematic approach for doing this was not evi- dent, resulting in the lower score.

Since the entire team was made up of specialists, their score in Specialist Collaboration should come as no surprise. Similarly, the involvement of diverse social scientists as well as the involvement of external peers foreordained that the Explanation Critique would be rigorous. All the individuals brought differing perspectives as they identified the strengths and weaknesses of each other’s inferences and conclusions. Finally, an explicit multi-methodological approach forced consideration of diverse interpretations of the evidence—a highly rigorous example of Information Synthesis.

Observations and Discussion

It is no accident that the traditional means by which assessments of such issues are created, as evidenced by the NIC-Eurasia Group effort, resulted in a relatively weak score, whereas the highly rigorous, critical-thinking based, multimethodological effort by a collaborative team of diverse experts led to a relatively high score (a comparison highlighted by figure 15).

Another advantage of graphic analysis using Zelik et alia’s metric is that more information can be clearly conveyed. For example, in examining the composite efforts of all three groups of participants in the non-state actor study, it is evident that Information Validation could have been improved through the use of a more rigorous systematic approach, ensuring that the sources were deemed valid and “closest to the areas of interest.”415

there appear to be several rea- sons why information is often not fully validated in intelligence work. First, validation is difficult and the intelligence professional may decide the result is not worth the effort, or that initial conditions suggest validity. “Information hubris”—arising when similar information has without negative con- sequence been presumed or found to be valid, may compound this effort. Wishful thinking and belief in the infallibility of the source are other factors that may contribute to this pathology. Finally, information uncertainty may allow it to resist validation. Unfortunately any or all of these can lead to intelligence errors and failures, suggesting that information validation, despite its inclusion in the rigor metric itself, may require a transcendent application of rigor.

In applying the metric it becomes apparent that some disambiguation is necessary between several of the individual considerations. The differences between high rigor assessments involving Stance Analysis and Sensitivity Analysis at first glance appear to be unusually subtle, suggesting a need for an explanatory critique as part of the standard process assessment. Additionally, because several of the specific metrics are process-related, assessors need to be present to observe the process or otherwise have access to appropriate and sometimes-scarce process-associated materials. Alternately, a formal means for capturing applied (and omitted) sensemaking processes needs to be developed.

In developing the rigor metric, Zelik et alia note that it is “grounded largely in the domain of intelligence analysis.”417 Looking at the metric from a generalizing point of view, Zelik et alia are interested in whether it can be broadened to other disciplines such “as information search by students working on educational projects, medical diagnosis using automated algorithms for x-ray interpretation, and accident investigation analyses.”418 For those of us within the domain of intelligence, however, that this model “emerged from studies of how experts ‘critique’ analysis (rather than how experts ‘perform’ analysis.” is a strength.419 The rigor metric has been empirically, if tentatively, shown (according to Daniel Zelik) to “reveal [some of the] “critical attributes to consider in judging analytical rigor” in intelligence sensemaking.420 In so doing, Zelik and his colleagues also validated the usefulness of the model within the intelligence domain.

A danger inherent in any process of making sense of an issue exists when the “process is prematurely concluded and is subsequently of inadequate depth relative to the demands of [the] situation.”

As Zelik et alia conclude, “the concept of analytical rigor in information analysis warrants continued exploration and diverse application as a macrocognitive measure of analytical sensemaking activity.”424 But what does it mean if such a model is adopted, or not adopted? The final chapter examines the implications of either outcome.

CHAPTER 9
In Search of Foresight: Implications, Limitations, and Conclusions

Considering Foresight

We turn in conclusion to a discussion of the purpose of mindful, critical sensemaking for intelligence. The discussion may be best framed by pertinent questions: To what end is intelligence intended? In other words, intelligence professionals and their overseers critically ask “knowledge of what?” and, secondly, “knowledge for whom?” One answer to these questions is embodied in the concept of foresight: Intelligence knowledge advises policymakers and decisionmakers about what phenomena are likely precursors of events of interest before they occur. Such foresight—in light of the discussions in this book—does not entail specific predictions. Rather, it allows us to anticipate a range of alternative event sequences.

Foresight informs policy and decisionmakers about what could happen so that those individuals can improve the quality of their decisions. Done mindfully, its vision shifts and evolves apace with the phenomena about which it makes sense. Done wisely in such a manner as presented here, it augments the vision of leaders, enabling mobilization and discouraging two traits that often handicap visionaries: recklessness and intolerance. Done rigorously, it cannot be accused of failing to be imaginative. This prospective approach contrasts with the current practice and paradigm for intelligence production.

In the Kent tradition, as has been noted (and is summarized in figure 16), intelligence knowledge of “analyzed” issues becomes and tends to remain disaggregated into constituent parts—oriented, as Treverton notes, toward solving isolated “puzzles” rather than the more holistic “mysteries” of the intelligence world. On the other hand, we have discussed how Kendall’s competing idea, that intelligence knowledge should “paint a picture” (by way of a macrocognitive, holistic approach) for the policy maker as a fellow “insider,” is consistent with a model of intelligence where the predominant method and motive of intelligence sensemaking is through aggregation and the articulation of a fact-based “vision” recognizable by national-level policymakers (figure 16). Even in an operational military scenario, where isolated, specific facts are essential to successful employment of mission knowledge, a larger intelligence sense of who the ultimate commanders are and why they are doing what they’re doing, remains the essence of useful, foresightful strategic knowledge—also known as national intelligence.

Implications

Creating intelligence as presented here is a mindful process of sensemaking, encompassing the activities of planning, foraging, marshaling, understanding, and communicating. It is critical of itself and the means that are employed in bringing it about. It lies within the largely overlooked Kendallian vision of what national intelligence ought to accomplish. This approach allows a focus on better and worse solutions, and anticipation of likely futures, instead of a more narrow focus on right and wrong answers in an intellectual environment trained on predictive and specific warning. It can make sense of wicked problems.

By contrast, intelligence as it is currently practiced is still somewhat akin to the practice of medicine in the 14th Century.

Intelligence practitioners find themselves in a similar situation. They often do not know why they do what they do, only that the last time, it “worked”—or that it is an “accepted practice.” They do not acknowledge that they have “forgotten” all the times it did not work. Yet, intelligence practitioners who would wear the “professional” label need to know what they are doing and why.427

One means set forth for “improving intelligence” is to capture the processes by which sense is made of an issue. It is certainly true that imposing audit trails is a critical step because they encourage process improvement in the light of serious errors, and stimulate repetitive analysis, synthesis, and interpretation for validation in the full course of sensemaking. However, auditing trails remain inadequate when the Community cannot understand from an epistemological point of view what does and does not work and in what situations.

The major intelligence failures of the first years of the present decade, as well as repeated failures over at least six decades, demonstrate what hap- pens when there is a formal failure to synthesize and interpret beyond what is popularly believed or even to recognize that a situation exists that requires new synthesis and interpretation. A popular hypothesis is that tradecraft can minimize the likelihood of such failures of imagination. Yet this hypothesis remains untested except in some anecdotal cases which, given the Type 2, wicked nature of the intelligence issues now often faced by the Community, is inadequate.

Limitations

It should be noted that a tradecraft of mindful understanding does not guarantee accurate findings. Any of the components of sensemaking can be done poorly yet “correct” answers can be reached. Disaggregating phenomena can be done well yet yield faulty results. Synthesis and interpretation of analyzed phenomena can still lead to faulty conclusions. However, analysis, synthesis and interpretation within the framework of appropriately applied, multimethod tradecraft does guarantee more rigorous sensemaking.

Conclusions

As has been noted repeatedly in this book, many 21st-Century intelligence issues are wicked problems: They are ill-defined and poorly understood issues with multiple goals that must be made sense of within severe time constraints; the stakes and risks are high and there exists no tolerance for failure. As a means of increasing situational awareness, merely creating mindfulness about such complex issues falls short. On the other hand, a mindful sense- making approach to situational awareness accomplishes more by enabling the intense, holistic scrutiny of a complex developing scenario, as suggested in the case study in chapter 7. This macrocognitive approach ensures that the knowledge created also evolves.

If intelligence is to rise above the noise and get the attention of policy and then be acted upon it must be both. A critical, mindful process of sensemaking offers a means for this to occur. As we have seen, it covers the issue broadly, takes into account its complexity, is systematic and rigorous. It offers the best means currently understood for making sense of what is known and knowable.

Aggressive, mindful sensemaking is one path- way to this new paradigm, and may require a different mix of skills and abilities than is currently present. It certainly requires greater, authentic diversity. Considering the present community, one is reminded of Kent’s quip, “When an intelligence staff has been screened through [too fine a mesh], its members will be as alike as tiles on a bathroom floor—and about as capable of meaningful and original thought.”439 In contrast, making sense of the 21st Century’s intelligence challenges requires as much rigorous, “meaningful and original thought” as we can muster. Sensemaking, as it has been developed here, offers us a means of creating that desperately needed thought.