Transcription
Crew chief with 36th Aircraft Maintenance Unit, OsanAir Base, South Korea, checks computer data duringRed Flag-Alaska 14-2, ensuring F-16 Fighting Falconreadiness (U.S. Air Force/Peter Reft)Defense Intelligence Analysis inthe Age of Big DataBy Paul B. Symon and Arzan Taraporever the past decade, the U.S.and Australian intelligence communities have evolved rapidlyto perform new missions. They havedeveloped new capabilities and adaptedtheir business processes, especially insupport of joint and complex militaryoperations. But in the coming decade,Otheir greatest challenge will be todevelop new capabilities to manageand exploit big data.1 We use the termbig data to mean the exponentiallyincreasing amount of digital information being created by new informationtechnologies (IT)—such as mobileInternet, cloud storage, social network-Major General Paul B. Symon, Australian Army, served as Director of the Defence IntelligenceOrganisation from 2011 to 2014. Arzan Tarapore is a Doctoral Student in the War Studies Departmentat King’s College London. The authors thank Josh Kerbel and Peter Mattis for comments on an earlierversion of this article.4 Forum / Defense Intelligence and Big Dataing, and the “Internet of things”—andthe advanced analytics used to processthat data. Big data yields not simply aquantitative increase in information,but a qualitative change in how wecreate new knowledge and understandthe world. These data-related information technologies have already begunto revolutionize commerce and science,transforming the economy and actingas enablers for other game-changingtechnology trends, from next-generation genomics to energy exploration.2In defense intelligence communities,JFQ 79, 4th Quarter 2015
some of these technologies have beenadopted for tasks, including technicalcollection and operational intelligencefusion—but big data’s impact on allsource intelligence analysis has scarcelybeen examined.This article offers a view on how thesedisruptive information technologies couldtransform defense intelligence analysis andthe functions of the all-source enterprise.It is not a comprehensive study on trendsin technology or in the intelligenceprofession, nor is it a deterministicscenario of a high-tech future.Rather, here we seek to identify someopportunities and risks of the disruptivetechnologies at hand. First, we sketch abackground of the most important ITtrends that are shaping today’s economyand society. Second, we outline how bigdata could transform intelligence analysis;it has the potential to unlock enormousproductivity gains and effectivenessby automating some currently laborintensive tasks, enabling new formsof analysis and creating new forms ofpresentation. Third, we argue big datacannot do it all; its utility in makingsense of complex systems and addressingknowledge gaps is limited. Finally, weoutline how big data could transform thewider assessment agency enterprise. Weargue that the explosion in data supplyand demand will incentivize assessmentagencies to reposition their roles moretoward service-delivery functions and torebalance their workforces.None of this is inevitable. In bothanalytic operations and enterprisemanagement, much of how the scenarioactually unfolds will be determined by thevision and agility of our leadership, ourpartners, and our adversaries. Defenseand intelligence community (IC) leadersmust play an active but balanced role,exploiting big data’s potential, butunderstanding its limitations.Today’s Tech TrendsThe big data phenomenon presentsdefense intelligence with a range ofopportunities, from off-the-shelf toolsto complex business-process reforms.Some tools can be absorbed wholesaleby the IC; for example, social network-JFQ 79, 4th Quarter 2015ing tools such as Wikis and Chat arealready being used to facilitate bettercollaboration between analysts. Beyondsimple software acquisitions, however,disruptive information technologieshave birthed a number of trends in howdata are collected, moved, stored, andorganized. Four of the most salientprevailing concepts, which are alreadytransforming the economy and society,could reshape all-source intelligence.Everything Is Social, Mobile, andLocal. Much of the explosion of big datahas been driven by the fact that information is increasingly social (generated andtransmitted by many users, rather than afew big producers), mobile (collected bysensors on ubiquitous Internet-connectedmobile devices), and local (geospatiallytagged). These trends have irreversiblytransformed IT; mobile devices in particular have become the primary means ofconnecting to the Internet and have thusbecome the primary market for much ITinnovation. This has already created newopportunities not only for collection,but also for intelligence processing, exploitation, and dissemination (PED), andanalysis.Data Are Useless Without DataScience. The exponential creation ofdigital data holds enormous potential forcreating insight and knowledge throughPED and data analytics. The burgeoningfield of data science—at the intersectionof statistics, computer science, and otherrelated fields—is increasingly being usedby the private sector to realize the commercial potential of big data, often forprosaic tasks such as tracking a person’sconsumption patterns to better targetadvertising campaigns. The IC’s routinework of collection, PED, and analysis isstill largely organized on the Cold Warmodel of seeking out sparse and secretinformation. Now, however, it must copewith the inverse challenge (and exploitthe opportunities) of managing andanalyzing massive quantities of data and,in the process, compete with the lucrativeprivate sector to attract the highly specialized skills of data scientists.3IT Solutions Are Customized andIntuitive. The accelerating pace ofinnovation and the need to best harnessbig data are both enabling and drivingthe creation of IT solutions that are customized and intuitive for the user. Goneare the days of hefty user manuals or obscure text-based user interfaces. Specificapplications perform specific functions.Even major platforms such as Palantir aredelivered with bespoke service support,both in tailoring the product to customerrequirements and in providing ongoingsoftware development support. Complexdata-driven analysis demands a menu ofapps or even dedicated software developers integrated into analyst teams—as theyalready are in some parts of the IC.The Internet Is Everywhere. The rateof increase in big data will only grow asmore devices join the Internet. Thesedevices not only provide an interface forusers, but are also creating a growing“Internet of things”—everything fromhousehold appliances to industrial robots—that generate and use more data,in turn creating more potential knowledge and vulnerabilities. At the sametime, emerging technologies (such asfree-space optical communications, whichuse lasers to transmit data through the atmosphere) are allowing users to bring theInternet into austere communications environments in order to enable the widermilitary use of Internet-connected IT andgreater resilience to network failures.These technology trends have beendriven by the commercial and scientificsectors, but they also have powerfulimplications for the IC; they are rapidlychallenging long-held conceptionsof intelligence collection targets,business processes, required IT tools,and workforce skill sets. But the IC’scapacity to adopt these technologiesremains inadequate; fully exploiting thesetrends would require a deep revisionof innovation policy and IT-acquisitionbusiness models. To adequately exploitthese opportunities, the IC wouldneed to incorporate a “technologypush” acquisition model alongside thecustomary “demand pull” model. In today’s IT environment of faster innovationand more disruptive and unpredictabletechnologies, where government lacksthe speed or vision to lead innovation,the IC’s best option may be to monitorSymon and Tarapore5
and leverage incipient innovation insteadof attempting to drive it. Rather thandictating requirements to firms through abyzantine acquisitions process (as in mostdefense procurement programs), the IC’sgreatest potential for IT adoption may liein injecting its “use cases” (and resources)in the start-up or development phases offuture technologies. And in a data-intensive information environment, assessmentagency leaders would need to recognizethat adaptive IT is integral to analyticoperations and no longer an ancillarysupport function toiling in the basement.The analysis mission-owner shouldtherefore be responsible for shaping theagency’s IT architecture as never before.Even if imperfectly realized, today’stechnology trends hold enormouspotential to transform all-sourceintelligence.Transforming AnalysisAcross intelligence problems, big data’sgreatest promise is its potential to integrate and organize information. Newtechnologies for collecting, moving,storing, and organizing data could giveall-source analysts access to vastly moreinformation with more automation andproductivity, thereby allowing themto concentrate their finite cognitivecapacity on the hardest, highest-priority problems. But rather than simplybolting new technologies onto currentprocesses, assessment agencies nowhave an opportunity to incorporate newtechnological trends in ways that fundamentally reshape how data are used forall-source analysis. The new technologiescould be usefully applied to a range ofdefense intelligence problems, includingsocial network analysis, weapons systemsmodeling, trend analysis for tacticalmilitary intelligence or nontraditionalwarning problems, and nascent analyticconstructs such as “object-based production” and “activity-based intelligence.”4Thus, they not only improve our capacity to execute existing intelligence missions, but they also create entirely newdata-intensive types of analysis.More Information with Less Effort.Big data and data analytics rely heavilyon automation. Once the architecture6Forum / Defense Intelligence and Big Dataand algorithms are set, the data could bemanaged—collected, moved, stored, andorganized—with relatively little additionaleffort. Applied to all-source intelligence,the exponential increase in data and analytics would render manual informationretrieval impractical and unnecessary; theheavy lifting of data management could belargely automated. Already-existing toolscan create an automatic and persistentpush of data to analysts, obviating thelabor-intensive requirement to manuallypull data from various sources. That pushof data could be more processed andvaluable—for example, collated across different sources or formats—before it evenreaches the analyst.Automated data collation and analytics would both save analyst effort andenable powerful new capabilities. Dataanalytics could, with varying levels ofhuman supervision, characterize datainto meaningful clusters or categories,categorize and file new data into existingclusters, and detect outliers or new datathat do not fit into existing clusters.5 Forall-source analysis, new methods such asobject-based production could enableseamless integration of data from multiplesources and in multiple formats, therebybuilding comprehensive libraries of dataon given targets. Analysts could use thatmass of data and associated analytics tomore quickly identify intelligence gaps,unexpected correlations and associations,or anomalies or irregular behavior. Thisrange of capabilities could be profitablyused, for example, for everything fromfinding patterns or anomalies in a terrorist target’s pattern of life, to trackingmilitary targets automatically in widearea surveillance, to tipping and cueingfor humanitarian assistance and disasterrecovery support. In such cases, humanintervention—especially expert analysisof the target—is still critical, but big datacould empower those analysts to knowmore and to know it more quickly andwith less effort.Big data technologies allow intelligence to move quickly, be storedindefinitely, and yield more valuableinsights over time. Much of the newlycollected data would arrive at or nearreal-time, compressing the latency ofcollection, PED, and analysis, and cueingfurther collection. Vast quantities ofdata—unprocessed and unseen by anyanalyst—would be stored, available tobe mined later in the context of futuredata or requirements or to discoveror recognize associations or trends.Machine learning would allow this entire process to improve with time. Theaccumulation of data and the refinementof algorithms would allow for dynamicand progressively more accurate modelsor more robust and adaptive normalcypatterns, and would enable the detectionof finer or more meaningful anomaliesaccordingly.There are significant challenges tofielding these new capabilities. Someof these challenges are technical—forexample, optimizing ways to ingest andcollate data from different sources and indifferent formats, especially unstructureddata from text and media. The thorniestchallenges, however, are associated withpolicy settings and governance frameworks. For example, intelligence agencieswill need to set standards for the vettingand quality assurance of data they sourcefrom interagency or other partners;establish security and legal complianceprotocols for sharing data across organizations; establish robust security measuresto protect data from spoofing, cyberexploitation, or insider leaks; and standardize the tagging and coding of datafor use in analytics. Once mission-ownersset these frameworks to govern the effective and secure use of big data, all-sourceanalysis should yield unprecedented gainsin productivity and capability.Presentation Is Everything. Oncecollated, managed, and applied to gainnew insights, data must be presentedeffectively to the customer. Here, too,big data carry risks and opportunities.Customers will never lose the temptationto acquire and interpret their own data,and big data, plentiful and apparently authoritative, will exacerbate that problem.The IC f
Internet, cloud storage, social network-ing, and the “Internet of things”—and the advanced analytics used to process that data. Big data yields not simply a quantitative increase in information, but a qualitative change in how we create new knowledge and understand the world. These data-related informa-tion technologies have already begun to revolutionize commerce and science .
