WIXLIB

Operational NeuroscienceIntelligence Community ForumDr. Amy KruseProgram ManagerDARPA/DSONov 5, 2008

DARPA’s MissionDARPA’s mission is to prevent technological surprisefrom harming our national security by sponsoringrevolutionary, high-payoff research that bridgesthe gap between fundamental discoveries andtheir military use.Approved for Public Release, Distribution Unlimited

DARPA Strategic VisionStrategic Thrusts Precision detection, tracking, anddestruction of elusive targets Characterization of undergroundstructures Urban Operations Networked manned & unmannedattack operations Assured use of space Cognitive systems Bio-Revolution Robust, secure self-forming networksBio-DetectionChlorine DioxideEnduring Foundations MaterialsMicrosystemsInformation TechnologiesApproved for Public Release, Distribution Unlimited

DARPA Accomplishments1960SaturnVelaHotelCommand Post ofthe FutureX-45M-16 RifleALTAIR1970Mobile Robots2000MEMSATACMSJSF EngineGroundSurveillance RadarGlobal HawkARPANET1980Uncooled IR1990Stealth FighterPredatorJSTARSSea ShadowApproved for Public Release, Distribution UnlimitedGPSTaurus LaunchVehicleBAT

DARPA Accomplishments1960SaturnVelaHotelX-45Operational NeuroscienceM-16 RifleALTAIR1970Mobile Robots2000MEMSATACMSJSF EngineGroundSurveillance RadarGlobal HawkARPANET1980Uncooled IR1990Stealth FighterPredatorJSTARSSea ShadowGPSTaurus LaunchVehicleBAT

DARPA Technical OfficesDirector, Tony TetherDeputy Director, Bob LehenyStrategic TechnologyTactical TechnologyBarbara McQuistonDave NeylandLarry Stotts, Brian PierceSteve Walker Space Sensors/Structures Air/Space/Land/Sea Platforms Strategic & Tactical Networks Unmanned Systems Information Assurance Space Operations Underground Facility Detection& Characterization Laser Systems Chem/Bio Defense Precision Strike Maritime OperationsDefense SciencesInformation Processing TechniquesMicrosystems TechnologyBob Leheny (Acting)Chuck MorefieldGreg KovacsLeo ChristodoulouCharlie HollandDean Collins Physical Sciences Cognitive Systems Materials Command & Control Systems Electronics Biology Computer Language Translation Photonics Mathematics High Productivity Computing MEMS Human Effectiveness Sensors & Processing Algorithms Bio Warfare DefenseApproved for Public Release, Distribution Unlimited Integrated Microsystems

Defense Sciences OfficeQuantum MechanicsTraining and ionizingProstheticsArmor SystemsZoologyPhysicalSciencesMine the “Far Sideof the Far Side”BiologicalSciencesParadigm shiftsin defense capabilitiesEngineeringStructural BiologyNanotechnologyBig DogVaccine ManufactureFuel CellsNeuroscienceSolid State PhysicsApproved for Public Release, Distribution UnlimitedWASP UAV

Methods to Understand Neural Basisof Human Cognition uropsychologyneuropsychology recordingsininprimatesprimates(mammals)(mammals) studiesstudies nstimulation FunctionalFunctionalNeuroimagingNeuroimaging ethodsincludingincludingPET/fMRIPET/fMRI cludingincludingERPERPandandMEGMEGFunctional neuroimaging allows the activity of all brain regions to be seensimultaneously – true network analysis is possibleApproved for Public Release, Distribution Unlimited

Non-Invasive Sensor ncySpatialResolutionmsmscmMeasures electrical activity in the brain. Practicaltool for applications - real time monitoring or braincomputer-interface.YesmmMeasures magnetic fields generated as a result ofthe brain’s electrical activity. Research tool forinvestigating temporal properties of neuronal andcognitive ?fMRIsminmmMeasures the BOLD signal in neuronal tissue –oxygen uptake. Excellent structural localization ofbrain function. Important tool for foundationalcognition research.PETminhmmMeasures uptake of specially tagged molecules(e.g. glucose) in brain tissue following stimulus.Excellent spatial localization, poor temporally.NommMeasures the ratio of oxygenation in corticalregions using near infrared light. Permits spatialand temporal measurements from the samevolume of brain tissue.YesfNIRmsApproved for Public Release, Distribution Unlimitedms

My Vision of NeuroscienceFundamentalsRestoration& RepairOperationalNeuroscienceAI andCognitiveSystemsDeviceDevelopment& ProcessingMethods

Operational sessmentindesertassessment in desertstationsstationsApproved for Public Release, Distribution isesexerciseslastinglasting7 7 hourshoursatataatimetimeImages Courtesy of Honeywell, Aberdeen Test Center, and Advanced Brain Monitoring Inc.

Operational NeuroscienceDESIGNTRAINApproved for Public Release, Distribution UnlimitedOPS

Amy Kruse

Program Vision iseexpertisethroughthrough learninglearning erwarfighter Learning is a continuous challenge in the operationalenvironmentCurrent methods of learning fail to capitalize on basiclessons from neuroscienceMeasures of learning on key skills only as good asqualitative and subjective assessmentsUsingUsing aa neuroscienceneuroscience basedbased gm oved for Public Release, Distribution Unlimited

Phase I GoalsUtilize neuroscience to understand the development of expertise through learning todirectly facilitate and accelerate task learning for the warfighterProgram Goals: pertexpertperformanceperformance Track fofintermediaryintermediarystagesstagesExpertTask Performance Demonstrate gesofofthethenovice-to-expertpathnovice-to-expert pathExpertNoviceT2Approved for Public Release, Distribution UnlimitedT1TimeNovice

Phase I ApproachesOptimized TaskQualities Apply multiplebiological andcognitive findingsabout the user tocustomize thelearning environmentNeurofeedback Present the userwith real-timefeedback onbrainwaveactivity in theform of a hapticfeedback duringtrainingGlobal Network Measures Determine andfacilitate the neuralmechanisms ofconsolidation(declarative andproceduralmemory, attentionnetworks, memorychunking)InnovationApproved for Public Release, Distribution UnlimitedDirect Stimulation Utilize techniquessuch as tDCS incombination withfunctional imagingto directlystimulate neuralpathways criticalfor learning

Accomplishment: fMRI Signatures ofNovice Vs. Expert BehaviorThreat - Nonthreat StimuliNoviceExpertMetric 1: Establish baseline measures of theindependent novice and expert statesApproved for Public Release, Distribution Unlimited17

enoid (right temple) at 2 nanimprovementinlearningvs.shamimprovement in learning vs. shamininthreatdetectiontrainingthreat detection training 2.1x 2.1ximprovementimprovement(p 0.0093)(p acyand non-threat detection accuracy 3.1x ealone(p 0.0004)(p 0.0004)Approved for Public Release, Distribution Unlimitedpercentage accuracyAccomplishment: Direct Stimulationleads to 2x Improvement

Problem: Imagery OverloadOperational Challenges Collections capabilities are increasing Quantity Modality Exploitation can require manualCredit: nps.navy.milsearch through terabytes of overheadimagery Current brute-force method(Broad Area Search) is time- andlabor-intensive054,012 Machine vision and ATR have notmatched the detection sensitivity andflexibility of the human visual systemSQKCredit: DigitalGlobeResources Exploitation requires manuallysearching through terabytes ofoverhead imageryCredit: schreiver.af.milResult: Strategic analytical requirement exceeds available and foreseeable resourcesApproved for Public Release, Distribution Unlimited20

One Solution:Leverage Human PerceptionHuman vision is fast, accurate and robust to changes Speed: Earliest brain visual responses are 250 msecAccuracy: Human detection rate can be 99%Robustness: Detection persists across a wide range na:LightIT: Objects,Faces“WHAT”V4: Shapesand ColorsForm,ColorNIA Solution: Brain-assisted exploitation – but how?Approved for Public Release, Distribution Unlimited21

Science Behind NIABasic Science: Human visual detection events can be identified in EEG EEG signatures during Rapid Serial Visual Presentation Brain events can be detected in one-third of the time needed to press a button(Thorpe et al, Nature, 1996) Signature in neurophysiological recordings demonstrates detection of target images forpresentation rates of up to 72 images/sec (Keysers et al, JCN, 2001) Demonstrated with faces, natural scenes, multiple presentationsReal Time Signal Detection :Individual visual target identification events can be detected in real-timewith an average accuracy of 83%!Detection Event Based on RealTime EEG, @ 200msecImage Onset“Detection” Based on MotorResponse @ 550msecApproved for Public Release, Distribution Unlimited22

Experimental Approach: BaselineBaseline (Broad Area Search) Task Example:Total Area (urban, riverine, maritime) 314.86 km2Targets: HelipadsImagery Analysts were instructed to exploit thefull scene manually in a geospatial softwareenvironment and to mark a defined target.Exploitation was self-paced and analysts wereallowed to search the image until satisfied alltargets had been located.Credit: DigitalGlobeAnalyst-marked target set was compared toground truth (by 2 or more IAs) for utes135 analyst indicated target during searchwww.darpa.mil/dso23

Experimental Approach: NIA TriageIA with neurophysiologicalsensors for image chip viewing(prototype display)Full scenes were chipped into 512 pixels2 (e.g.3300 chips for 314.86 km2 total area)Chips were at a magnification suitable forexploitation (1:1) but sized to minimize eyemovements during triageAll images: Credit: DigitalGlobeAll chips were displayed rapidly (5-12 Hz) to IA.Neural signals were collected and classified inreal time for target or non target propertiesTriage mode time (10 Hz, including breaks): 10 minutes targets indicated vianeural signals during vityassessmentassessmentsensitivity24

Phase I ResultsEvaluate neuralactivity to identifylikely targetsPresent chips to users inhigh speed bursts (10chips a second)Decompose into chips fewhundred pixels wide andtallOriginal image tens ofthousands of pixels wideand tallAverage % Sensitivity for Target DetectionExperimentalBaselineNIA triage555%improvementTarget Type / Rate ofImage Presentation:Approach:SAM / 10 Hz1.1,696%improvementHelipad / 10 HzSingle-trial event 2.related potentialclassification640%improvementHelipad / 5 HzMulti-sensor (EEG 3. Single-trial linear& pupil) fusiondiscriminant approachapproachResult:Result: 300% thancurrentcurrentSOASOA123

NIA Phase 2 VisionPhase 2 GoalIntegrate modern neuroscientific techniquesinto imagery analysis workflow to improvethroughput and quality of imagery analysisPhase 2 Metrics Maintain 300% throughput increase in imageryexploitation in a realistic analystsoftware/hardware environment.Demonstrate greater than or equal to unassistedimage analyst sensitivity.Maintain performance across 3 complex targetclasses and under variable operating conditions.Phase 2 Technical Challenges Capture brain signals in real time during realisticimagery analysis on baseline imageryexploitation systemsCategorize target detection brain signals basedon object / scene complexityIntegrate neuromorphic computational imageanalysis and physiological brain signalsApproved for Public Release, Distribution UnlimitedPhase 2 Applied ScienceApply Phase 1 breakthrough sciencein operational contexts Extend capture of brain signals fortarget detection to: Multiple imagery types Diverse target and scenecomplexity Integrate brain-assisted search intostandard imagery analysis software Leverage/converge with automatedmachine vision technologies Demonstrate with trained analysts withrealistic tasks and environment

Accomplishment: RemoteviewIntegrationCredit: cationTriage PreviewRSVP TriagePost-Triage agepost-triageexploitationexploitationApproved for Public Release, Distribution Unlimited

Accomplishment: Uncovering impactof image rget 8, T008 Tthersh 35.0230.6Probability CorrectProbability CorrectCredit: DigitalGlobe0.80.40.2110.80.8Probability Correct10Target 16, T016 Tthersh 198.973Target 9, T009 Tthersh 126.1450.60.450100150200Stimulus Duration (msec)Approved for Public Release, Distribution Unlimited25030000.40.20.200.60050100150200Stimulus Duration (msec)250300050100150200Stimulus Duration (msec)250300