Transcription
2017 ITEA Cyber Security Workshop: Challenges Facing Test and EvaluationEnabling Operationally Realistic Cyber T&E,Training, and Mission Rehearsal for the DoDMr. Derrick HintonActing DirectorTest Resource Management CenterMarch 29, 20171
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Topics of Discussion TRMC Overview National Cyber Range Technology Challenge Concluding Remarks Points of ContactUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12742
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274TRMC OverviewUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12743
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274TRMC OrganizationUnder Secretary of Defense for Acquisition, Technology & LogisticsMr. James A. MacStravic (Performing Duties Of)As of 15 Feb 2017Acting Director, TRMCMr. G. Derrick HintonChief Financial OfficerDeputy EA for Cyber Test RangesActing Principal Deputy, TRMCMr. Paul D. MannChief Operating OfficerRange Director, NCRDeputy Range Director, NCRDD, T&E Range OversightAgency ROAF ROArmy RONavy ROPM, CTEIPPM, REPDeputy PM, CTEIPDD, Major Initiatives and Technical AnalysisPM, T&E/S&TPM, JMETCDeputy PM, T&E/S&TDirector, TENA SDAUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12744
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12742016 Strategic Plan for DoD T&E ResourcesGoal: ActionableStrategic Plan to GuideDoD T&E Spending 9B in FY16Source: Institute for Defense Analyses (IDA) Report“Cost of Testing Analysis Origin, Description, Data Sources, Assumptions andLimitations, and Results” June 2015UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12745
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274TRMC MissionsDoD (Charter) Directive 5105.71MRTFB OVERSIGHT / T&E INFRASTRUCTURE Plan for and assess the adequacy of the MRTFB to provide adequate testing in support of the development,acquisition, fielding, and sustainment of defense systems Support the Department’s objective of ensuring compliance with DoDD 7000.14-R Review proposed significant changes to T&E facilities and resources of the MRTFB before they are implementedby the DoD Components Issue guidance to the DoD Components, through the USD(AT&L), with respect to MRTFB planning Maintain an awareness of other T&E facilities and resources, within and outside the DoD, and their impacts onDoD requirements Serve as Executive Agent for Cyber Test RangesSTRATEGIC PLAN Complete a strategic plan for T&E not less often than once every 2 fiscal yearsBUDGET CERTIFICATION Submit report to the SECDEF containing the comments of the Director concerning all such proposed budgets,together with the Director’s certification as to whether such proposed budgets are adequatePROGRAMS Administer the CTEIP (Central Test and Evaluation Investment Program) and T&E/S&T ProgramCAPABILITIES Manage and operate the JMETC (Joint Mission Environment Test Capability) Multiple Independent Levels of Security(MILS) Network and the Regional Service Delivery Points (RSDP) cloud computing environments Manage and operate the NCR (National Cyber Range) (IAW RMD 407A1, Issue #1, Title: Cyber, Jan 12, 2015) toprovide test capability and capacity for the T&E CommunityStatutoryRegulatoryUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12746
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Executive Agent (EA) for Cyber TestRanges Responsibilities1) Develop the Biennial Integrated Plan which includes:a. Maintaining comprehensive list of test capabilities (DoD and non-DoD)b. Organizing and managing designated test capabilitiesc.Certify DoD cyber range investmentsd.Perform assessments/analysis per the SD direction2) Certify component cyber test infrastructure investments3) Generate requirements and standards for cyber security testinfrastructure. (i.e. cyber event and threat data language)4) Maintain a list of government and non-government cyber test ranges5) Use the T&E Board of Directors (BoD) as an advisory boardKey: Statute PolicyUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12747
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274FY 2016 NCR Utilization(Range Weeks) By Event TypeMDAP Cybersecurity DT&EMission Rehersal13.8%3.4%Training/Exercises29.3%Cyberspace CapabilityOT&E6.9%46.6%Training and MissionRehearsal – 53.4%Note: In FY16, NCR was offline for 5 weeksto support A&A and RecapitalizationUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12748
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Capacity vs. NeedCyber T&E Range Capacity vs.Actual, Scheduled, and Projected Test Events Trend analysis firstconducted in FY14 Updated in FY16 A year and a halfof actual data1682959(485)(509)(461)(422)(352)(268)82FY11 – FY16 Actual Events ExecutedFY17 – FY22 Projected Events AnticipatedFY15 JS J6 CBA CMFtraining requirementsare much larger, butthis chart reflects ourconcept for deliveringpersistent cybertraining environmentsUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-12749
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Situational AwarenessNDAA 2016 (PUBLIC LAW 114–92—25 NOV 2015) TITLE XVI, SUBTITLE C,SECTION 1647 “The Secretary of Defense shall, in accordance with the plan undersubsection (b), complete an evaluation of the cyber vulnerabilities of eachmajor weapon system of the Department of Defense by not later thanDecember 31, 2019.”DOD CYBER STRATEGY (APRIL 2015) – STRATEGIC GOAL 1 “Maintain a persistent training environment. DoD requires an individual andcollective training capability to achieve the goals outlined in this strategyand to meet future operational requirements. U.S. Cyber Command willwork with other components, agencies, and military departments to definethe requirements for and create a training environment that will enable thetotal cyber force to conduct joint training (including exercises and missionrehearsals), experimentation, certification, as well as the assessment anddevelopment of cyber capabilities and tactics, techniques, and proceduresfor missions that cross boundaries and networks.”“Build viable career paths” – “Improve civilian recruitment and retention”UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127410
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274National Cyber RangeTRMC has a mission tomanage and operate the National Cyber Range to providetest capability and capacity for the T&E Community(IAW RMD 407A1, Issue #1, Title: Cyber, Jan 12, 2015)UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127411
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274National Cyber Range ComplexBackground Impetus—DoD is facing a challenge in aligning funding andknowledgeable personnel to design and conduct operationallyrealistic cyber events that will improve the mission resilience ofour warfighters Concordantly, TRMC obtained the resources to build theinfrastructure and employ the skilled workforce to satisfycustomer requirements The goal is to establish and refine best practices; amass arepository of reusable models and tools; and cultivate organiccapabilities within the program management offices and theCyber Mission Force TRMC currently operates the National Cyber Range (NCR) inOrlandoTRMC institutionally funds the National Cyber Range ComplexUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127412
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Key Demand Drivers for Buildout ofCapacity and Capability to SupportCyber T&E, Training, and Mission RehearsalAcquisition Program TestingSystem of Systems (SoS) Testing Rapid growth in number of customers Many systems are part of a larger SoS and do notoperate in the field alone Connectivity to a larger SoS presents a cyberattack surface that is NOT covered by individualsystem testing Demand for integrated SoS testing will continue toincrease Every customer has been a repeat customer Emerging testing rhythm of every six months– 15 programs have established a twice yearly event 60% growth in new customers from year to year Returning FY16 test customers have accounted forapproximately 80% utilization of the existing capacityDemand will expand as threats evolve and SoS risks arecharacterized and proactively addressedEnable shift from reactive to proactive approach to ensuringcyber mission resilience in DoD programsCyber Mission Rehearsal & OCO/DCOTesting Testing can not be accomplished on the internet NCR has demonstrated the ability to replicate therealistic cyberspace environment necessary tosupport effective testing and mission rehearsal Demand is shifting from single test events todedicated environments for both test and missionrehearsalRealistic cyberspace environments developed at the NCR can beshared across the ServicesCyber Mission Force (CMF) Training NCR provides a customized, high-fidelity red and grayenvironment for training and mission rehearsal As CMF shift from training individual to training team arealistic environment is critical to effective trainingCMF TeamOperationsCMFTeam ofTeamsCOCOMIntegrationDemand for NCR will grow asmore teams become FOCand are integrated intoOperational CommandsIndividualSkills /refresherThe NCR provides a realistic training environment that will berequired for team, teams of teams, and COCOM trainingUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127413
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274National Cyber Range (NCR)Mission: Improve the resiliency of our warfighters in the cyber-contested battlespace by conducting testing, training,and mission rehearsal events in operationally-representative cyberspace environmentsComputing Assets/Facility(LMCO Orlando, FL) 2009-2012 – DefenseAdvanced ResearchProjects Agency (DARPA)Encapsulation Architecture &Operational Procedures Oct 2012 – Transitionedfrom DARPA to TRMCCyber Test TeamIntegrated Cyber Event Tool SuiteSecure Distributed ConnectivityRealistic MissionEnvironments Provides secure facilities, innovative technologies, repeatable processes Creates high-fidelity, mission-representative cyberspace environments Facilitates the integration of the cyberspace T&E infrastructure through partnerships with keystakeholders across government, industry, and academiaUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127414
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274National Cyber Range ComplexMission: Improve the resiliency of our warfighters in the cyber-contested battlespace by conductingtesting, training, and mission rehearsal in operationally-representative cyberspace environmentsNationwide NCR NetworkAirLandNCR Complex VisionDistributed Access toPersistent Cyber T&E,Training, and MissionRehearsalSeaSpaceKey NCR Flagship Planned NCR Instance Existing NCR Network Connectivitya b g d WThe Right TeamEnabled by theRight TechnologyAugmenting existing capabilities to deploy currentand future systems to include avionics HW&SW,ICS/SCADA, and more.UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127415
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Technology ChallengeUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127416
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Sanitization is an Example of Multi-ProngedApproach to Mature Best-of-BreedTechnologies for Utilization Communicating technology development needs in multiple forumspromotes best-of-breed solutions Multi-pronged approach to augmenting sanitization capabilities includesthe following initiatives: Developing broadermethodologies andtools to sanitize arange of missionsystem hardware,provides severalbenefits:– Framework for Automated and Verified Sanitization (FAVS)(Lockheed Martin) Effort progressing to establish and demonstrate tools and proceduresto restore systems to fully mission capable status (with confidence) Initial emphasis is on T&E, training, and mission rehearsal—in thefuture, this could provide real-time sanitization of systems in theoperational environment– Avionics Attestation (TRMC / AF/TE, AFRL)– Increased throughput /rapid turnaround Determine whether or not an aircraft preparing for a mission containsmalware or maldata– Increased event scope Minor modifications in avionics memory enable attestation to detect achange before a cyber attack occurs– Greater operationalrealism– Higher fidelity– Customized server components for NCR (Dell; unsolicited) Tailored hardware and software to reduce sanitization time for NCRinfrastructure assetsUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127417
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Improved Sanitization Capabilities MayReduce Reliance on Virtualization for T&E In order to conduct system cybersecurity evaluations, it can bedesirable to virtualize mission hardware– No hardware procurement necessary– No concern regarding destruction of assets– Sanitization is straightforward By design, a greater number of DoD mission systems nowinclude virtualized components However, in some cases, new development of virtualrepresentations of mission systems to support T&E activitiesmay be labor intensive By developing sanitization tools and techniques with broaderapplicability, we reduce the need to virtualize system hardwarefor the purpose of avoiding destructive testingUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127418
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Concluding RemarksUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127419
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Concluding Remarks TRMC is working collaboratively with the Services to augment theexisting capacity and capabilities to satisfy growing customerdemands The end state for the buildout of the DoD’s cyber capabilities willbe a National Cyber Range Complex that is– Seamlessly interoperable with HWIL, SIL, ISTF, and OAR facilities– Rapidly configurable– Distributed– Flexible– ExpandableServices will own and operate the new instantiations of the NCRUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127420
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Points of ContactUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127421
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Points of ContactMr. G. Derrick HintonActing Director, TRMCgeorge.d.hinton2.civ@mail.mil(571) 372-2761Mr. Paul MannActing Principal Deputy Director, TRMCpaul.d.mann2.civ@mail.mil(571) 372-2773Dr. Robert TamburelloDeputy Range Director, NCRrobert.n.tamburello.civ@mail.mil(571) 372-2753http://www.acq.osd.mil/dte-trmc/UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127422
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274BackupUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127423
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Customer Requirements Drive Development ofInfrastructure, Methodologies, and Tools T&E Customer Domain– Aviation Systems Emulate commonly utilized MIL-STD-1553 and ARINC 429 bus architectures insupport of cyberspace T&E– Industrial Control Systems OASD Energy, Installations, and Environment (EI&E) and NCR collaborating tobuild-out a complex/to-scale representation of a designated industrial controlsystem environment– Sanitization of mission system components Training / Mission Rehearsal Customer Domain– Internet-based technologies designed to function in closed loopcyberspace environment to provide operational realism (train as you fight)UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127424
UNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-1274Cyberspace T&E Infrastructure RoadmapAs of Jan 3, 2017Drivers Increasing volume of acquisition programs (i) ready to commencecybersecurity T&E activities and (ii) scheduling iterative and incrementalcybersecurity T&E activities Shift from reactive to proactive approach to ensuring cyber missionresilience in DoD programsGrowing scale and complexity of operationally-representative cyberspaceenvironments requested by customers T&E, training, and mission rehearsal (OCO & DCO) Emerging demand for distributed testing of integrated systems-of-systems Progression from supporting training of individual Cyber Mission Forcemembers to training teamsGaps Standardized, secure, agile and operationally relevant blue, red andgray cyberspace T&E environments Common and specialized defensive and offensive capabilitiesinstrumentation Standardized metrics to improve assessment of security architectures Processes and capabilities to optimize operations and sustainment ofthe cyberspace T&E infrastructure Effective Cyber T&E governance process to ensure cross-DoDcollaboration, convergence, and advocacy Cyber T&E Workforce Development ProgramTest Concepts & Methodologies This roadmapo Describes the current and target states of Cyberspace T&Ecapabilitieso Proposes incremental activities to improve capabilitieso Highlights the top-level strategy for managing requirements Result of analysis conducted by DoD communities of interest, industry,academia, and non-DoD organizations identified key challenges, gaps,and shortfalls with current Cyberspace T&E capabilities Focused on the enterprise level Cyberspace T&E infrastructureinitiatives and enabling capabilitieso Intent is to guide DoD investments and way forward activitiesRecommendations TRMC, as the DoD Executive Agent for Cyber Test Ranges, will:o Develop, coordinate, and integrate plans to synchronizeactivities across designated cyber test rangeso Establish appropriate test infrastructure architectures andstandardso Utilize this roadmap as the underlying foundation forCyberspace T&E Continue updating the Cyberspace T&E Infrastructure Roadmapthrough a collaborative process with Services and AgenciesUNCLASS // CLEARED FOR OPEN PUBLICATION – DOD OFFICE OF PREPUBLICATION AND SECURITY REVIEW #17-S-127425
c. Certify DoD cyber range investments d. Perform assessments/analysis per the SD direction 2) Certify component cyber test infrastructure investments 3) Generate requirements and standards for cyber security test infrastructure. (i.e. cyber event and threat data language) 4) Maintain a list of government and non-government cyber test ranges
