Transcription
Security Measurement and AnalysisSecurity Measurement andAnalysisChristopher AlbertsJulia AllenRobert StoddardSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213 2011 Carnegie Mellon UniversityThis presentation is entitled “Security Measurement and Analysis.” It describes work being performed bythe Software Engineering Institute in the area of security measurement and analysis. 2011 Carnegie Mellon University1
Security Measurement and Analysis 2011 Carnegie Mellon UniversityNO WARRANTYTHIS MATERIAL OF CARNEGIE MELLON UNIVERSITY AND ITS SOFTWARE ENGINEERINGINSTITUTE IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NOWARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING,BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY,EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLONUNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROMPATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.This presentation may be reproduced in its entirety, without modification, and freely distributed in written orelectronic form without requesting formal permission. Permission is required for any other use. Requests forpermission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 withCarnegie Mellon University for the operation of the Software Engineering Institute, a federally fundedresearch and development center. The government of the United States has a royalty-free governmentpurpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have orpermit others to do so, for government purposes pursuant to the copyright license under the clause at252.227-7013.CERT is a registered mark owned by Carnegie Mellon University.Security Measurement and Analysis 2011 Carnegie Mellon University2 2011 Carnegie Mellon University2
Security Measurement and AnalysisTopicsSecurity Measurement and Analysis (SMA) ProjectSoftware Security AssuranceFrameworks and ProtocolsMethods and ToolsSummarySecurity Measurement and Analysis 2011 Carnegie Mellon University3The following five topics will be covered in this presentation: Security Measurement and Analysis (SMA) Project Software Security Assurance Frameworks and Protocols Methods and Tools Summary 2011 Carnegie Mellon University3
Security Measurement and AnalysisSecurity Measurement andAnalysis (SMA) ProjectSecurity Measurement and Analysis 2011 Carnegie Mellon University4Topic 1: Security Measurement and Analysis (SMA) Project 2011 Carnegie Mellon University4
Security Measurement and AnalysisSecurity Measurement and Analysis (SMA) Project:ObjectiveTo develop frameworks, methods, and tools for measuring andmonitoring the security of large-scale, networked systems across thelife cycle and supply chainSecurity Measurement and Analysis 2011 Carnegie Mellon University5For several years, the software engineering community has been working to identify practices aimed atdeveloping more secure software. Although some foundational work has been performed, efforts tomeasure software security assurance have yet to materialize in any substantive fashion. As a result,decision makers (e.g., development program and project managers, acquisition program offices) lackconfidence in the security characteristics of their software infrastructures.The CERT Program at Carnegie Mellon University’s Software Engineering Institute (SEI) has charteredthe Security Measurement and Analysis (SMA) Project to advance the state of the practice in securitymeasurement and analysis.The objective of the SMA Project is to develop frameworks, methods, and tools for measuring andmonitoring the security of large-scale, networked systems across the life cycle and supply chain.The SMA Project is focused on measuring and monitoring interactively complex socio-technical systemsthat span multiple organizational entities. Here, a socio-technical system is defined as interrelatedtechnical and social elements that are engaged in goal-oriented behavior. Elements of a socio-technicalsystem include the people who are organized in teams or departments to do their work tasks and thetechnical systems on which people rely when performing work tasks. 2011 Carnegie Mellon University5
Security Measurement and AnalysisTraditional Measurement and Analysis: SystemDecomposition and Component AnalysisDecompose a system into its constituent components.Prioritize the components.Analyze the most critical components.Security Measurement and Analysis 2011 Carnegie Mellon University6Traditional measurement and analysis approaches are based on the principle of system decompositionand component analysis, where the first step is to decompose a system into its constituent components.Next, the individual components are prioritized, and only the most critical components are analyzed indetail. 2011 Carnegie Mellon University6
Security Measurement and AnalysisSystem Decomposition and Component Analysis:LimitationsOnly critical components are analyzed. Noncritical components are not examined. Interdependencies among components are not addressed.Nonlinear relationships (e.g., feedback) are not analyzed. Causalrelationships are presumed to be simple direct linearEstablishing confidence in the performance of individual componentsis not sufficient for establishing confidence in the performance of theparent system.Security Measurement and Analysis 2011 Carnegie Mellon University7Limitations of system decomposition and component analysis include the following: Only critical components are analyzed; noncritical components and interdependencies amongcomponents are not addressed. Causal relationships are presumed to be simple, direct, and linear. Nonlinear relationships, such asfeedback, are not analyzed. Confidence in the performance of critical components is not sufficient for establishing confidence inthe performance of the parent system (or the parent system of systems). 2011 Carnegie Mellon University7
Security Measurement and AnalysisSMA Project: Problem SpaceLow ControlCERT Security Measurement andAnalysis (SMA) itionalMeasurement andAnalysisHigh ControlSecurity Measurement and Analysis 2011 Carnegie Mellon University8Note: This slide is a build.The problem space is defined by the following two dimensions: (1) the degree of management controlover a system and (2) the extent to which the system is interconnected.Traditional measurement and analysis approaches, which employ system decomposition and componentanalysis, are extremely effective in high-control environments with few interconnections. However,traditional approaches also scale to (1) high-control environments with many interconnections (e.g., usingmodeling and simulation) and (2) low-control environments with few interconnections (e.g., usingcollaborative approaches and information sharing among participants).The SMA Project is focusing on the upper right quadrant in the grid, low-control environments with manyinterconnections. Traditional measurement and analysis approaches do not readily scale to low-control,highly interconnected environments. 2011 Carnegie Mellon University8
Security Measurement and AnalysisA distributed management environment is defined as multiple, independently managedorganizational entities working collaboratively to achieve a common mission or purpose.In general, no single administrative structure or set of policies governs all organizations in adistributed management environment such as a software supply chain. In addition, no singlemanager has authority over all organizations within the environment. Multiple points ofmanagement control (i.e., multiple decision makers) exist, which creates a degree ofprogrammatic complexity that can be difficult to manage effectively.Examples of distributed management environments include large Department of Defense(DoD) acquisition and development programs as well as software supply chains.Software products produced by distributed management environments tend to comprisemany integrated components, which leads to programmatic complexity that can be difficult tomanage effectively. 2011 Carnegie Mellon University9
Security Measurement and AnalysisThe SMA Project takes a life-cycle approach to addressing security, where security must beeffectively integrated with day-to-day acquisition, development, and operational activities. Asa result, security is not viewed as a separate, add-on activity that is addressed duringoperations, which differs from current practice. 2011 Carnegie Mellon University10
Security Measurement and AnalysisSMA Project: Initial Application AreaSoftware security assurance is the project’s initial application area.Future work might address other aspects of security, for example incident management operational security othersSecurity Measurement and Analysis 2011 Carnegie Mellon University11The SMA project is initially focused on measuring and monitoring within a software security assurancecontext. However, SMA frameworks, methods, and tools can be applied in other contexts as well. Futurework might address other aspects of security, for example, incident management or operational security. 2011 Carnegie Mellon University11
Security Measurement and AnalysisSoftware Security AssuranceSecurity Measurement and Analysis 2011 Carnegie Mellon University12Topic 2: Software Security Assurance 2011 Carnegie Mellon University12
Security Measurement and AnalysisSoftware Security Assurance: SEI DefinitionThe level of confidence that software-reliant systems are adequatelyplanned, acquired, built, and fielded with sufficient security to meetoperational needs, even in the presence of attacks Failures accidents unexpected eventsSoftware security assurance is focused on the security aspect ofsoftware assurance.Security Measurement and Analysis 2011 Carnegie Mellon University13A common definition of software assurance is “the level of confidence that software is free fromvulnerabilities, either intentionally designed into the software or accidentally inserted at any time during itslife cycle, and that the software functions in the intended manner.” [1]An expanded definition of software assurance is “the application of technologies and processes toachieve a required level of confidence that software systems and services function in the intendedmanner, are free from accidental or intentional vulnerabilities, provide security capabilities appropriate tothe threat environment, and recover from intrusions and failures.” [2]The SMA Project defines software security assurance as justified confidence that software-reliantsystems are adequately planned, acquired, built, and fielded with sufficient security to meet operationalneeds, even in the presence of attacks, failures, accidents, and unexpected events. Software securityassurance is thus focused on the security aspect of software assurance.For several years, various groups within the software engineering community have been workingdiligently to identify practices aimed at developing more secure software. However, efforts to measuresoftware security assurance have yet to materialize in any substantive fashion, although somefoundational work has been performed.[1]Committee on National Security Systems. Information Assurance Glossary. CNSS Instruction No. 4009.http://www.cnss.gov/Assets/pdf/cnssi 4009.pdf[2]Mead, N., Allen, J., Ardis, M., Hilburn, T., Kornecki, A., Linger, R., and McDonald, J. Software AssuranceCurriculum Project Volume I: Master of Software Assurance Reference Curriculum (CMU/SEI-2010-TR-005).Software Engineering Institute, Carnegie Mellon University, rts/10tr005.cfm 2011 Carnegie Mellon University13
Security Measurement and AnalysisSoftware Security Assurance ActivitiesAssessEstablish the current level of softwaresecurity assurance.AssessPlanPlaTake planned action to maintain orimprove the current level of softwaresecurity assurance, and track the plan tocompletion.veproImImprovenDevelop a plan to maintain or improvethe current level of software securityassurance.Security Measurement and Analysis 2011 Carnegie Mellon University14Management of software security assurance comprises the following three activities: Assess – Establish the current level of software security assurance. Plan – Develop a plan to maintain or improve the current level of software security assurance. Improve – Take planned action to maintain or improve the current level of software securityassurance, and track the plan to completion. 2011 Carnegie Mellon University14
Security Measurement and AnalysisRole of Assessment in Software Security AssuranceIndependent assessment is the vehicle for establishing confidencethat large-scale, networked systems will be adequately secure to meetoperational needs.AssessnPlaeovprImSecurity Measurement and Analysis 2011 Carnegie Mellon University15The foundation of a software security assurance capability is the ability to assess assurance effectively.Independent assessment is the vehicle for establishing confidence that large-scale, networked systemswill be adequately secure to meet operational needs. 2011 Carnegie Mellon University15
Security Measurement and AnalysisSMA Project: Products and ServicesTypeFrameworksand ProtocolsMethods andToolsCurriculum andCertificationObjectiveProducts and ServicesTo define the foundationalelements of software securityassurance measurement andanalysis Integrated Measurement andAnalysis Framework (IMAF) Mission-Objective-Driver (MOD)Protocol Practice-and-StandardMappings Software Security Review (SSR) Multi-View Decision Making(MVDM) Software Security Measurement Model-Based SSR Software Security AssuranceCoursesTo provide software securityassurance measurement-andanalysis solutions thatpractitioners can applyTo enable practitioners to applysoftware security assurancemeasurement-and-analysissolutionsSecurity Measurement and Analysis 2011 Carnegie Mellon University16The SMA Project is developing the following three types of products and services: Frameworks and Protocols – the foundational elements of software security assurancemeasurement and analysis Methods and Tools – software security assurance measurement-and-analysis solutions thatpractitioners can apply Curriculum and Certification – courses and certification programs that enable practitioners toapply software security assurance measurement-and-analysis solutionsThis presentation examines the first two types, (1) Frameworks and Protocols and (2) Methods andTools. 2011 Carnegie Mellon University16
Security Measurement and AnalysisFrameworks and ProtocolsSecurity Measurement and Analysis 2011 Carnegie Mellon University17Topic 3: Frameworks and Protocols 2011 Carnegie Mellon University17
Security Measurement and AnalysisIntegrated Measurement and Analysis Framework(IMAF)Decision MakerSystemicAnalysisTargeted AnalysisReportsStatus ReportingMeasurementIMAF provides decision makers with insight into the mission.Security Measurement and Analysis 2011 Carnegie Mellon University18The SMA Project defines a framework as a basic conceptual structure that highlights the relationshipsamong a collection of components. A protocol is defined as the sequence of activities that must becompleted when conducting a method. While a protocol defines what needs to be accomplished, it doesnot specify how to perform those activities.The SEI Integrated Measurement and Analysis Framework (IMAF) employs systemic analysis to integratesubjective and objective data from a variety of sources, including targeted analysis, status reporting, andmeasurement, to provide decision makers with a consolidated view of the performance of large-scale,networked systems. 2011 Carnegie Mellon University18
Security Measurement and AnalysisSystemic AnalysisSystemic analysis is based on system theory.The goal is to analyze a system as a whole.Some system properties are best analyzed by considering the entiresystem, including influences of environmental factors feedback and nonlinearity among causal factors systemic causes of failure (as opposed to proximate causes) emergent propertiesSecurity Measurement and Analysis 2011 Carnegie Mellon University19Systemic analysis is based on system theory. The underlying goal of system theory is to analyze asystem as a whole rather than decomposing it into individual components and then analyzing eachcomponent separately [3]. In fact, some properties of a system are best analyzed by considering theentire system, including influences of environmental factors feedback and nonlinearity among causal factors systemic causes of failure (as opposed to proximate causes) emergent properties[3]Leveson, Nancy. “A New Accident Model for Engineering Safer Systems.” Safety Science 42, 4 (April 2004):237-270. 2011 Carnegie Mellon University19
Security Measurement and AnalysisMission-Objective-Driver (MOD) Protocol: EnablingSystemic Analysis of Complex SystemsMissionDriver Identification1.Identify the mission.2.Identify the objective(s).3.Identify drivers (i.e., criticalfactors that have a stronginfluence on outcome orresult).Objective(s) Probability Driver 1Driver 2Driver 3 Driver MDriver Analysis4.Evaluate drivers.5.Document evidence.6.Establish driver profile.StrengthsWeaknessesTactical OpportunitiesTactical RisksUnknownsAssumptionsHighMediumLow.Driver 8.Driver MDriver 7Driver 3Driver 6Driver 2Driver 5Driver 4MinimalDriver 1Probability of Success StateDriver ProfileMaximumSecurity Measurement and Analysis 2011 Carnegie Mellon University20The Mission-Objective-Driver (MOD) Protocol specifies an approach for performing systemic analysis ofinteractively complex socio-technical systems. The following two activities form the foundation of the MODProtocol: (1) driver identification and (2) driver analysis.The main goal of driver identification is to identify a set of factors, called drivers, that can be used tomeasure performance in relation to a program’s mission and objectives. Refer to slide 32 of thispresentation for the standard set of drivers for software security.Once the set of drivers is identified, analysts can then evaluate each driver in the set to gain insight intothe likelihood of achieving the mission and objectives. To measure performance effectively, analysts mustensure that the set of drivers conveys sufficient information about the mission and objectives beingevaluated.Driver identification comprises the following protocol activities: Identify the mission. Identify the objective(s). Identify drivers (i.e., critical factors that have a strong influence on outcome or result).The goal of driver analysis is to determine how each driver is influencing the objectives. Analysts mustdetermine whether each driver is guiding the system toward its objectives (success driver) or away fromits objectives (failure state).Driver analysis comprises the following protocol activities: Evaluate drivers. Document evidence. Establish driver profile. 2011 Carnegie Mellon University20
Security Measurement and AnalysisMOD Protocol: ExampleMissionMissionThe XYZ Program is providing a new,web-based payroll system for ourorganization.Objective(s)ObjectiveWhen the system is deployed, securityrisks to the deployed system will bewithin an acceptable tolerance.Probability Driver 1 Driver 2Driver 3 Driver MDriver ProbabilitiesUnknownsAssumptionsDriver ProfileMaximumHighMediumLow.Driver 8.Driver MDriver 7Driver 3Driver 6Driver 2Driver 5MinimalDriver 410% probability that the code will besufficiently secure.Tactical OpportunitiesTactical RisksDriver 140% probability that security-relatedtasks and activities are performedeffectively and efficiently.StrengthsWeaknessesProbability of Success State75% probability that the program’ssecurity objectives are realistic andachievable.Security Measurement and Analysis 2011 Carnegie Mellon University21The term mission is defined as the fundamental purpose of an individual, group, or operation. An exampleof a mission is: The XYZ Program is providing a new, web-based payroll system for our organization.An objective is defined as a tangible outcome or result that must be achieved when pursuing a mission.An example of an objective is: When the system is deployed, security risks to the deployed system will bewithin an acceptable tolerance.A driver is a factor that has a strong influence on the eventual outcome or result (i.e., whether or notobjectives will be achieved). The following are examples of evaluated drivers: 75% probability that the program’s security objectives are realistic and achievable. 40% probability that security-related tasks and activities are performed effectively and efficiently. 10% probability that the code will be sufficiently secure. 2011 Carnegie Mellon University21
Security Measurement and AnalysisPractice-and-Standard MappingsDecision MakerMissionSystemicAnalysisObjective(s)Targeted AnalysisReportsDriversStatus tices and standards are mapped to drivers and measures.Security Measurement and Analysis 2011 Carnegie Mellon University22Performing meaningful measurement and analysis based on carefully considered and defined softwaresecurity measures requires a clear statement of the mission or purpose. This statement is furtherexpanded into a set of objectives that reflect the mission. A set of drivers can be derived from theobjectives to define a set of factors that has a strong influence on the eventual outcome or result (i.e.,whether or not objectives will be achieved).A measurement is an observation that results in information (reduction of uncertainty) about a quantity. [4]A measure is the value assigned to a variable that is used to provide a decision maker with insight into agiven characteristic or property of an entity. Measures can be linked to drivers. As shown in the slide,IMAF provides a line of sight from mission to measures.As illustrated on the slide, drivers and measures can also be mapped to practices and standards. For agiven mission and objectives, decision makers can (1) assess confidence in achieving the mission andobjectives and (2) gauge performance in relation to practices and standards.The SMA Project team has begun to develop mappings for the following two standards: NIST 800-53 andISO 27002.[4]Hubbard, Douglas. Applied Information Economics Seminar: Executive Overview. Hubbard Decision Research,2010. http://www.hubbardresearch.com/ 2011 Carnegie Mellon University22
Security Measurement and AnalysisPractice Mapping: ExampleDriver 10. Security Requirements: Requirements sufficientlyaddress roduct security requirementsare documented.% of software products for whichsecurity requirements are/are notdocumentedEffectivenessMeasureProduct security requirementsadequately address customer,user, and stakeholderrequirements and needs.% of security requirements thatmeet (do not meet) customer-,user-, and stakeholder-definedthresholds for adequacyProcessPerformanceMeasuresThe process used to specifysecurity requirements performsas expected.Extent to which the definedprocess for specifying securityrequirements meets itsperformance criteriaSecurity Measurement and Analysis 2011 Carnegie Mellon University23This slide presents an example practice mapping for Driver 10, Security Requirements. (This example isnot complete.) The following three types of measures are included in this example: implementation measure effectiveness measure process performance measure 2011 Carnegie Mellon University23
Security Measurement and AnalysisExample: NIST 800-53 (a)Family and ClassControlRelated ControlsSI. System andInformation IntegritySI-2 Flaw RemediationCA-2, CA-7, CM-3,MA-2, IR-4, RA-5,SA-11, SI-11The organizationa) identifies, reports, and correctsinformation system flawsb) tests software updates related to flawremediation for effectiveness andpotential side effects on organizationalinformation systems before installationc) incorporates flaw remediation into theorganizational configurationmanagement processSecurity Measurement and Analysis 2011 Carnegie Mellon University24This slide and the next present an example mapping for NIST 800-53. (This example is not complete.)This slide presents information from the NIST 800-53 standard related to family and class, control, andrelated controls. 2011 Carnegie Mellon University24
Security Measurement and AnalysisExample: NIST 800-53 (b)GuidanceRelated DriversPracticesMeasures2. The organization(including anycontractor to theorganization) promptlyinstalls securityrelevant softwareupdates (e.g., patches,service packs, and hotfixes).16. OperationalSecurityPreparednessSecurity-relevant software updatesare installed for all softwarecomponents with software flawsand vulnerabilities where correctiveaction is required. % of softwarecomponentsrequiring securityrelevant softwareupdates % of softwarecomponentsrequiring securityrelevant softwareupdates where suchupdates have beeninstalledOrganizations areencouraged to useresources such as theCommon WeaknessEnumeration.7. ExternalInterfacesSecurity-relevant software updatesare installed in a timely manner.“Updates” as used here may alsoinclude other mitigating actions thatdo not involve a change to thesoftware.Security Measurement and Analysis 2011 Carnegie Mellon University25This slide continues the example from the previous slide. Here, the first column presents specificguidance for the control that was featured on the previous slide.The three columns to the right of the Guidance column show our mapping of the guidance to related drivers practices measuresRefer to slide 32 of this presentation for the standard set of drivers for software security. 2011 Carnegie Mellon University25
Security Measurement and AnalysisExample: ISO 27002 (a)Security ClauseSecurity TopicControl ObjectiveControl12. Informationsystemsacquisition,development, andmaintenance12.1. Securityrequirements ofinformation systemsTo ensure that12.1.1 Securitysecurity is an integral requirements analysispart of informationand specificationsystemsStatements of businessrequirements for newinformation systems, orenhancements toexisting informationsystems should specifythe requirements forsecurity controls.Security Measurement and Analysis 2011 Carnegie Mellon University26This slide and the next present an example mapping for ISO 27002. (This example is not complete.)This slide presents information from the ISO 27002 standard related to security clause, security topic,control objective, and control. 2011 Carnegie Mellon University26
Security Measurement and AnalysisExample: ISO 27002 (b)GuidanceRelated DriverPracticeMeasures2. Securityrequirementsjustified, agreed,and documented aspart of the businesscase for aninformation system(Objective).10. SecurityRequirementsSecurity % of system componentsrequirements arefor which securitydocumented as partrequirements are/are notof the business casedocumented as part ofthe business case for theinformation system % of business cases forinformation systems thatinclude/do not includesecurity requirements forthe system componentsthat reside on the systemSecurity Measurement and Analysis 2011 Carnegie Mellon University27This slide continues the example from the previous slide. Here, the first column presents specificguidance for the control that was featured on the previous slide.The three columns to the right of the Guidance column show our mapping of the guidance to related drivers practices measures 2011 Carnegie Mellon University27
Security Measurement and AnalysisMethods and ToolsSecurity Measurement and Analysis 2011 Carnegie Mellon University28Topic 4: Methods and Tools 2011 Carnegie Mellon University28
Security Measurement and AnalysisSoftware Security Review (SSR)Decision MakerSoftware SecurityMission and ObjectivesSystemicAnalysisTargeted AnalysisReportsStatus ReportingMeasurementSSR establishes confidence in the security of large-scale, networked,software-reliant systems across the life cycle and supply chain.Security Measurement and Analysis 2011 Carnegie Mellon University29The Software Security Review (SSR) is a method conducted by independent teams to assess the securitycharacteristics of software-reliant systems. SSR is a driver-based approach that can be used to measureand monitor software security assurance across the life cycle and supply chain (including acquisition,development, and operations). 2011 Carnegie Mellon University29
Security Measurement and AnalysisSSR: Family of AssessmentsSupply Chain Activity?Use LegacySoftwareProgramOfficeContractorReuseSSR defines a family of assessments that can beapplied across the life cycle and supply ouseDevelopIn-HouseDevelopOffshoreAcquireDevelopin ersDevelopin USUSDevelopersSupplierAcquireSSR Outsource?Life-Cycle PhaseEarly AcquisitionRequest for ProposalProposal nCodingTestDeploymentOperation and Sustainment.SSR NSecurity Measurement and Analysis 2011 Carnegie Mellon University30SSR defines a family of assessments that can be applied across the life cycle and supply chain. An SSRmethod can be tailored based on life-cycle phase and supply chain activity. 2011 Carnegie Mellon University30
Security Measurement and AnalysisPhase 1 of the SSR Method, Prepare for the Assessment, is focused on getting ready toconduct the assessment. This includes all of the
Security Measurement and Analysis Christopher Alberts Julia Allen Robert Stoddard Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213. This presentation is entitled "Security Measurement and Analysis." It describes work being performed by the Software Engineering Institute in the area of security measurement and .
