WIXLIB

Joe Andrews, MsIA, CISSP-ISSEP, ISSAP, ISSMP, CISASr. Compliance Auditor – Cyber SecurityImproved Compliance through Metrics andMeasurementJanuary 30th, 2012Mesa, Arizona

Speaker Introduction Joseph A. Andrewso 21 years DoD IT Security / Network Engineering(Federal Civilian) Information Systems Security Engineer Information Assurance Manager Network Security Engineer Information Systems Security Officero Academic Master of Science in Information Assurance Bachelor of Science in Information Technology Professional Certifications: CISSP-ISSEP, ISSAP,ISSMP, CISA, CAP, GCIH, CEH, CBRM, CGEIT, CNDA2

Metrics: Why do we collect data? Justify the value of our activities Improve our ability to control and secure theinfrastructure To better understand our Cyber Securityinfrastructure3

Reality: Data Collection Programs Compliance driven (e.g., NERC CIP, FISMA, ISO) Data retention requirements; cyber security systemevent logs (i.e., 90 days, 3 years for incidents, etc.) Collecting data that sometimes has no real context No real analysis of all the data collected Data collection activities that are not measurementdriven4

Metrics and Measurement Metrics- Records of ourobservations5 Measurement- The activity ofmaking observationsand collecting data inan effort to gainpractical insight intowhat we areattempting tounderstand.

Metrics: Typical Examples Records of our observationso Risk Matrix (Likelihood x Severity)o Annualized Loss Expectancy ALEo Total Cost of Ownership TCOo Return on Investment ROI6

(Basic) Risk Matrix ExampleSeverity of ImpactLikelihood of mBadNot GoodErrorLowAnnoyanceTypicalImprobable

Annualized Loss Expectancy ALE Annualized Loss Expectancy ARO Annualized Rate of Occurrence SLE Single Loss ExpectancyALE ARO x SLEWindows Server cost 10K (system and data) with 25%chance of compromise (ARO .25) if compromised youexpect 5K in losses (SLE 5000)ALE 0.25 x 5000 1250Annual Security Budget for server is 12508

Total Cost of Ownership Total Cost of Ownership TCO- Hardware and Software- License and support fees- Installation and maintenance- Training- Security and Audit- Other hidden cost (i.e., utility costs)9

Return on Investment Return on Investment ROI- Expected loss for security incident is 10K- You spend 1K to prevent loss- Your ROI is now 9K- If you spend 20K to prevent the loss- Your ROI is now a negative return of 10K10

Measuring opinion not actual risk Cyber Security metrics & statistics are in infancy stages- Industries are currently measuring limited areas of Security- Very limited industry sharing of Cyber Security data/statistics No common standardized central data repositories for CyberSecurity data- Computer Security Institute (CSI) and other orgs. collect datafrom various industries (Computer Crime and Security Survey)- Not enough willing participants – due to security concerns –fear of data and ultimately infrastructure compromise Mature industries have been collecting and sharing data forcenturies (i.e., insurance, manufacturing, transportation, etc.)11

Example Cyber Security Metrics Percentage of systems compliant with NERCCIP standards (CIP-005, CIP-006, CIP-007) Ratio of systems containing vulnerabilities as aresult of a Cyber Vulnerability Assessment Percentage of budget devoted to compliance Number of configuration changes orexceptions request per time period Average time required to remediatevulnerabilities12

Security Process Mgmt. (SPM) Framework Goal Question Metric method Security Measurement Projects Security Improvement Program Security Process Management13

14

Goal Question Metric methodThree step process Invented in 1970, by Victor Basili and was originallydeveloped for NASA: software engineering practices;aligning software metrics with software goalso 1. Define goals and objectives the measurement issupposed to achieveo 2. Translate goals into specific questions that mustbe answeredo 3. The questions are answered by identifying anddeveloping the appropriate metrics15

GQM diagram16

(Good) Goal attributes Specifico Bad: We’re going to improve Cyber Securityo Good: We’re going to reduce response times to CyberSecurity incidents by 10% Defined boundaries and attributeso business unit, system, concept Attainable Verifiable** Note: Keeping goals too vague diminishesyour accomplishments17

GQM – Enforcement of Security PolicyCIP-3, R1CIP-7, R5.2- Example 1 Goal Components- Outcome: Increase- Element: Enforcement of Cyber Security Policy- Element: User Awareness of the Policy- Element: User Acknowledgement of the Policy- Perspective: Compliance manager Goal statementThe goal of this project is to increase the enforcementand user awareness of the Cyber Security policy, byincreasing user acknowledgements of the company’ssecurity policy documents from the perspective of thecompliance manager.18

GQM – Enforcement of Security PolicyCIP-3, R1CIP-7, R5.2- Example 1 Question 1: What is the current level ofenforcement of the Cyber Security policy? Metric: Number of reported security policyviolations in the previous 12 months Metric: Number of enforcement actionstaken against policy violations in previous12 months19

GQM – Enforcement of Security PolicyCIP-3, R1CIP-7, R5.2- Example 1 Question 2: What is the current structure of theCyber Security policy? Metrics:- Number documents that make up the securitypolicy- Format(s) of security policy documents- Location(s) of security policy documents- Types of acknowledgement mechanisms- Length of time since last review by mgmt.20

GQM – Tailgating - Example 2CIP-6, R5 Goal Components- Outcome: Understand, observe, elicit, improve- Element: Physical security practices and behaviors- Element: Employee explanations and opinions- Element: Physical and Cyber Security posture- Perspective: Compliance, Physical and Cyber Security teams Goal statementThe goal of this project is to understand the physicalsecurity practices and behaviors within the company byobserving physical activities and eliciting employeeexplanations and opinions regarding these activities, inorder to improve the company’s physical and CyberSecurity posture from the perspective of theCompliance, Physical and Cyber Security21 teams

GQM – Tailgating - Example 2CIP-6, R5 Question 1: What are the physical securitypractices and behaviors taking placethroughout the company? Metric: Ethnographic/human observation ofcompany facilities (entryways) and employeeactivities (e.g., tailgating, facilitating tailgating) Question 2: Why are physical securitypractices and behaviors undertaken? Metric: Observations, interviews, anddiscussions with employees and otherstakeholders of the company22Ethnography qualitative research involving theobservation in behavior of groups and/or societies

GQM – Tailgating - Example 2CIP-6, R5 Question 3: How is physical securityperceived and enacted by the members ofthe company? Metric: Qualitative analysis of the datagathered (interviews, observations) toidentify categories, patterns, and themesregarding the practices of physical security23

GQM – Tailgating - Example 2CIP-6, R5 Tailgating narratives identified during projecto Tailgating is understandable- Culture of Trust (Company fosters trusting environment)- Avoiding confrontation- Matter of convenienceo Tailgating must be prevented- Theft, and other loss potential (acknowledged)- Keeping people safe- Physical access may allow “hackers” to potentially compromise systemso Tailgating is hard to prevent- Too expensive (badge readers, cameras, guards)- Lack of compatibility (doors, badge readers, difficult to centrally manage)- Physical locations encourage tailgating24

Tailgating diagram25

GQM – Policy Readability - Example 3CIP-002 - 009 Goal Components- Outcome: Improve, assess- Element: Compliance rates- Element: Readability and difficulty- Element: Security policy documents- Perspective: Security policy user Goal statementThe goal of this project is to improve the securitypolicy compliance rates, by assessing the readabilityand difficulty levels of different policy documents fromthe perspective of the general security policy user26

GQM – Policy Readability - Example 3CIP-002 - 009 Question 1: How difficult is it to read andunderstand company security policydocuments? Metric: Readability software & test standards(e.g., Readability Studio & Flesch ReadabilityEase Test; sentence length and # of syllables) Question 2: Are the readability levels for thesecurity policy documents appropriate for thespecific policy user audience? Metric: Estimated reading levels for policydocument users (based on known educationlevels)27

GQM – Policy Readability - Example 3CIP-002 - ---------------------------Number of sentences 110Number of difficult sentences 55 (50%)Average sentence length 23.5 wordsMinimum grade level (suitable) 16 graduatelevel education28

GQM – Policy Readability - Example 3CIP-002 - 009Readability software resultsTotal Words875Monosyllabic Words7003 Syllable Words10856 Character Words2290Total Words02950010001500**Online tools are available20002500

GQM – Policy Readability - Example 3CIP-002 - 009Flesch Readability Ease Test scriptionVery EasyEasyFairly EasyStandardFairly DifficultDifficultVery Confusing

GQM – Policy Readability - Example 3CIP-002 - 009 Potential liabilities – Complex Readabilityo Security breaches by employees whocouldn’t understand the policyo Potential lawsuits for wrongfultermination of employees fired for policyviolations31

Don’t dumb down policies Too Much!BEWARE: Special & Mail Room Employees!32

Don’t dumb down policies too much!33

Don’t dumb down policies too much!34

35

Establish Security Metrics CatalogSecurity Metrics CatalogGoals and ProjectsPerimeter SecurityEndpoint SecuritySecurity Policy36#As s o c i a t e d M e t r i c s1CVA Results: (# or %) vulnerable Access Points2# or % of Access Points with TFEs3Ports and Services: % Compliant Access Points1CVA results: Vulnerable CCAs (# or %)2# or % of Cyber Assets with TFEs3Ports and Services: % Compliant Access Points1Time between Cyber Security Policy reviews2Readability of Cyber Security Policy3# of Cyber Security policy violations (prev. 6 mo.)

Security Measurement Project (SMP) Logistics and organizational structureo Decide project type (descriptive,experimental, compliance)o Conduct Goal-Question-Metric analysiso Conduct a review for previous effortso Consider data sources & analysisrequirementso Get buy in from management andstakeholders37

Security Improvement Program (SIP) Interconnects tactical SecurityMeasurement Projects over time Measuring security operations becomes astrategic effort (tactical to strategic) Forms a knowledge loop Defining, managing and improvingcollaborations between projects Making informed decisions that improveCyber Security38

SIP document structureSIP Document Number: SIP2013.01-30General Project Data:Completed Projects 3Active Projects 1Proposed Projects 1Security Measurement Project AProject Name / Number: Policy Readability AssessmentSMP2013.01Project Sponsor / Lead: Sponsor - B. Castagnetto – CIOLead - J. Andrews – CISOProject Begin / End:Begin: 11.04.12End: 01.30.13SMP GQM Goal(s):Questions: 39 Metrics: Assess the readability of the corporate securitypolicy from the perspective of the policy user

SIP Diagram40

SIP DiagramDATA INFORMATION KNOWLEDGE WISDOM41

42

3rd CVA Project2nd CVA Projectw/ new vulnerabilityremediationtime concerns1st CVA Project431st Budget Project2nd Budget Project

Business Case for Metrics 44Identify stakeholders and sponsorsGoal, questions and metrics analysisProject cost and project benefitsRisk analysis results (be upfront)Formal acceptance (formal signoff bystakeholders

Cyber Security Business Process Deconstruct all the activities that make upyour Cyber Security Business Processo Who owns the process?o Who completes each process?o What systems are involved with eachactivity?o How much does each activity cost?o How long does each activity take?45

46