WIXLIB

WHITEPAPERMETRICS AND ANALYSISIN SECURITY MANAGEMENTBy Brian McIlravey, CPPand Peter Ohlhausen

About the Authors:Brian McIlravey, CPP, is Co-CEO of PPM 2000 Inc. (www.ppm2000.com) and isresponsible for driving strategic planning and product direction. He is a member ofASIS International’s Information Security Technology Council and has experience in bothcorporate security and public law enforcement.Peter Ohlhausen is president of Ohlhausen Research, Inc. (www.ohlhausen.com), whichfor more than 20 years has provided research and consulting to the security, technology,and criminal justice fields. He formerly served as editor of Security Management, themonthly magazine of ASIS International.Published by PPM 2000 Inc.www.ppm2000.comFor over twenty years, PPM has worked with organizations around the world—using theirknowledge of risk management, security management and loss prevention—to providehigh quality subject matter expertise in the design and application of Incident Reportingand Investigation Management software. Thousands of organizations have implemented aPPM solution, and the company’s clients span all industries and the Fortune 1000. PPMis recognized by Microsoft as a Gold Independent Software Vendor.From incident reporting, to investigation management, to actionable businessintelligence, PPM offers end-to-end Incident Management solutions for—and from—security professionals. For more information on Perspective by PPM 2000, contact PPMtoll-free at 1-888-776-9776 or email information@ppm2000.com.Copyright 2012 PPM 2000 Inc.

ContentsExecutive Summary7The Power and Importance of Metrics and Analysis8Fortified Decision Making11Metrics as a Security Operations Tool12Metrics as Marketing for the Security Program14Developing Specific Metrics17Essential Ingredient: Data20From Data to Information: Analyzing Metrics21Getting Started23ReferencesPPM 2000 Inc.10088 ‑ 102 Avenue, Suite 13071‑888‑776‑9776 Edmonton, Albertainformation@ppm2000.com T5J 2Z1www.ppm2000.com25

Executive SummaryThe use of metrics and analysis (MA) is a sophisticated practicein security management that takes advantage of data to produceusable, objective information and insights that guide decisions.In addition, MA provides chief security officers (CSOs) with clearevidence of their operations’ value, expressed in the language oftop management.As Carnegie Mellon University notes, “metrics are quantifiablemeasurements of some aspect of a system or enterprise Security metrics focus on the actions (and results of thoseactions) that organizations take to reduce and manage the risksof loss of reputation, theft of information or money, and businessdiscontinuities that arise when security defenses are breached.”Through MA, a CSO or other security professional can betterunderstand risks and losses, discern trends and manageperformance. He or she can also report clearly and accurately toexecutive management. These uses of MA all work to support theorganization’s strategic goals.Software designed specifically for the security field can make thegathering of security and risk-significant data orderly, convenientand accurate—and hold the data in a format that facilitatesanalysis. Security and risk-focused incident management softwareoffers both the standardization and consolidation of data. Suchsoftware also automates the task of analysis through trending andpredictive analysis and the generation of customized statisticalreports.This paper synthesizes the current MA literature in the securitymanagement field. It describes the use of metrics and analysis to: Improve decision making; Strengthen security operations; and Gain support for the security and risk management operation.It then describes the process of developing specific metrics,collecting and managing data and performing useful analyses withsecurity risk-focused software.Metrics and Analysis in Security Management7

The Power and Importance ofMetrics and AnalysisThis paper examines key themes and thinking in the field ofmetrics and analysis (MA), focusing on applications in thedomain of security management. The aim is to inform securityprofessionals about a powerful practice that is becomingincreasingly essential in competitive business environments—and,in fact, is often demanded by executive management.The use of MA is part of a serious approach to securitymanagement. In contrast to more casual, gut-oriented approachesto security decision making, MA takes advantage of data toproduce usable, objective information and insights that guidedecisions. In addition, MA provides CSOs with clear evidenceof their operations’ value, expressed in the language of topmanagement.The Systems Security Engineering Capability Maturity Model,developed by a team headed by Carnegie Mellon University toadvance security engineering, provides an especially clear view ofmetrics:At a high level, metrics are quantifiable measurements of someaspect of a system or enterprise. For an entity (system, product, orother) for which security is a meaningful concept, there are someidentifiable attributes that collectively characterize the security ofthat entity. Further, a security metric (or combination of securitymetrics) is a quantitative measure of how much of that attributethe entity possesses Security metrics focus on the actions (and results of thoseactions) that organizations take to reduce and manage the risksof loss of reputation, theft of information or money, and businessdiscontinuities that arise when security defenses are breached.They are useful to senior management, decision makers, users,administrators, or other stakeholders who face a difficult andcomplex set of questions regarding security, such as:8 How much money/resources should be spent on security? Which system components or other aspects should betargeted first? How can the system be effectively configured? How much improvement is gained by securityexpenditures, including improvements to securityprocesses?Metrics and Analysis in Security ManagementWHYUSEMETRICS?Metrics and analysisprovides CSOs withclear evidence of theiroperations’ value, expressedin the language of topmanagement.“What’s the benefit of usingmetrics? Basically, toimprove overall security andreduce costs.”Raymond Musser, CPPVice President, SecurityGeneral Dynamics(Musser, 2011)

How do we measure the improvements? Are we reducing our exposure?The MA approach results in business intelligence, which has beendefined as (PPM 2000 Webinar, 2009):The collection, integration, analysis, interpretation andpresentation of business information to provide historical, currentand predictive views of business operations, [and] the use of thisinformation through extraction, analysis and reporting to supportbetter business decision making.The insights and findings a CSO gains through MA can supportactivities both inside and outside the corporate securitydepartment. Inside the department, the CSO can betterunderstand risks and losses, discern trends and manageperformance based on actual measurements. Outside thedepartment, the CSO can report clearly and accurately toexecutive management. Both the internal and external uses of MAwork to support the organization’s strategic goals.The related concept of benchmarking—comparing one’sorganization with others in the same industry—relies in parton using metrics. That comparison relies first of all on anunderstanding of one’s own organization, and that understandingmust be developed through MA. According to Hayes and Kotwica(2011),Business leaders recognize benchmarking as a proven businesspractice that can identify competitive strengths and vulnerabilitiesas well as opportunities for improvement But while the demandfor performance measures has trickled down to the securityfunction, the appreciation for them hasn’t always come along forthe ride. Too many security leaders create or find benchmarks forthe sole purpose of appeasing their bosses rather than from anearnest desire to use these tools to explore what others are doing,address potential gaps and add value.ALIGNSTRATEGY ANDPERFORMANCE[C]orporate performancemetrics [was] the topictackled by the most recentBlue Ribbon Commission atthe National Association ofCorporate Directors (NACD).Why corporate performancemetrics? Because theylink corporate strategy andcorporate performance Strategy is about the future,performance is about the pastand metrics align the two.Financial Executive(Daly, 2011)It is important to remember that MA consists of both metricsand analysis. Hayes and Kotwica emphasize that point withthe example of benchmarking on corporate ethics hotlines. Thebenchmark report may suggest that the average organization ofa certain size and industry receives eight to nine calls to thecorporate ethics hotline per thousand employees. If a particularcompany receives only three calls per thousand employees,analysis is warranted. Does the company have fewer ethicsproblems than its peers? Are employees intimidated into notreporting their concerns? Is the hotline underpublicized?Metrics and Analysis in Security Management9

In the MA approach, which is relatively new, key terminology isnot completely settled. On one hand, Payne (2006) observes:Measurements provide single-point-in-time views of specific,discrete factors, while metrics are derived by comparing to apredetermined baseline of two or more measurements takenover time. Measurements are generated by counting; metricsare generated from analysis. In other words, measurements areobjective raw data and metrics are either objective or subjectivehuman interpretations of those data.In Security Metrics Management: How to Manage the Costs of anAssets Protection Program, Kovacich and Halibozek (2005) definea metric as “a standard of measurement using quantitative,statistical, and/or mathematical analyses.” In their taxonomy, asecurity metric is,The application of quantitative, statistical, and/or mathematicalanalyses to measuring security functional costs, benefits,successes, failures, trends and workload—in other words, trackingthe status of each security function in those terms.On the other hand, the National Institute of Standards andTechnology (2008) states that “while a case can be made forusing different terms for more detailed and aggregated items,such as ‘metrics’ and ‘measures,’ [this report] standardizeson ‘measures’ to mean the results of data collection, analysis,and reporting.” The same source refers to the process of datacollection, analysis and reporting as “measurement.” HarvardBusiness Review refers to analytics rather than metrics andanalysis (Davenport & Harris, 2010). The terminology will likelycontinue to evolve.Despite the clear value of MA, one source suggests that onlyabout a third of CSOs collect and analyze metrics (Kohl, 2009).Specifically, in a survey by the Security Executive Council (SEC),only 31 percent of survey respondents “gather security programdata in order to create statistical reports to present to seniormanagement.”Regarding the significance of that finding, Kohl quotes SECspokesmen as follows:[I]t should be more than a wake-up call that 69 percent saidthey don’t collect information—it should be an alarm. [A] largepercentage didn’t collect data because management hadn’tasked for it. That may mean management isn’t even awarethat security has metrics that may impact the business, or it10Metrics and Analysis in Security ManagementMAKEBETTERDECISIONSAnalytics: Using data andquantitative analysis tosupport decision making.Benefits: Decisions are morelikely to be correct. The scientific methodadds rigor.Caution: Correct assumptionsare crucial.If you don’t assess theresults of your changes,you’re unlikely to achievebetter decisions.Harvard Business Review(Davenport, 2009)