Transcription
Threat Modeling Tools: ATaxonomyZhenpeng Shi, Kalman Graffi, David Starobinski, Nikolay MatyuninAbstract—Threat modeling tools allow to identify weaknesses in a system design. Yet, understandingconceptual differences between the tools is not trivial. We propose a taxonomy of threatmodeling tools, and use it to compare several popular ones. Alongside, we illustrate differencesbetween the tools with a usage example.1. IntroductionThreat modeling is a structured process foridentifying and understanding potential threats, aswell as developing and prioritizing mitigations,so that valuable assets in the system can be protected. Performing threat modeling can not onlyhelp to identify and mitigate security issues at anearly stage, but also guide investment on systemsecurity. For instance, the work in [1] presentsa case study of threat modeling conducted atNew York City Cyber Command, a large-scaleand high-risk enterprise environment. The resultsof the case study suggest that, when properlyconducted, threat modeling is effective at theenterprise level and results in positive feedbackfrom the involved participants.Many threat modeling tools have been developed over the years to help with the threat modeling process. Although the tools share the sameoverarching goal, they vary significantly in termsof functionalities and workflow. Different toolsmight be effective for different threat modelingsteps. Thus, the most suitable tool depends onthe specific use case.In this paper, we propose a set of criteria forevaluating threat modeling tools. We conceptually classify current threat modeling tools, anddiscuss their properties and design implications.We then apply the criteria to provide a taxonomyof popular threat modeling tools, and qualitativelyevaluate and compare the capabilities offered bythe tools. Note that comparing the performanceof the tools is difficult, since most tools requiremanual configurations and design. Hence, we donot compare the effectiveness of the tools, andleave quantitative evaluation to future work.Our proposed evaluation and comparison ofthreat modeling tools are based on the followingcriteria, which are discussed in detail in Section 5.1) Model form. What form of system model isadopted by the tool (e.g., diagram vs. textbased)?2) Model reusability. Is the tool able to reuseparts of existing models when building newones?3) Threat library. What libraries are used forthreat identification? What are the sizes ofthose libraries?4) Threat evaluation. How does the tool evaluate the identified threats?5) Mitigation suggestion. Does the tool suggest effective countermeasures to the identified threats?6) Availability. How does the tool run,as a desktop/web application or throughcommand-line interface? Is the tool opensource or closed-source?7) Development stage. Is the tool still in development or in a relatively mature state?8) Software development life cycle (SDLC)Published by the IEEE Computer Society IEEE1
integration. Does the tool offer integrationwith other development tools regularly usedin SDLC (e.g., issue tracking systems)?We envision that our proposed taxonomy ofthreat modeling tools will help academic researchers and security practitioners to find thetools most suited for their needs. In summary, ourwork has the following main contributions:1) We propose a set of criteria for qualitativeevaluation of threat modeling tools. Thecriteria cover different features and capabilities of the tools.2) We use the criteria proposed in this paperto compare current threat modeling tools.The comparison results are summarized ina table to help understand the strengths andweaknesses of the different tools.3) We perform threat modeling for an example use case to investigate in more detailthree popular threat modeling tools (i.e.,Microsoft Threat Modeling Tool, OWASPpytm, and IriusRisk community edition).We illustrate the workflow of the threatmodeling process with each tool, and pointout interesting findings.4) We share insights from the evaluation andcomparison of the threat modeling tools.2. BackgroundIn this section, we provide background onthreat modeling, including detailing its mainsteps. Further, we describe popular methods andknowledge bases that help implement each stepin practice.Threat modeling is a structured process foridentifying potential threats and developing mitigations, with the purpose of protecting valuableassets in a system. We define threat modeling asa process consisting of the following six steps:(i) Define security requirements. Security requirements usually include a set of standards tomeet and functionalities that are related to thesecurity of the system. The requirements helpdetermine what data assets in the system are themost important, and to what extent those dataassets need protection from threats. A referenceto help with this step includes the OWASP TopTen1 , which represents the most critical 10 typesof security risks to web applications. Such references provide developers with a quick and effective way to reason about fundamental securityrequirements that should be satisfied.(ii) Model the system. The system model isa graph that represents system artifacts as nodes,which are grouped in trust zones, and data flowsbetween the nodes as edges. This model caneither be written in a text-based form following acertain syntax, or visualized in a diagram form,for example, a data flow diagram (DFD). A DFDis a common way to model system entities, eventsand boundaries of the system. It consists of fourtypes of elements: processes, data flows, datastores, and external entities.(iii) Identify potential threats based on thesystem model. Typical ways to implement thisstep include the STRIDE method and using threatknowledge bases. STRIDE is a widely adoptedand mature method for threat identification. Theword “STRIDE” represents six categories ofthreats: spoofing identity, tampering with data,repudiation, information disclosure, denial of service, and elevation of privilege. STRIDE analysis is performed from the defender’s perspectivebased on DFDs. Additionally, STRIDE provideschecklists to help identify threats that fall in eachcategory [2].Prominent knowledge bases include the Common Vulnerabilities and Exposures (CVE)2 ,Common Weakness Enumeration (CWE)3 , andCommon Attack Pattern Enumeration and Classification (CAPEC)4 , which can further help identify threats. The CVE lists publicly disclosedcybersecurity vulnerabilities, while the CWE listssoftware and hardware weakness types. A weakness can lead to a specific instance of a vulnerability within a product or system. The CAPECenumerates common attack patterns that helpunderstand how weaknesses can be exploited byadversaries.(iv) Evaluate the identified threats based onmultiple aspects, such as accessibility to attackers, attack complexity, privileges required, etc.In order to quantify the impact of threats, scoring systems are employed. Moreover, since thethreats are typically inter-connected, attack trees2 https://cve.mitre.org/3 https://cwe.mitre.org/1 https://owasp.org/www-project-top-ten/24 https://capec.mitre.org/
and attack graphs can be used to capture therelationships between the different threats.To assist with evaluation, the Common Vulnerability Scoring System (CVSS)5 and CommonWeakness Scoring System (CWSS)6 are designedto provide numerical scores of known vulnerabilities or weaknesses. CVSS and CWSS dividethe evaluation metrics into multiple groups, andcompute a score based on the metric groups. Thescore computed by CVSS or CWSS indicates theseverity of a vulnerability or weakness, respectively.Attack trees are conceptual diagrams that represent attacks in a tree form, where the root noderepresents the attack goal, and leaves representways to achieve the goal [3]. Attack graphs aredirected graphs, where each vertex denotes a state(e.g., indicating that a certain vulnerability hasbeen exploited) and each edge represents a statetransition [4]. Attack trees and attack graphs arewidely used for network security evaluation.(v) Mitigate the potential threats accordingto the security requirements. The most suitablemitigations vary with the specific threat and thesystem where it resides. Nonetheless, some general suggestions for threat mitigation can be foundin knowledge bases such as CWE and CAPEC.(vi) Validate that the threats are mitigated afterapplying mitigation techniques. The OWASP Application Security Verification Standard (ASVS)7can help with this step by providing a basis fortesting web application security controls.Note that our definition of threat modelingsteps is a variation of the five typical steps introduced by Microsoft on threat modeling [5]. Here,we introduced the additional “evaluate” step, toensure that scarce security resources are allocatedin a timely manner to the most important vulnerabilities.The six steps defined in this section are tobe iteratively applied during the SDLC. In someother definitions of threat modeling steps, the“define” step is omitted [6]. Since security requirements usually change over time, we arguethat it is necessary to perform the “define” stepeach time to appropriately prioritize threats.5 https://www.first.org/cvss/6 https://cwe.mitre.org/cwss/cwssv1.0.1.html7 -verification-standard/3. Overview of Threat Modeling ToolsMany threat modeling tools have been developed over the years. In this section, we give ataxonomy of current threat modeling tools. Moredetailed evaluation and comparison are presentedin Section 5.At the core of a threat modeling tool lies itslibrary. The threat library lists a set of systemartifacts to select from, threats associated to eachartifact and mitigations associated to the threats.A system developer can design a system using theavailable artifacts either through a visual diagramdrawing plane, or through text-based representation (e.g., in YAML format). The threat modelingtool then parses the system model and generatesa threat report. Some tools further provide an integration of the weaknesses and their mitigationsin a ticketing system. This allows tracking of themitigations in the SDLC.The tools can be categorized in several ways,for instance, non-commercial versus commercial, diagram-based versus text-based. Commercial tools are usually tailored for the needs of largecompanies (e.g., scalability and collaboration) butcome at higher costs, while all the functionalitiesof non-commercial tools are available for free.The non-commercial tools introduced in thissection are entirely free to use without limitationson functionalities. All of them are open-source,except for the Microsoft Threat Modeling Tool.Commercial tools emphasize usability, agiledevelopment, and collaboration. Almost all commercial tools are diagram-based. In the “model”step, commercial tools usually take measures tosimplify the process of drawing the system diagram, for example, by providing questionnaires.Moreover, they typically maintain large threatlibraries. Overall, commercial tools offer a rathercomplete solution for threat modeling, thoughtheir effectiveness remains to be compared withother tools.3.1. Diagram-based toolsThreat modeling with diagram-based tools isintuitive: the tools provide users with an easy-touse interface to draw system diagrams, and thenapply rules from the library to detect potentialthreats. A report is then generated as a summaryof threat modeling results.3
Microsoft Threat Modeling Tool. The Microsoft Threat Modeling Tool (TMT) is a maturetool that has evolved over several years. Forstarters, it provides a default library (here calledtemplate). Once the model is built based on thetemplate, the tool identifies threats by checkingthreat conditions in the template. Threats in thedefault template are categorized by STRIDE. Thetemplate can be customized to suit specific usecases. Elements in the diagram, threat types,and threat properties are all customizable. Threatproperties include description, priority, and countermeasures. Users can also add communitycontributed templates or build their own onesfrom scratch.OWASP Threat Dragon. OWASP ThreatDragon (TD) shares a similar workflow to TMT.It is designed as a relatively light-weight tool.Hence, its threat library (here called threat ruleengine) is not as customizable as TMT. TD onlysuggests generic potential threats, while details ofthe threats are left for users to fill in. Its advantages include workflow integration with Github,and support for more threat categories (LINDDUN and CIA) besides STRIDE.OVVL. The Open Weakness and Vulnerability Modeler (OVVL) works as a web application.OVVL distinguishes itself from other diagrambased tools by identifying threats at both thedesign level and operation level [7]. At the designlevel, OVVL identifies STRIDE threats stored inits self-defined library (akin to TMT). At the operation level, OVVL can query the CVE knowledgebase to find more specific vulnerabilities. Thisquery is based on a description of system artifactsthat follows a standard naming scheme.IriusRisk (commercial). IriusRisk embedsthe Draw.io diagram editor in its web applicationto enhance user experience when drawing theDFDs of a system. When initializing the diagramand adding new elements, questionnaires are provided to help users with configuration. Its threatlibrary covers not only common knowledge baseslike CVE and CWE, but also self-defined threatsfrom typical enterprise use cases, such as AWSdeployments. IriusRisk provides rather comprehensive threat evaluation, and offers priority andcost estimation of the suggested countermeasures.ThreatModeler (commercial). The functionalities offered by ThreatModeler are similar to4those of IriusRisk. However, in the “model” step,it applies so-called process flow diagrams (PFDs).Compared to the commonly used DFDs whichmodel the system from a defender’s perspective,PFDs aim to provide a visual decomposition ofthe system from an attacker’s perspective.SecuriCAD (commercial). SecuriCAD features AI-based predictive attack simulations. Itincludes a potential attacker in the diagram-basedsystem model, and uses it to simulate possibleways (“attack paths”) to compromise a system. Bymeasuring the time-to-compromise, the criticalattack paths to high-value assets can be found,and risk can be evaluated.3.2. Text-based toolsTo build a model with text-based tools, usersare required to express their model in a structuredlanguage, instead of drawing the system diagramdirectly.OWASP pytm. OWASP pytm describes thesystem model in Python language. It requiresusers to create a Python object for each of the system components based on its pre-defined Pythonclasses. The tool then generates a system diagramand identifies threats from the text-based model.Its threat library contains comprehensive information about each potential threat. To mention afew, the “condition” attribute for threat identification, the “likelihood” and “severity” attributes forsimple evaluation, and the “mitigation” attributesuggesting countermeasures.Threagile. Threagile is designed for agilethreat modeling. The models in Threagile arerepresented in YAML format. This format is moreconcise and abstract than the Python representation used in OWASP pytm. Threagile evaluatesthreats in a more rigorous way: instead of providing pre-defined evaluation information from thethreat library, it calculates two metrics based onthe system model: relative attacker attractivenessand data loss probability.3.3. Other toolsA few other tools have been designed withslightly different purposes or expected outcomes.As a result, they do not strictly follow the sixthreat modeling steps defined in this paper.CAIRIS. CAIRIS is a platform for specifyingand modeling secure and usable systems. It takes
information about different aspects of the system,including assets, roles, and tasks. Users can thenuse CAIRIS to manually model the system asDFDs, or as a top-down representation similarto attack trees, and perform risk analysis andresponse.Threatspec. Threatspec performs threat modeling directly on the code. It requires that developers and security engineers write threat modelingannotations as comments inside their source code.It then generates DFDs and reports from thecommented source code.SD Elements (commercial). SD Elementsdoes not require users to draw a system diagram.Instead, it gathers information about the systemvia a survey. It claims to identify threats at a fasterrate than traditional threat modeling processes.Kenna (commercial). Kenna emphasizesdata-driven threat modeling. It uses data sciencetechniques, such as natural language processingand predictive modeling, to assess, prioritize, andpredict risks. It introduces its own risk scoreengine for quantitative threat evaluation.Tutamen Threat Model Automator (commercial). The Tutamen Threat Model Automaterleverages existing project design diagrams insteadof letting users create separate DFDs. Users entermetadata as values into each of the diagramelements to describe the threat model.4. Tools Usage ExampleIn this section, we investigate three representative threat modeling tools through an exampleapplication. The selected tools include the community edition of a commercial tool (IriusRisk), anon-commercial diagram-based tool (TMT) and anon-commercial text-based tool (OWASP pytm).We model the example application with the selected tools, compare the results, and discuss ourexperience in using the tools. The main purposeof this section is to show how the tools areactually used for threat modeling, and how theselected tools differ.The example application is one provided byOWASP pytm. The DFD of the application isshown in Fig. 1. The application is a simple webapplication, where a user logs into the application and posts comments on the app. The appserver uses the real identity database to verifythe user’s identity, stores the comments into theSQL database, and shows the comments backto the user when requested. In the meantime, aseparate AWS Lambda service periodically cleansthe database.We next model this application with TMT andthe IriusRisk community edition. When modelingthe application with TMT, we use the defaulttemplate. Both TMT and IriusRisk use a diagrambased model. When drawing the diagram with thetools, users can select from pre-defined systemcomponents. Fig. 2 and Fig. 3 show the interfaceof drawing system diagrams using TMT andIriusRisk, respectively.When building the model in OWASP pytm, auser needs to create a Python object for each system component, then set the component attributes(or leave them as default values). For example, thePython code used for defining the SQL databasein Figure 1 is as follows:1234567db Datastore("SQL Database")db.OS "CentOS"db.isHardened Falsedb.inBoundary server dbdb.isSQL Truedb.inScope Truedb.maxClassification Classification.RESTRICTEDAll three tools generate reports of threat modeling results. We compare next these reports andnotice the following differences (a more completeevaluation of the three tools is provided in Section 5, and here we only list additional findingsfrom our usage example):1) Duplicate threats. Each tool lists the samethreat, if it occurs for different components.For example, OWASP pytm identifies the“privilege abuse” threat for both the webserver and the real identity database. Thisis because the tools iterate through eachcomponent and check threat conditions inits library. In other words, each componentis examined independently when identifying threats.2) Threat identification. Data flow diagramsconsist of processes, data stores, data flow,and external entities. TMT finds threats bychecking the data flows, while IriusRiskmainly reasons about the other three typesof components. In comparison, OWASPpytm examines all components in the diagram.5
Figure 1. The example application modeled with threat modeling tools.3) Threat information provided to users. Withits default template, TMT provides the leastinformation about the identified threats: nomitigations are suggested, and priority ofthreats are left for users to set. OWASPpytm offers basic threat evaluation and mitigation based on its threat library. IriusRiskprovides further information (e.g., costs ofcountermeasures).4) Tracking threat status. OWASP pytm lacksthe functionality of tracking threats. TMTallows users to select the state of each threatfrom “not started”, “needs investigation”,“not applicable”, and “mitigated”. IriusRiskhas a relatively more complete trackingsystem, and can also utilize external issuetrackers such as Jira.We find that IriusRisk provides a more complete support for the threat modeling process. Incomparison, TMT and OWASP pytm fall short ofcertain capabilities. We also noted subtle differences in the ways used by the tools to identifythreats. Regarding user experience, we find thatthe two diagram-based tools are more intuitive touse, and have more pre-defined types of systemcomponents. The questionnaires and abundantchoice of pre-defined components provided byIriusRisk also make it easier to build the systemmodel.5. Evaluation of Threat Modeling ToolsIn this section, we compare six threat modeling tools under a certain set of criteria. We mainly6focus on freely available tools, since we couldindependently verify their capabilities. The selected tools are: Microsoft Threat Modeling Tool(TMT), OWASP Threat Dragon (TD), OVVL,OWASP pytm, Threagile, and IriusRisk. Amongthese tools, TMT, TD, and OVVL are noncommercial diagram-based tools, OWASP pytmand Threagile are non-commercial text-basedtools, and IriusRisk is a commercial diagrambased tool. We evaluate the tools based on thecriteria introduced in Section 1, and summarizethe results in a table.Note that TMT offers one default template(threat library), and two additional templates forthreat modeling of Azure and medical devices,but we only evaluated the default one, since it issuitable for general use. Regarding IriusRisk, wechoose the community edition for our comparison. This version can be used for free but haslimited functionalities.We stress that quantitative comparison of theperformance of the threat modeling tools is difficult, because the tools require manual configuration and support customization (e.g., customizingrules for threat identification). Thus, users cangreatly influence the results provided by each tool.As a result, we compare the tools qualitativelyusing the default library and threat rules providedby each tool. In other words, we only evaluate thecapabilities offered by the tools, instead of theireffectiveness. Still, the comparison is based onobjective metrics.We note that the recent work in [8] comparesthe TMT and TD tools. Our work differs in that
Figure 2. System diagrams drawn with the Microsoft Threat Modeling Tool.we evaluate more tools, including commercialtools and text-based tools. Further, the evaluationin [8] is primarily based on user experiencewith the tools, while our evaluation is based onobjective criteria regarding the tools’ capabilities.5.1. Evaluation criteria justificationIn this subsection, we introduce our evaluationcriteria, and explain their rationale.There exists significant literature on evaluation criteria for software. Some work proposescriteria for general software [9], while othersfocus on specific types of software, such ascommercial software [10], [11] and free/opensource software [12], [13]. Common concernsinclude functionality, usability, sustainability, andmaintainability. In [14], common threat modelingmethods are also evaluated and compared, whichprovides insights into requirements for threatmodeling tools.Note that some software evaluation criteriaare subjective. For example, users might thinkweb applications have better usability, or prefercommand-line tools. Here, we only select criteriathat can be evaluated objectively, such as whethera tool is available as a web application or througha command-line interface. Our evaluation canserve as a reference, while the final choice ofa tool depends on the needs and preferences ofusers.Our proposed criteria fall into two groups:(i) model form, model reusability, threat libraryand threat evaluation; (ii) availability, development stage and SDLC integration. The first groupexamines the functionality of the tools for threatmodeling; the second evaluates other importantaspects of the tools as a software. Evaluatingfunctionality helps characterize the capability ofa tool to assist with the threat modeling process.Other important aspects of a tool include its form(e.g., open-source or commercial), whether it isregularly maintained, and how well the tool fitsinto SDLC [9], [12], [13].Model form. While a diagram-based modelis often more intuitive to build and understand, atext-based model has several advantages, including: (i) performing version control of the modelwith common tools such as Git; (ii) enabling for-7
Figure 3. System diagrams drawn with the IriusRisk community edition.mal and rigorous analysis; (iii) generating different forms of diagrams from the text-based model,for instance, DFDs and sequence diagrams.Model reusability. Light-weight tools mightwork well for simple and small systems, butfall short of handling models of workflow atlarge companies. Some tools strive for modelreusability, making it easier to scale the model.For example, allowing users to define a nestedsystem component that can be reused, or allowingto import or export part of a diagram that can beused by other models.Threat library. Threat modeling tools identify threats by following associations of systemcomponents to threats in their libraries. As aresult, the threat library used by a tool determinesthe types and numbers of threats that can befound. In general, most of the tools allow extending the basic library, with commercial tools oftenproviding a much larger and sophisticated threatlibrary.Threat evaluation. Some threats are more severe or urgent than others. By evaluating threats,a tool can help users make decisions on how tojudiciously allocate security resources. The evaluation of a threat can be static or dynamic, wherestatic evaluation shows pre-defined information inthe threat library, and dynamic evaluation involvesmetric calculation based on the system model.Mitigation suggestion. Suggestions for threatmitigation are valuable information when usersdecide to handle a threat. Common threat knowl-8edge bases such as CWE usually include simplecountermeasures to corresponding threats, making it easier for tools to propose mitigations.Some, but not all threat libraries of the consideredtools link this mitigation information to threats intheir libraries.Availability. This criterion covers three subcriteria: (i) how can users run the tool; (ii) onwhich platform the tool is available; (iii) whetherthe tool is open-source. Sub-criteria (i) and (ii)are related to the usability of the tool, while (iii)influences sustainability and maintainability [9].Threat modeling tools can run as desktop or webapplications, or through a command-line interface(CLI). Tools that run as desktop or web applications are usually more intuitive to use, while toolsproviding only command-line interface might bemore efficient for experienced users and consumefewer resources. Some tools only work on specificplatforms, which may cause problems to userson other platforms. Open-source tools are accessible to everyone for free, and tend to have acommunity that continually maintains the tools.Moreover, experienced users can extend thesetools to suit their own needs. On the other hand,the adoption of commercial tools can be costly.Development stage. It is important to consider whether a tool has long-term support andevolves over time [12]. Recent updates of a toolcan indicate such information. Moreover, the maturity of the tool is reflected by its developmentstage. Some tools were released several years
ago, and have become relatively mature in termsof functionalities. In comparison, some newlyreleased tools are still in development, with morefeature updates planned in the future.SDLC integration. Threat modeling tools aretypically used in the SDLC in conjunction withother development software. During the SDLC,threat modeling needs to be performed repeatedlywhen new changes are made to the software. Asa result, tools are required to be interoperablewith other software [9]. SDLC integration is notonly an important concern for threat modelingmethods [14], but also for the tools themselves.Providing integration with tools in SDLC, such asissue tracking systems, can reduce manual workand make a tool easier to use.5.2. Evaluation of the toolsIn this subsection, we compare the selectedtools based on our proposed criteria.Model form. Most tools, including TMT, TD,OVVL and IriusRisk, model the system in adiagram form, which makes it easier for nonexperts to build and understand models. However,a large diagram might be difficult to maintain.As an alternative approach, OWASP pytm andThreagile model the system in a text form following a specific syntax, which is more suitable forformal analysis and deliberate extensions usingtext-processing scripts, as both the system modeland the threat library are available in structuredtext formats.Model reusability. With TMT, TD andOVVL, system models are usually built fromscratch, and the diagrams for each system arestored in individual files, which makes it difficultto reuse parts of the existing diagrams whenbuilding new models. OWASP pytm and Threagile, too, lack built-in support for reusing systemmodules. Nonetheless, since system componentsare all defined in text forms, it is easier to reusemodules of an existing model with these two textbased tools. For example, in OWASP pytm, userscan import existing system modules from otherfiles as Python objects. IriusRisk allows usersto import and export parts of system diagramsas reusable modul
Microsoft Threat Modeling Tool, OWASP pytm, and IriusRisk community edition). We illustrate the workflow of the threat modeling process with each tool, and point out interesting findings. 4)We share insights from the evaluation and comparison of the threat modeling tools. 2. Background In this section, we provide background on
