WIXLIB

SOFTWARETECHNOLOGYEditor: Christof EbertVector Consulting Serviceschristof.eber t@vector.comModel-Based TestingIna SchieferdeckerTest consumes a large share of development efforts. If you don’tcontinuously trim validation efforts and improve test efficiency whilemaintaining quality, your overall costs won’t remain competitive.Model-based testing (MBT) strives to automatically and systematicallygenerate test cases. In this column, Ina Schieferdecker introducesMBT technologies and methods. I look forward to hearing fromboth readers and prospective authors about this column and thetechnologies you want to know more about. — Christof EbertIN MODEL-BASED TESTING (MBT), manually selected algorithms automatically andsystematically generate test cases from a setof models of the system under test or its environment. Whereas test automation replacesmanual test execution with automated testscripts, MBT replaces manual test designswith automated test designs and test generation. However, a recent survey about the“Practice on Software Testing” shows thatMBT hasn’t yet arrived in industry, althoughits potential gains could be enormous.1 Itsprospective benefits include early and explicit specification, modeling, and review of system behavior, andearly discovery of specification errors; better documentation of test cases, increased transparency, and enhancedcommunication between developers andtesters; the ability to automatically generate useful tests and measure and optimize testcoverage; the ability to evaluate and select regression test suites;14I E E E S O F T WA R E PUBLISHED BY THE IEEE COMPUTER SOCIE T Y easier updates of test models and suitesfor changed requirements and designsand for new product versions, and improved maintenance of test cases; higher test quality through model-basedquality analysis; and shorter schedules and lower costs.This column provides an overview of thestate of the practice and MBT methods andtools.MBT SynopsisFirst-generation MBT consisted of test generation from system models. This researchdeveloped the main concepts and algorithmsfor test behavior and test-data generation.However, it showed various shortcomingssuch as using a single model for code andtest generation. This meant that errors in themodel got propagated to the code and testsand were thus impossible to detect.Second-generation MBT uses separate testmodels, which are sometimes called scenariomodels, usage models, environmental models, and so on. Test models support specifica0 74 0 -74 5 9 / 12 / 3 1. 0 0 2 0 12 I E E E

SOFTWARE eedbackGenerateTest selectionTest adaptationControlCriteriaIssueObserveSystem under testVerdict ]Abstract test?]tion of testing concerns in a dedicatedmodel. This approach propagates theduality of the system and test systemto the model level. This extends to therequirements level, which distinguishesbetween system requirements and testrequirements and the models thereof.Second-generation MBT makes explicit the tester’s expertise in denotingtesting’s essential concerns. Tests aredesigned by methods and tools established as industrial practice in modeldriven software engineering and aredocumented during high-level reviewand discussion. The tests’ quality—including correctness and coverage—canbe predetermined at the model level.2FeedbackFIGURE 1. Model-based testing (MBT) elements.3 This process consists of three main steps:designing a functional test model, determining test generation criteria, and generating thetests.MBT’s Three Main TasksThe generated tests can be executedmanually or automatically, but combining automated test generation withautomated test execution achieves thebiggest gain. Key aspects of MBT encompass modeling notation and principles for test models, strategies, andalgorithms to guide test generation and,optionally, on-the-fly generation or offline realization of executable tests. Figure 1 shows a typical MBT process inindustry, which consists of the following three tasks.Design a Functional Test ModelThe test model represents the expectedoperational behavior of the system under test (SUT) or its environment. Testdesigners can use standard modelinglanguages such as UML to formalizethe points of control and observationof the SUT, the system’s expected dynamic behavior, the entities associatedwith the test, and test data for varioustest configurations. They link modelelements such as states, transitions,and decisions to the requirements toensure bidirectional traceability between the requirements and the modeland later to the generated test casesand test results. The test models mustbe precise and complete enough toallow automated derivation of testsfrom them.Determine Test Generation CriteriaModels can usually generate an infinitenumber of tests, so test designers selectcriteria to limit the number of generated tests (for example, by selectingthe highest-priority tests or ensuringspecific coverage of system behaviors).A common approach for test selectionis based on structural-model coverage(for example, determining the tests’coverage of model elements). Two examples of this approach are equivalencepartitioning on the basis of the model’sdecisions or constraints and path coverage on the basis of k-pair transitionsor selected execution sequences in themodel.Another useful type of criteria ensures that the generated test cases coverall the requirements, possibly withmore tests generated for requirementswith higher risk. In this way, testers canuse model-based testing to implementrequirements-oriented or a risk-driventest.Generate the TestsTypically, test generation in MBT isfully automated. The generated testcases are sequences of high-level eventsor actions on or by the SUT, with inputparameters, expected output parameters, and return values for each eventor action. These sequences are similarto high-level test sequences that testerswould design manually in keyworddriven testing (see Figure 1). Typically,the test sequences are easy to understand and complete enough for manualor automated test execution. If necessary, testers refine the tests to a moreconcrete level or adapt them to the SUTto support their automated execution.MBT ToolsIn the past, three commercial toolshave led the MBT scene: Smartesting’sCertifyIT, the Conformiq Tool Suite,and Microsoft’s Spec Explorer. Thenext generation of more sophisticatedMBT tools include Tedeso by Siemens/Imbus, Elvior’s TestCast Generator,and All4Tec’s MaTeLo. Many tools useTTCN-3 (Testing and Test Control Notation) as the language and executiontechnology for generated tests.4 Table 1compares these tools.Companies are also deploying MBTtools for dedicated application domains.J A N U A R Y/ F E B R U A R Y 2 0 1 2 IEEES O F T W A R E 15

TABLE 1SOFTWARE TECHNOLOGYGeneral-purpose model-based-testing tools.ToolURLTarget domainsTest modelTest generation criteriaTest usiness ProcessModel and Notationor Unified ModelingLanguage (UML)Test data and verificationpointsTextual test plansand executable testscripts in QuicksetProfessional and so nications andtelecommunicationsState chartsRequirements-driven testgeneration, black-box testdesign heuristicsTextual test plans andexecutable test casesin Java, and so jects/specexplorerSoftwareSpec#Transition coverageExecutable test casesin C# or on-the-flytestingTedeso naged-model-based-testingSoftwareUML activity andsequence diagramsModel and data coverageExecutable test casesin C , and so torTelecommunications,transportation,defenseUML state machinesState, transition, anddecision coverageExecutable test casesin TTCN-3 (Testingand Test ControlNotation)MaTeLo4.7.5www.all4tec.netEmbedded systemsEnhanced MarkovchainsProbabilities for transitionsand inputsTextual test plans andexecutable test casesin TTCN-3, and so onFor example, embedded-systems MBTtools include Reactis by Reactive Systems (www.reactive-systems.com) andModena by Berner & Mattner (http://www.berner-mattner.com/en/berner-m at t ner-home /produc t s /moden a /index.html). Research and open sourceMBT tools include /verification/gtcb/index.html),GraphWalker (www.graphwalker.org), Auto Focus (http://autofocus.in.tum.de/index.php/Main Page), Fokus!MBT hnologien/fokusmbt/index.html), and Uppaal-CoVer (www.hessel.nu/CoVer).MBT in StandardizationA key issue in MBT is the plethora ofconcepts and methods that don’t followa common convention. Since 2005, theObject Management Group (OMG) hasoffered the UML Testing Profile (UTP)specification to support model-based16I E E E S O F T WA R E testing that’s seamlessly integrated withUML. Today, several open source andcommercial solutions implement UTP.Several research papers and companies(IBM, Microsoft, and so on) also reference it. OMG completed a UTP revision in July 2011.Other standardization bodies haveinitiated approaches to a common nomenclature for testing. For example,several MBT tool vendors and majorindustrial users have developed a European Telecommunications StandardsInstitute (ETSI) standard to unify terminology and define a common set ofconcepts that MBT tools should support.3 The MBT special interest groupfrom the International Software Quality Institute has begun a training andqualification initiative This technology-, tool-, and vendorindependent MBT qualification hasfour goals. The first is to let testers apply model-driven engineering methodsW W W. C O M P U T E R . O R G / S O F T W A R Efor test automation that are well established in software development. Thesecond goal is to build the corresponding engineering skills. The third goal isto establish a qualification that servesas a standardized skill set for recruitment by providing a profound MBTcertification rather than a number ofspecific tool qualifications. The finalgoal is to establish a qualification forcareer and training planning.MBT RolloutMBT’s successful application dependsgreatly on proper adaption to a company’s development infrastructures andprocesses. One potential cause for disappointment is expecting too much fromMBT. Companies often underestimateits rollout costs (see the “Model-BasedTesting Costs” sidebar). In addition,they often misclassify requirementsfor MBT infrastructure and personnelqualifications.Studies on the application of MBT