WIXLIB

1.1 MotivationThe Tieto office in Karlstad develops telecommunication products and provides testingservices within the same area. Tieto is a supplier, or subcontractor, of telecommunicationsolutions and services for global actors, such as Ericsson. Thus projects at Tieto incorporatecomplex systems and products. In the development process of large and complex systemstesting is a critical activity in order to obtain a system of high quality. Testing must be donecontinuously during the development of a system, as its functionality is gradually increasing,and is necessary on a daily or at least weekly basis. Systems developed at Tieto typically havea large number of states as well as a large number of input and output signals. Thus thenumber of test cases is very high. Normally it is not possible to test all combinations of statetransitions so a representative subset of the important use cases has to be tested.Currently much of the software testing at Tieto makes use of a script-based testingapproach, where the test scripts are written manually by testers. For complex systems thisapproach results in a large number of test cases which have to be manually designed andmaintained as the system evolves. Furthermore, the test scripts are dependent on the testerimplementing the tests, hence test scripts may be difficult to maintain. Recent developmentwithin software testing has led to the development of methods and tools that can generate testcode automatically, based on a model of the system to be tested. This testing approach iscalled MBT.The MBT approach is a black-box testing technique which automates testing tasks such astest generation and test result evaluation [12] and has when deployed shown to yieldeconomic benefits [2]. Moreover, reports claim that software defects can be found earlier inthe development process using a MBT approach compared to the use of manual testingpractices [5]. The use of models to depict the behavior of a system is a proven and majoradvantage in software development [11] as well as in software testing [2]. Software modelsare now accepted as a part of modern object orientated analysis and design. Modeling is agood way of capturing knowledge about a system and then reusing this information as thesystem grows. The model may also be used as a means of communication between differentteams in an organization during development and the model may be reviewed by new teammembers to quickly come up to speed. For test teams, models provide a useful mechanism forstructured analysis of the system [2].1.2 GoalsThe general goal of this thesis is to evaluate the concept of MBT. This general goal includes apre-study of the concept as well as a feasibility study. The initial goals of the feasibility studyare to learn the MBT tool Qtronic and to implement a test object. The general goal of thefeasibility study is to apply and evaluate MBT, to document experiences and to makerecommendations for future work. To summarize, the purpose of the feasibility study is toprove the concept but also to produce guidelines for future reference, as if MBT were to beadopted at Tieto.2

The general goal of the thesis work was further divided into sub-goals.1. Learn the concepts of model-based testing2. Learn to use a tool (Qtronic) for modeling, test code generation and test execution3. Develop a framework for a finite state machine and implement a logic on theframework (with incrementally increased complexity) to be used as a test object.4. Develop a model for the test object in the test tool and try different criteria forgeneration of test code5. Establish the test environment including development of “glue” between the testcode and the test object.6. Execute the tests and incrementally make the test object and the model morecomplex.7. Evaluate the result, document the experience gained and make recommendations.The project goals listed above will be performed as they are listed, with the exception thatall goals except the initial goal, to study the theory behind the MBT concept, will beperformed in an iterative and incremental fashion.1.3 Feasibility studyThe general purpose of the feasibility study is to evaluate the concept of MBT as well as aspecific MBT tool, namely Qtronic. Specifically, the purpose is to develop a test object and totest this test object using Qtronic. The idea is that the test object will be developedincrementally while documenting experiences and results from the testing process. The testobject, and consequently the model, will be extended and modified for different experiments.Each experiment will focus on different aspects for evaluation. Since the test object will beincrementally developed the results and the experiences of the experiments will be compared.The test object is a part of a simplified model for an ATM client-server system, using anapplication layer protocol. The complete system specification includes the ATM (the client)and the central bank computer (the server), which communicates over a network. Both theclient and the server are divided into subsystems. The client consists of the user interface,where the user can withdraw money and request account balance information, and a protocolmodule. The server has a corresponding structure. The test object in the feasibility study islimited to the protocol module of the client, which consists of a finite state machine whichhandles incoming messages from two interfaces and responds with outgoing interfaces forboth interfaces. Thus the finite state machine describes the state of the protocol module andwhich outgoing message that follows an incoming message. The specifications for the finitestate machines are defined in Appendix C.1.Qtronic is the MBT tool that will be used. Qtronic automatically designs test cases given amodel of the system under test (SUT) as input. The generated test cases are black-box tests,which mean that they evaluate the SUT based only on the system’s external behavior. Thusthe models do not need to reflect the test object implementation structurally given that theydescribe the intended outwardly observable behavior [8]. The design models are expressed inthe Qtronic Modeling Language (QML), and its graphical notation, which is defined in aseparate tool, namely Qtronic Modeler. The generated test cases in Qtronic are abstract testcases that then are rendered to an executable format. In this thesis the test cases will berendered to TCL scripts.3

Qtronic renders the test cases to executable test scripts. Glue code, or a test harness, is thenimplemented using a test execution environment, i.e. a means of communicating with theSUT. The test harness is necessary to execute the test cases against the SUT.Feasibility studyExp 1Exp 2Exp 3Exp 4TimeInitial acted frommodel,implemented intest harnessFigure 1.2: Feasibility studyAs Figure 1.2 indicates, this thesis includes four experiments. The approach of the thesiswork is to incrementally develop the test object and the model for each experiment as if MBTwere deployed in a new project within the organization. Since the experiments are based onincremental development of the test object each experiment is analyzed individually but willalso be compared to the previous experiment, except for the initial experiment. The purposeof the initial experiment is to create a working model of the test object and to successfullyexecute the generated test cases. The purpose of the second experiment is to extend thespecification, thus also the test object and the model, to evaluate how added requirementspropagate through the process. The purpose of the third experiment is to change theauthentication requirements for the protocol module to see how changed requirementspropagate through the process. The specification of this experiment will require a successfulbiometric authentication to complete account requests against the central bank computer. Asecond goal of the third experiment is to model a reusable structure, i.e. encapsulate logic thatcan be reused in multiple state machines in the model. The purpose of the fourth experimentis to extract some logic from the model and implement this logic in the test harness, whileusing the same version of the specification and testing the same version of the SUT as for thethird experiment. For the specifications see Appendix C.1.1.4 Thesis OrganizationThis thesis consists of five chapters. In chapter 2 software testing is discussed in generalbefore introducing the concept of MBT. Software testing is discussed to place MBT in theperspective of traditional testing approaches and its testing scope. Chapter 3 includes adescription of the whole test system, an introductory example of the testing process in thisproject as well as experiment descriptions and corresponding results. The analysis and thediscussion of each experiment are provided in chapter 4. Chapter 4 also includes an analysisof this project and evaluations of Qtronic and Qtronic Modeler. In chapter 5 the conclusions4

from the project are described. The conclusions include the results, a discussion of the projectand suggestions for future work that may be performed in the context of this project area.5

2 Background2.1 IntroductionThis background chapter includes the pre-study of the project. The chapter starts with section2.2 by briefly discussing the process of software development and the roles of requirementsand specifications in the process. Section 2.3 is an introduction to software testing, includingcharacteristics and levels of testing as well as testing techniques. The concept of model-basedtesting (MBT) is introduced in section 2.5 and includes the different approaches, the processas well as benefits and limitations. The chapter ends with section 2.6 which briefly describesthe interfaces used in this project.2.2 Software DevelopmentAs an introduction to software testing some basic concepts of software development will bediscussed. These will be described to place software testing in an overall perspective of thesoftware development process as well as to introduce the requirements and basic ideas fortesting.2.2.1 ProcessA software process is a set of activities and associated results which lead to the production ofa software product. Such activities may include software development from scratch or byextending and modifying existing systems [33].Today there are a number of software development processes deployed in the industry,such as Scrum [32], eXtreme Programming [3] and Rational Unified Process [23]. However,there is no ideal process and different organizations have developed different approaches tosoftware development. Although there are many different software processes used, there arefundamental activities which are common to all software processes [33]:1. Software specification: The functionality of the software and constraints on itsoperation must be defined.2. Software design and implementation: The software to meet the specification mustbe produced.3. Software validation: The software produced must be validated to ensure that itfulfills its purpose.4. Software evolution: In many cases the software must evolve to meet the changingcustomer requirements.2.2.2 RequirementsSoftware engineers often have to solve complex problems. Understanding the nature of theproblems can be very difficult, especially if the system is new, and hence it is difficult toestablish exactly what the system should do. The descriptions of the services and

model-based testing tool Qtronic, developed by Conformiq, which automatically designs test cases given a model of the system under test as input. The experiments for the feasibility study were based on the incremental development of a test object, which was the client protocol module of a simplified model for an ATM (Automated Teller Machine .