Transcription
MTAT.03.159: Software TestingLecture 07:Tools, Metrics and TestProcess Improvement / TMMi(Textbook Ch. 14, 9, 16)Spring 2013MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Dietmar Pfahlemail: dietmar.pfahl@ut.ee
Structure of Lecture 07 Test ToolsTest MeasurementTest Process ImprovementSWT ExamMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Tools – the Workbench Good at repeatingtasks Good at organisingdata Requires training Introducedincrementally No “silver bullet”MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Evaluation criteria Ease of usePowerRobustnessFunctionalityEase of insertionQuality of supportCostCompany policies and goals
Test Tools – in the tiontoolsandcomparisonSystem gnTest designtoolsDetaileddesignStatic analysis toolsIntegrationtestCodeTest management toolsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Unit testCoverage toolsDynamicanalysis toolsDebuggingtools
TMM level 5Test Tools– by Test MaturityTMM level 3TMM level 1TMM level 4Code checkersCode comprehension toolsTest harness generatorsPerform./network analyzersSimulators/emulatorsTest management toolsConfiguration managementRequirements toolsCapture-replay toolsComparatorDefect trackerComplexity measurerTMM level 2 Load generatorsTest/project plannersRun-time error checkersTest preparation toolsCoverage analyzersCross-reference toolsDebuggersConfiguration buildersLOC countersMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Test library toolsAdvanced
There is no shortage of Test Tools Defect Tracking (98)GUI Test Drivers (71)Load and Performance (52)Static Analysis (38)Test Coverage (22)Test Design Tools (24)Test Drivers (17)Test Implementation (35) assist with testing at runtime - memoryleak checkers, comparators, and a widevariety of othersTest case Management (24)Unit Test Tools (63)3 different categories of othersFrom http://www.testingfaqs.org/MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Other links to test tool overviews: twareqatest.com/qatweb1.html
Test data generator – Generatedata.comonline service for generating dataMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Output of the previous input fieldMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Evolution of System Testing approaches1. Recorded Scripts2. Engineered Scripts3. Data-driven 72454. Keyword-driven Testing5. Model-based TestingMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Recorded Scripts Unstructured Scripts generated using capture andreplay tools Relatively quick to set up Mostly used for regression testing Scripts non-maintainable, in practice– If the system changes they need to becaptured again Capture Replay Tools– Record user’s actions to a script(keyboard, mouse) – Scripts access the (user) interfaceof the software Input fields, buttons and otherwidgets– Simple checks can be created inthe scripts MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Tool specific scripting languageExistence of texts and objects inthe UIData of GUI objects
Engineered Scripts Scripts are well-designed (following a systematic approach),modular, robust, documented, and maintainable Separation of common tasks– E.g. setup, cleanup/teardown, and defect detection Test data is still embedded into the scripts– One driver script per test case Code is mostly written manually Implementation and maintenance require programming skillswhich testers (test engineers) might not have “Just like any other software development project”MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Data-Driven 7245 Test inputs and expected outcomes stored as data– Normally in a tabular format– Test data are read from an external data source One driver script can execute all of the designed test cases External test data can be edited without programming skills– Test design and framework implementation are now separate tasks– former can be given to someone with the domain knowledge(business people, customers) and latter to someone withprogramming skills. Avoids the problems of embedded test data– Data are hard to understand in the middle of all scripting details Updating tests or creating similar tests with slightly differenttest data always requires programming– Leads to copy-paste scriptingMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Data-Driven TestingMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Keyword-Driven Testing Keywords also known as action words Keyword-driven testing improves data-driven testing:– Keywords abstract the navigation and actions from the script– Keywords and test data are read from an external data source When test cases are executed keywords are interpreted by a test librarywhich is called by a test automation framework The test library the test scripts Example:Login: admin, t5t56y;AddCustomers: newCustomers.txtRemoveCustomer: Pekka Pukaro More keywords ( action words) can be defined based on existing keywords Keyword driven testing domain specific languages (DSL) Details: keyword.driven.testing.html Another tool: 9 / Lecture 07 / Dietmar Pfahl 2013
Architecture of a Keyword-DrivenFramework Pekka Laukkanen. Data-Driven and Keyword-Driven Test AutomationFrameworks. Master’s Thesis. Helsinki University of Technology. 2006.MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Model-based TestingMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Model-based Testing System under test is modelled– UML-state machines, domain specific languages (DSL) Test cases are automatically generated from the model– The model can provide also the expected results for the generatedtest cases– More accurate model - better test cases Generate a large amount of tests that cover the model– Many different criteria for covering the model– Execution time of test cases might be a factor Challenges:– Personnel competencies– Data-intensive systems (cannot be modelled as a state-machine) Simple MBT tool http://graphwalker.org/MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Evolution of System Testing approaches1. Recorded Scripts– Cheap to set up, quick & dirty2. Engineered Scripts– Structured3. Data-driven Testing– Data separation4. Keyword-driven Testing– Action separation, DSL5. Model-based Testing– Modeling & Automatictest case generationMTAT.03.159 / Lecture 07 / Dietmar Pfahl 5
Automation and Oracles Automated testing depends on the ability to detectautomatically (via a program) when the software fails An automated test is not equivalent to a similar manual test– Automatic comparison is typically more precise– Automatic comparison will be tripped by irrelevantdiscrepancies– The skilled human comparison will sample a wider rangeof dimensions, noting oddities that one wouldn't programthe computer to detect “Our ability to automate testing is fundamentally constrainedby our ability to create and use oracles” (Cem Kaner)MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Types of outcome to compare Screen-based– Character-basedapplications– GUI applications Correct message, displayattributes, displayedcorrectly GUI components and theirattributes– Graphical images Avoid bitmap comparisonsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013 Disk-based– Comparing text files– Comparing non-textualforms of data– Comparing databases andbinary files Others– Multimedia applications Sounds, video clips,animated pictures– Communicating applications Simple vs. complex comparison
Test case sensitivity in comparisonsRobust testsSensitive testsSusceptibility to changeImplementation effortMiss defectsFailure analysis effortSensitive test case compares manyelements and is likely to notice thatsomething breaks. However, it is alsomore sensitive to change and causesrework in test automation.A robust test checks less and is morechange-resilient, but also missespotential defects. Striking a balance isthe challenge.Storage space Redrawn from Fewster and Graham, Software Test Automation, 1999.MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Effect of automation on ”goodness” of a test caseEffectiveAutomated testafter many runsManualtestEconomicEvolvability (maintainability)of automated test does notchange, but economicsincreases the sizes and the“goodness” of the test caseEvolvableFirst run ofautomated testsExemplaryMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013 Redrawn from Fewster and GrahamSoftware Test Automation, 1999.
Scope: Automating different stepsAutomated testsAutomated testingSelect/Identify test cases to runSelect/Identify test cases to runSet up test environment- create test environment- load test dataSet up test environment:- create test environment- load test dataRepeat for each test case:- set up test prerequisitesRepeat for each test case:- set up test prerequisites- execute-compare results- log results-clear up after test case- execute- compare results- log results-analyze test failures-report defect(s)- clear up after test caseClear up test environment:- delete unwanted data- save important dataSummarize resultsClear up test environment:- delete unwanted data-save important dataSummarize resultsManual processAnalyze test failuresReport defectsAutomated process Redrawn from Fewster et al. Software Test Automation, 1999.MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Relationship of testing activities Edit tests(maintenance) Set up Execute Analyze failures Clear up Manualtesting Same testsautomated More matureautomation Redrawn from Fewster et al.Software Test Automation, 1999. TimeMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Test automation promises1.2.3.4.5.6.7.8.Efficient regression testRun tests more oftenPerform difficult tests (e.g. load, outcome check)Better use of resourcesConsistency and repeatabilityReuse of testsEarlier time to marketIncreased confidenceMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Common problems1. Unrealistic expectations2. Poor testing practice”Automatic chaos just gives faster chaos”3. Expected effectiveness4. False sense of security5. Maintenance of automatic tests6. Technical problems (e.g. Interoperability)7. Organizational issuesMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
What can be automated?1. Identify2. Design3. Build4. Execute5. CheckMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013IntellectualPerformedonceRepeatedClerical
Limits of automated testing Does not replace manual testing Manual tests find more defects than automated tests– Does not improve effectiveness Greater reliance on quality of tests– Oracle problem Test automation may limit the software development– Costs of maintaining automated testsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
What to automate first? Most important tests A set of breadth tests (sample each system areaoverall) Test for the most important functions Tests that are easiest to automate Tests that will give the quickest payback Test that are run the most oftenMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Structure of Lecture 07 Test ToolsTest MeasurementTest Process ImprovementSWT ExamMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Test Management Monitoring (or tracking)– Check status– Reports– Metrics Controlling– Corrective actionsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Purpose of Measurement Test monitoring – check the statusTest controlling – corrective actionsPlan new testingMeasure and analyze results–––––The benefit/profit of testingThe cost of testingThe quality of testingThe quality of the productBasis of improvement, not only for the test processMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Cost of Testing How much does testing cost? As much as resources we have!MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Ericcssonmobile phonesdid not do toowell becausedid too much oftesting instad ofgetting theproduct tomarket
Test Monitoring Status– Coverage metrics– Test case metrics: development and execution– Test harness development Efficiency / Cost metricsMetricsEstimation– How much time have we spent?– How much money/effort have we spent?Cost Failure / Fault metrics– How much have we accomplished?– What is the quality status of the software? Effectiveness metricsStop?– How effective is the testing techniques in detecting defects?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Selecting the right metrics What is the purpose of the collected data?– What kinds of questions can they answer?– Who will use the data?– How is the data used? When and who needs the data?––––Which forms and tools are used to collect the data?Who will collect them?Who will analyse the data?Who have access to the data?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Goal-Question-Metric Paradigm (GQM) Goals– What is the organization tryingto achieve?– The objective of processimprovement is to satisfy thesegoals Questions– Questions about areas ofuncertainty related to the goals– You need process knowledge toderive the questions Metrics– Measurements to be collectedto answer the questions[van Solingen, Berghout, The Goal/Question/Metric Method,McGraw-Hill, 1999]MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Goal example: Analyze object(s) of study – the detection of design faults usinginspection and testing for the purpose of purpose – evaluation with respect to their quality focus – effectiveness and efficiency from the point of view of the perspective – managers in the context of context – developers, and in a real applicationdomain
Measurement BasicsBasic data: Time and Effort(calendar- and staff-hours) Failures / Faults Size / FunctionalityMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Basic rule: Feedback to origin Use data or don’tmeasure
Test metrics: CoverageWhat?% statements covered% branches covered% data flow% requirements% equivalence classesMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Why? Trackcompleteness of test
Test metrics: Development status Test case development status– Planned– Available– Unplanned (not planned for, butneeded) Test harness developmentstatus– Planned– Available– UnplannedMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Test metrics: Test execution statusWhat? # faults/hour # executed tests RequirementscoverageMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Why? Track progress of testproject Decide stoppingcriteria
Test metrics: Size/complexity/lengthWhat?Why? Size/Length – LOC Estimate test effort Functionality –Function Points Complexity – McCabe Difficulty – Halstead Cohesion, Coupling,.MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Test metrics: EfficiencyWhat? # faults/hour # faults/test caseMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Why? Evaluate efficiency ofV&V activities
Test metrics: Faults/Failures(Trouble reports)What? # faults/size repair time root causeMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Why? Monitor quality Monitor efficiency Improve
Test metrics: EffectivenessWhat?% found faults perphase% missed faultsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Why? Evaluate effectivenessof V&V activities
How good are we at testing?Test qualityMany faultsAre we here?Few faultsProduct qualityFew faultsOr are we here?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Few faults
When to stop testing? All planned tests are executed and passedAll coverage goals met (requirements, code, .)Detection of specific number of failuresRates of failure detection fallen below a specifiedlevel Fault seeding ratios are favourable Reliability above a certain value Cost has reached the limitMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
ExampleNumber ofdetectedfailuresNumber ofexecutedtest casesNumber of failures per dayInterpretation?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
ExampleNumber ofdetectedfailuresNumber ofexecutedtest casesInterpretation?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
ExampleNumber ofdetectedfailuresNumber ofexecutedtest casesInterpretation?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Structure of Lecture 07 Test ToolsTest MeasurementTest Process ImprovementSWT ExamMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Process quality and product quality Quality in process - Quality in product Project: instantiatedprocess Quality according toISO 9126– Process quality contributes toimproving product quality,which in turn contributes toimproving quality in useMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013ProcessProjectProduct
PrinciplesAssessTestorganisationMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013ImproveMaturityModel
MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Process improvement models (Integrated) Capability maturity model (CMM, CMMI) Software process improvement and capability determination(SPICE) ISO 9001, Bootstrap, Test maturity model (TMM)Test process improvement model (TPI)Test improvement model (TIM)Minimal Test Practice Framework (MTPF) MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
rocess change managementTechnology change managementDefect preventionManagedSoftware quality managementQuantitative process managementDefinedPeer reviewsIntergroup coordinationSoftware product engineeringIntegrated software managementTraining programmeOrganization process definitionOrganization process focusRepeatableSoftware configuration managementSoftware quality assuranceSoftware subcontract managementSoftware project tracking and oversightSoftware project planningRequirements managementInitialMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Test Maturity Model (TMM) Levels Maturity goals and sub-goals– Scope, boundaries, accomplishments– Activities, tasks, responsibilities Assessment model– Maturity goals– Assessment guidelines– Assessment procedureMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Level 2: Phase Definition Institutionalize basic testing techniques andmethods Initiate a test planning process Develop testing and debugging toolsMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Level 3: Integration Control and monitor the testingprocess Integrate testing into software lifecycle Establish a technical trainingprogram Establish a software testorganizationMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Level 4: Management and Measurement Software quality evaluation Establish a test managementprogram Establish an organization-widereview programMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Level 5: Optimizing, Defect Prevention,and Quality Control Test process optimization Quality control Application of process data fordefect preventionMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Can the organization be too ’mature’?MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Clausewitz: Armor and mobilityalternate dominance (DeMarco)GreeksRomansVandals, HunsFranksMongolsCastlesField ArtilleryMaginot LineTanksMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Birth of the castle (CMMI) and the tiger (Agile)U.S Department of defenseScientific managementStatistical process controlManagementControlLarge team & low skillMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Leading industry consultantsTeam creates own processWorking softwareSoftware craftsmanshipProductivitySmall team & high skill
Plan-driven vs. Agile (Boehm & Turner,2003, IEEE Computer, 36(6), pp 64-69) 103MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Software quality assurance comparison:castle vs. tigerOrganisationIndependent QA teamIntegrated into the project teamEnsuringCompliance to documented processesApplicability and improvement ofthe current processes and practicesEvaluation CriteriaAgainst predefined criteriaIdentifying issues and problemsFocusDocuments & processes & controlProductivity & quality & customerCommunicationFormal: Reporting to managementMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013Informal: Supporting the team
General advice Identify the real problems before starting animprovement program “What the customer wants is not always what itneeds” Implement “easy changes” first Involve people Changes take time!MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
RecommendedTextbook Exercises Chapter 14– 2, 4, 5, 6, 9 Chapter 9– 2, 3, 4, 5, 8, 12 Chapter 16– No exercisesMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Structure of Lecture 07 Test ToolsTest MeasurementTest Process ImprovementSWT ExamMTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Final Exam Written exam (40%)– Based on textbook, lectures and lab sessions– Open book– 90 min Dates:– Exam 1: 30-May-2013 10:15-11:45 (J. Liivi 2-405)– Exam 2: 10-June-2013 14:15-15:45 (J. Liivi 2-403)MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Thank You!MTAT.03.159 / Lecture 07 / Dietmar Pfahl 2013
Keyword-Driven Testing Keywords also known as action words Keyword-driven testing improves data-driven testing: - Keywords abstract the navigation and actions from the script - Keywords and test data are read from an external data source When test cases are executed keywords are interpreted by a test library
