WIXLIB

Software Testing:Models, Patterns, ToolsGuest Lecture, UIC CS 540November 16, 2010Robert V. Binder

Overview Test design pattern fly-byLevels of testingCase study: CBOE DirectQ&A

TEST DESIGN PATTERNS

Test Design Patterns Software testing, c. 1995– A large and fragmented body of knowledge– Few ideas about testing OO software Challenges––––Re-interpret wealth of knowledge for OOAddress unique OO considerationsSystematic presentationUniform analytical framework Patterns looked like a useful schema– Existing templates didn’t address unique testingissues4

Some Footprints1995 Design Patterns2003 Briand’s Experiments1995 Beizer, Black BoxTesting2003 Dot Net Test Objects1995 Firesmith PLOOT2003 Microsoft Patterns Group1995 McGregor2004 Java Testing Patterns1999 TOOSMPT2005 JUnit Anti Patterns2000 Tutorial Experiment2007 Test Object Anti Patterns2001 POST Workshops (4)2007 Software QS-TAG5

Test Design Patterns Pattern schema fortest design– Methods– Classes– Package and SystemIntegration– Regression– Test Automation– Oracles6

Test Design Patterns Pattern schema fortest designName/IntentContextTest ModelFault ModelTest ProcedureStrategyOracleEntry CriteriaAutomationExit CriteriaConsequencesKnown Uses7

Test Design Patterns Method Scope–––– Class/Cluster l FunctionRecursive FunctionPolymorphic Message 2000 Robert V. Binder, allrights reserved8Invariant BoundariesModal ClassQuasi-Modal ClassPolymorphic ServerModal Hierarchy

Modal Class:Implementation and Test ModelsTwoPlayerGameTwoPlayerGame TwoPlayerGame() p1 Start( ) p1 WinsVolley( )-p1 AddPoint( ) p1 IsWinner( ) p1 IsServer( ) p1 Points( ) p2 Start( ) p2 WinsVolley( )-p2 AddPoint( ) p2 IsWinner( ) p2 IsServer( ) p2 Points( ) ( )Two Play erG am e ( )αp1 S tart ( ) /s im u lat eV olle y( )p1 W ins V olle y ( )[th is .p 1 Sc ore ( ) 20 ] /th is .p 1A ddP oin t( )s im u lat eV olle y( )G am e S tarte dp2 S tart ( ) /s im u lat eV olle y( )αp1 Start( ) /simulateVolley( )p2 W ins V olle y ( )[th is .p 2 Sc ore ( ) 20 ] /th is .p 2A ddP oin t( )s im u lat eV olle y( )p1 W ins V olle y ( ) /s im u lat eV olle y( )P la ye r 1S erv edP la ye r 2S erv edp1 WinsVolley( ) /simulateVolley( )p2 W ins V olle y ( ) /s im u lat eV olle y( )p1 W ins V olle y ( )[th is .p 1 Sc ore ( ) 20] /th is .p 1A ddP oin t( )p2 W ins V olle y ( )[th is .p 2 Sc ore ( ) 20] /th is .p 1A ddP oin t( )P la ye r 1Wonp1 Is W in ner( ) /retu rn TR UE ;P la ye r 2Won ( )ωThreePlayerGame( ) /TwoPlayerGame( )p2 Is W in ner( ) /retu rn TR UE ;Game Startedp2 Start( ) /simulateVolley( )p2 WinsVolley( )[this.p2 Score( ) 20] /this.p2AddPoint( )simulateVolley( )p1 WinsVolley( )[this.p1 Score( ) 20] /this.p1AddPoint( )simulateVolley( )p1 WinsVolley( ) /simulateVolley( ) ( )Player 2ServedPlayer 1Servedp1 WinsVolley( )[this.p1 Score( ) 20] /this.p1AddPoint( )αTh ree P la y erG a m e ( ) / Two P la y erG am e ( )ThreePlayerGame ThreePlayerGame() p3 Start( ) p3 WinsVolley( )-p3 AddPoint( ) p3 IsWinner( ) p3 IsServer( ) p3 Points( ) ( )G a m e S ta rt edp3 WinsVolley( ) /simulateVolley( )p 3 S tart ( ) /s im ulat eV o lley ( )p 3 W ins V o lle y( ) /s im ulat eV o lley ( )p 3 W ins V o lle y( )[t his .p 3 S co re ( ) 2 0] /th is . p3 A dd P oint ( )s im ulat eV o lley ( )Tw oP lay erG am e ( )p 1 W ins V o lle y( ) /s im ulat eV o lley ( )P la y er 3S erv e dp 2 W ins V o lle y( ) /s im ulat eV o lley ( )p 3 W ins V o lle y( )[t his .p 3 S co re ( ) 2 0] /th is . p3 A dd P oint ( )P la y er 3W onωp1 IsWinner( ) /return TRUE;Player 1Wonp2 IsWinner( ) /return TRUE;p2 WinsVolley( ) /simulateVolley( )p3 WinsVolley( )[this.p3 Score( ) 20] /this.p3AddPoint( )simulateVolley( )Player 3Servedp3 WinsVolley( ) /simulateVolley( )p2 WinsVolley( ) /simulateVolley( )ThreePlayerGamep3 Start( )/simulateVolley( )p3 WinsVolley( )[this.p3 Score( ) 20] /this.p3AddPoint( )p2 WinsVolley( )[this.p2 Score( ) 20] /this.p1AddPoint( )Player 2WonPlayer 3Wonp3 IsWinner( ) /return TRUE; ( ) ( )ω ( )p 3 Is W in ne r( ) /ret urn TR UE ; ( )9

Test Plan and Test Size K events N states With LSIFs12345678910ThreePlayerGame( )p1 Start( )p2 Start( )p3 Start( )p1 WinsVolley( )p1 WinsVolley( )[this.p1 Score( ) 20]p1 WinsVolley( ) [this.p1 Score( ) 20]p2 WinsVolley( )p2 WinsVolley( ) [this.p2 Score( ) 20]p2 WinsVolley( ) [this.p2 Score( ) 20]811Player 1 Served*7Player 2 ServedPlayer 3 Served17Player 1 W onomega14Player 1 W on*6Player 1 Served*9Player 2 Served11Player 3 Served2– KN tests1alpha3Gam eStartedPlayer 2 Served17* 10Player 2 W onomega15Player 2 W on5 No LSIFs– K N3Player 1 Served4* 12testsPlayer 3 Served1711121314151617p3 WinsVolley( )p3 WinsVolley( ) [this.p3 Score( ) 20]p3 WinsVolley( ) [this.p3 Score( ) 20]p1 IsWinner( )p2 IsWinner( )p3 IsWinner( ) ( )* 13Player 3 ServedPlayer 3 W onomega16Player 3 W on8Player 2 Served5Player 1 Served10

Test Design Patterns Subsystem Scope–––– Reusable Components––––Class AssociationsRound-Trip ScenariosMode MachineControlled Exceptions 2000 Robert V. Binder, allrights reserved11Abstract ClassGeneric ClassNew FrameworkPopular Framework

Test Design Patterns Intra-class Integration Integration Strategy– Small Pop– Alpha-Omega Cycle 2000 Robert V. Binder, allrights reserved–––––––––12Big BangBottom upTop ibuted ServicesHigh Frequency

Test Design Patterns System Scope Regression Testing– Extended Use Cases– Covered in CRUD– Allocate by Profile 2000 Robert V. Binder, allrights reserved–––––13Retest AllRetest Risky Use CasesRetest ProfileRetest Changed CodeRetest Within Firewall

Test Oracle Patterns Smoke Test Judging – Testing By Poking Around– Code-Based Testing– Post Test Analysis Pre-Production Built-in Test Gold Standard––––Custom Test SuiteRandom Input GenerationLive InputParallel System 2000 Robert V. Binder, allrights onVotingSubstitutionEquivalency

Test Automation Patterns Test Case Implementation Test Drivers– Test Case/Test SuiteMethod– Test Case /Test Suite Class– Catch All Exceptions– TestDriver Super Class– Percolate the Object UnderTest– Symmetric Driver– Subclass Driver– Private Access Driver– Test Control Interface– Drone– Built-in Test Driver Test Control– Server Stub– Server Proxy 2000 Robert V. Binder, allrights reserved15

Test Automation Patterns Test Execution Built-in Test– Command Line TestBundle– Incremental TestingFramework (e.g. Junit)– Fresh Objects 2000 Robert V. Binder, allrights reserved– Coherence idiom– Percolation– Built-in Test Driver16

Percolation Pattern Enforces Liskov Subsitutability Implement with No Code Left BehindBase Base() arPre()barPost()Derived1 Derived1() d2 Derived2() )fooPost()barPre()barPost()feePre()feePost()17

Ten Years After Many new design patterns for hand-crafted testautomation– Elaboration of Incremental Test Framework (e.g. JUnit)– Platform-specific or application-specific– Narrow scope 18Few new test design patternsNo new oracle patternsAttempts to generate tests from design patternsTo date 10,000 copies of TOOSMPT

What Have We Learned? TP are effective for articulation of insight andpractice– Requires discipline to develop– Supports research and tool implementation Do not “work out of the box”– Requires discipline in application– Enabling factors Irrelevant to the uninterested, undisciplined– Low incremental benefit– Readily available substitutes Broadly influential, but not compelling19

TEST AUTOMATION LEVELS

What is good testing? Value creation (not technical merit)– Effectiveness (reliability/quality increase)– Efficiency (average cost per test) Levels– 1: Testing by poking around– 2: Manual Testing– 3: Automated Test Script/Test Objects– 4: Model-based– 5: Full Test AutomationEach Level 10x Improvement21 2004 mVerify Corporation

Level 1: Testing by Poking AroundManual“Exploratory”Testing Low Coverage Not Repeatable Can’t Scale Inconsistent 2004 mVerify CorporationSystem Under Test22

Level 2: Manual TestingTest SetupManualTest Design/Generation 1 test per hour Not repeatable 2004 mVerify CorporationManualTest InputTest Results System Under TestEvaluation23

Level 3: Automated Test ScriptTest SetupManualTest Design/GenerationTest ScriptProgramming 10 tests per hour Repeatable High change cost 2004 mVerify CorporationTest Results System Under TestEvaluation24

Level 4: Automated Model-basedTest SetupModel-basedTest Design/GenerationAutomaticTestExecution 1000 tests per hour High fidelityTest Results System Under TestEvaluation 2004 mVerify Corporation25

Level 5: Total AutomationAutomatedTest SetupModel-basedTest Design/Generation 10,000 TPH 2004 mVerify CorporationAutomaticTestExecutionAutomatedTest ResultsEvaluation26System Under Test

MODEL-BASED TESTING OFCBOE DIRECT

CBOE Direct Electronic technology platform built and maintained in-house byChicago Board Options Exchange (CBOE)– Multiple trading models configurable by product– Multiple matching algorithms (options, futures, stocks, warrants,single stock futures)– Best features of screen-based trading and floor-based markets Electronic trading on CBOE, the CBOE Futures Exchange (CFX), andthe CBOE Stock Exchange (CBSX), others As of April 2008:––––––More than 188,000 listed productsMore than 3.8 billion industry quotes handled from OPRA on peak dayMore than two billion quotes on peak dayMore than 684,000 orders on peak dayMore than 124,000 peak quotes per secondLess than 5 ms response time for quotes

Development Rational Unified process– Six development increments– 3 to 5 months– Test design/implementation parallel with app dev Three years, version 1.0 live Q4 2001About 90 use-cases, 650 KLOC JavaCORBA/IDL distributed objectsJava (services and GUI), some XMLOracle DBMSHA Sun server farmMany legacy interfaces 2004 mVerify Corporation29

Test Models Used Extended Use Case– Defines feature usage profile– Input conditions, output actions Mode Machine– Use case sequencing Invariant BoundariesStealth Requirements Engineering 2004 mVerify Corporation30

Behavior Model Extended Use Case pattern12345ConditionsTest InputConditions forautomatic testinput generationVariable/ObjectWidget 1Widget 2Widget 3Host Name PickHost Name EnterValue/StateQuerySet TimeDELValidHost NameTTTTTFTLogiccombinationsTF controlDC test inputdata selectionTTTTActionsRequired Actionsfor automaticresult checkingVariable/Interface Value/ResultHost Name Display No ChangeDeletedAddedHost Time Display No ChangeHost TimeCE Time DisplayError MessageTTLast Local Time THost TimeFRelative Frequency 2004 mVerify ols statisticaldistributionof testTcasesF0.100.05

Load Model Vary input rate, any quantifiable pattern–––––––––ArcFlatInternet FractalNegative rampPositive rampRandomSpikesSquare waveWavesActual “Waves” Load Profile 2004 mVerify Corporation32

MBT Challenges/Solutions One time sample noteffective, but fresh testsuites too expense Simulator generates fresh,accurate sample on demand Too expensive to developexpected results Oracle generatesexpected on demand Too many test cases toevaluate Comparator automateschecking Profile/Requirementschange Incremental changes torule base SUT Interfaces change Common agentinterface

Simulator Discrete event simulation of user behavior 25 KLOC, Prolog– Rule inversion– “Speaks” Load Profile– Time domain variation– Orthogonal to operational profile Each event assigned a "port" and submit time 2004 mVerify Corporation34

Test Environment Simulator, etc. on typical desktop Dedicated, but reduced server farm Live data links 10 client workstations for automatic test agents– Adapter, each System Under Test (SUT) Interface– Test Agents execute independently Distributed processing/serialization challenges– Loosely coupled, best-effort strategy– Embed sever-side serialization monitor 2004 mVerify Corporation35