Transcription
Agile Test Automation StrategyFor Anyone and Everyone!Gerard MeszarosAgile2012ATAS@gerardm.comMuch Ado About Agile 20111Copyright 2011 Gerard Meszaros
My Background Software developer80’s Development manager----- Project Manager Software architect90’s OOA/OOD Mentor----00’sEmbeddedTelecom Requirements (Use Case) Mentor XP/TDD MentorI.T. Agile PM Mentor Test Automation Consultant & Trainer Lean/Agile Coach/ConsultantProduct & I.T.Much Ado About Agile 20112Gerard MeszarosATAS2012@gerardm.comCopyright 2011 Gerard Meszaros
Agenda Motivation– The Agile Test Problem– The Fragile Test Problem Approaches to Test Automation Test Automation StrategyRough timings for Agile Test Automation StrategyMuch Ado About Agile 20113Slide ##TopicTime Slides Start EndMotivation11.2829Exercise 1 - Automation Motivation101 10 10Intro to Automation75 11 15Exercise 2 - Why not Record & Playback?101 16 16Why Automated Tests are Fragile8.46 17 22How Agile Automation Changes Things9.87 24 30Intro to Example-Driven Development75 32 36Managing Scope vs Detail in Examples15.411 38 48How to specify workflows8.46 50 55Exercise 3 - Workflow Tests (Keyword-Driven)151 56 56Using Data-Driven Tests to specify business rules8.46 55 60Exercise 4 - Business Rules Test (Data-Driven)151 61 61How Tests Interact With the SUT75 62 66Test-Driven Architecture5.64 67 70Legacy Systems (if time permits)19.614 71 84The Role of Unit Tests8.46 85 90Test Automation Strategy1410 91 100180.297Copyright 2011 Gerard MeszarosTime per slide:1.4 # of
Product Owner Goal Goal: Maximize business value t OwnerQuality is Assumed; Not ManagedMuch Ado About Agile 20114Copyright 2011 Gerard Meszaros
Why Quality Often Sucks Iron Triangle of Software Engineering:ResourcesQualityTimeFunctionality What about Quality?You can fix any three; the fourth is the outcomeMuch Ado About Agile 20115Copyright 2011 Gerard Meszaros
Why Quality Often Sucks Iron Triangle of Software Engineering:ResourcesQualityTimeIn Agile, we“Pin” qualityusingautomatedtestsFunctionality What about Quality?You can fix any three; the fourth is the outcomeMuch Ado About Agile 20116Copyright 2011 Gerard Meszaros
Speaking of Quality,would you . ask your doctor to reduce the costof the operation . by skipping the sterile technique ?Test Automation is like hand washing:Improved results but an upfront cost.Much Ado About Agile 20117Copyright 2011 Gerard Meszaros
Minimizing Cost of ProductTotal cost includes: developing the software verifying the newly built functionality verifying old functionality still works fixing any bugs found Verifying noting was broken by fixesAgile Test Automation can reduce thecost of all of these activities.Much Ado About Agile 20118Copyright 2011 Gerard Meszaros
Incremental DevelopmentInitialConceptNFVersion 1.0EvolvedConceptNFVersion 1.x Type NF bugs: New Func. is wrong Type RB bugs: New bugs in old func.(Regression Bugs)Much Ado About Agile 20119Version 1.0RBCopyright 2011 Gerard Meszaros
Exercise 1 Time to test our little application Oh, new build, please retest! Another build, please retest!Much Ado About Agile 201110Copyright 2011 Gerard Meszaros
The Agile Test ProblemRequirementsMuch Ado About Agile 2011Development11TestingCopyright 2011 Gerard Meszaros
The Agile Test evelopmentTesting As development increments reduce induration, testing needs to be reducedaccordinglyMuch Ado About Agile 201112Copyright 2011 Gerard Meszaros
The Agile Test uirementsDevelopmentTesting. and traditional approaches to testing nolonger workMuch Ado About Agile 201113Copyright 2011 Gerard Meszaros
Anatomy of an Automated TestPreconditions(State)Other SystemGiven.TestSetup1&2 May berepeatedTestTeardownTest Scenario Name-----------------------Our SystemPreconditions---------------------1. Do SomethingTest2. Check Something----------------------Clean UpInterfaceBusinessLogicDatabaseContainer ServicesOther SystemWhen .Then .Much Ado About Agile 2011Other SystemAdapter14Copyright 2011 Gerard Meszaros
(C)OTS Record&Playback User executes tests manually; tool records as tests Tool replays tests later without user interventionThe tests areare code/datainterpretedby the tsExpectedOutputsTest Script RepositoryTestScript 1TestScript 2TestScript nTestResultsScript nResultTest Result RepositoryMuch Ado About Agile 201115Copyright 2011 Gerard Meszaros
Exercise 2 Record & Playback Test Automation– Please record a test against the System Under Test– Then, run the test to make sure it works New build has been delivered– Please run the test against new buildMuch Ado About Agile 201116Copyright 2011 Gerard Meszaros
Agenda Motivation– The Agile Test Problem– The Fragile Test Problem Changing the Role of Test Automation Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201117Copyright 2011 Gerard Meszaros
The Fragile Test ProblemWhat, when changed,may break our testsaccidentally:– Behavior SensitivityOther SystemOther SystemOur System» Business logic– Interface Sensitivity» User or system– Data SensitivityInterfaceTest» Database contentsBusinessLogicDatabaseContainer Services– Context Sensitivity» Other system stateOther SystemIn Agile, these are all changing all the time!Much Ado About Agile 201118Copyright 2011 Gerard Meszaros
Interface Sensitivity Tests must interact withthe SUT through someinterface Any changes to interfacemay cause tests to fail.Other SystemOther SystemWindowOur System– User Interfaces:Title» Renamed/deleted windows ormessages» New/renamed/deleted fields» New/renamed/deleted data valuesin listsCaption FieldInterfaceLink ButtonData GridWindow– Machine-Machine Interfaces:» Renamed/deleted functions in API» Renamed/deleted messages» New/changed/deleted functionparameters or message fieldsMuch Ado About Agile 2011BusinessLogicDatabaseContainer ServicesOther SystemE.g.: Move tax fieldto new popup window19Copyright 2011 Gerard Meszaros
Behavior Sensitivity Tests must verify thebehavior of the system.Other System– Behavior also involved in testset up & tear down Any changes to businesslogic may cause tests tofail.– New/renamed/deleted states– New/changed/removedbusiness rules– Changes to businessalgorithms– Additional data requirementsMuch Ado About Agile 201120Other SystemOur orithmRuleDatabaseObjectContainerServicesOther SystemE.g.: Change fromGST PST to HSTCopyright 2011 Gerard Meszaros
Data Sensitivity All tests depend on“test data” which are:– Preconditions of test– Often stored in databases– May be in other systems Changing the contentsof the database maycause tests to fail.– Added/changed/deletedrecords– Changed SchemaOther SystemOther SystemOur Container ServicesOther SystemE.g.: Change customer’sbilling termsMuch Ado About Agile 201121Copyright 2011 Gerard Meszaros
Context Sensitivity Tests may depend on inputsfrom another system– State stored outside theapplication being tested– Logic which may changeindependently of our systemOther SystemCustomerXOtherSystemOur System Changing the state of thecontext may cause tests tofail.– State of the containerInterfaceBusinessLogicContainer ServicesContainer ServicesDate» e.g. time/date– State of related systemsDatabaseTimeSecuritySystemOtherSystem» Availability, data contentsUser: XPermissions: noneE.g.: Run test in ashorter/longer monthMuch Ado About Agile 201122Copyright 2011 Gerard Meszaros
Agenda Motivation Changing the Role of Test Automation– From Defect Detection to Defect Prevention– Different Tests for Different Purposes Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201123Copyright 2011 Gerard Meszaros
The Role of Automation in Agile Provide a Safety Net for Change & Innovation– Provide rapid feedback to reduce cost of fixing defects.» On demand (Developer) and event driven (CI build)– Rapid feedback enables experimentation» Don’t have to choose between Quick and Safe Guide Development of the Product– Provide executable examples of what “done” looks like Support Manual Testing– Remove the repetitive drudgery so testers can focus onhigh value activity by:– Automating entire tests, or by– automating the steps that can be automated.Much Ado About Agile 201124Copyright 2011 Gerard Meszaros
How is Agile Test Automation Different? We automate the tests for a different reason– Defect Prevention vs. Detection– To communicate requirements– To “Pin” the functionality once it’s built We automate the tests a different way– Many different kinds of tests» E.g. We don’t rely solely on GUI-based automation– Using tools that support collaboration & communication» in addition to confirmation We plan the automation based on ROI– Goal isn’t: 100% automation– Goal is: To maximize benefit while minimizing costMuch Ado About Agile 201125Copyright 2011 Gerard Meszaros
Traditional Role of TestingCritique ProductBusinessFacingAcceptance TestsRegression TestsTechnologyUnit TestsFacingComponent TestsInspectionto finddefects isWasteShigeo ShingoCo-inventor ofToyota Production SystemMuch Ado About Agile 2011Usability TestsExploratory TestsReportCardFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)Inspectionto preventdefects isessential26 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Changing the Role of TestingRequirementsBusinessFacingCritique ProductDefine ProductAcceptance TestsRegression TestsUnit TestsTechnologyComponent TestsFacingSoftwareDesignMuch Ado About Agile 2011Preventanticipatabledefects fromhappeningReportCardUsability TestsExploratory TestsFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)Find nonanticipatableDefects, ASAP!27 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Changing the Role of TestingRequirementsBusinessFacingDefine ProductAcceptance TestsRegression TestsUnit TestsTechnologyComponent TestsFacingSoftwareDesignMuch Ado About Agile 2011Critique ProductUsability TestsExploratory TestsReportCardFunctionality BUsabilityCScalability AResponseBAvailability CProperty Tests(Response Time,Security, Scalability)For effective prevention:1. Tests must be available beforedevelopment2. Developers must be able to run testsbefore check-in28 Marrick and Mary PoppendieckCopyright 2011 Gerard MeszarosQuadrants courtesy of Brian
Reducing the Cost to Fix DefectsCost to understandand fix a defect goesup with the time ittakes to discover it. Why? We can remember where we put the newlyinserted defect because1. We know what code we were working on2. The design of the code is still fresh in our minds We may have to change less code– Because we wrote less codebased on the defect29Copyright 2011 Gerard MeszarosMuch Ado About Agile 2011
Continuous Acceptance Testing! Defines what “Done Looks Like”– Several to many tests per User Story / Feature Tests executed as soon developer says “It’sReady”– End-of-iteration: OK– Mid-iteration : BetterWrite StoryTest Build Code Test Code Test StoryWrite StoryTest Build Code Test Code Test Story30Much Ado About Agile 201130Copyright 2011 Gerard Meszaros
Continuous Readiness Assessment! Defines what “Done Looks Like”ReadinessAssessment– Several to many tests per User Story / Feature Executed by developers during development– To make sure all cases are implemented– To make sure it works before showing to business Tests executed as soon developer says “It’sReady”– End-of-iteration: OK– Mid-iteration : BetterWrite StoryTest Build Code Test Code Test StoryWrite StoryTest Build Code Test Code Test Story31Much Ado About Agile 201131Copyright 2011 Gerard Meszaros
Prevention: - Building the Right ProductWhat the customer thought they wantedWhat the customer actually asked forWhat the customer realized they actually neededWhat development thought the customer asked forWhat development actually builtWhat testing thought the customer asked forWhat testing actually tested forMuch Ado About Agile 201132Copyright 2011 Gerard Meszaros
Building the Right Product How do we eliminatethe waste caused bybuilding the wrongproduct?– Missed requirements?– Misunderstoodrequirements?– Unneeded functionality?Much Ado About Agile 201133Copyright 2011 Gerard Meszaros
Building the Right Product How do we eliminatethe waste caused bybuilding the wrongproduct?– Missed requirements?– Misunderstoodrequirements?Much Ado About Agile 201134Copyright 2011 Gerard Meszaros
Example-Driven Development A.K.A.– Acceptance Test Driven Development– Behaviour-Driven Development– Executable Specification– StoryTest-Driven Development Concrete examples flesh out requirements Testers flush out missed scenarios.before development startsMuch Ado About Agile 201135Copyright 2011 Gerard Meszaros
Life Cycle of an Example / TestStoryExamplesMake ConcreteBy Adding xamplesMuch Ado About Agile 201136Copyright 2011 Gerard Meszaros
Agenda Motivation Changing the Role of Test Automation– From Defect Detection to Defect Prevention– Different Tests for Different Purposes Approaches to Test Automation Test Automation StrategyMuch Ado About Agile 201137Copyright 2011 Gerard Meszaros
Test Automation Pyramid Large numbers of verysmall unit testsExploratory Tests– Ensures integrit
Agenda Motivation –The Agile Test Problem –The Fragile Test Problem Approaches to Test Automation Test Automation Strategy Rough timings for Agile Test Automation StrategyFile Size: 2MBPage Count: 105
