WIXLIB

1

2

3

4

Objectives:In this talk, I want to talk about:a. What does it mean when folks say, “I am doing Exploratory Testing”, why isit important in Agile, etcb. Along the road, debunk some of the myths surrounding Exploratory TestingKey Takeaways:1. How to make your Exploratory testing sessions more effective so that youachieve customer value5

James Bach defines it succinctly as simultaneous test definition and execution. Ofcourse, exploratory testing has been very popular and most testers do exploratorytesting, perhaps calling it by another name or not calling it anything at all6

What pops in your mind as you think about exploratory testing?There are many reasons why exploratory testing might be interesting to testers intoday’s world:Find bugs fast and early without test overhead – exploratory testing is lightweightand reasonably good test hunches will let you find defects quick and early withoutthe documentation time overheadCentered on customer value rather than spec compliance – Tester is an end userproxy on the team and is key to the product being useful to the end user. Exploratorytesting is done on working software and focused on customer value.Crucial component of Agile testing – Everyone tests on an Agile team : thedevelopers, Product owner, SCRUM master. The iterations are typically short anddefects need to be found and fixed to deliver optimal customer value within a shortturnaround time. This is an ideal situation to use a lightweight and effective style oftesting like exploratory testingWhen you know less about a system and testing is open endedWhen you have to find maximum bugs in minimum time7

So, despite so many advantages, why do many people still have reservations withexploratory testing?Exploratory testing has rapidly gained popularity over the past few years as one of themost efficient forms of testing in an agile lifecycle. Exploratory testing has a provenadvantage of finding bugs faster, with less wastage in terms of test documentationand covering large areas of the application under test in a shorter time period. Whythen, do we still find exploratory testing not being a mainstay in large scale enterpriseproduct testing?Here are some of the myths associated with Exploratory testing:a. Exploratory testing is ad-hoc, monkey testingb. Exploratory testing creates poorly documented bugsc. There is no way to track exploratory testingLets look at each one of them in detail.8

Exploratory testing is NOT random, undirected testing. On the contrary, unlikescripted testing, you cannot just follow a set of pre-determined steps that mightarrive at a defect. Testers often have hunches about where defects could be – aparticular dev tends to be bad at exception handling, or an app that is weak in dealingwith frequently disconnecting connections, or a website doesn’t do effective loadbalancing when stressed. As you learn more about the system under test, you getnew hunches, new test ideas that could lead to unfound defects.Vectoring exploratory testing with some of these test ideas will help explore thesystem more effectively and completely. Two great ways of vectoring exploratorytesting can be found in session based test management by James Bach andexploratory testing tours by James Whittaker. SBTM will help direct exploratorytesting by getting the tester to define a mission upfront and help keep explorationfocused in that area. Testing tours help define various paths to explore in the systemwith each tour leading to different kinds of defects in different areas.9

How we use XT within our product development – using Charters/Themes10

We use the concept of testing tours extensively in our in house testing – check outsome of our testing tours look like:My favorite tour is the Back Alley tour where you exercise some of the lesser knownpaths in the application,the Fedex tour where you can track a data structure from point to point to see how itgets transmuted,the All Nighter tour that has caught some nasty memory leaks we fixed recentlyThe Landmarks tour where sales folks do DEMOs of the productThe Guidebook tour or F1 tour – following documentation to the letterThe Intellectual tour – asking the software hard questions – Inputs & data that systemaccepts that streches its limitsThe Anti-social tour – providing least likely inputs or bad inputsThe couch potato tour – accepting all default valuesThe sabotage tour – find out the resources used, then restrict access to thoseresourcesThe collectors tour – collect up all the outputs that can be generated from the systemThe Supermodel tour – UI Interface11

Apart from the tours we discussed, we also use XT for the user stories that gotcompleted in this sprint. This provides us the traceability to answer questions such as:did all the user stories in the sprint got covered in the Testing; which stories yieldedthe most number of bugs etc. This allows us to tie the progress back to the customervalue instead of just the deviations from the spec.We also use XT for our Bug bashes – we divide up the team into different roles andeither use the Tours Or User Stories to drive the bug bash.Demo of VS11 for doing exploratory testing on user stories12

Lets look at the 2nd myth – XT creates poorly documented bugs13

Since in exploratory testing, there is no prescribed path/spec that you are following, itbecomes critical to have all the information needed to reproduce the bug, once it hasbeen found. So, how do you capture all the data that helps you in filing a Rich bug.Using some of the tools that capture your actions as you are exploring the applicationwould be very nice. Similarly collecting all the event logs, system info logs,screenshots that show what you just saw, an entire video recording of what paths youwere running through are some of the items that could be captured to generate aRich bug. Well, it might even include an entire debugging log to analyze what youwere doing for the developer – effectively capturing the entire state of the bug.14

Here is some of the information we collect as part of bug filing during exploratorytestingDemo of VS11 for rich bug filing15

Some of the key things you saw in the Demo include:- How easy it is to create test cases from you exploratory session – this could beeither during the session or post the session completion- How the tests that you create are automatically added to the right Test Suiteswhich get rolled up into the feature area- How each iteration of exploration generates test cases which get automaticallyrolled up in the Test plan you are executing- How with each sprint, regression suites are automatically being created16

Lets look at the 3rd myth – there is no way to track exploratory testing17

Here are some of the metrics that can be used to track Exploratory testing:- Trending of bug priorities and bug types that were filed during exploratory testing- Ratio of time spent on exploration to the number of bugs that were filed- Ratio of time spen on exploration to the number of test cases that were created- Coverage of functionality across charters/themes- Coverage of Code while doing exploratory testing- Coverage across User stories that have been taken up for current / past sprints- The main thing is that there are NO absolute metrics – you have to finetune/tailorto suit the quality of your application18

Here are some of the metrics we use to track our XT workDemo of VS11 for tracking purposes (Code Coverage, Bugs files, Test cases filed etc)19

Exploratory testing done with the right tools enables you to:Define and test what is appropriate for your appTest more frequently and find bugs fasterCreate a trail of structured test cases for regression20

Objectives:In this talk, I want to talk about:a. What does it mean when folks say, “I am doing Exploratory Testing”, why isit important in Agile, etcb. Along the road, debunk some of the myths surrounding Exploratory TestingKey Takeaways:1. How to make your Exploratory testing sessions more effective so that youachieve customer value21

most efficient forms of testing in an agile lifecycle. Exploratory testing has a proven advantage of finding bugs faster, with less wastage in terms of test documentation and covering large areas of the application under test in a shorter time period. Why then, do we still find exploratory testing not being a mainstay in large scale enterprise