WIXLIB

Rigorous EvaluationUsability Testing

What is Usability Testing? Formal and rigorous testing using a structured processValidate adherence to interaction requirements“Actual” users who perform realistic and representative tasksUtilize a functional prototypeQuantitative and qualitative usability measures

Chuckism Usability Evaluation instead of Usability TestingWant to put participants at easeThe word test automatically invokes feelings of anxietyDon’t use the word test on- Recruiting artifacts (flyers)- Consent forms- Label on door to testing room- first thing the user sees when they get to the evaluation Your conversation (“We are evaluating the software and notyou.”)

Usability Measures Ease of learning (learnability)—how fast can a user learn toaccomplish basic tasks?Ease of remembering (memorability)—can a user rememberenough to be effective the next time?Efficiency of use—how fast can an experienced user accomplishtasks?Error frequency and severity (understandability/comprehensibility) how often do users make errors, how serious are they, and how dousers recover from them?Engaging (Subjective satisfaction)—how much does the user likeusing the system?

Usability Evaluation - Ethics

Awareness of Regulations Human Subjects Protocols You must be fully aware of the regulations imposed by the variousinstitutions and regulatory bodies that pertain to your experimentaldesign Health and well being of subjectsThe U.S. Department of Health and Human Services Web site http://www.hhs.gov/ohrp/ Informed consent form – all participant users shouldread and sign Explains the study, risks (if any), and contact name for questions/issuesWritten consent to record participants (video and audio)How you may or may not use their image/voice

Advantages and Limitations of UsabilityEvaluation Advantages Discover usability issues before deploymentParticularly important for a market driven productBegin to build user loyaltyGain knowledge for future releasesDisadvantages Artificial contextNo guarantee of product acceptanceResult skew if true user demographic missedMay not be the most efficient and cost effective method for usabilityevaluation

Comparative Can be formative or summative- Formative: what do we need to change vs competition,with which design should we proceed- Summative: verify we are better than competition Between your own designs/prototypesVs. competitionMethodology- Compare to pre-determined standard or benchmark- Use performance and preference criteria- Between or within-subjects design

Format of the Test Plan1.2.3.4.5.6.7.Purpose/goalsResearch questions (usability requirements)Target AudienceDesign of the Usability EvaluationLogisticsData Collection MethodologyDeliverables Description

1. Purpose High level reasoning for performing the test at this timeNeed sound reasons – NOT Everyone else has a testing programGoal examples from 3.3 of Tullis/Albert- Comparing products (you vs. competition) (summative)- Comparing designs- Evaluate Information Architecture (IA)- Problem discovery- Evaluate critical feature or product- Impact of change on established user base

2. Research Questions / Usability Requirements Describes issues/questions that need to beresolved and focuses the researchPrecise, accurate, clear and measurable (orobservable)Helps you plan your test designYour test results will need to answer these questions If a test task scenario isn’t related to one of the questions, itshould be removed If you don’t have a test task scenario to answer one of these,need to add one

2. Research Questions / Usability Requirements Too unfocused and vague- Is the current product usable?- Is product ready for release? Better - Can users successfully install the software using the setup guidewithout assistance?- Do the screens reflect the end user’s conceptual model?- Can users perform task xyz in under 3 minutes?- Can users perform task abc without instruction (first-use intuitive.)?- Do users prefer prototype A vs. prototype B?- How quickly & easily can users perform picture upload from amobile device?

ActivityOutline a first version of your project test plan: Define the purpose of the usability evaluation- Which system will you compare your design against? Define the usability evaluation goalsUse the Test Plan template

3. Target Audience Participants should be real users You don’t need a large sample (8-15 or so) to get goodfeedback Recruit users with the following characteristics: AvailabilityResponsivenessObjectivityDiversity – background, experience, responsibility, Represent primary user roles

4. Design of the Usability Evaluation Introductory paragraph- Layperson’s explanation of how you will conduct the test Test Design- Type of study (within-, or between- subjects)- Number of participants- Duration of each session- Number and types of tasks performed by each participant- Matrix design

Test Design: Intro Paragraph ExampleIn this between-subjects test, each participant will sit through aone-hour usability study session. Approximately 15 minutes ofeach session will be used to explain the session to the participant,review basic background information with the participant, andconduct a pre-test questionnaire. During the middle 30 minutes ofthe session, participants will use the XYZ software to complete 4tasks, which the moderator will administer. Each participant willbe asked to use the think-aloud protocol while conducting thetasks. The last 15 minutes will be used to conduct the post-testquestionnaire and debriefing.

Within-Subjects Test Design A single group of participantsEssentially comparing each participant to themselvesCompare Designs- Each participant uses both interfaces- Preferred interface measures Repeated measures- Example: Multiple trials of same task, Compare time on taskbetween trials- Can be used to measure “learnability” of product Advantage: fewer participants requiredDisadvantage: carryover/transfer of learning- Mitigate via counterbalancing

2 Products – 1 Group – 1 Set of TasksABWithin-Subjects Test Design

Between-Subjects Comparing Interfaces- Randomly assign participants to groups- Randomly assign all participants- Randomly assign males to each group then females to ensureequal distribution between groups- Or balance novice and expert between groups Comparing groups- Manually assign to group based upon characteristic- group of experts vs. group of novices

2 Products – 2 Groups – 1 Set of TasksABBetween-Subjects TestDesign

Disadvantage of Within-SubjectsWhat if Product A leaves a bad taste?What if participants learned something usingProduct A that made Product B easier to use?*Carryover/transfer of learningABThink: Taste Test

Counterbalancing Alternate order of tasks Alternate order of interface use Alternate order of both interfaces and tasks

Within-Subjects: Comparing Interfaces*More TypicalCompare 2A2B2A2An Tasks for Interface ABn Tasks for Interface BTasks An equivalent to task Bn in theirrespective A4B4A4B4B4A4B4A4B4A4B4A4

Session Outline General description of what will happen duringsessionContains high-level outline with timings- Pre-session setup (5 minutes)- Intro and informed consent (5 minutes)- Background questionnaire (5 minutes)- Tasks (not a list of tasks) (30 minutes)- Post task questionnaire (5 minutes)- Post-session debrief and questionnaire (10 minutes)

ActivityWork on the design of the usability evaluation section Define the user tasksDesign the task matrixDesign the session outlineUse the Test Plan template

5. Logistics ScheduleLocationResources – people and equipmentMaterials, e.g. task scenarios – how will tasks be used in the userenvironment?

Constraints on Usability Testing Time to Financial Design, prepare, and administer the testAnalyze the resultsEquipment and softwareLaboratory timeRecording media[Participant compensation ]Space—to perform the usability test A dedicated laboratory or room is recommended.

People Tester roles Test project leader, expertModerator – interacts with the participant during the testData logger / Note taker[Technician] – operational responsibility Optional observers: Other development team members not involved in the testOther stakeholders

Test Scripts Write test scripts – to avoid bias due to inconsistentmoderator-participant interaction Greet the participant – introductions, set the stagePreliminary interview – warm-up questionsProvide instructionsMonitor the test – record observations, capture participant’simpressions and commentsDebrief the participant – wrap-up discussionScript test and task execution details Length and orderBreaks to minimize user fatigueIntervals between testsFlexibility for the unexpected

Activity Work on the Logistics section of your project test plan. Write test scripts Fill in Appendix A.

6. Data Collection Methodology (Plan) Define the measurements – reflect usability goalsQuantitative - objective, measurable Performance data - times, error rates, etc.Subjective ratings, from post test surveys Qualitative: subjective Participant comments, survey answersTest team comments, observationsBackground participant data from user profiles, surveys,questionnaires List of problems (known and/or suspected)

6. Data Collection Methodology(During the test) Maintain a log or observation check list for eachtask Create a problem list to capture anything that isnot covered by the check list Note any ideas or theories that occur to you aboutthe problems Usability measurements Critical incident observation – emotional impact

Usability Specification TableUserRole UX LevelObservedResultsUser role – user categoryUX goal – quality measure, e.g., learnabilityMeasuring instrument – the benchmark task(s) or survey to generate testdataUX Metric – test measurement values to be collected; e.g., response timesBaseline level – performance of current system if relevantTarget level – minimum value for successObserved results – measured valuesUsability engineering: Our experience and evolution M. Helander Handbook ofHuman-Computer Interaction , J.A. Whiteside J. Bennett K. Holtzblatt 1988

7. Deliverables: Process the Data Identify problems (known and/or suspected) SeverityFrequencyErrors of omissionErrors of commissionPrioritize problemsTheorize reasons and solutionsIdentify successes and areas of uncertainty

Run A Pilot Test To Rehearse Be organized Be presentable for a good first impression

ActivityWork on the Data Collection section of your project testplan. Fill in Appendix B and Appendix C.UserRoleUX LevelObservedResults

References Albert, William, and Thomas Tullis.Measuring the user experience: collecting,analyzing, and presenting usability metrics.Newnes, 2013.

Use the Test Plan template . 3. Target Audience Participants should be real users You don’t need a large sample (8-15 or so) to get good feedback Recruit users with the following characteristics: Availability Responsiveness Objectivity Diversity – background, experience, responsibility, Represent primary user roles . 4. Design of the Usability .