More Than I Ever Wanted Or Just Good Enough? User .

Transcription

Qual User Exp (2017) CH ARTICLEMore than I ever wanted or just good enough? User expectationsand subjective quality perception in the context of networkedmultimedia servicesA. Sackl1,2 R. Schatz1,2 A. Raake3Received: 28 April 2016 / Published online: 9 February 2017 Springer International Publishing Switzerland 2017Abstract This article addresses the overall relevance andthe influence of user expectations in the context of perceptual quality evaluation of technical systems, and howinformation about individual expectations can be integratedinto research on Quality of Experience (QoE). Based on aliterature survey on expectations in the fields of psychology, service quality and consumer satisfaction theory, anexisting conceptual QoE perception model is extended byexplicitly including desired and adequate expectations inthe quality perception process. A direct expectationassessment paradigm is proposed that uses dedicatedquestionnaires. Further, it is shown how expectation-related knowledge can be used to increase the accuracy ofquality prediction models. In addition, experimental setupsare presented that enable controlled triggering of expectations in user studies. It is discussed how the role ofexpectations for QoE can be investigated in future researchas well as how to further validate the proposed methods.& A. Sacklandreas.sackl@ait.ac.atR. Schatzschatz@ait.ac.atA. Raakealexander.raake@tu-ilmenau.de1FTW Telecommunications Research Center Vienna,Donau-City Straße 1, 1220 Vienna, Austria2AIT Austrian Institute of Technology, Donau-City Straße 1,1220 Vienna, Austria3Department of Audiovisual Technology, Institute for MediaTechnology, Technical University Ilmenau, Ilmenau,GermanyKeywords Quality of experience Expectations Userstudies ModelingIntroductionIn general, Quality of Experience (QoE) research investigates the relationship between technical quality parameterslike video frame rate, video bit rate, downlink bandwidthetc. and the subjectively perceived quality of a technicalsystem (Möller and Raake 2014). Various non-technicalinfluencing factors have an effect on this relationship, forexample, human-factors like the socio-cultural backgroundor demographic variables and context-factors like an individual usage vs. a usage within a social context likevideoconferencing (see Reiter et al. 2014 for more information). In this article, we focus on one crucial qualityrelated aspect which has been hitherto largely neglected inempirical QoE-related research: the expectations of theperceiving subject.Although the term ‘‘expectations’’ is frequently used inthe context of quality perception and QoE-related research,we witness a lack of applicable concepts and methods thatenable the operationalization of expectations and the utilization of related findings in empirical research. Albeitcommonly used QoE frameworks and definitions highlightthe importance of expectations, clear guidance on how toactually address this influencing factor is missing. Forexample, in Möller and Raake (2014), QoE relates to ‘‘. . .person’s evaluation of the fulfillment of his or her expectations . . .’’, i.e. expectations as the perceiving subject’sframe of reference. Similarly, in the Qualinet1 QoE1COST Action IC1003, European Cooperation in Science andTechnology, http://www.qualinet.eu/, last access: 29th July 2015.123

3 Page 2 of 27Qual User Exp (2017) 2:3definition white paper expectations are described as keyfactor determining the end users perceptions and relatedwith emotional state: ‘‘Quality of Experience QoE is thedegree of delight or annoyance of the user of an applicationor service. It results from the fulfillment of his or herexpectations with respect to the utility and/or enjoyment ofthe application or service in the light of the user’s personality and current state’’ (Le Callet et al. 2012). Despiteits primarily technical focus, the ITU-T RecommendationP.10 also highlights the relevance of user expectations asan influencing factor. Here, QoE is defined as ‘‘the overallacceptability of an application or service, as perceivedsubjectively by the end user [which] may be influenced byuser expectations and context.’’ Also conceptual QoEmodels and frameworks like the one used in Schatz et al.(2013) explicitly include expectations as a main userspecific influencing factor.To address the observed gap in existing research—thatis, a frequent mention of the high relevance of expectationsto QoE but at the same time lacking empirical and conceptual research foundations (see ‘‘Practical Inclusion ofExpectations in QoE related research’’ for details)—thisarticle provides a comprehensive analysis of the concept ofexpectations and makes it methodologically applicable toQoE research. To this end, the remainder of this paper isstructured as follows: first, we provide an overview of howother domains (consumer satisfaction research, servicequality research) define, operationalize and utilize userexpectations in various application fields. Based on thesefindings, suitable existing concepts that aid in the operationalization of expectations (such as adequate and desiredexpectations) are used to extend an existing conceptualQoE-model. Building on this, a survey on existing QoEliterature clarifies which aspects of expectations are actually relevant for QoE research. The main part of thisjournal paper focuses on discussing our empirical findingsregarding triggering and quantifying expectations in thecontext of QoE assessment and further describes how touse these findings to increase the prediction accuracy ofQoE models. Finally, we discuss further open researchquestions and required future work on expectations andQoE.Based on that, existing QoE-related literature is discussedto identify open research questions in the context of QoEand expectations.Related work: towards a conceptualQoE/expectations model1.In this chapter we present the findings of our literaturesurvey regarding expectations in the context of servicequality and consumer satisfaction research. The resultingfindings are then used to extend existing QoE-models tofinally propose a conceptual QoE/Expectations model.2.123Expectations in socio-psychology, service qualityand consumer satisfaction researchA good starting point for learning about the nature ofexpectations is the field of psychology in which the conceptof expectations is often used to describe the processes ofunderstanding and cognition because ‘‘in perception, weconsider prior expectations’’ (Sternberg 2011, p. 108). Forexample, when someone reads a piece of written text, shehas to make assumptions regarding the content, the contextof the work’s creation, the author’s purpose, etc. Consequently, the ‘‘understanding at each point in the [text] wasinfluenced by [. . .] existing knowledge and expectationsbased on [. . .] own experiences within a particular context’’(Sternberg 2011, p. 393).Additionally, expectations play a critical role in psychological research in the context of decision making, seeGerrig (2012, p. 243), and also in motivation and behavioral changes, see Gerrig (2012, p. 300). In the field ofsocio-psychology, expectations are considered as animportant aspect which determines how subjects interactwith others to fulfill social norms, cf. Gerrig(2012, p. 447, 453). Additionally, the well-know concept‘‘self-fulfilling prophecy’’ describes how triggered expectations can lead to unforeseen output, see Gerrig(2012, p. 450).To obtain a more practical and applicable definition ofexpectations, research fields like human–computer-interaction, economics, etc. yield additional informative results.The authors of Bonito et al. (1999) define expectations as‘‘a kind of schemata that focuses interpretation processeson specific meanings and functions of communicativeaction’’ (Bonito et al. 1999). As described in Higgs et al.(2005), expectations are ‘‘pre-trial beliefs about a productor service and its performance at some future time’’ andexpectations ‘‘form the frame of reference for satisfactionjudgments’’ (Higgs et al. 2005). Additionally, the authorsof Higgs et al. (2005) divide expectations into four maincategories:3.Forecast (or expected/predictive): user beliefs aboutwhat will occur in specific forthcoming action regarding a specific provider.Normative (or deserved/desired): consumer perceptionof what should occur based on an assessment of what isrealistic and feasible regarding a specific provider.Ideal (or wished): highest level of performance attainable, independent from specific provider or brand.

Qual User Exp (2017) 2:34.Minimum tolerable (or adequate): minimum baselineperformance acceptable, independent from specificprovider or brand.According to the authors of Higgs et al. (2005), practicalimplications of expectations have been investigated mostlyby two different research traditions:1.Consumer satisfaction research: here, the primary goalis to understand the user’s cognitive processes, whichleads to customer’s satisfaction.2. Service quality research: here, the primary object is tounderstand and to measure quality in serviceenvironments.2In this context, one of the most widely used modelsregarding perceived service quality and expectations is theGAPS model and, building on that, the SERVQUAL modeldeveloped by Parasuraman et al. (1988). For example, inthe context of information system evaluation, the authors ofWatson et al. (1998) used the SERVQUAL model tomeasure existing expectations to compare it with gainedexperiences: Test users had to indicate their ideal information system and to evaluate 22 questions via a 7-pointanswering scale with answering options ranging from1 ¼ ‘‘Strongly disagree’’ to 7 ¼ ‘‘Strongly agree’’, e.g.,‘‘The employees of these Information Systems units willunderstand the specific needs of their users’’. After that, thetest participants used a particular information systemimplementation to perform some tasks. Then, the users hadto evaluate their experiences by answering slightlyrephrased questions, e.g., ‘‘Employees of this InformationSystem understand the specific needs of its users.’’ Withthis information, it is possible to calculate the expectationgap.Similar to SERVQUAL but in the context of e-commerce, the authors of Kim et al. (2003) used the Expectation Confirmation Theory (ECT), originally developed bythe authors of Kristensen et al. (1999), to measure the usersatisfaction of Web services. The satisfaction level wasgenerated by a comparison of post-purchase evaluations ofa product or service with pre-purchase expectations. Following the ECT, users generate specific expectationsregarding a desired product. After a trial phase, the usersform perceptions about its performance. Then the participants determine if their expectations were confirmedregarding the perceived performance. Finally, the users’satisfaction level results from the previous confirmationand the underlying expectations. In the end, a reuse orrepurchase is considered or not.In general, existing literature points out an importantdifference regarding asking about expectations: in thePage 3 of 27 3service quality tradition the subject states expectationsabout what the service provider should offer. In contrast, inthe consumer satisfaction literature the subject revealsmore about his/her expectations what will be offered.When it comes to QoE assessment, inquiring expectations can be problematic because for the user it is often noteasy to verbalize expectations. For example, what does a‘fast’ Internet connection actually refer to? How shouldend users quantify the expected quality of a video transmission? In this context, the author of Ojasalo (2001)focuses on an understudied aspect of service quality anduser satisfaction, namely fuzzy, implicit and unrealisticexpectations. According to the author, users have fuzzyexpectations ‘‘when they expect a change but do not have aprecise picture of what this change should be (Ojasalo2001)’’. If these expectations are not fulfilled, for theconcerned users the experienced service was unsatisfactorybut they do not know why. The opposite of fuzzy expectations are precise expectations. According to the authorsof Ojasalo (2001), implicit expectations are so self-evidentusers do not actively think about them. They only becomerelevant and explicit for the users when these expectationsare not fulfilled. Finally, there are also unrealistic expectations which are obviously unable to be fulfilled. Theauthors of Ojasalo (2001) argue, that fuzzy expectationscan be converted into more precise expectations via adialog between the user and the service provider. Obviously, this qualitative approach is not appropriate in thecontext of quantifying expectations for QoE relatedresearch.In the field of consumer satisfaction research, brandsplay an important role for expectations. Here, examinedexpectations—and their fulfillment or non-fulfillment—arebased on concrete products or services, cf. Cadotte et al.(1987). In contrast, assessing the influence of brands orother marketing-related aspects is not very common in QoEresearch. Here, the focus is mainly on assessing the impactof the quality/performance of the technical system which istypically evaluated by test participants without having anybackground information, e.g., in common video studies, theresponsible video content provider delivering the streamedvideo is considered to be irrelevant.3According to the authors of Cadotte et al. (1987), in thecontext of service quality research it is not alwaysstraightforward to quantify expectations. For example, it isfairly simple to quantify the speed of the service in arestaurant by means of asking about expected seconds orminutes of waiting time. Similarly, menu variety can be32For more information about these two traditions, please seeBoulding et al. (1993), Buttle et al. (1996) and Oliver (1996).Most QoE researchers would agree that a broad range of influencingfactors impact the perceived quality, but when it comes to empiricalresearch or modeling attempts, softer influencing factors like brandsor other marketing factors are mostly neglected.123

3 Page 4 of 27determined by the number of food items on the menu.However, it is considerably more complicated to measureand operationalize employee-friendliness. Obviously, thisparticular issue can be solved via appropriate questionnairedesigns. Nevertheless, quantifying individual expectationsis difficult and depends on the context and the evaluatedservice.In the context of usability research, expectations can beused to identify critical usability problems. Briefly speaking, test-users in a usability experiment are asked beforehand how difficult a certain task would be, e.g., find item Xon website Y. Subsequently, the users have to evaluate thedifficulty of this activity after they have fulfilled this particular task. The ensuing comparison shows which designissues should be fixed immediately and which are notcrucial. For example, if task A was expected by most of theusers to be very easy and the ratings after the usage weremostly ‘‘very difficult’’, this issue should be fixed veryquickly, see Albert and Dixon (2003) for more details.Obviously, this approach is not appropriate for QoEbecause for the users it is often difficult to verbalizequality-related expectations, which has been discussedbeforehand.According to Zeithaml et al. (1993), there are two different types of expectations: desired and adequate expectations. The authors of Zeithaml et al. (1993) stated that thedesired expectations are rather stable and invariant, e.g.,some users are always concerned about high quality orsome users are always concerned about low prices, andthese basic needs do not change over time. In contrast tothis, adequate expectations are more flexible and they arestrongly influenced by the context. Between these twokinds of expectations, there is the so-called zone of tolerance: if the perceived service is in-between the slowlyadapting desired expectations and the more variable adequate expectations, the user/customer accepts the perceivedQual User Exp (2017) 2:3service, see Fig. 1b for a graphical representation. In thecontext of typical business/customer relations, adequateexpectations are influenced by (see also Fig. 1b):1.Transitory service intensifiers: The urgency of asituation can lower the adequate expectations.Perceived service alternatives: If there are alternativesavailable in the current situation or if it is possible tosolve an issue without external support, the adequateexpectations get higher, which leads to a smaller zoneof tolerance.Self-perceived service role: The customer tries tofulfill her role in the current process, e.g., it is notalways possible to blame others for non-fulfilledexpectations. Therefore, the more pretentious the levelof a customer’s view on the self-perceived service roleis, the higher is the level of adequate service.Situational factors: These factors can lower the level ofadequate service if the environmental influences areindependent from a service provider. Hence, customersrealize that this is not the fault of the provider and theyaccept a lower service level.Predicted service: This is the service quality customersbelieve they are likely to get.Contextual circumstances: For example, economicaspects are included here. A participant in the studypresented in Zeithaml et al. (1993) stated that ‘‘priceincreases do not really drive up expectations. But mytolerance level will become more stringent/less flexiblewith an increase.’’2.3.4.5.6.There are some attempts to quantify adequate and desireduser expectations in current literature. For example, theauthors of Hsieh and Yuan (2009) use the Weber/Fechnerlaw—which originates from psycho-physics and is alsoused for QoE modeling—to generate a quantitativeexpectation measurement model which ccepted(rather x)adequateexpectationsnotaccepted(variable)(a) Mathematical relationship between desiredand adequate expectations, originally publishedby [19], see also next page.(b)perceivedservicein uences1) Transitory service intensi ers2) Perceived service alternatives3) Self-perceived service role4) Situational factors5) Predicted service6) Contextual tationsmodel.Own graphical representation of the desiredand adequate expectation concept of [20].Fig. 1 Mathematical relationship and model of desired/adequate-expectations123zone oftolerance determinesacceptance

Qual User Exp (2017) 2:3Page 5 of 27 3describes the relationship between the desired and theadequate expectations of customers regarding service providers. Figure 1a shows that desired expectations ED arerather stable even if the stimulus magnitude I of theexpectation determinants increases (e.g. personal needs,transitory service intensifiers, perceived service alternatives, customer self-perceived service role, situationalfactors, etc., see also Zeithaml et al. 1993). In contrast tothis, adequate expectations EA are more flexible and alsoincrease when the stimulus magnitude I of the expectationdeterminants increases. Nevertheless, the authors of Hsiehand Yuan (2009) do not describe how to measure orquantify expectation determinants and therefore theircontribution to the challenge of quantifying expectations israther limited.In common expectation and service quality researchapproaches, it is usual to capture expectations in the postconsumption phase instead of getting expectation information before a certain action is initiated, cf. Oliver (1996).Nevertheless, in the context of QoE it is more relevant toget information about users without conducting a certainexperiment, so that this information can be used, forexample, for MOS prediction modeling.It is also a common assumption that test subjects havehad prior experiences in a way that they can articulateexpectations for current evaluation tasks, see Zeithamlet al. (1990). Nevertheless, some researchers assume thatexpectations exist even when no prior experience has beengained (see for examples McGill and Iacobucci 1992;Shirai and Meyer 1997), whereas the authors of O’Neil andPalmer (2003) state that expectations can not be generatedwithout prior usage (Table 1).To summarize, the nature of expectations has beenexamined in various research areas, e.g., human–computerinteraction, economics, psychology, etc., with the findingslisted below being essential for understanding the role ofexpectations in the field of QoE: Expectations depend on a broad variety of influences,and understanding how they emerge and how theyinfluence quality perception is not trivial.Expectations can have negative effects on perceivedquality when being under-fulfilled, but also positiveones when being over-fulfilled. Therefore, empiricalQoE research should also include test conditions inwhich expectations are over-fulfilled (currently, thefocus of QoE research is on situations in whichexpectations are not fulfilled).In research fields dealing with service quality expectations are generally considered as measurable, e.g., bymeans of questionnaires. But this approach requiresexpectations which can be verbalized and which arequantifiable, which is not always possible in the field ofQoE. For example, it might be challenging for anaverage end user to verbalize her expectations regarding Internet connection speed in Mbit/s.It is essential to distinguish between relatively stable,higher desired expectations and variable, adequateexpectations, which both together influence the acceptance of a certain service.Conceptual model QoE/expectation modelIn the previous section, the current state of the art regardingexpectation classification, assessment and quantification inthe field of service quality and customer satisfactionresearch has been discussed. For the next steps it is necessary to include expectations in common conceptual QoEmodels in an appropriate way.For the present work, Zeithaml’s concept of desired andadequate expectations Zeithaml et al. (1993) (see alsoprevious section) is used to integrate expectations into QoEresearch. As depicted in Fig. 2a, current QoE models (e.g.see Schatz et al. 2013) often consider the specificTable 1 Overview of the different research fields addressing expectationsResearch fieldExample referencesTopicPsychologySternberg (2011), Gerrig (2012)Processes of understanding and cognition, decision making, motivation andbehavioral changes, fulfilling of social normsHuman computerinteractionBonito et al. (1999)Communicative action, interpretation processes, expectations as pre-trialbeliefsConsumersatisfactionresearchRust and Oliver (1994), Boulding et al.(1993)User’s cognitive processes, tends towards brand cued expectations how iscustomer’s satisfaction reachedService qualityresearchOliver (1996), Parasuraman et al. (1988),Zeithaml et al. (1990)Focus on expectations and perceptions, measurement of service quality,various models e.g. SERVQUALQuality ofexperienceMenkovski et al. (2010), Monath et al.(2010)Expectations as a relevant, but vague factor in the context of qualityassessment123

3 Page 6 of 27Qual User Exp (2017) 2:3HighexpectationsUserQoE modeldesiredexpectationsacceptedzone oftoleranceadequateexpectations2) Perceived service alternatives3) Self-perceived service role4) Situational factors5) Predicted service6) Contextual circumstancesnotaccepted(variable)(a)The term expectations as an important userrelated influencing factor on QoEperceivedservice determinesacceptanceContextQoE modelLowexpectations(b) Adapted QoE-Adequate/Desired QoEmodel. Own graphical representation of thedesired/adequate expectation concept of [20]extend with QoE aspectsFig. 2 Generic QoE framework with influence factors and QoE/expectations model based on Zeithaml et al. (1993)Quality user decisionDesired QualityFeauresAdequateExpectationsComparison &JudgementPerceived QualityFeaturesExternal Factors(which triggersexpectations)noneconomicOptionse.g. use PCor TabletContexte.g. at cqualityrelateddecisions)Reflection ionse.g. 3 forHD videoTasksRecognized objectsof perceptionPerceptionProcess [Fig 2.2]e.g. changing the volume of a CD trackSignalFig. 3 QoE-Expectations-Decision-Model. Circles represents processes, two horizontal parallel lines storages and rectangles the inputs andoutputs to the userinfluences of context-related and user-intrinsic variables onQoE. In terms of Zeithaml’s expectation concept, thephrase ‘‘user’’ may be related to the rather stable ‘‘desiredexpectations’’, whereas the phrase ‘‘context’’ may berelated with the context-sensitive term ‘‘adequate expectations’’. Hence, a combination of the two influencingfactors with Zeithaml’s model can be conceived, seeFig. 2b.In Raake and Egger (2014, p. 23), a conceptual modelof the quality-formation process—based on Jekosch (2006)123and Raake (2006)—is presented and discussed. Figure 3depicts a (slightly simplified) representation of this model,extended by the concepts of desired and adequate expectations as part of the user-internal ‘‘Assumptions’’. Theperception process is triggered by signals (for exampleaudiovisual information) and various external factors, forexample contextual information representing the use caseof consuming a video at home. This perception processtriggers adequate expectations, that is, expectations that arecontext-specific. In perceptual terms, the adequate and the

Qual User Exp (2017) 2:3more stable desired expectations are instantiated by thedesired quality features used in the comparison andjudgement process as the internal references that perceptualfeatures are compared with, see Zeithaml et al. (1993).Depending on the given task, the quality formation processmay lead to an explicit quality rating, e.g., resulting from avideo quality questionnaire. Any kind of evaluation performed by humans may ultimately lead to a decision, basedon the underlying adequate and desired expectations: Forexample, a Video-on-Demand content may be consumed instandard definition (SD) resolution, but for the examplesituation—watching a movie at home on a large TVscreen—the perceived quality is too low, that is, theexpectations are not fulfilled and the user decides to switchfrom SD to HD video content which directly influences thesignal. Many different types of decisions and actions maybe triggered by different signal or content-related features,such as increasing the volume, dimming the room light, orpausing or stopping the video playback.The detailed description of the quality formation processcan be found in Raake and Egger (2014, p. 23). In thisjournal article we want to give only a brief overview andexplain how expectations are included in this process asdepicted in Fig. 3. The ‘‘Signal’’ represents any audiovisualinformation which is perceived by the user, e.g., a consumed video, a Web site interaction etc. The fine-grained‘‘process of perception’’—which is described in detail inRaake and Egger (2014, p. 20)—results in recognizedobjects of perception, which furthermore influences the‘‘experience’’ character of the situation, e.g., a disturbanceis recognized by the user like a too long page load timeduring Web browsing. The so called ‘‘Quality Awareness’’is triggered4 which results in a ‘‘Reflection & Attribution’’process, i.e., the user is aware of a quality problem, whichresults in ‘‘Comparison & Judgement’’. During ‘‘Comparison & Judgement’’, the ‘‘Perceived quality features’’ andthe ‘‘Desired quality features’’ are compared which resultsvia ‘‘Encoding’’ to a ‘‘Quality Rating’’, e.g., the user couldstate ‘‘I would not accept this video quality at home’’, arating could be made on a 5 point scale etc. The ‘‘Desiredquality features’’ highly depend on desired and both adequate expectations. We assume that desired expectationsare rather stable and do not change over time.5 Also‘‘External Factors’’ influence the ‘‘Perception Process’’:4Additionally to disturbance situations, quality awareness can alsobe triggered externally in a laboratory setup, i.e., a researcher asks aexperiment participant to evaluate the perceived quality in a particularsetup, and internally if the user wants to buy a new TV set andcompares the quality of several panels.5Obviously, even desired expectations adopt over time, but weassume that desired expectations evolve over long-time periods likeyears or decades. Hence, in our model we suppose that they remainconstant.Page 7 of 27 3end device used (e.g. is a video consumed via Laptop or viaSmartphone), situational context (e.g. is a Website accessed at home or during a train ride), is there a specific task tofulfill (e.g. booking a flight) or not (e.g. relaxing whilelisting to music) etc. These external factors also influencethe adequate expectations—e.g. a user might be more tolerant regarding network outages during a train ride incontrast to surfing at home via a fixed-line network—whichdetermine the ‘‘Desired quality features’’ for ‘‘Comparison& Judgement’’. One output of the ‘‘Comparison & Judgement’’ could be, that the user is not satisfied with thecurrent (quality) situation and something has to change.Hence, a ‘‘Decision’’ is made e.g. changing from 3G toWifi if page load times are too high (non-economic decision) or changing from SD to HD quality during a Video onDemand Session—which might implies additional fees(economic decision). Of course, these user decisionsinfluences the adequate expectations, e.g., switching fromHD to SD video streaming lowers my adequate expectations regarding video quality, but my adequate expectationsregarding interruptions are higher, i.e., I am less tolerantregarding stallings.6 Additionally, some decisions alsoinfluences the ‘‘Signal’’: if a user raises the Bitrate of amusic streaming service

More than I ever wanted or just good enough? User expectations and subjective quality perception in the context of networked multimedia services A. Sackl1,2 R. Schatz1,2 A. Raake3 Received: 28 April 2016/Published online: 9 February 2017 Springer International Publishing Switzer