Transcription
7/21/2011Examining Service Quality andData Quality in TelephoneInterviewsJulia CoombsU.S. Census BureauOutline Introduction to the Service Quality Assuranceoperation and the Data Quality operation Service Quality Assurance operation– Operational overview– Scoring criteria– Outcomes Data Quality operation– Operational overview– Scoring criteria– Outcomes How the two operations relate2CFU Quality OperationsDuring the 2010 CFU operation, two independentoperations were implemented to evaluate thequality of the interviews and of the data beingcollected– The Service Quality Assurance (SQA)operation measured interviewer performance– The Data Quality (DQ) operation measuredthe accuracy of the data collected31
7/21/2011Call Recordings Calls were recorded randomly throughoutthe day by EyeQ360 Audio and video were recorded Interviewers did not know which calls werebeing recorded Both SQA and DQ took all scored callsfrom the recordings in EyeQ3604SQA Introduction Each call center had an SQA team SQA monitors ensured that at least twocalls per interviewer were scored everyday SQA managers at all call centers hadweekly calibration sessions with the SQAteam in the operational command center(OCC)5SQA Critical CriteriaCritical CriteriaCapture Eligiblity AccuratelyCapture Correct Household AccuratelyModule A/PBCapture Correct Housing Unit AccuratelyCCapture Roster AccuratelyDCapture Movers AccuratelyECapture Other Addresses AccuratelyFCapture Demographics AccuratelyGUtilize Approved Scripts and Job AidsAllCode of Conduct ViolationAll The first seven criticalcriteria corresponded tomodules in the interview;these were scored as“Pass”, “Fail”, or “N/A”,depending on howaccurately the interviewercollected informationThe last two critical criteriawere evaluated throughoutthe call; these were scoredas “Pass” or “Fail”Failing any of these resultedin a failed call. Failing thelast one automaticallyresulted in a failed call with ascore of 062
7/21/2011SQA Universal CriteriaUniversal CriteriaDisplay courtesy and professionalismDisplay enthusiasm and confidenceProvide accurate and complete informationEffectively control the callEffectively use active listening and probing questionsEffectively and efficiently navigate systemsAppropriately document and disposition the call The universal criteriawere evaluatedthroughout the call A call could receive a“Meets Standard”,“Needs Improvement”, or“Needs SignificantImprovement” score inany of these categories Even if a call was scoredas “Needs SignificantImprovement” in allseven categories, if thatcall passed all criticalcriteria, that call passed7SQA Score The SQA score was calculated by dividing thesum of the number of points awarded by thenumber of possible points and multiplying thequotient by 100 A call failed if the SQA score was below 94%–If a call failed, the supervisor of the interviewer wascontacted immediately, and a coaching session betweenthe interviewer and the supervisor would occur to reviewdeductions–Interviewers who consistently failed calls were disciplined8SQA Outcomes 550,829 calls were evaluated 99.7% of interviewers were monitoredtwice a day The overall SQA score was 99% 1.6% of the monitored calls failed Interviewers were most commonly failed inthe Utilize Approved Scripts and Job Aidscritical criteria93
7/21/2011SQA ScoresSQA Quality Score - Actual Daily & Cumulative PTD vs TargetActual Cumulative PTDGreen Threshold100.0%99.0%Average Evaluation 090.0%Date10DQ Introduction The Data Quality (DQ) operation measured theaccuracy of the data collected DQ monitors at the OCC scored a smallpercentage of all English and Spanish completedcalls Monitors scored the accuracy of the data collectedfor 15 specific questions as “Accurate”,“Inaccurate”, “Uncertain”, or “Not Scored” If a question was scored as “Inaccurate” or“Uncertain”, monitors selected a reason for a givenscore11DQ ScorecardAccurateAccurateInterviewer read the critical question verbatim andthe response matched the outputInaccurateQuestion not ReadInterviewer did not read the critical questionNo MatchRespondent’s response did not match what theinterviewer selected in the toolData ErrorRespondent’s response matches what theinterviewer selected but not the output on thescorecard124
7/21/2011DQ Scorecard (continued)UncertainNot Read VerbatimCritical question was not read verbatimNo Clear AnswerCritical question had a complex exchange betweenthe interviewer and respondent where no clearanswer was given by the respondentInaudibleAudio from either the interviewer or respondent wasinaudibleNot ScoredQuestion AbsentCritical question interaction for this question was notcaptured in the eyeQ360 video13DQ Score and Outcome 11,583 calls were evaluated The DQ score was calculated by the numberof questions scored as “Accurate” divided bythe number of questions scored as “Accurate”or “Inaccurate” for a call– Questions that were not scored were not includedin the DQ score– There is no “fail” score because DQ was not usedto evaluate interviewer performance The overall DQ score was 0.99414SQA and DQ ComparisonSQADQPurposeEvaluate each interviewerEvaluate data collectedNumber ScoredTwo per interviewer per dayAbout 5% of completed calls per dayScoring ElementsAt the module level throughout all At the question level within twomodulesmodulesScore ReportingReported to each interviewer andReported at an operational level toat an operational level; meant tothe central team to track generaldirectly impact each interviewer'strendsperformance155
7/21/2011Operational Utility Both operations allowed an enterprisewide monitoring of trending interviewissues– SQA scores allowed for some tracking ofinterviewer performance, and DQ scoresshowed how interviewer behaviors may havebeen affecting data– Job aids and extra training were releasedbecause of trends observed in SQA and DQ16Any Questions?Julia Coombsjulia.coombs@census.gov17Backup186
7/21/2011DQ ER(pn): Is there anyone I’ve mentioned that you don’t know?DWHODK: Who is the person(s) you don’t know?DROSTER(p1): Is your name correct?MISSBABY: I’d like to make sure we are not missing anyone who lived or stayed here at {filladdress} on {fill date}? Other than the people we’ve already mentioned, were there: Anynewborns or babies?MISSREL: Any other relatives who lived or stayed here?MISSOFTEN: Anyone else who stayed here often?FCOLYN: In Spring of 2010, was anyone attending college?COLNAME: Who was attending college?COLWHERE: Where did COLNAME stay while attending college?FOTH: In April or May, did {fill if person count 1 “you”, else “anyone”} stay somewhere else foran extended time or live part of the time at another residence?OTHNAME: Who was staying elsewhere for an extended time during April or May?MOSTTIME: In April or May, where did you live or stay most of the time?STAYAPR1: Were you staying at {fill ADDRESS} or at the other place on April 1, 2010?FGQYN: {Were you/Was Full Name} staying in any of those places on April 1, 2010?ADDAGE: What was {fill FULL NAME’s} age on April 1, 2010?197
Data Quality operation -Operational overview -Scoring criteria -Outcomes How the two operations relate 2 CFU Quality Operations During the 2010 CFU operation, two independent operations were implemented to evaluate the quality of the interviews and of the data being collected -The Service Quality Assurance (SQA)
