Transcription
Project EpsomManagement Summary Paper
This management summary contains somecontent which was originally presented inProfessor Peter Saville’s keynote speeches at:The British Psychological Society Division of OccupationalPsychology Conference, Stratford-upon-Avon, January2008: “Personality Questionnaires – Valid Inferences,False Prophecies”The Psychological Society of South Africa AnnualConference, Johannesburg, August 2008: “Does Your TestWork?”A&DC Conference, Institute of Directors, London,November 2008:“A Comparison of Leadership in Business and EliteAthletes”1 2017 Saville Assessment, a Willis Towers Watson company. All rights reserved
Project Epsom: How Valid is Your Questionnaire?Management Summary: A New Comparative Study ofthe Major Personality Questionnaires in Predicting JobPerformanceProfessor Peter Saville, Rab MacIver, Dr Rainer Kurz & Tom HoptonOverviewThe Study 2The Saville Consulting Wave Professional Stylesquestionnaire was the most valid assessment formeasuring work performance, in a major study of308 participants who completed a range of popularpersonality questionnairesQuestionnaires were compared against the sameindependent work performance criteria. These were: A global performance measure of AccomplishingObjectives, Applying Specialist Knowledge andDemonstrating Potential The SHL Great Eight work competenciesWave Focus Styles and the newly-developed SavillePersonality Questionnaire (Saville PQ ), comparablewith the Occupational Personality Questionnaire(OPQ ), were found to be at least as valid, if not moreso, than the OPQ32i Compared to the OPQ32i, the Saville PQ uses moremodern technology, takes about 13 rather than 60minutes to complete, gives both normative and ipsativescores and measures both workplace motives andtalentsMeasures of participants’ work performance wereestablished by asking third-parties to independentlyrate how effectively the participants performed in thework competencies covered by the Great Eight andglobal performance criteria The more accurately a personality questionnairepredicts how independent raters have judged thework performance of the participant in a completelyseparate rating form, the more valid the personalityquestionnaire The most important aspect of a questionnaire is itsvalidity for measuring and predicting work performance Validity is shown as a correlation coefficient between -1and 1. A validity of 0 indicates chance measurementand is as effective as flipping a coin to predict aperson’s job performance. A validity of 1 is perfectmeasurement of job performance but is impossibleto achieve in practice. A validity of 0.3, althoughseemingly low, can offer high financial gain and can cutselection errors by half or more Studies generally indicate that a good personalityquestionnaire can have a validity of 0.3. Validitiesabove 0.7 for any technique are virtually unknown inthe literatureUsing more valid questionnaires can dramaticallyreduce selection errors with great impact on companyprofitabilityProject Epsom
Higher validities significantly reduce selection errors Ability tests have validities of around 0.5; unstructured interviewsaround 0.2. Educational qualifications are surprisingly poorpredictors of performance, at around 0.1 For the purposes of this summary, we report validity results fromseven key questionnaires: Saville Consulting Wave Professional Styles (WaveProfessional) Saville Consulting Wave Focus Styles (Wave Focus) Saville Personality Questionnaire (Saville PQ) Occupational Personality Questionnaire (OPQ32i) 16 Personality Factor Questionnaire (16PF5) Hogan Personality Inventory (Hogan PI) NEO Personality Inventory (NEO) These seven key questionnaires were compared against the externalratings of global work performance They were then compared against the external ratings of the SHLGreat Eight work performance competencies3 2017 Saville Assessment, a Willis Towers Watson company. All rights reserved
Figure 1: The validity of seven key questionnaires in measuring totaljob performance.Matched Sample of N 308 against external ratings on Global Performance(Applying Specialist Expertise, Accomplishing Objectives, Demonstrating Potential)Results.6 4With the DISC questionnaire, no evidence could befound for the validities of 0.75-0.95 which wereclaimed on the Thomas International WebsiteThe Hogan Development Survey, which assessesthe “Dark Side” of personality, failed to relatepositively or negatively to work performance.Only 8% of participants retained their exact DarkSide profile when they re-completed the samequestionnaire after just one weekProject EpsomWaveProfessionalWave FocusNEOSaville PQHogan PI16PFOPQ32i.0Figure 2: The average validity of seven key questionnaires inmeasuring the Great Eight competencies.Validity - Average CompetencyThe average validity in predicting SHL’s Great Eight. Matched Sample of ssionalWave FocusSaville PQ.0NEO.1OPQ32iChanceFigure 3: The power of seven key questionnaires in terms of theirdelivery of validity in 15 minutes.Valid Power - Validity per 15minsMatched Sample of 308.6.5.4.3.2.1.0Wave Focus .1Saville PQOther questionnaires in Project Epsom are the subjectof future reports, academic papers and presentations,but briefly:ChanceWaveProfessional The Wave Focus Styles and Saville PQ questionnairesare the most powerful, offering high validity in theshortest completion times.216PF Power” relates to measuring effectiveness oroutput in a given unit of time. In terms of personalityquestionnaires, it can relate to the questionnaire whichprovides the greatest validity per unit of time. Figure 3compares the power of the tests in terms of how muchvalidity can be achieved by each in 15 minutes.3Hogan PI Saville Consulting questionnaires are strong in termsof validity compared with the OPQ, even against SHL’sown model of workModerateValidityHogan PI The Saville Consulting questionnaires are the mostvalid questionnaires for measuring work performancein Project Epsom, even when validity is defined bythe independent SHL Great Eight measures of workperformance.4NEO Wave Focus Styles and the Saville PQ take under 15minutes each to complete yet compare favourably interms of validity with much longer questionnaires suchas the OPQ32i, the Hogan Personality Inventory and16PF5.516PF Saville Consulting Wave Professional Stylescomprehensively outperforms all other questionnairesin terms of validityHighValidityOPQ32i All of the seven questionnaires show at leasta moderate level of validity in predicting workperformance according to the global work performancecriteriaValid Power
Increasing Validity IncreasesReturn on InvestmentThe Full Project Epsom PhaseOne Paper Questionnaires of high validity increase the chanceof selecting better work performers; considerablyreducing selection errors An example of a serious selection error is selecting acandidate from the bottom 20% of performers whenyou mean to select from the top 20% of performersThis document offers a brief management summary ofthe major Saville Consulting research programme, ProjectEpsom. A full version of this research paper is availableonline and in hard-copy, and further details can be found atwww.savilleassessment.com, or by calling 44(0)20 8619 9000. If a questionnaire has a validity of 0.3, one personin every 10 that you select will prove to be in thebottom 20% of performersReferences to accompany this Management Summaryare in the full version of this paper, “How Valid is YourQuestionnaire?”, available from www.savilleassessment.com. If a questionnaire has a validity of 0.6, one personin every 50 that you select will prove to be in thebottom 20% of performersCopyright PermissionsMoving from recruiting using a questionnaire with avalidity of 0.2 to using a questionnaire with a validity of 0.4 can double the cost-benefit to an organisationIt can reduce the number of serious selection errorsfive-fold, remarkably improving the accuracy of theselection processConclusions The Saville Consulting questionnaires maximise themeasurement of performance at work, assessing workspecific competencies which are relevant to modernbusiness lifeSome other questionnaires, perhaps developed half acentury ago measure out-of-date aspects of work performance have work-irrelevant clinical scales of poor validity use obscure and badly-written questions Cost savings from using better tests can be “startlinglylarge”. Neither employer nor employee gains from poorselection When the participants were followed up after sixmonths, essentially the same questionnaire rank orderof validity was maintained with Saville ConsultingProfessional Styles and Focus Styles outperforming theother questionnaires5 2017 Saville Assessment, a Willis Towers Watson company. All rights reserved 2017 Saville Assessment, a Willis Towers Watsoncompany. All rights reserved.This summary is available in hard-copy format and online atwww.savilleassessment.com. The authors give permissionto copy, print, or distribute this research paper providedthat:1.each copy makes clear who the research paper’sauthors are;2.each copy acknowledges that this paper was producedthrough research endeavour by Saville Assessment;3.copies are not altered without the consent of the seniorauthor.
6Project Epsom
About Saville Assessment, a Willis Towers WatsonCompanyOur mission is to transform assessment around the world. We enable organisationsto identify potential, accelerate performance, and achieve outstanding results. Ourportfolio of leading-edge assessments are designed based on extensive researchinto successful workplace performance and the critical relationship between motive,talent and workplace culture. With representatives in over 80 countries we aretransforming how organisations Hire, Build and Lead talent globally. Learn more atwww.savilleassessment.comCopyright 2017 Saville Assessment, a Willis Towers Watson Company.All rights reserved.savilleassessment.com
16PF Hogan PI The average validity in predicting SHL's Great Eight. Matched Sample of 308. Validity - Average Competency.0.1.2.3.4.5.6 OPQ32i NEO Wave Professional Saville PQ Wave Focus Valid Power Hogan PI 16PF Matched Sample of 308. Valid Power - Validity per 15mins Figure 1: The validity of seven key questionnaires in measuring total job .
