WIXLIB

Education in Medicine Journal 2017; 9(3): 89–102The major statistics used in Delphi studiesare measures of central tendency andlevel of dispersion (standard deviationand inter-quartile range) in order topresentinformationconcerningthecollective judgements of respondents (36).Generally, the uses of median and modeare favoured. In the literature, the use ofmedian score, based on Likert-type scale,is strongly favoured (37–39). One criterionrecommends that consensus is achieved byhaving 80% of subjects votes falling withintwo categories on a seven-point scale (32).Green (40) suggests that at least 70% ofDelphi subjects need to rate three or higheron a four point Likert-type scale (41, 42)and the median has to be at 3.25 or higher(42).Delphi Round OneThe questionnaires were emailed and mailto all ten experts together with an officialletter of invitation and feedback form. Inthe first round, respondents were given astructured questionnaire regarding item ofconcept mapping care plan and MCQ inDiabetic Mellitus subject. The researcherprovided a guideline for the expert’s panelregarding the scores to given. The totalscores are 100%. In the first round, theDelphi panels were provided with closedended, 5-point Likert scale questions inorder to elicit their level of agreement witha series of statements regarding the relativeimportance of concept mapping care planand MCQ that developed by researcher. Inround one, offers a suitable environmentto the experts to anonymously discussand express themselves. After receivingparticipants’ responses, researcher needsto convert the collected information into awell-structured questionnaire. Participantswere asked to rate the categorised responsesfrom Round 1 on a scale of 1 to 5, with 1 Very irrelevant; 2 Not relevant; 3 Less relevant; 4 Relevant; and 5 Veryrelevant). This questionnaire is used as thesurvey instrument for the second roundof data collection. It should be noted thatit is both an acceptable and a common92www.eduimed.commodification of the Delphi process format touse a structured questionnaire in Round 1that is based upon an extensive review of theliterature. Kerlinger (43) noted that the useof a modified Delphi process is appropriateif basic information concerning the targetissue is available and usable.The returns of the Round 1 questionnaireswere analysed. The return of the Round 2questionnaires was analysed by applyingSPSS version 23 for descriptive statistics.Basically, consensus on a topic can bedecided if a certain percentage of the votesfalls within a prescribed range (44). Themajor statistics used in Delphi studiesare measures of central tendency (means,median, and mode) and level of dispersion(standard deviation and inter-quartile range)in order to present information concerningthe collective judgements of respondents(36). Generally, the uses of median andmode are favoured.The degree of importance and consensusare justified after each Delphi round beforemaking interpretation. The group responsemedian value and the inter quartile rangedistribution are usually referred as thereference for the degree of importance andconsensus in the past research (45–47).For the example of this study, the analysisof consensus data of the experts was donebased on median, inter quartile range andquartile deviation on Round 1, 2 and 3 data.After the median value, inter quartile rangeand quartile deviations are identified, thesubsequent analysis technique is classifyingitems according to the consensus level andimportance level.Delphi Round TwoIn Round 2, again two weeks were givento the panel to respond. After a given adate, a few follow-up emails, messagesvia short messaging service (SMS) ortelephone calls was made. In Round 2,the question was modified based onthe experts’ panel comments. Resultson Round 1 also indicated that most ofexperts were give scores between four to

EDUCATIONAL RESOURCE The Delphi Techniquefive. Due to many valuable responses theformat of questionnaire was changed fromportrait to the landscape layout, so thateasier for the experts understanding, wherein this Round 2 the researcher add resultof min and median that got from respondRound 1. When get the feedback fromparticipants’, the researcher will combinetogether all the feedback in one summaryof the comments. In the second round,each Delphi participant received a secondquestionnaire and they were asked to reviewthe items summarised by the investigatorbased on the information provided in thefirst round. Accordingly, Delphi panellistsmay be required to rate or “rank-order itemsto establish preliminary priorities amongitems”. As a result of Round 2, “areas ofdisagreement and agreement are identified”(28). In some cases, Delphi panellists wereasked to state the rationale concerning ratingpriorities among items (39). In this round,consensus begins forming and the actualoutcomes can be presented among theparticipants’ responses (39).Delphi Round ThreeAs with Round 1 and Round 2, two weekswere given to the panel members to respondin Round 3. In the third round and oftenfinal round, each Delphi panellist receiveda questionnaire included the items andratings summarised by the researcher inthe previous round and were asked torevise his/her judgements in order to getthe consensus. Participants were askedto review their response, respond againusing the same rating scale, and add anycomments regarding the responses. Someof these comments have been cited in thetext and some others are presented in theDelphi technique Round 3. The survey wassuccessful in providing a general consensusregarding concept mapping care plan andMCQ.This round gives Delphi panellists anopportunity to make further clarificationson both the information and theirjudgements of the relative importanceof the items. The list of remaining items,their ratings, minority opinions, and itemsachieving consensus are distributed to thepanellists. In the third round, the expertscan retain their original answer as given inRound 2 where their answers are given asinterquartile ranges. Experts might changetheir answer in the third round if their initialresponses fell outside the interquartile rangeor the experts may choose to retain theiranswers that fall outside the interquartilerange, and give their reasons for retainingtheir answers. The third round is aimedat achieving consensus and narrowing therange of differences in opinion among theexperts. After the third round, the datawere analysed and the median as well asinterquartile range calculated. Findings fromthe Delphi third round were used to answerthe research question.The degree of importance and consensusare justified after each Delphi round beforemaking interpretation. In this study, theanalysis of consensus data of the expertswas done based on median, inter quartilerange and quartile deviation on round one,two and three data. After the median value,inter quartile range and quartile deviationsare identified, the subsequent analysistechnique is classifying items accordingto the consensus level and importancelevel. In this study, the consensus level isdivided into three levels (high, mediumand no consensus) and importance levelis divided into two levels (very high andlow). The consensus level was determinedas high if quartile deviation is less than orequal to 0.5, medium if quartile deviationis in between 0.5 and 1 and no consensusif quartile deviation is more than 1. Theimportance level are very high if the medianvalue was 4 and above and low if the medianvalue is less than 3.5.RESULT AND ANALYSISIn the validation process, the researcherusing three rounds Delphi techniques tovalidate and concept mapping care plan(9 items) and MCQ (20 items). In thiswww.eduimed.com93

Education in Medicine Journal 2017; 9(3): 89–102Delphi technique ten experts were invitedto participate. The experts were contactedthrough personal phone calls or email. Inthis study, the researcher used formulafrom Norizan (45) as a guideline to get theconsensus and importance of items (referTables 1 and 2). The value of inter quartilerange using the formula (Q3-Q1) weredetermined using Microsoft SPSS version23.0 and reported in the round threequestionnaire. The data from the roundthree were treated in a similar way. Theformula for identifying deviation (QD) is asfollows:Inter - quartile rangeFormula: QD 2 Q3 - Q1 h 2Table 1: Level of consensus and importanceQuartile deviation (QD)Level of consensusMedianLevel ofimportanceLess or equal to 0.5 (QD 0.5)High4 and above (M 4)HighMore than 0.5 and less than or equal to1.0 (0.5 QD 1.0)Moderate3.5 and less (M 3.5)LowMore than 1.0 (QD 1.0)Low and no consensus––Note: Formula by Norizan (45) on classifications of consensus was determined at three levels.Table 2: Description of the classificationsLevelDescriptionHigh importance – high consensusItems that achieved high consensus with QD value of less orequal to 0.5, but are regarded as important and very importantwith median of 4 and above [(QD 0.5) and (M 4)]High importance – moderateconsensusItems that achieved moderate consensus with QD value ofmore than 0.5 and less of equal to 1.0, but are regarded asimportant and very important with median 4 and above [(0.5 QD 1.0) and (M 4)]High importance – no consensusItems that did not achieve consensus with QD value of morethan 1.0, but are regarded as important and very importantwith median of 4 and above [(QD 1.0) and (M 4)]Low importance – high consensusItems that achieved high consensus with QD value of less orequal to 0.5, but are regarded as moderate and not importantwith median of 3.5 and less [(QD 0.5) and (M 3.5)]Low importance – moderateconsensusItems that achieved moderate consensus with QD value ofmore than 0.5 and less of equal to 1.0, but are regarded asmoderate and not important with median of 3.5 and less [(QD 0.5) and (M 3.5)]Low importance – no consensusItems that did not achieve consensus with QD value of morethan 1.0, but are regarded as moderate and not important withmedian of 3.5 and less [(QD 0.5) and (M 3.5)]Source: Adapted from Norizan (45).94www.eduimed.com

EDUCATIONAL RESOURCE The Delphi TechniqueTable 3: Consensus in concept mapping care plan through three round Delphi techniqueItemRound of DelphiRound 1Round 2Round 3MedianMeanQDMedianMeanQDMedianMeanQDStatement 15.04.70.55.04.70.55.04.70.5Statement 25.04.70.55.04.70.55.04.70.5Statement 35.05.005.04.70.55.04.70.5Statement 45.05.005.04.41.05.04.70.5Statement 55.05.005.04.41.05.04.70.5Statement 65.05.005.04.70.55.04.70.5Statement 75.04.70.55.04.11.55.04.70.5Statement 85.05.005.04.70.55.04.70.5Statement 95.05.005.04.70.55.04.70.5Table 4: Consensus in multiple choice questions (MCQ) through three round Delphi techniqueItemRound of DelphiRound 1Round 2Round 3MedianMeanQDMedianMeanQDMedianMeanQDStatement 14.03.52.04.04.01.05.04.60.5Statement 24.03.81.54.04.01.04.54.50.5Statement 34.03.81.54.04.30.54.54.50.5Statement 44.04.40.53.03.81.04.04.40.5Statement 54.04.11.05.04.41.05.04.60.5Statement 64.03.524.03.70.55.04.80.5Statement 74.03.81.53.03.40.54.54.70.5Statement 84.03.81.54.03.70.54.54.60.5Statement 94.04.11.05.04.41.05.04.70.5Statement 104.04.11.05.04.41.04.54.50.5Statement 114.04.004.04.40.55.04.80.5Statement 125.04.70.54.04.11.04.54.50.5Statement 133.03.30.55.04.70.54.04.40.5Statement 144.04.005.04.41.04.54.50.5Statement 153.03.30.54.04.40.54.04.30.5Statement 164.04.40.55.04.41.05.04.60.5Statement 174.03.52.04.03.41.04.04.11.0Statement 184.03.52.04.04.004.04.40.5Statement 194.03.81.54.03.70.54.54.50.5Statement 203.03.22.04.04.004.04.21.0www.eduimed.com95

Education in Medicine Journal 2017; 9(3): 89–10296Table 3 show that in first, second and thirdDelphi rounds, all the Quartile Deviation(QD) of the statements was less or equalto 0.5 (QD 0.5), it indicate that the levelof consensus was high. In others word allexpert panels’ responses lying into scale of5 (very relevant). The median scores wereused to analyse the level of consensus ofexperts and result shows that in all threerounds Delphi the medium result morethan 4. It reported that level of importanceof the statements were high. In conclusionthe concept mapping care plan reachingconsensus in Delphi technique and provideda reliable manner to conclude that ten ofexpert panels overall agreement upon thenine statements assumed.DISCUSSIONTable 4 show that in the first Delphiround, there are statements have low andno consensus which were Statements1, 2, 3, 6, 7, 8, 17, 18, 19 and 20. TheQuartile Deviation (QD) may be above 1.Respectively, there may be a case where theexperts panel responses lying into scale of 2(not relevant). However after the correctionwere made, they were denoting overallconsensus among ten statements. This inturn can conclude that ten of expert panelsoverall agreement upon the 20 statementsassumed. The median score was used toanalyse the level of consensus of experts. Infirst Delphi round, the statements was lessthan 3.5 were Statements 13, 15 and 20. InDelphi Round Two, after the modified wasdone based on the comments of experts,these statements got the medium result of3.5. However the statements of Item 4 and 7got the median less 3.5. In Delphi Round 3,all the statements got the value of medium4 and above, which reported that level ofimportance of the statements were high,except Statements 17 and 20, item thatachieved moderate consensus with QD valueequal to 1.0, and very important with thevalue of median was 4.After three rounds of Delphi, researcherfound that the concept mapping structureshows at agreed upon by all the expertpanels (ten experts) have get a positivecomment. The Delphi techniques resultshows that the Quartile Deviation (QD)of the statements was less or equal to 0.5(QD 0.5), it indicate that the level ofconsensus was high. In other word all expertpanel responses lying into scale of 5 (veryrelevant).www.eduimed.comConsensus in Concept Mapping Care Planthrough Three Rounds Delphi TechniqueThe Delphi techniques are widely used andaccepted method for gathering data fromparticipants within domain of expertise (2).The Delphi process can be continuouslyiterated until consensus is determined tohave been achieved (2). In the validationprocess, the researcher using three roundsDelphi techniques to validate conceptmapping structure (9 items). The threerounds survey took place between twomonths and was conducted via emails andmail.The development of concept mappingcare plan at clinical practices was used toevaluate the students academic achievementin clinical pratices. Concept mapping careplan is useful for students’ preparation forclinical practices. In nursing education,concept mapping has been used as ateaching strategy to provide students theopportunity to visualise and integratetheories with the nursing process. Thisconcept mapping care plan allowed thestudents to determine the patient problemsand interrelationships based on analysethe patient data and plan comprehensivenursing care. By using concept mappingcare plan, the students have a total viewof patient care, where patient’s medicalconditions and nursing education can bemade to be related. It promotes a holisticview of the patient care and preparestudent to think critically when the student

EDUCATIONAL RESOURCE The Delphi Techniqueunderstand the concept, relationship to planand evaluate nursing care. The studentswhose using concept mapping can seethe holistic view of the patients (48). Thisrepresentation allows the students to see apatient’s problem and visually connect withnursing interventions (49).Consensus in Multiple Choice Questions(MCQ) through Three Rounds DelphiTechniqueThe most common and frequently usedby the teacher-constructed tests to testingstudents’ knowledge was multiple choicequestions (MCQ) (50). In nursing researchand education, multiple choice questionsare used extensively and play a fundamentalrole in the design of research studies oreducational programs (51). In the contextof nursing research and/or education, theliterature review regarding format, structure,validity and reliability in MCQ is not manyand most of the current literature in thisarea is based on opinion or consensus.In the validation process, the researcherusing three rounds Delphi techniques tovalidate MCQ (20 items). After completedthree rounds of Delphi techniques, resultshows that Quartile Deviation (QD) of thestatements was less or equal to 0.5 (QD 0.5), except the Statements of 17 and 20, itindicate that the level of consensus was high.In other word all expert panel responseslying into scale of 5 (very relevant). Themedian score was used to analyse the levelof consensus of experts and result showsthat in all three rounds Delphi the mediumresult more than 4. It was reported that levelof importance of the statements were high.Similar finding in Miller (52) studyto ascertain the opinion of experts onindicators considered to measure themovement towards sustainable tourism.In these statements, experts were asked toprovide their opinion choosing a value from1 (strongly disagree) to 5 (strongly agree). Insuch a case, the consensus is proposed to beassessed using three measures combinatory:(1) The 51% of experts responding to thecategory ‘strongly’, (2) the interquartilerange below 1 (53, 54); and (3) the standarddeviation below 1.5 (55).MCQ must be developing base on orabove the students’ cognitive level. Theobjective of MCQ test is to enhance andpromote students’ critical thinking. For thatreason, the development of the questionsat analyse, synthesise and evaluation levels.The questions format cover must be analyse,synthesise and evaluate by the students.MCQ have long been applied in assessingthe student critical thinking ability usedto assess student academic performance,and to develop critical thinking ability (56).According to Morrison & Free (57) andYoungblood & Beitz (58), the students canmake a connection learning in theoreticalconcept in the classroom and the clinicalpractices if the MCQ questions relate to theclinical situations.CONCLUSIONThe Delphi technique becomes animportant in data collection methodologywith a wide variety of applications and usesfor people who want to gather informationof the study interest. Delphi technique wasused because this is the best method toobtain consensus in health care educationresearch. This technique uses a judgement toprove the accuracy of the content item whenevaluating instrument. However, subjectselection and the time frames for conductingand completing a Delphi study are two areaswhich should be considered carefully priorto initiating the study. As a conclusion, theconcept mapping care plan and MCQ havemeets the consensus by using three roundsof Delphi techniques. Concept mappingcare plan enhanced the knowledge andthe understanding of the nursing studentsand also improving the quality of clinicaleducation.www.eduimed.com97

Education in Medicine Journal 2017; 9(3): 89–10298www.eduimed.com

EDUCATIONAL RESOURCE The Delphi TechniqueREFERENCES1. Dalkey NC, Helmer O. An experimentalapplication of the Delphi method to theuse of experts. Management Science.1963;9(3):458–67. https://doi.org/10.1287/mnsc.9.3.458.2. Ab Latif R, Mohamed R, Dahlan A, MatNor MZ. Using Delphi technique: makingsense of consensus in concept mappingstructure and multiple choice questions(MCQ). Education in Medicine eimj.v8i3.421.3. Gordon TJ. The Delphi method. TheMillennium Project Futures ResearchMethodology

experts. This technique uses a judgement to prove the accuracy of the content item when evaluating instrument. The objective of this study is to discuss the process of the three rounds Delphi technique to obtain consensus of concept mapping care plan and multiple choice questions (MCQ) in Diabetic Mellitus subject.