WIXLIB

REPLICATION IN PARAPSYCHOLOGYOne of the first American researchers touse statistical methods in parapsychologywasJohnEdgar Coover,who was the Thomas WeltonStanfordPsychicalResearchFellow in the PsychologyDepartmentat StanfordUniversityfrom1912to 1937 (Dommeyer,1975). In 1917, Coover published a large volume summarizinghis work(Coover, 1917). Coover believed that his resultswere consistentwith chance,but othershave argued that Coover's definitionof significancewastoo strict(Dommeyer,1975). For example,in oneevaluation of his telepathyexperiments,Cooverfounda two-tailedp-valueof0.0062. He concluded,"Since this value, then, lies within the field ofchance deviation,althoughthe probabilityof itsoccurrenceby chance is fairlylow, it cannot beaccepted as a decisive indicationof some causebeyondchancewhichoperatedin favorofsuccessinguessing" (Coover, 1917, page 82). On the nextpage, he made it explicitthat he would requireap-value of 0.0000221 to declare that somethingotherthan chancewas operating.It was during the summerof 1930, with theexperimentsof J. B. Rhine at Dukecard-guessingthatparapsychologybegan to take holdUniversity,as a laboratoryscience. Rhine's laboratorystillexists under the name of the FoundationforResearchon the NatureofMan, housedat the edge ofthe Duke Universitycampus.It wasn't long after Rhine published his firstbook, Extrasensory Perception in 1934, that thebegan. Since his claimsattackson his methodologywere whollybased on statisticalanalyses of histhe statisticalmethodswere closelyexperiments,scrutinizedby criticsanxiousto finda he most persistentcritic was a psychologistfromMcGill Universitynamed Chester Kellogg(Mauskopfand McVaugh, 1979). Kellogg's mainargumentwas that Rhine was using the binomialdistribution(and normal approximation)on a seThe expeririesoftrialsthatwerenotindependent.ments in question consistedof having a subjectguessthe orderofa deckof25 cards,withfiveeachoffivesymbols,so technicallyKelloggwas correct.By 1937, several mathematiciansand statisticians had come to Rhine's aid. MauskopfandMcVaugh(1979) speculatedthatsincestatisticswasitselfa youngdiscipline,"a numberofstatisticianswere equally outraged by Kellogg, whose argumentsthey saw as discreditingtheir profession"(page 258). The major technicalwork,which acwereaccurateknowledgedthatKellogg'scriticismsbut did little to change the significanceof theresults, was conductedby Charles Stuart andJosephA. Greenwoodand publishedin the first(Stuartvolume of the Journalof Parapsychology365and Greenwood,1937). Stuart,who had been anundergraduatein mathematicsat Duke, was one ofRhine's early subjectsand continuedto workwithhim as a researcheruntil Stuart's death in 1947.whoapparGreenwoodwas a Duke mathematician,entlyconvertedto a statisticianat the urgingofRhine.Anotherprominentfigurewho was distressedwith Kellogg's attack was E. V. Huntington,amathematicianat Harvard. After correspondingwith sethe publicwitha technicalreplytoKellogg'sarguments,a simplestatementshouldbemade to the effectthat the mathematicalissues inRhine's workhad been resolved.Huntingtonmusthave successfullyconvincedhis formerstudent,Burton Camp of Wesleyan,that this was a wiseapproach.Camp was the 1937 Presidentof IMS.Whenthe annual meetingswere held in Decemberof 1937 (jointly with AMS and AAAS), Campreleaseda statementto the pressthatread:Dr. Rhine's investigationshave two aspects:experimentaland statistical. On the experimental side mathematicians, of course,have nothingto say. On the statisticalside,however, recent mathematical work hasestablishedthe fact that, assuming that theexperimentshave been properlyperformed,the statisticalanalysis is essentiallyvalid. Ifthe Rhineinvestigationis to be fairlyattacked,it mustbe on otherthanmathematicalgrounds[Camp,1937].One statisticianwho did emergeas a criticwasWilliamFeller. In a talk at the Duke Mathematical Seminaron April 24, 1940, Feller raised threecriticismsto Rhine'swork(Feller, 1940). Theyhadbeen raised beforeby others(and continueto beraised even today). The firstwas that inadequateof the cards resultedin additionalinforshufflingmationfromone seriesto the next.The secondwaswhat is now known as the "file-drawereffect,"namely,that if one combinesthe results of published studies only,there is sure to be a bias infavorof successfulstudies.The thirdwas that theresultswere enhancedby the use of optionalstopthe numberoftrialsping,that is, by notspecifyingin advance. All threeof these criticismswere addressed in a rejoinderby Greenwoodand Stuart(1940), but Feller was neverconvinced.Even in itsthirdeditionpublishedin 1968,his book An Introduction to Probability Theory and Its Applicationsstill containshis conclusionabout GreenwoodandStuart: "Both their arithmeticand their experimentshave a distincttinge of the supernatural"(Feller,1968,page 407). In his discussionofFeller'sposition, Diaconis (1978) remarked, "I believeThis content downloaded from 128.114.163.7 on Fri, 1 Aug 2014 23:40:46 PMAll use subject to JSTOR Terms and Conditions

366J.UTTSFeller was confused. he seemedto have decidedthe oppositionwas wrongand thatwas that."Several statisticianshave contributedto theliteraturein parapsychologyto greateror lesserdegrees. T. N. E. Greville developed applicablestatisticalmethodsformanyofthe experimentsinand was StatisticalEditor of theparapsychologyJournal of Parapsychology(with J. A. Greenwood)fromits startin 1937 throughVolume31 in 1967;Fisher (1924, 1929) addressedsome specificprobWilks(1965a,b)lemsin card-guessingexperiments;describedvarious statisticalmethodsforparapsychology;Lindley(1957) presenteda Bayesian analysis of some parapsychologydata; and Diaconis(1978) pointedout someproblemswithcertainexperimentsand presenteda methodfor analyzingwhenfeedbackis given.experimentsOccasionally,attacks on parapsychologyhaveintakentheformofattackson statisticalinferencegeneral, at least as it is applied to real data.Spencer-Brown(1957) attemptedto showthat truerandomnessis impossible,at least in finite sequences,and thatthis couldbe the explanationforthe resultsin parapsychology.That argumentreemergedin a recentdebate on the role ofrandomness in parapsychology,initiatedby psychologistJ.BarnardGilmore(Gilmore,1989, 1990; Utts,1989;Palmer,1989, 1990). Gilmorestatedthat "The agnostic statistician,advising on research in psi,shouldtake accountof the possibleinappropriateness of classical inferentialstatistics"(1989, page338). In his studies showingpurportedlytems that do not behave as they should underrandomness (e.g., Iversen, Longcor, Mosteller,Gilbert and Youtz, 1971; Spencer-Brown,1957).Gilmore concluded that "Anomalous data.should not be found nearly so oftenif classicalstatisticsoffersa valid model of reality" (1990,page 54), thusrejectingthe use ofclassical statistiforreal-worldcal inferenceapplicationsin general.3. REPLICATIONImplicitand explicitin the literatureon parapsychologyis the assumptionthat, in orderto trulyestablish itself,the field needs to find a repeatable experiment.For example, Diaconis (1978)startedthe summaryofhis articlein Science withthe words "In search of repeatable ESP experiments,moderninvestigators." (page 131). OnOctober28-29, 1983, the 32nd InternationalConFoundationwas heldferenceofthe Parapsychologyin San Antonio,Texas, to address"The Repeatability Problemin Parapsychology."The ConferenceProceedings(Shapin and Coly, 1985) reflectthediverseviews amongparapsychologistson the nature of the problem.Honorton(1985a) and on is uncommonin most branchesofscienceandshould not be singledout asthat parapsychologyunique in this regard. Other authors expressedin the lack of a single repeatabledisappointmentexperimentin parapsychology,with titles suchas "Unrepeatability:Parapsychology'sOnly Finding" (Blackmore,1985), and "Research Strategiesfor Dealing with Unstable Phenomena" (Beloff,1985).It has never been clear, however,just exactlywhat would constituteacceptableevidenceofa reIn the earlydaysofinvestigapeatableexperiment.tion, the major critics"insisted that it would besufficientforRhine and Soal to convincethem ofESP if a parapsychologistcould performsuccessfullya single 'fraud-proof'experiment"(Hyman,1985a, page 71). However,as soonas well-designedexperiments showing statistical significanceemerged,the criticsrealized that a single experiment could be statisticallysignificantjust bychance.BritishpsychologistC. E. M. Hansel quantifiedthe new expectation,that the experimentshouldbe repeateda fewtimes,as follows:If a result is significantat the .01 level andthisresultis notdue to chancebut to information reachingthe subject,it may be expectedthat by makingtwo furthersets of trials theantichanceodds of one hundredto one will beincreasedto arounda millionto one, thus enabling the effectsof ESP-or whateveris responsibleforthe originalresult-to manifestitselfto such an extentthattherewill be littledoubt that the result is not due to chance[Hansel,1980,page 298].In onvinceHanselthat somethingpcotherthan chancewas at work.This argumentimpliesthatifa particularexperimentproducesa ationsfail to attain significance,thentheoriginalresultwas probablydue to chance,or at least remainsunconvincing.The problemwiththis line of reasoningis that there is no consideration given to sample size or power. Only anexperimentwith extremelyhigh power shouldbe expected to be "successful" three times insuccession.It is perhaps a failure of the way statisticsistaughtthat manyscientistsdo not understandtheimportanceofpowerin definingsuccessfulreplication.To illustratethispoint,psychologistsTverskyand Kahnemann(1982) distributeda questionnaireThis content downloaded from 128.114.163.7 on Fri, 1 Aug 2014 23:40:46 PMAll use subject to JSTOR Terms and Conditions

REPLICATION IN PARAPSYCHOLOGYto theircolleaguesat a professionalmeeting,withthe question:has reporteda resultthatyouAn investigatorconsiderimplausible.He ran 15 subjects,andvalue, t 2.46. Anotherreporteda significanthas attemptedto duplicatehis proinvestigatorvaluecedure,and he obtaineda nonsignificantof t with the same numberof subjects.Thedirectionwas the same in both sets of data.You are reviewingthe literature.What is thehighestvalue of t in the secondset ofdata thatyou would describeas a failureto replicate?[1982,page 28].In reportingtheir results, Tversky and Kahnemannstated:The majorityof our respondentsregardedt 1.70 as a failureto replicate.If the data oftwosuch studies (t 2.46 and t 1.70) are pooled,the value of t forthe combineddata is about3.00 (assumingequal variances).Thus,we arefaced with a paradoxical state of affairs,inwhichthe same data that would increaseourin the findingwhenviewedas partconfidenceof the original study, shake our confidencewhen viewed as an independentstudy[1982,page 28].At a recentpresentationto the Historyand Philosophyof Science Seminar at the UniversityofCaliforniaat Davis, I asked the followingquestion.Two scientists,ProfessorsA and B, each have atheorytheywouldlike to demonstrate.Each plansto run a fixednumberofBernoullitrials and thenAtest Ho: p 0.25 versilsHa: p 0.25. Professorhas access to large numbers of students eachsemesterto use as subjects.In his firstexperiment,he runs 100 subjects,and there are 33 successes(p 0.04, one-tailed).Knowingthe importanceofA runsan additional100 subProfessorreplication,jects as a secondexperiment.He finds36 successes(p 0.009, one-tailed).ProfessorB only teaches small classes. Eachon her studentstoquarter,she runsan experimenttest her theory.She carries out ten studies thisway,withthe resultsin Table 1.I asked the audience by a show of hands toindicatewhetheror nottheyfeltthe ccessfullyA's theoryreceived overwhelmingsupport,with20 votes,while ProfessorB's theoryapproximatelyreceivedonlyone vote.If you aggregatethe resultsof the experimentsforeach professor,you will noticethat each conducted200 trials,and ProfessorB actuallydemonstrateda higherlevel ofsuccessthan ProfessorA,367with 71 as opposed to 69 successfultrials. Theone-tailed p-values for the combinedtrials are0.0017 forProfessorA and 0.0006 forProfessorB.To address the questionof replicationmoreexplicitly,I also posed the followingscenario. InDecemberof 1987, it was decidedto prematurelyterminatea studyon the effectsofaspirinin reducingheartattacksbecause the data wereso convincing (see, e.g., Greenhouseand Greenhouse,1988;Rosenthal,1990a). The physician-subjectshad beenrandomlyassigned to take aspirin or a placebo.There were 104 heart attacks among the 11,037subjectsin the aspiringroup,and 189 heartattacksamong the 11,034 subjects in the placebo group(chi-square 25.01, p 0.00001).Aftershowingthe results of that study,I presentedthe audience with two hypotheticalexperiments conductedto try to replicate the originalresult,withoutcomesin Table 2.I asked the audienceto indicatewhichone theythoughtwas a moresuccessfulreplication.The audiencechosethe secondone, as wouldmostjournaleditors,because of the "significantp-value." Infact,the firstreplicationhas almost exactlytheofheartattacksin the two groupssame proportionas the originalstudyand is thus a veryclose replication of that result. The second replicationhasTABLE potheticalreplicationsof the aspirin/ heartattack areYesNo1119115610902.596, p 0.11This content downloaded from 128.114.163.7 on Fri, 1 Aug 2014 23:40:46 PMAll use subject to JSTOR Terms and 013.206, p 0.0003

368J.UTTSand in factthe relativeproportions,verydifferentriskfromthe secondstudyis noteven containedinintervalforrelativeriskfromthea 95% confidenceoriginal study. The magnitudeof the effecthasbeen much more closelymatchedby the stsare beginningto noastice that replicationis not as straightforwardtheywereoriginallyled to believe. A special issueof the Journal of Social Behavior and Personalitywas entirelydevotedto the questionofreplication(Neuliep, 1990). In one of the articles,Rosenthalcautionedhis colleagues: "Given the levels of statistical power at which we normallyoperate,weofsignificanthave no rightto expectthe proportionresultsthat we typicallydo expect,even if in naeffect"turethereis a veryreal and veryimportant(Rosenthal,1990b,page 16).Jacob Cohen, in his insightfularticle titled"Things I Have Learned (So Far)," identifiedanothermisconceptioncommonamong social scientists: "Despite widespreadmisconceptionsto thecontrary,the rejectionof a given null hypothesisthatgivesus no basis forestimatingtheprobabilitya replicationof the researchwill again result inrejectingthat null hypothesis"(Cohen,1990, page1307).Cohen and Rosenthalboth advocate the use ofeffectsizes as opposedto significancelevels whendefiningthe strengthofan experimentaleffect.Ingeneral,effectsizes measurethe amountby whichthe data deviatefromthe null hypothesisin termsof standardizedunits. For instance,the effectsizefora two-samplet-testis usually definedto be thein the two means, dividedby the standifferencedard deviationforthe controlgroup.This measurecan be comparedacross studieswithoutthe dependence on sample size inherentin significancelevels. (Of coursetherewill still be variabilityin theofsamsampleeffectsizes,decreasingas a functionple size.) Comparisonofeffectsizes acrossstudiesisofmeta-analysis.one ofthe majorcomponentsSimilar argumentshave recentlybeen made inthe medicalliterature.For example,GardnerandAltman(1986) statedthat the use of p-values "todefinetwo alternativeoutcomes-significantandnot significant-isnothelpfuland encourageslazythinking"(page 746). They advocatedthe use ofintervalsinstead.confidenceAs discussedin the next section,the argumentshas failedtoused to concludethat parapsychologya replicableeffecthingeon these misdemonstrateconceptionsof replicationand failureto examinepower.A moreappropriateanalysiswouldcomparethe effectsizes forsimilarexperimentsacross experimentersand across time to see if there havebeen consistenteffectsof the same magnitude.Rosenthalalso advocatesthis view ofreplication:The traditionalview of replicationfocusesonsignificancelevel as the relevant summarystatisticofa studyand evaluatesthe successofa replicationin a dichotomousfashion. Thenewer,moreusefulview of replicationfocuseson effectsize as the moreimportantsummarystatisticofa studyand evaluates the successofa replicationnot in a dichotomousbut in acontinuousfashion[Rosenthal,1990b,page 28].The dichotomousview of replicationhas beenbythe historyof parapsychology,used throughoutand critics(Utts,1988). Forbothparapsychologistsexample,the National AcademyofSciencesreportbutcriticallyevaluated ed"nonsignificant"In the nextthreesections,we will examinesomeusingthe broader,ofthe resultsin parapsychologyofreplication.In doingmoreappropriatedefinitionso, we will show that the results are far morethan the criticswouldhave us believe.interesting4. THE GANZFELD DEBATE INPARAPSYCHOLOGYAn extensivedebate tookplace in the mid-1980sand critic,questioningbetweena parapsychologistwhetheror nota particularbodyofparapsychologipsi abilities.The experical data had demonstratedments in question were all conductedusing theganzfeldsetting(describedbelow). Several authorson the debate.were invitedto writecommentariesAs a result,this data base has been more thoroughly analyzed by both critics and proponentsthan any other and provides a good source forstudyingreplicationin parapsychology.The debate concludedwith a detailed series ofand ornotwhetherofthepsiquestionopenhad been demonstrated.A new series of experiments that followedthe recommendationswereconductedover the next few years. The resultsofwill be presentedin Section5.the newexperiments4.1 Free-Response ExperimentsRecent experimentsin parapsychologytend touse more complextargetmaterial than the cardspartiallyand dice used in the tiallyresemblethe conditionsof spontaneouspsi occurrences"(Burdickand Kelly,1977,page 109). Theseexperimentsfall under the general heading ofbecause the subjectisexperiments,"free-response"asked to give a verbalor writtendescriptionoftheThis content downloaded from 128.114.163.7 on Fri, 1 Aug 2014 23:40:46 PMAll use subject to JSTOR Terms and Conditions

REPLICATION IN PARAPSYCHOLOGYtarget,ratherthan being forcedto make a choicefroma small discreteset of possibilities.Varioustypesoftargetmaterialhave been used, includingpictures,shortsegmentsof movieson video tapes,actual locationsand small objects.Despite the more complextarget material,thestatisticalmethodsused to analyze these experimentsare similarto thoseforforced-choiceexperiments.A typicalexperimentproceedsas follows.Beforeconductingany trials,a large pool ofpotential targetsis assembled,usuallyin packetsoffour.Similarityof targetswithina packet is kept to aminimum,for reasons made clear below. At thestartofan experimentalsession,afterthe subjectissequesteredin an isolatedroom,a targetis selectedat randomfromthe pool. A sender is placed inanotherroomwiththe target.The subjectis askedto providea verbal or writtendescriptionof whathe or she thinksis in the target,knowingonlythatit is a photograph,an object,etc.Afterthe subject'sdescriptionhas been recordedand securedagainst the potentialforlater alteration,a judge (whomay or may notbe the subject)is givena copyofthe subject'sdescriptionand thefourpossibletargetsthat were in the packetwiththe correcttarget. A properlyconductedexperimenteitheruses video tapes or has two identicalsets of targetmaterial and uses the duplicatesetforthis part of the process,to ensure that cluessuch as fingerprintsdon't give away the answer.Based on the subject'sdescription,and ofcourseona blindbasis, thejudge is asked to eitherrank thefourchoicesfrommostto lea

controversies in parapsychology have focused on statistical issues, and statistical models have played an integral role in the experimental work. Recently, parapsychologists have been using meta-analysis as a tool for synthesizing large bodies of work. This paper presents an overview of the use of statistics in parapsychology and offers a summary