P-hacking: What It Is, How To Prevent It

1y ago

55 Views

1 Downloads

7.77 MB

50 Pages

Report/dmca

Download PDF

Transcription

2017-12-05p-hacking:What it is, how to prevent itThis presentation is licensed under a CC-BY 4.0 license.You may copy, distribute, and use the slides in your own work, as longas you give attribution to the original author at each slide that you use.PD Dr. Felix read303

Much of the scientific literature,perhaps half, may simply be untrue.Part of the problem is that no one isincentivised to be right.Richard Horton,Editor von The Lancet2

Researchers are not rewarded for being right,but rather for publishing a lot.Nelson, Simmons, & Simonsohn (2012); Nosek, Spies, Motyl (2012); Munafo (2016)3

How can we publish a lot?Psychology/Psychiatry92%!Fanelli, D. (2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. doi:10.1007/s11192-011-0494-74

p-hack your wayto scientific glory!

p-hacking (n.). Tune your data analysis in a way that youachieve a significant p-value in situations where it wouldhave been non-significant.Questionable research practices (QRPs) (n.).Practices of data collection and data analysis that are notoutright fraud, but also not really kosher.

Tool 1: Outcome switchinghttp://compare-trials.org/ 5KJM7

Tool 1: Outcome switching 2outcome variables:false positive rate 55% 9.5%outcome variables with one-sided testing:false positive rate5% 41%8

Tool 2: Many conditions, reportonly those that 258386051072Best-practiceexample:Transform aboringdissertation intoa groundbreakingpublication9

Tool 3: Optional stoppingWith long enoughsampling and optionalstopping, it isguaranteed to get asignificant result!100%Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A(General), 132, 235–244.10

Tool 4: Analytical flexibility11

http://crtt.flexiblemeasures.com by Malte Elson12

in psychology13

Pro-Tip: Build the p-hacking intothe software!14

How bad is it?http://shinyapps.org/apps/p-hacker/15

Kahneman: Open LetterI believe that you shouldcollectively do somethingabout this mess.I see a train wrecklooming.Daniel Kahneman, Nobel prize 2002http://www.nature.com/polopoly er.pdf16

Which part of published research can beindependently logy(2015; n 97)79%Economics*(2015; n 67)21%Cancer Research 1 Cancer research 2(2011; n 53)(2012; n 67)* The data on economics is about reproducibility; i.e. the attempt to get the same resultsif you apply the original data analysis on the original data set.Open Science Collaboration (2015); Chang & Li (2015); Begley, C. G., & Ellis, L. M. (2012). Prinz, F., Schlange, T., & Asadullah, K. (2011).17

90%: t-the-lid-on-reproducibility-1.1997018

Retractions: 1000% in 10 years„In the past decade, the number of retraction noticeshas shot up 10-fold.“6845004672013 hould-they-go-to-jail/19

Scientific misconduct: 1200% in 4 yearsU.S. Oﬃce of Research Integrity:Confirmed cases of scientific misconduct3632009-2011https://ori.hhs.gov/case summary2012-201520

„Innovative, unprecedented, transformative!“ 880% from 1974- 2014Groundbreaking!!!Amazing!!Enormous!!Vinkers, C. H., Tijdink, J. K., & Otte, W. M. (2015). Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014:retrospective analysis. Bmj, 351, h6467–6. http://doi.org/10.1136/bmj.h646721

Thesis:Anti-Thesis:Our current incentives fosterquestionable researchpractices, which decrease thetruth value of our sharedknowledge.Society pays for us that wegenerate valid and robustknowledge.What is good for theindividual careers ofresearchers leads to acollective fiasko.Our incentives should bechosen in a way that theyfoster good science.Researchers who do it right(i.e., high power, no QRPs,transparency) have a clearcompetitive disadvantage.Researchers who do it rightshould be supported andpromoted.22

How to prevent p-hacking1. Reduce analytical degrees of freedom2. Embrace analytical degrees of freedom3. Change the incentives

Registered Reportshttps://cos.io/rr/26

Registered Reportshttps://cos.io/rr/AIMS NeuroscienceAttention, Perception, and PsychophysicsCognition and EmotionComprehensive Results in Social PsychologyCortexDrug and Alcohol DependenceExperimental PsychologyJournal of Accounting ResearchJournal of Business and PsychologyJournal of Personnel PsychologyJournal of Media Psychologyoffer Registered ReportsNFS Journal(by December 2017)Perspectives on Psychological ScienceRoyal Society Open ScienceSocial PsychologyWorking, Aging and Retirement82 journals27

Slides by Anne Scheel

Where can I preregister? RPR "gold standard"internal only–better planning of not externallystudiesrecognisedonline repository–/ (depends)UPRpubliconline repository supportingpublic PR, e.g. OSF no scoopingcommentnot offered by everyjournalRegistered Reportsprivate!–potential option forstudents' projectsnot hack-safe(misconduct) ,not encourageddoesn't prevent filedrawer effectScooping possible?find(But: time-stampcollaborators, get proofs date offeedback,creation; scoopingfile-drawer safe prevented byembargo)embargo possible(OSF: 4 years max)all UPRs can be done with and without a template (exception: AsPredicted is template-only)Slides by Anne Scheel

no prereg:prereg:57%8%success rate!success rate http://chrisblattman.com/2016/03/01/13719/Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Eﬀects of Large NHLBI Clinical Trials Has Increased over Time. PLoS ONE, 10(8), e0132382–12. http:// 34doi.org/10.1371/journal.pone.0132382

How to prevent p-hacking1. Reduce analytical degrees of freedom2. Embrace analytical degrees of freedom3. Change the incentivesPrevent p-hacking byreducing the data analysis toa single pipeline.

How to prevent p-hacking1. Reduce analytical degrees of freedom2. Embrace analytical degrees of freedom3. Change the incentives

Many names for the same idea Sensitivity/robustness analysis Multiverseanalysis (Steegen et al., 2016) Specificationcurve (Simonsohn et al., 2015) Vibrationof effects (Patel et al., 2015) Ensembleapproach (e.g. climatology) use a set of models with the same input data to produce a range ofoutcomescf. Boulesteix, Elsas, & Schönbrodt (in prep)37

A “multiverse analysis” (Steegen, Tuerlinchx, Gelman, & Vanpaemel, 2016):Report results for all plausible analytical decisions Check robustness of results: Do several analytical paths lead tocomparable conclusions? Based on open data by Carney et al. (2010)38

Of 54 plausible analyses exactly one was significant.Guess which has been reported in the original paper?39

Open Letter by Dana Carney (2016)http://faculty.haas.berkeley.edu/dana carney/pdf my%20position%20on%20power%20poses.pdf40

Rohrer, J. M., Egloﬀ, B., & Schmukle, S. C. (in press). Probing birth-order eﬀects on narrow traits using Specification CurveAnalysis. Psychological Science. https://osf.io/vg2un/41

Rohrer, J. M., Egloﬀ, B., & Schmukle, S. C. (in press). Probing birth-order eﬀects on narrow traits using Specification CurveAnalysis. Psychological Science. https://osf.io/vg2un/42

Prerequisite: Open Data“dass die Daten unmittelbar nach Abschluss der Forschungen oder nachwenigen Monaten der Öﬀentlichkeit frei zur Verfügung gestellt werden.“„Das Engagement [ ] von Wissenschaftlern und Wissenschaftlerinnen um dieVerfügbarmachung von Forschungsdaten sollten bei der Würdigung vonwissenschaftlichen [ ] Leistungen zukünftig stärker berücksichtigt gramme/lis/ua inf empfehlungen 200901.pdf, ellung/forschungsdaten/richtlinien cuments/Empfehlungen/Datenmanagement deu.pdf, n/Data Management eng.pdf43

Journals with mandatory open data(or justification why not) Advancesin Methods and Practices in Psychological ations/ampps/ampps-submission-guidelines#DISC) Collabra: -integrity/) /doi/10.1027/1618-3169/a000355) Journalof Research in rticle/pii/S0092656617300211) Judgmentand Decision Making(http://journal.sjdm.org/) Journalof /editorialpolicies/) PLOSONE RoyalSociety Open alsocietypublishing.org/author-information#Open data) editorial-policies)44

How to prevent p-hacking1. Reduce analytical degrees of freedom2. Embrace analytical degrees of freedom3. Change the incentivesShow robustness againstp-hacking by computing all(sensible) analytical pipelines

How to prevent p-hacking1. Reduce analytical degrees of freedom2. Embrace analytical degrees of freedom3. Change the incentives

Proximate incentives Whydo we p-hack? To get a publishable result. Hypothesis: Publicationbias as the driving forcebehind p-hacking. Carter, Schönbrodt, Gervais, & Hilgard (2017): p-hacking (withoutpublication bias) has minor impact; but quite problematic incombination with publication bias (see https://psyarxiv.com/9h3nu/) Butalso see Simonsohn (2016): „P-hacking is easy to stop. File-drawering nearly impossible. Fortunately,while p-hacking is a real problem, file-drawering is not.“ http://datacolada.org/5547

Ultimate incentives The Department of Psychology at the Faculty of Human Sciences of the University ofCologne (UoC) seeks to appoint a Full Professor (W3)of Social Psychology 3 additional professorship jobto be filled as soon as possible.The successful candidate is expected to have a record of excellence in social cognition,TheDepartmentof Psychologyat theFaculty oforHumanSciencesof the University ofand/orrelated areassuch as cognitivepsychologymotivationscience.Cologne (UoC) seeks to appoint aThe candidate is also expected to strongly contribute to the UoC’s Center for Social andEconomic Behavior and the Social Cognition Center Cologne of the Department ofFull(W3) Area II, „Behavioral EconomicPsychology. Both structures arepart Professorof UoC’s Key ProfileEngineering and Social Cognition“.of Social PsychologyThecandidate’strack record should show an excellent fit with these interrelatedto be idealfilled assoon as possible.structures and a strong interest to bridge the fields of social cognition and behavioraleconomics.The successful candidate is expected to have a record of excellence in social cognition,and/or related areas such as cognitive psychology or motivation science.The Department of Psychology aims for transparent and reproducible research (includingOpenData, Openand Preregistrations).Applicantsaskedto illustratehow theyThe candidateis Materials,also expectedto strongly contributeto theareUoC’sCenterfor eyplantodosointhefuture.Economic Behavior and the Social Cognition Center Cologne of the Department ofPsychology. Both structures are part of UoC’s Key Profile Area II, „Behavioral EconomicWe strongly encourage international applicants. Salaries and working conditions at the UoC https://osf.io/dbkva/48

17 members of 10 disciplines:Psychology, sociology, computer science, statistics,geography, medicine, veterinary medicine, economics, 3 entire faculties as members:Faculty of Medicine, Faculty of Veterinary Medicine,Faculty of Psychology and Educational Science MissionStatement: Education(from PhD student to professor) Meta-science Changeresearchthe incentive structure http://www.osc.lmu.de49

Thanks for your attention!

p-hacking: What it is, how to prevent it 2017-12-05 This presentation is licensed under a CC-BY 4.0 license. You may copy, distribute, and use the slides in your own work, as long