WIXLIB

1White Paper: The Effects of OnlineDisinformation Detection Training forOlder AdultsRyan C. MooreJeffrey T. HancockStanford UniversityRecent research has identified olderadults as a demographic groupespecially susceptible to disinformationonline. During the 2016 U.S. presidentialcampaign, people 65 and older weretwice as likely to be exposed to fakenews on Twitter and seven times morelikely to share fake news on Facebookthan 18-29 year olds (Grinberg et al.,2019; Guess et al., 2019). One possibleexplanation for this phenomenon is thatolder adults are less digitally literatecompared to younger individuals(Brashier & Schacter, 2020). For thatreason, organizations have begun todevelop programs for older adults whichfocus on the skills and knowledgeneeded to identify disinformation online.Here, we report the results of ourcollaborationwithMediaWise,anon-profit journalism organization whoran such a course for older adults (aged50 ) in the lead up to the 2020 U.S.presidential election (Span, 2020).MethodsFrom September 24 - October 26, 2020,145 older adults recruited by MediaWisecompleted MediaWise’s 1-hour courseand both our pre- and post-coursesurveys ( M age 66.9, 69% female, 84.8% white). MediaWise’s course wasa one-hour, self-directed series ofinteractive modules which focused onhow to identify disinformation online.From October 1 - October 10, 227 olderadults recruited from online surveypurveyor Lucid also completed both ourpre and post surveys, without takingMediaWise’s course ( M age 63.3, 61% female, 89.2% white). The Lucidrespondents served as a control groupto compare to the MediaWise courseenrollee group.Of primary interest in our surveys was adeception detection task for newsheadlines. In both the pre and postsurveys, individuals were asked to ratethe veracity of 6 distinct headlines from1:definitely false - 7:definitely true. Eachsurvey contained 3 true and 3 falsenews headlines. Per survey, one false(true)headlinewasRepublican-congruent, one false (true)headlinewasconsistentwithDemocrat-congruent, and one false(true) headline was more non-partisan innature (see Table 1). All 12 headlineswere taken from articles or postsfact-checked by Snopes. After providingveracity judgements on the headlines ineach survey, respondents were asked ifthey did any research online to informtheir judgement of the veracity of eachheadline.In addition, we measured levels of skillon the concepts and skills taught in thecourse important for being able toidentify disinformation online. Weadopted a widely-used measure of

2internet skills to measure respondents’level of skill for these techniques andconcepts (Hargittai & Hsieh, 2011), inwhich individuals are asked to rate theirunderstanding of internet-related terms(e.g., firewall) from 1:no understanding 5:full understanding. We replacedHargittai & Hsieh’s terms with 6 termstaught in MediaWise’s course andimportant for detecting disinformationonline (lateral reading, click restraint,reverse image searching, Wikipediapage features, search engine keywordoptimization, search engine filters;Wineburg & McGrew, 2019). We alsomeasuredrespondents’politicalideology from 1:very liberal - 5:veryconservative.Preliminary ResultsWe preregistered our analyses on theOpenScienceFramework: 1https://osf.io/jv8hk?view only 51e12172bc8542ee86211d58ae150bdb .To evaluate individuals’ ability toaccurately judge the veracity of newsheadlines, we estimated mixed effectslogistic regression models in which theDV was a binary variable indicating if arespondents’ veracity judgement for aparticular headline was correct orincorrect. To create that DV, wedichotomizedrespondents’7-pointveracity judgements such that, for trueheadlines, ratings of 5-7 were coded “1”for correct and ratings of 1-3 werecoded “0” for incorrect. For false1Not all preregistered analyses are presentedhere in this short white paper.headlines, ratings of 1-3 were coded “1”while ratings of 5-7 were coded “0”.Neutral ratings of 4 were excluded fromanalysis. The IVs included a binaryindicatorvariableforwhetherjudgements came from the MediaWisecourse enrollee group or from thecontrol group, a binary indicator ofwhether judgements came from theposttest or pretest survey, and our keyvariable of interest was the interaction ofthose two variables. Subjects and eachof the news headlines were modeled asrandom effects (see Baayen et al.,2008).As can be seen in Table 2, we obtaineda positive and significant interactionbetween the course enrollee andposttest indicator variables for all newsheadlines, as well as both true and falseheadlines separately (All news: β 1.145, SE 0.165, p .01; True news:β 1.478, SE 0.236, p .01; Falsenews: β 0.941, SE 0.251, p .01).Thisinteractionsuggeststhatimprovement in the ability to accuratelyjudge the veracity of news headlinesamong the MediaWise enrollee groupfrom pretest to posttest was significantlygreater than the change in ability amongthe control group. Models 2, 4, and 6show that this effect holds even whencontrolling for respondents’ politicalideology. To better interpret thisinteraction, we created predictedprobability plots (see Figure 1).MediaWise course enrollees rose from apretest probability of accurately judgingthe veracity of a headline of 63% to a

3posttest probability of 87% (Figure 1,Among all news). The control group didnot experience a significant increase(pretest probability 57%; posttestprobability 59%).In addition to more accurately beingable to judge the veracity of newsheadlines, MediaWise enrollees alsobecame more likely to report doingresearch online to inform theirjudgement of the headlines in thesurvey. Pretest, the average rate ofresearching headlines prior to makingjudgements was 2.9% and posttest theaverage rate jumped to 69.6%. Becausewe do not observe this pattern in thecontrol group (their pretest averageresearching rate 2.7%; posttest rate 2%), it is unlikely this effect in theMediaWise enrollee group was drivenby priming in the survey instrument.Finally, the average level of skill on thesix terms important for tly among the MediaWisegroup from pretest to posttest, while nosignificant increase was observedamong the control group (see Figure 2).Significance/Next StepsThe findings reported here indicate thata relatively brief disinformation detectionprogram can significantly improve olderadults’ abilities to detect disinformation.It will be crucial to assess the scalabilityof this program and to test its effects onlarger, more representative samples ofolder adults. Next steps with the datawehavecollectedincludeunderstanding which changes in specificdisinformation detection skills frompretest to posttest best explainimprovements in the ability to detectdisinformation.

4Table 1: News headlines used in deception detection ublicannor DemocratcongruentTrue NewsFalse NewsDonald Trump Sends 10,000 to Hero BusDriver after Driver Saves Suicidal Womanfrom Jumping off BridgeBus driver Darnell Barton took a detour fromhis normal route in order to stop a woman fromjumping off an overpass. Trump said he wasmoved by the gesture and rewarded Bartonwith a 10,000 check.Joe Biden Waves at Empty Field as HeDeparts Plane in TampaThe 77-year-old was caught waving to anempty field after departing a plane inTampa on Tuesday. He was in Tampaholding a veterans roundtable.Child with Rare Ailment Rescued AboardTrump's Private JetDonald Trump's private jet carried a critically ill3-year-old Jewish boy from California to NewYork for medical treatment after commercialairlines refused to carry the boy.Biden Caught Using TeleprompterDuring InterviewDemocratic presidential nominee JoeBiden is caught red-handed using ateleprompter during a recent interviewwith Telemundo anchor Jose Diaz-Balart.Trump Administration Secretly WithheldMillions from FDNY 9/11 Health ProgramThe Trump administration has secretlysiphoned nearly 4 million away from aprogram that tracks and treats FDNYfirefighters and medics suffering from 9/11related illnesses.Semi Truck Bearing the Words "AllAboard the Trump Train" Crashes IntoOverpassThe truck wedged itself under theMamaroneck Avenue Overpass in NewYork on Monday.Republican Candidate Made UnbelievableStatement About RapeClayton Williams, former Republican candidatefor Texas Governor, likened rape to badweather, stating, "Rape is kinda like theweather. If it's inevitable, relax and enjoy it."Williams made the remark to reporters at hisranch in West Texas.Trump Refuses to Send Wildfire Aid toCA, Offers it to PutinDonald Trump has refused to providefederal assistance to California to helpthem fight against September 2020wildfires. In July, however, Trump offeredUS aid to Russian President VladimirPutin to help fight against fires that brokeout in Russia.Netflix Releases Film Poster PortrayingYoung Girls in Sexualized MannerThe movie, titled "Mignonnes", follows an11-year-old Senegalese Muslim girl who iscaught up in the clash of her traditional familyvalues and modern, internet culture.Dozens of Countries OrderedCOVID-19 Tests in 2018Data displayed on the World Bank'sWorld Integrated Trade Solution websitein September 2020 proved that dozens ofnations imported or exported COVID-19Test Kits in 2018 — more than a yearbefore the pandemic occurred.Walmart Customer’s Change DonatedWithout ConsentIn July, a Walmart customer in Massachusettshad the change from her purchase donated tocharity without her consent. The customer saysa Walmart cashier refused to give her changewith no advanced warning.Adolf Hitler Found to Have Inventedthe Inflatable Sex DollHitler's Borghild Project, which began inlate 1940, was a secretive attempt to stopthe spread of syphilis by providing Nazisoldiers with inflatable sex dolls.

5Table 2: Correctly judging news headline veracityNote: “MediaWise Course” variable: control group 0; MediaWise course enrolleegroup 1. “Posttest” variable: post survey 1; pre survey 0. “Political Ideology”variable: Very liberal 1; Liberal 2; Moderate 3; Conservative 4; Veryconservative 5. Regression coefficients are shown with standard errors inparentheses. Subjects and news headlines are modeled as random effects in all models(Baayen et al., 2008).

6Figure 1: Predicted probabilities of correctly judging news headline veracityNote: Estimated predicted probability of accurately judging the veracity of newsheadlines broken down by all news headlines, among only true news headlines, andamong only false news headlines (Table 2 Models 2, 4, and 6, respectively). Highervalues indicate greater predicted probability of correctly judging the veracity of a givenheadline. Bars are 95% confidence intervals.

7Figure 2: Levels of skill on techniques important for identifying disinformation onlineNote: Bars represent the average level of reported skill (ranging from 1-5) on each of 6skills important for identifying misinformation online that were taught in the course. Errorbars represent standard errors. “Before” indicates that the average level of skill wascomputed from pretest measurements and “After” indicates that the average level ofskill was computed from posttest measurements. Blue bars represent those in theMediaWise enrollee group and red bars represent those in the control group.