WIXLIB

r e s u lt sI. Supplying computers and internet alone generally do notimprove students’ academic outcomes, but do increasecomputer usage and improve computer proficiency.Disparities in access to information and communicationtechnologies can exacerbate existing educational inequalities.Students without access at school or at home may struggleto complete web-based assignments and may have a hardtime developing digital literacy skills. Ever since technology’sincorporation in the classroom took off during the 1990s,governments and other stakeholders have invested heavily intechnology distribution and subsidy initiatives to expand access.8At the same time, increasing access to technology may haveadverse impacts on academic achievement, for example ifstudents end up using technology only for recreational purposes.When it comes to academic achievement, computerdistribution and internet subsidy programs generally didnot improve grades and test scores at the K-12 level. In theUnited States, the Netherlands, and Romania, distributingfree computers to primary and secondary students did notimprove—and sometimes harmed—test scores.9 In studiesthat found negative results, researchers find suggestiveevidence that family rules regarding computer use andhomework appear to mitigate some of the negative effects.10Experimental studies conducted in developing countries have,for the most part, come up with similar results.11 However,one program in China that combined computer distributionwith educational software boosted test scores, suggestingdistributing hardware while sharing specific learning toolsmay be a promising approach.12At the postsecondary level, computer distribution programsappear to be more promising, although evidence comes mainlyfrom one randomized evaluation at a community college.Distributing laptops to low-income students at a northernCalifornia community college had modest but positive effectson passing rates, graduation rates, and likelihood of taking atransfer course for a four-year college, at least in part becauseit saved time previously spent accessing computer labs.138White House Office of the Press Secretary. “President Obama AnnouncesConnectALL Initiative.” Accessed December 21, 2018. sconnectall-initiative.9Fairlie and Robinson 2013; Leuven et al. 2007; Malamud and Pop-Eleches 2011.10Malamud and Pop-Eleches 2011.11Beuermann et al. 2015; Cristia et al. 2012; Piper et al. 2016.12Mo et al. 2015.13Fairlie and London 2012.4A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a bLaptop distribution also increased computer skills. Computerskills rose more meaningfully for minorities, women, lowerincome, and younger students.14 More research is needed todetermine whether these results would successfully replicateto other contexts.Broadly, programs to expand access to technology havebeen effective at increasing use of computers and improvingcomputer skills.15 Though perhaps intuitive, this is noteworthygiven the logistical challenges of technology distribution,the potential reluctance of students and educators to adopttechnology into daily practice, and the increasing importanceof digital literacy skills.Evidence base: 13 experimental papersII. Educational software (or “computer-assisted learning”)programs designed to help students develop particularskills have shown enormous promise in improvinglearning outcomes, particularly in math.Targeting instruction to meet students’ learning levels hasbeen found to be effective in improving student learning, butlarge class sizes with a wide range of learning levels can makeit hard for teachers to personalize instruction.16 Software hasthe potential to overcome traditional classroom constraints bycustomizing activities for each student. Educational software–or “computer-assisted learning”–programs range from lighttouch homework support tools to more intensive interventionsthat re-orient the classroom around the use of software. Mosteducational software that have been evaluated experimentallyhelp students practice particular skills through “personalizedtutoring” approaches.17Computer-assisted learning programs have shown enormouspromise in improving academic achievement, especially inmath. Of all thirty studies of computer-assisted learningprograms, twenty reported statistically significant positiveeffects.18 Fifteen of the twenty programs found to be effective14Ibid.15Fairlie and Robinson 2013.16Banerjee et al. 2007; Banerjee et al. 2016.17Kulik and Fletcher 2015.18Barrow et al. 2009; Beal et al. 2013; Campuzano et al. 2009; Deault et al. 2009;Hegedus et al. 2015; Kelly et al. 2013; Mitchell and Fox 2001; Morgan and Ritter2002; Pane et al. 2014; Ragosta 1982; Ritter et al. 2007; Roschelle et al. 2010;Roschelle et al. 2016; Schenke et al. 2014; Singh et al. 2011; Snipes et al. 2015; Tataret al. 2008; Wang and Woodworth 2011; Wijekumar et al. 2012; and Wijekumar et al.2014 report positive effects in at least one treatment arm. Borman et al. 2009; Cabaloet al. 2007; Cavalluzzo et al. 2012; Dynarski et al. 2007; Faber and Visccher 2018;Pane et al. 2010; Rouse and Krueger 2004; Rutherford et al. 2014; and Van Klaverenet al. 2017 do not report positive effects. Pane 2014 only finds positive impacts onmath outcomes in the second year.

photo: shutterstock .comwere focused on improving math outcomes.19 A study of amath program that enabled students to control the motionsof animated characters by building or editing mathematicalfunctions showed the largest effect sizes of any large-scalestudy included in the review—0.63 and 0.56 standard deviationimprovements in math scores for seventh and eighth graders,respectively.20 While other studies of computer-assisted mathprograms demonstrated more modest effects, they continuedto show promise. A number of these programs adaptedinstruction to meet student needs by leveraging artificialintelligence and machine learning. Other effective programsprovided timely feedback to students and shared data onstudent performance with teachers to inform their approach.When it comes to computer-assisted reading programs, theevidence was limited and showed mixed results. A programthat taught students a technique for breaking down textsboosted middle school reading comprehension scores by 0.2to 0.53 standard deviations,21 demonstrating that computerassisted learning has the potential to support students inliteracy development as well as in math.com pute r - a s s i s te d le a r n i n gAn evaluation of a supplementary math homeworkprogram in Maine boosted average scores by 0.18standard deviations despite requiring less than thirty toforty minutes per week.22 This program gives studentsfeedback and guidance as they work through mathproblems and sends student data to teachers to help themmeet students’ needs. This program had a positive effecton student achievement, with a significantly larger effectsize for students at or below the median.Note that this program required access to a laptop ora tablet—programs that expand access to technology(described in section I) may sometimes be necessary togenerate the positive effects associated with computerassisted learning (described in section II).1920Barrow et al. 2009; Beal et al. 2013; Hegedus et al. 2015; Kelly et al. 2013; Morganand Ritter 2002; Pane et al. 2014; Ragosta 1982; Ritter et al. 2007; Roschelle etal. 2010; Roschelle et al. 2016; Schenke et al. 2014; Singh et al. 2011; Snipes et al.2015; Tatar et al. 2008; Wang and Woodworth 2011. Pane 2014 only finds positiveimpacts on math outcomes in the second year. Campuzano et al. 2009 did not focusexclusively on math outcomes and is therefore not included in this count.Evidence base: 30 experimental papers21Wijekumar et al. 2012; Wijekumar et al. 2014.Roschelle et al. 2010.22Roschelle et al. 2016.p ove r t y a c t i o n l a b.o r g5

figure 1. computer- assisted learning: impact on student learning in math0.00Reasoning Mind adaptive math program(Wang and Woodworth 2011)0.14DreamBox adaptive math program(Wang and Woodworth 2011)0.00Adaptive CAL program compared against a static oneacross multiple subjects (Van Klaveren et al. 2017)0.40digital tutoring progr amsASSISTments online math homework support(Singh et al. 2011)ASSISTments online math homework support(Roschelle et al. 2016)0.180.36Cognitive Tutor math(Ritter et al. 2007)0.00Cognitive Tutor Algebra I(Pane et al. 2014)0.20Cognitive Tutor Geometry -0.19(Pane et al. 2010)0.29Cognitive Tutor Algebra I(Morgan and Ritter 2002)0.56ASSISTments online math homework support(Kelly et al. 2013)-0.2-0.100.10.20.30.40.50.6estimated impacts on math performance ( as reported in standardized effect sizes)Year 1 Cohort(where applicable)Year 2 Cohort(where applicable)6Note: This graph only includes studies that looked exclusively at math software. Studies that looked at both mathand reading programs, including Campuzano et al. 2009 and Dynarski et al. 2007, are not included for this reason.These two Department of Education studies evaluated roughly a dozen computer-assisted learning programs and overtwo years. The studies found a general pattern of null effects. However multiple programs are aggregated together insome of the analyses, and the multi-program design generally makes it difficult to interpret these results in the contextof the other studies discussed here.A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b

figure 1. computer- assisted learning: impact on student learning in math (continued)0.00digital tutoring progr amsCognitive Tutor’s Bridge to Algebra program(Cabalo et al. 2007)0.30AnimalWatch web-based math tutoring program(Beal et al. 2013)0.17whole - schoolintegr ationI Can Learn aka “Interactive Computer Aided NaturalLearning” program for pre-algebra (Barrow et al. 2009)School of One middle school math program(Rockoff 2015)0.00Kentucky Virtual Schools hybrid program for Algebra 1(Cavalluzzo et al. 2012)0.000.14Spacial-Temporal (ST) Math(Schenke et al. 2014)0.00simulation progr amsSpacial-Temporal (ST) Math(Rutherford et al. 2014)0.56SimCalc interactive math software for 8th grade(Roschelle et al. 2010)0.63SimCalc interactive math software for 7th grade(Roschelle et al. 2010)0.35SimCalc interactive math software(Hegedus et al. 2015)*-0.2-0.100.10.20.30.40.50.6estimated impacts on math performance ( as reported in standardized effect sizes)Year 1 Cohort(where applicable)Year 2 Cohort(where applicable)* Standardized effect size backed out using post-test mean and standard deviation.p ove r t y a c t i o n l a b.o r g7

photo: shutterstock .com8A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b

III. Technology-based nudges—such as text messagereminders—can have meaningful, if modest, impactson a variety of education-related outcomes, often atextremely low costs.Technology can be used to help address systematic biases indecision-making and other psychological factors that lead tounintended outcomes, like high school graduates not enrollingin college as a result of missing financial aid deadlines. Lowcost interventions like text message reminders can successfullysupport students and families at each stage of schooling.Technology can make this communication easier, faster,and more systematic. Programs to facilitate school-parentcommunication—including sending grades and attendanceinformation and sharing personalized feedback—have shownpromising results. Eight of ten studies focused on improvingschool-family information flows demonstrated positiveeffects on student GPAs, test scores, assignment scores,and/or attendance.26m es s ag i n g m at te r s i n sc hool- fa m i lycom mu n i c ati onEarly Childhood and Elementary: Programs to Increase Literacy andLearning at Home (7 experimental papers)Keep your school community in mind when selecting anddesigning programs.Young children do better in school if their parents haveencouraged and participated in learning activities at home.23However, parents—especially low-income parents dealing withhigh stress, limited resources, and time constraints at home—do not always regularly dedicate time to these activities. Identify barriers to student engagement to assesswhether this approach makes sense in your context. Choose communication methods that parents canaccess easily, and select opt-out rather than opt-inprograms where possible.Text messages with reminders, tips, goal-setting tools, andencouragement can increase parental engagement in learningactivities, such as reading with their children. For example, apreschool program in San Francisco that texted suggestionsto parents of small, easy tasks, provided encouragement, andsent reminders increased parental engagement and boostedchildren’s literacy scores (with effect sizes ranging from 0.21to 0.34 standard deviations).24 While a similar standardizedprogram in San Francisco kindergartens showed no impact, textsto parents with specific recommendations matched to eachkindergartener’s reading level showed substantial benefits.25 Use language and translation options in schoolswith parents who are English Language Learners.Personalized feedback and specific action items canincrease student engagement.Think carefully about the tone and messaging toparents as family-school communication can affectstudent-teacher relationships.Middle and High School: Programs to Facilitate School-ParentCommunication (13 experimental papers)In middle and high schools in the U.S., the role of parentstypically shifts away from direct activities with children andtoward encouraging teenagers’ effort in school. Schools canhelp parents support their children by providing families withinformation about their children’s performance.23Levine, Susan C., Linda Suriyakham, Meredith Rowe, Jenellen Huttenlocher, &Elizabeth Gunderson. 2010. “What Counts in the Development of Young Children’sNumber Knowledge?” Developmental Psychology 46: 1309-1319; Price 2010; Sénéchaland LeFevre 2002.24York and Loeb 2018.25Doss et al. 2018.26Bergman 2015; Bergman 2016; Bergman and Chan 2017; Bergman et al. 2018;Bergman and Rogers 2016; Kraft and Dougherty 2013; Kraft and Rogers 2015; andRogers and Feller 2016 found positive effects. Balu et al. 2016 and Bergman and Hill2018 did not find positive effects.p ove r t y a c t i o n l a b.o r g9

Transitioning to College: Programs to Support the College ApplicationProcess, Financial Aid, and Enrollment (19 experimental papers)As students near the end of high school, they have theopportunity to pursue further education. However, thecollege application process can be complex and overwhelming.Technology-based programs to personalize support and sharereminders on specific tasks may help smooth this process.While interventions that provided generic information oneducation tax credits or financial aid did not increase collegeenrollment in the U.S.,27 programs that provided timely,specific, and personalized information were more consistentlyeffective. In particular, programs that leveraged technologyto suggest specific action items, streamline financial aidprocedures, and/or provide personalized support boostedcollege application and enrollment rates28 and encouragedbetter-informed financial aid decisions.29 For example,personalized text messages increased college matriculation by3.3 percentage points among students who had been acceptedto and planned to attend Georgia State University.30 Thisprogram sent reminders based on specific incomplete requiredtasks and leveraged artificial intelligence to automate responsesto common student questions. Programs like this one canreduce the proportion of students who register for college butthen do not show up. Programs that combined technology within-person supports also improved financial aid receipt, collegematriculation, and college persistence.31Social Psychology Interventions: Programs to DevelopResilience, Confidence, and Positive Learning Attitudes(15 experimental papers)Students’ educational performance can be heavily affected byemotions, beliefs, and attitudes. Technology-enabled socialpsychology interventions aim to alleviate psychological barriersand cultivate confidence and positive learning attitudes. Acommon social psychology intervention, for example, is toreinforce the idea that intelligence is not fixed and rather cangrow through hard work.32Despite promising evidence from small-scale studies, large-scalestudies have found that technology-enabled social psychologyinterventions do not improve academic outcomes on average,although they can lead to meaningful effects under somecircumstances.33 These effects tend to be concentrated withinsubsamples and, even then, tend to be quite small.34 In somecases where social psychology interventions did not improveacademic outcomes, they did have a positive impact onpsychological outcomes, for example, the likelihood oftaking academic risks.35 Findings from studies so far havegenerated hints that certain students may benefit more fromsocial psychology interventions. For instance, those who startout further behind in terms of academic performance and/orsocial-psychological attitudes tend to respond better to socialpsychology interventions. However, the current evidence is farfrom sufficient to state this conclusively.Evidence base: 54 experimental papers27Bergman et al. 2016; Darolia 2016; Hyman 2018; Page et al. 2016. Note that the evidencefrom outside the United States shows that information interventions can lead topositive effects on related outcomes, including views of higher education and knowledgeof financial aid. See Oreopoulos and Dunn 2013 and Dinkelman and Martinez 2014.28Castleman and Page 2015; Castleman and Page 2016. Oreopoulos and Petronijevic 2017found text-based advising was not effective for first year college students in Canada.29Barr et al. 2016; Bird et al. 2017; Castleman and Page 2015; Castleman andPage 201620162016A.30Page and Gehlbach, 2017.31Bettinger et al. 2012; Castleman et al. 2012; Castleman and Meyer 2016;Oreopoulos and Ford 2016.27Bergman et al. 2016; Darolia 2016; Hyman 2018; Page et al. 2016. Note that theevidence from outside the United States shows that information interventions canlead to positive effects on related outcomes, including views of higher educationand knowledge of financial aid. See Oreopoulos and Dunn 2013 and Dinkelman andMartinez 2014.28Castleman and Page 2015; Castleman and Page 2016. Oreopoulos and Petronijevic2017 found text-based advising was not effective for first year college students in Canada.29Barr et al. 2016; Bird et al. 2017; Castleman and Page 2015; Castleman and Page 2016.30Page and Gehlbach 2017.31Bettinger et al. 2012; Castleman et al. 2012; Castleman and Meyer 2016;Oreopoulos and Ford 2016.32Snipes et al. 2012.33Pauneksu et al. 2015; Yeager et al. 2016.34Ibid.35Unkovic et al. 2016; Forsyth et al. 2007.photo: shutterstock .com10A b d u l L a t i f J a m e e l Pove r t y A c t i o n L a b

photo: shutterstock .comIV. Online courses are developing a growing presencein education, but the limited experimental evidencesuggests that online courses lower student academicachievement compared to in-person courses. However,students perform similarly in courses with both in-personand online components compared to traditional face-toface classes. In Massive Open Online Courses (MOOCs),behavioral interventions (like the mindset interventionsdescribed in section III) increased course persistence andcompletion rates.Since their emergence in the 1990s, online courses havedeveloped a growing presence in education. Proponents ofconventional online courses and massive open online courses(MOOCs) highlight their potential to reduce costs and improveaccess. Post-secondary students who enroll in conventionalonline programs tend to be more likely to face educationaldisadvantages compared to students in traditional programs.36Conventional Online Courses (17 experimental studies)Conventional online courses—taught as part of entirely onlinedegree programs or degree programs that include online orpartially online courses—have grown in popularity in thelast decade. However, in four of six studies37 that directlycompared the impact of taking a course online versus in-person36Deming et al. 2015.37Alpert et al. 2016; Figlio 2013; Heppen et al. 2012; Keefe 2003; Poirier andFeldman 2004; Zhang 2005.only, found that student performance was lower in onlinecourses. It is possible that students taking online coursesmay struggle with the lack of accountability or miss outon motivating relationships with instructors and peers.Nonetheless, students generally performed similarly—andin some cases better—in courses that included both a faceto-face component and an online component and in courses thatwere entirely face-to-face.38One study did find that offering 8th grade students the optionto enroll in an online algebra course in schools where a standalonealgebra class was not offered improved algebra achievementand also increased the likelihood of participation in an advancedmath course sequence in high school.39 However, it is possiblethat students would have learned even more had they taken anin-person algebra course rather than an online course.One study assessed whether online programs expand accessto students who would not otherwise enroll in a degreeprograms, finding that Georgia Tech’s online master’s programin computer science did expand access, especially among midcareer prospective students.4038Alpert et al. 2016; Bowen et al. 2014; Esperanza et al. 2016; Foldnes 2016;Harrington et al. 2015; Joyce et al. 2015. Wozny et al. 2018. Positive effects fromblended learning were found only in three of the four studies that specifically testedthe flipped classroom model, which reverses traditional instruction by deliveringcontent that is typically taught in the classroom at home via the internet (Esperanzaet al. 2016; Foldnes 2016; Wozny et al. 2018.)39Heppen et al. 2011.40Goodman et al. 2016.p ove r t y a c t i o n l a b.o r g11

photo: shutterstock .comMassive Open Online Courses

will technology transform education for the better? This publication summarizes a forthcoming academic review paper on education technology, “Upgrading Education with Technology: Insights from Experimental Research.” evidence review povertyactionlab.org 1 Technology for Education