Transcription
MANUELA EKOWO AND IRIS PALMERPREDICTIVEANALYTICS INHIGHER EDUCATIONFive Guiding Practices for Ethical UseMARCH 2017
About the AuthorsAbout New AmericaManuela Ekowo is a policy analyst withthe Education Policy program at NewAmerica. She provides research andanalysis on policies related to highereducation including innovations in highereducation delivery, the use of technology, and ensuringequitable outcomes for underrepresented students. Herwriting has been featured in such media outlets asEdSurge, Pacific Standard, the EvoLLLution, and theAmerican Youth Policy Forum.New America is committed to renewing American politics,prosperity, and purpose in the Digital Age. We generate bigideas, bridge the gap between technology and policy, andcurate broad public conversation. We combine the best ofa policy research institute, technology laboratory, publicforum, media platform, and a venture capital fund forideas. We are a distinctive community of thinkers, writers,researchers, technologists, and community activists whobelieve deeply in the possibility of American renewal.Find out more at newamerica.org/our-story.Iris Palmer is a senior policy analyst withthe Education Policy program at NewAmerica. She is a member of the highereducation team, where she providesresearch and analysis on state policiesrelated to higher education including performance basedfunding, state student financial aid, and state datasystems. Palmer previously worked at the NationalGovernors Association on postsecondary issues. Thereshe helped states strengthen the connection betweenhigher education and the workforce, support competencybased systems, use data from effectiveness andefficiency metrics and improve licensure for veterans.Prior to joining NGA, she worked at HCM Strategists on theLumina Foundation’s initiative to develop innovative highereducation models, including new technologies andcompetency-based approaches. Before joining HCMStrategists, Palmer worked at the U.S. Department ofEducation in all of the offices related to higher education:the Office of Vocational and Adult Education, the Office ofPostsecondary Education, the Policy Office and the Officeof the Undersecretary.AcknowledgmentsWe would like to thank the Kresge Foundation, LuminaFoundation, and the Bill & Melinda Gates Foundationfor their generous support of this work. The viewsexpressed in this report are those of its authors and do notnecessarily represent the views of the foundations, theirofficers, or employees.About the Education Policy ProgramNew America’s Education Policy program uses originalresearch and policy analysis to solve the nation’s criticaleducation problems, serving as a trusted source ofobjective analysis and innovative ideas for policymakers,educators, and the public at large. We combine asteadfast concern for low-income and historicallydisadvantaged people with a belief that better informationabout education can vastly improve both the policiesthat govern educational institutions and the quality oflearning itself. Our work encompasses the full range ofeducational opportunities, from early learning to primaryand secondary education, college, and the workforce.Our work is made possible through generous grantsfrom the Alliance for Early Success; the Buffett EarlyChildhood Fund; the Foundation for Child Development;the Bill and Melinda Gates Foundation; the Evelyn andWalter Haas, Jr. Fund; the Heising-Simons Foundation;the William and Flora Hewlett Foundation; the JoyceFoundation; the George Kaiser Family Foundation; theW.K. Kellogg Foundation; the Kresge Foundation; LuminaFoundation; the McKnight Foundation; the CharlesStewart Mott Foundation; the David and Lucile PackardFoundation; the Pritzker Children’s Initiative; the SmithRichardson Foundation; the W. Clement and Jessie V.Stone Foundation; and the Berkshire Taconic CommunityFoundation.Find out more at newamerica.org/education-policy.
ContentsIntroduction2Guiding Practice 1: Have a Vision and Plan4Guiding Practice 2: Build a Supportive Infrastructure6Guiding Practice 3: Work to Ensure Proper Use of Data7Guiding Practice 4: Design Predictive Analytics Models andAlgorithms that Avoid Bias10Guiding Practice 5: Meet Institutional Goals and ImproveStudent Outcomes by Intervening with Care12Advisory Council17Notes18
INTRODUCTIONColleges are under increasing pressure to retain theirstudents. Federal and state officials are demandingthat those who enter their public institutions—especially students from underrepresented groups—earn a degree. Over two dozen states disburse somestate funding on how many students an institutiongraduates, rather than how many it enrolls.Students and families are more anxious than everbefore about crossing the degree finish line, as thefinancial burden of paying for college has increasedsignificantly in recent years. And retaining studentsis becoming more crucial to the university bottomline. As recruiting and educating students becomesincreasingly expensive, colleges hope to balance theresources they use to recruit students with revenuegenerated when those students are retained.Because of these pressures, institutions havebegun analyzing demographic and performancedata to predict whether a student will enroll at aninstitution, stay on track in her courses, or requiresupport so that she does not fall behind. Usingdata in this way is known as predictive analytics.Analyzing past student data to predict what currentand prospective students might do has helpedinstitutions meet their annual enrollment andrevenue goals with more targeted recruiting andmore strategic use of institutional aid. Predictiveanalytics has also allowed colleges to better tailortheir advising services and personalize learning inorder to improve student outcomes.2But while these are worthwhile efforts, it is crucialfor institutions to use predictive analytics ethically.Without ethical practices, student data could beused to curtail academic success rather than helpensure it. For example, without a clear plan in place,an institution could use predictive analytics to justifyusing fewer resources to recruit low-income studentsbecause their chances of enrolling are less sure thanfor more affluent prospective students.Last spring, New America interviewed more than30 college administrators, experts, and educationtechnology vendors, conducted an extensive reviewof existing literature, and visited a college campus(Georgia State University) to write The Promise andPeril of Predictive Analytics in Higher Education:A Landscape Analysis. That report, published inOctober 2016, examined how colleges are usingpredictive analytics and outlined the challengesschools face in ensuring that they are doing soethically. Last fall we also convened an advisorycouncil to discuss important considerations whenusing predictive analytics in higher education (seepage 17 for a list of council members).Our framework here aims to lay out someimportant questions to consider as administratorsformulate how to use predictive analytics ethically.Examining the ethical use of data is an iterativeprocess; colleges will continue to use student andinstitutional data in new and innovative waysEDUCATION POLICY
and will therefore have to occasionally reassesswhether their ethical standards address currentdata practices.Using data ethically is complex, and no magicformula exists. This ethical framework is meant tostart conversations on campus. It cannot address allpossible issues surrounding the use—and potentialabuse—of institutional data.We recognize that student and institutional datamay be used for various reasons and at variouslevels within an institution. For example, data onhow students learn captured in an adaptive toollater used by faculty is likely governed differentlythan data collected in institutional enrollmentprocesses used by admissions officers. There maybe different rules about who has access to whatinformation and whether students have the optionto object to having their data collected or analyzed.In writing this guidance, we have tried to takethese differences into account and believe that ourrecommendations are relevant for the variety ofways institutions may use student and institutionalinformation for predictive analytics.Colleges use data for predictive analytics in thefollowing ways: Early-Alert Systems. In an early-alert system,flags are triggered based on academic and nonacademic data from students that signal whenthey may need additional support. Academicinterventions may include tutoring, meetingswith an adviser, or assigning a coach ormentor to the student. For non-academic flags,colleges can deploy financial supports (i.e.,emergency grants) or referrals to other supports(i.e., mental and medical health, child care,transportation, housing, and food). Recommender Systems. Recommendersystems allow students to plan or map theirdegree, and integrate transfer credits or priorlearning assessments into that plan. Onecommon use for recommender systems ishelping students choose courses to take nextand/or choose a major based on data abouttheir previous academic performance. Adaptive Technologies. Adaptive tools use dataon how students learn to customize the learningenvironment for each individual student byidentifying gaps in knowledge, skills, andabilities and adjusting content delivery tosupport deeper and more efficient learning. Enrollment Management. Enrollmentmanagers use algorithms (computer-basedrules) to decide how best to target recruitmentefforts and distribute financial aid.Predictive Analytics in Higher Education: Five Guiding Practices for Ethical Use3
GUIDING PRACTICE 1:HAVE A VISION AND PLANDeveloping a vision and plan for data use will helpsteer the direction of a predictive analytics effort.Without such planning, predictive analytics maybe used in a way that does more harm than goodfor students, leaves out key staff who should beincluded in the planning process, and/or fails toidentify how success of this effort will be measured.To develop a vision and plan, take the followingsteps:Convene key staff to make importantdecisions.In developing a plan, include key staff andstakeholders in decision making, and get theirsupport. Including these individuals in theplanning process can help ensure that you areusing predictive analytics in a way that does notintentionally harm those whose data are being usedand analyzed. Your team can include: head of advising/student success provost/chief academic officer head(s) of admissions, enrollmentmanagement, and financial aid head of student affairs head of institutional research4 head of communications, marketing, or publicaffairs key faculty other key staff (for example, chief diversityofficer, privacy officer, information technologyofficer, data officer, digital learning officer,instructional designers, directors of centersfor teaching and learning, career services,registrars, athletics, etc.) data scientists, modelers, and third-partyvendors studentsUltimately, the staff and stakeholders included willdepend on the way in which you plan to use data.Consider the following three factors whendeveloping the plan.1. The purposes of predictive analyticsThe plan should include the questions you hope toanswer and the goals you aim to achieve. It shouldalso explore the potential pitfalls of using studentand institutional data for the purposes intended.The team should make sure that data will not beused for discriminatory purposes. For example,using predictive analytics to determine whatEDUCATION POLICY
amount of institutional aid might discourage lowincome students from enrolling at your institution.2. The unintended consequences of predictiveanalyticsThe plan should also include a discussion aboutany possible unintended consequences and stepsyour institution and its partners (such as third-partyvendors) can take to mitigate them. For example,using predictive analytics could lead to removinghuman judgment from decision making. This mayresult in decisions that typically require holisticreview becoming partly or completely automatedby data alone. Teams will also want to examine thatthey are not underestimating or overlooking howimplicit bias—beliefs we may not be consciouslyaware of—may become more frequent withpredictive analytics. In addition, they must guardagainst using biased algorithms that discriminateagainst certain students by limiting their access tocollege or to opportunities to pursue their interestsonce enrolled.3. The outcomes to measureThe plan should also lay out the measurableoutcomes you hope to achieve as a result of usingpredictive analytics.Questions to AskBelow are three questions to ask when creating a plan for using predictive analytics ethically in higher education. Have we set a goal/vision for using predictive analytics and/or adaptive technologies? Can our goals, methods, and measurable outcomes be explicitly stated in our institution’s strategic planand get support from key institutional officials? What would it take to make this happen? Is approval fromthe Institutional Review Board (IRB) necessary? Have we considered the unintended consequences predictive analytics may introduce? When drafting ourvision and goals, have we made any assumptions about students or data?Predictive Analytics in Higher Education: Five Guiding Practices for Ethical Use5
GUIDING PRACTICE 2: BUILD ASUPPORTIVE INFRASTRUCTUREA supportive infrastructure ensures the benefits ofpredictive analytics are understood and welcomed bycampus stakeholders, and that processes and othersupports are put in place to assist the data effort.Communicate the benefits of usingpredictive analytics and create a climatewhere it can be embraced.Predictive analytics uses student and institutionaldata to create change almost immediately. Manyinstitutions may not be experienced with using datain this way, at this pace, and perhaps with suchhigh stakes like ensuring students complete theirdegree in a timely manner. You should take thelead in communicating with campus leaders, staff,and students about why using predictive analyticsis critical to institutional and student success. Thehead of communications and marketing could helpin these efforts. Without a clear articulation of howusing predictive analytics can benefit the campus,well-devised plans may fail to receive the supportthey need to be successful.6Develop robust change managementprocesses.With new tools often come new processes, reportingstructures, people, and partners who bring newskills. This can, at best, create confusion for thosecharged with rolling out predictive analytics on acampus, and, at worst, chaos. Leaders convenedto make important decisions about data use couldalso help ensure that processes are put into place tosupport the change taking place on campus.Assess institutional capacity.Assess your school’s capacity to use predictiveanalytics. Having the appropriate technology, datainfrastructure, talent, services, financial resources,and data analysis skills are essential. Maintaining asound infrastructure can help ensure that cleaning,sharing, and using large amounts of data formaking decisions institution-wide can be carriedout smoothly and that different data systems can“speak” to one another.1 Experts in informationtechnology, student data laws, and staff withexperience drafting contracts with vendors wouldhelp ensure the success of the project.EDUCATION POLICY
Questions to AskBelow are two questions to ask to determine whether your institution has the infrastructure needed to embracethe changes that predictive analytics can bring. Have we communicated with stakeholders about why using predictive analytics in our decision making isimportant? Do we have the right messengers to build support for these new policies? Do we have the processes, structures, tools, infrastructure, people, and knowledge required to supportthe successful use of predictive analytics?GUIDING PRACTICE 3: WORK TOENSURE PROPER USE OF DATAPredictive models (showing how different datapoints are related) and algorithms need data tobuild predictive tools that will support enrollmentefforts or help students make academic progress.To build and use these tools ethically, consider thequality of your data and data interpretation, as wellas issues around privacy and security.Ensure data are complete and of highenough quality to answer targetedquestions.2Data about students and institutionalprocesses should not only be accurate but alsocomprehensive. Consider for example, developingan adaptive learning tool for a developmentalscience course based on eight semesters’ worthof information captured in the app about howstudents attempted questions or worked theirway through course content—on students whotested out of the developmental course. ThePredictive Analytics in Higher Education: Five Guiding Practices for Ethical Use7
information collected may be accurate, but it isnot comprehensive because the app was not testedon the population of students for whom the tool isdesigned.Comprehensiveness also means considering allrelevant data about the students who are beingexamined. For example, consider an early-alertsystem that flags at-risk students solely on theirpast grades and demographics. The system mayhave accurate data, but could potentially be missingimportant information such as students’ currentperformance that may be key to determiningwhether they are struggling academically.Beyond being accurate and comprehensive, qualitydata is also timely, derived using consistent toolsand processes, and is well defined.Ensure data are accurately interpreted.Include staff members who are knowledgeableabout your institution’s data and can accuratelyinterpret predictive models derived from thisinformation. It is essential that those analyzingthe data take context into consideration. It is alsoimportant to train faculty so that they can easilyinterpret dashboards that explain how studentsusing adaptive tools are faring in their courses.Lastly, look for ways to ensure that data used solelyfor reporting purposes is sound even though theymay also be included in data sets that are used forpredictive analytics. If institutional researchersare responsible for both compiling data sets forreporting purposes as well as for conductinganalysis for predictive analytics projects, theintegrity of data for reporting should not come intoquestion because information is being used oncampus in innovative ways. Put simply, predictiveanalytics should not diminish the quality of datayour institution is required to report to remain incompliance for federal funding.8Guarantee data privacy.Communicate with students, staff, and others whosedata are collected about their rights, including themethods used to obtain consent to use the data forpredictive analytics and how long the informationwill be stored.3 Make students and staff awarethat their data are going to be used for predictiveanalytics and get consent to use highly sensitiveinformation like health records.Be vigilant that data are well protected so that theinformation does not get into the hands of thosewho intend to misuse it. It is especially importantto protect the data privacy of vulnerable studentgroups, such as high school students who areminors and enrolled in dual-enrollment programs,undocumented students, and students withdisabilities.4In addition, make school policies on ownership ofand access to student and institutional data clear.For example:1. Advisers may have access to analyses of theirstudents’ performance data, and allow studentsto see these only and in a way that encouragesprogress.2. Faculty may be allowed access to only selectiveinformation to enable interventions for studentsin their classes, but they can access all learninganalytics data from learning managementsystems (LMS), adaptive technologies, or otherdigital tools required for class.3. The president and provost may have accessto only high-level analyses of students anddepartment-level data that is devoid ofpersonally identifiable information.4. Enrollment managers and financial aid officersmay have exclusive access to predictiveanalytics of prospective students for recruitingpurposes.EDUCATION POLICY
5. If students will not be able to see a predictivetool, they should know what their rights are toaccess their data.Monitor data security.Security threats occur without notice. As collegescollect and store more data on students and staff,and more devices that store data on the teachingand learning process are used in classrooms,security becomes an ever more pressing issue.For this reason, schools need to be vigilant about5assessing data privacy and security. Monitoringthreats and risks should be a regular undertaking.Data security requires you and your vendors to havesecurity protocols that adhere to student privacylaws and meet industry best practices.6To keep institutional data secure, involveyour information technology (IT) department.Information security and privacy officers help keepinstitutional data safe. Providing regular training toIT and other staff about keeping these data secureshould be a top priority.Questions to AskBelow are four questions to ask to determine whether your institution is using and maintaining data properly. How do we ensure that the data collected and the collection methodology are appropriate for thequestions we intend to answer or the tools we plan to build? What role will students’ current performance data play in predictive models? What role will qualitative dataplay? How will we get consent to collect student and staff data and analyze it, or is consent not necessary? Isawareness and transparency about how data are used sufficient? How will we safeguard student and institutional data and work to ensure that they are not used—internallyand externally—for non-educational purposes?7Predictive Analytics in Higher Education: Five Guiding Practices for Ethical Use9
GUIDING PRACTICE 4:DESIGN PREDICTIVE MODELS ANDALGORITHMS THAT AVOID BIASPredictive models and algorithms can helpdetermine the interventions an institution usesto support students or meet recruiting goals.Therefore, it is crucial that predictive models andalgorithms are, at the very least, created to reducerather than amplify bias and are tested for theiraccuracy. You should also ensure that models andalgorithms are created in consort with vendors whocan commit to designing them in a way that doesnot intentionally codify bias and so that they areable to be tested for veracity.Design predictive models and algorithmsso that they produce desirableoutcomes.8It is crucial to address bias in predictive models,ensure the statistical significance of predictionsbeyond race, ethnicity, and socioeconomic status,and forbid the use of algorithms that producediscriminatory results. An algorithm should neverbe designed to pigeonhole any one group.Therefore, design or know how predictive modelsand algorithms are created in order to ensuredesirable outcomes as determined by their vision10and plan. Failing to take this approach may lead toinadvertent discrimination. For example, imaginea course recommender system with a predictivemodel that finds that low-income students orstudents of color are likely to struggle in collegelevel-math courses and recommends that thesestudents be kept out of Science, Technology,Engineering, and Math (STEM) courses andprograms. To avoid this self-defeating approach,consider using predictive analytics to help ensurethat students from these groups who are interestedin going into STEM are given appropriate support ifthey start falling off track.Test and be transparent about predictivemodels.Before predictive models can be used to developalgorithms, test them for accuracy, perhaps by anexternal evaluator. Predictive models should alsobe updated or refreshed to reflect new campusrealities and goals. You may also want to limit thevariables used in predictive models to those thatcan be easily explained and you should work toensure algorithms can be understood by those whowill be impacted by them. Such practices fosterEDUCATION POLICY
transparency and makes it easier to hold individualsaccountable for creating poorly designed models oralgorithms that produce discriminatory outcomes.Choose vendors wisely.Most colleges rely on an outside vendor to helpthem build models and predictive tools. To ensuremodels and algorithms are sound, transparent, andfree from bias, you must be intimately involved withor knowledgeable about how predictive models andalgorithms are built. Partnering with third-partyvendors may make this harder.Some vendors are transparent about their modelsand algorithms, and allow colleges to have a handson approach in the design process, or even letinstitutions take the lead.9 Not all vendors, however,take this approach. Many consider their models andalgorithms proprietary, meaning institutions are notinvolved in the design process or are deliberatelykept out. You should make transparency a keycriterion when choosing to work with any vendor.Questions to AskBelow are three questions to ask to determine whether your institution is positioned to address bias inpredictive models and algorithms, ensure open and transparent designs, and work with a cooperative vendor. How will we address bias in predictive models and algorithms? Will we discontinue their use, or considerhow to minimize the impact gap of interventions for different student groups? Do we understand how the algorithms work and are improved through machine learning, when acomputer—on its own—learns from data and algorithms it is fed to create new algorithms? What are our standards for working with vendors? Have we asked the right questions when selectingthem, such as: How will your company and/or product be a fit with our institution? How does your company ensure tools and services are developed and used transparently? How will you hold yourself accountable for student outcomes?Predictive Analytics in Higher Education: Five Guiding Practices for Ethical Use11
GUIDING PRACTICE 5:MEET INSTITUTIONAL GOALS ANDIMPROVE STUDENT OUTCOMES BYINTERVENING WITH CAREHow your institution acts as a result of what itlearns from predictive analytics is where the rubbermeets the road. Students will experience theseactions or interventions firsthand, even if they donot see or understand how the algorithmic-baseddecisions are made. Despite the use of technology,humans primarily still have to deliver interventions.Therefore, it is important that interventions arethought about in the context of other supportsoffered at your institution and are disseminatedwith carefully communicated messages. Staffdeploying interventions should be trained on howto intervene appropriately, and you should test theeffectiveness of interventions once deployed.Communicate to staff and students aboutthe change in intervention practices.Adding predictive analytics to the student successtoolbox may spark a culture change as interventionsinformed by data become central to your institution.To get the campus to embrace this change, it isimportant to communicate how faculty, staff, and12students will benefit from using interventions thatare informed by predictive analytics, and allowthem to guide the change as well. For example,faculty and advisors may be used to makingdecisions on how to intervene (or not to) based ontheir instincts about what a student needs, ratherthan looking at student-generated data to guidethem. Offer professional development opportunitiesto faculty and staff and information sessions tostudents to communicate the benefits broughtby using students’ data to inform interventions.Training could also address how resourceconstrained institutions should make decisionsabout who to help, starting with students with thehighest need first.10Embed predictive-driven interventionsinto other student success efforts.Despite being a powerful tool, predictive analyticsis still only one part of a suite of tools—like firstyear orientation programs—that can ensure studentand institutional success. Look for opportunitiesEDUCATION POLICY
to leverage predictive analytics in ways thatfurther advance other activities so that all studentsuccess efforts are connected and build upon oneanother. For example, it might be wise to help astudent flagged as at-risk in an early-alert systemsee the benefits of taking advantage of learningcommunities or other supports offered. A one-offpredictive analytics project that is not integrated orpositioned to play off your broader efforts to helpstudents succeed will likely not see the impact itcould have if it was leveraged appropriately.Recognize that predictive-driveninterventions can do harm if not usedwith care.Even when institutional data, predictive models,algorithms, institutional practices, and training areas good as they can be, mistakes can be made whenacting on information. This is why interventionsused in response to predictive analytics shouldbe carefully calibrated to avoid harming students.Interventions used in response to data generatedby students and predictive models can range fromtargeting a particular student for increased outreachbased on his predicted chances of enrolling,requiring a meeting with an adviser based onthe recommendation of an early-alert system, tochanging the type of practice problem a student isassigned based on an adaptive technology system.and institutional data, as well as how to rebuildtrust after a harmful incident has occurred.Predictive tools can be used with care in thefollowing ways:Early-Alert SystemsBecause interventions are deployed based on flags,they must be carried out with care to ensure thatsuggested meetings or external resources do notunintentionally communicate to students thatthey do not belong in college. Starting with smallpilots to gather student feedback and determine theimpact of early-alert systems and va
using predictive analytics in higher education (see page 17 for a list of council members). Our framework here aims to lay out some important questions to consider as administrators formulate how to use predictive analytics ethically. Examining the ethical use of data is an ite
