WIXLIB

Research question 3: What are the technical and political challenges (and strengths) ofeducation reforms, and what strategies have been used to overcome these challenges?Specifically, the analysis explores the role of information and knowledge, coalitions andincentives, and innovation in shaping education reforms.1.2. Strategy and organization of the paperWe followed a three-step approach to answer our research questions. First, we identified episodesof improving or declining learning trends over five-plus years using panel data from cross-nationalstandardized tests. Specifically, we used data for reading scores from the Program for InternationalStudent Assessment (PISA), the Southern and Eastern Africa Consortium for MonitoringEducational Quality (SACMEQ), the Progress in International Reading Literacy Study (PIRLS),and the Latin American Laboratory for the Assessment of the Quality of Education (LLECE).Second, we created a database of education reforms that were planned around the period of theidentified learning episodes. Specifically, we documented types of reforms that were eitherplanned or introduced during the period starting five years before the beginning of the learningepisode until the last year of the episode.7 In addition, we use data on economic growth, educationspending, and political conditions corresponding to the learning episodes. This database helpsanswer our first two research questions.Our third strategy included qualitative case studies on reform experiences for a diverse set of ninecountries. The case studies also helped us dig into more nuanced political factors and reformprocesses that the variables in the database may not have captured. A primary criterion used foridentifying a country for the case studies was the availability of learning outcomes and the abilityto plausibly link these learning outcomes to reforms.1.3. Summary of findingsFindings from the database do not establish strong and consistent correlations with indicators ofeconomic and political conditions believed to affect education quality and learning outcomes.However, some patterns emerge: While there were many similarities in the education reforms associated with episodes ofimproving or declining learning in low- and middle-income countries, there were some notabledifferences. For example, curriculum, school-based management, learning assessments, andstructural reforms were more common in episodes where learning had improved. There were also differences in the detail of broad reform efforts. While building institutionalcapacity was the most commonly introduced reform in low- and middle-income countries,activities to strengthen financial management systems were more common in episodes wherelearning improved.7For example, for a learning episode from 2000 to 2007, we recorded programs that were planned orintroduced between 1995 and 2007.7

Initial levels of gross domestic product (GDP) per capita and economic growth were notsignificantly different between episodes where learning had improved and episodes where ithad declined. There was no strong evidence of a trade-off between improvements in learning andimprovements in access. Spending per-student tended to rise more quickly in episodes where learning improved. Compared to episodes where learning outcomes deteriorated, democracy levels and thecharacteristics of a country’s political settlement (an explicit or implicit agreement amongpowerful groups) did not seem to have a stronger association with episodes of improvedlearning. While there seems to be some relationship between the timing of reforms in the political cycleand improvements in learning, our information was insufficient to untangle whether successfulreforms were introduced at the beginning or at the end of a cycle. However, governmentturnover was not more strongly associated with episodes of learning decline.Findings from the case studies showed some of the complexities that shaped governments’abilities to deliver quality reforms. Elite buy-in was essential for a leader to introduce technical reforms. Successful reformswere not contingent on one charismatic leader (though this could help). Rather, qualityreforms needed to be introduced by leaders who were informed by well-qualified experts. Governments committed to improving learning outcomes tended to have information drivetheir decision-making. Information and metrics helped build demand for reforms that couldtrack learning, improve the curriculum, assess performance, and so on. Information could also drive communication strategies, be used to make the argument forreforms, and gain the support of key actors, including the public. An effectivecommunication strategy could also counter the spread of misinformation by groups whooppose the reform. Effective coalitions enlisted the buy-in of stakeholders that would be vital for introducingand implementing the reform. Teachers’ unions were often a powerful opposition group.Taking a confrontational stance to fight unions ran the risk of alienating the group of teacherswho would implement the reform. In the country cases we looked at, teachers’ unions oftensucceeded in obstructing efforts to introduce quality reforms. The most effective reforms were focused but flexible. Policies were effective if they had aclear objective, but they could also be changed or re-envisioned as external conditions8

changed. Over a longer period, reforms could build on one another, gradually offering greaterlevels of sophistication to the system in a way that could incrementally improve learning. “Critical moments” could give rise to quality-focused reforms. Leaders could feel compelledto accept reports if there is, for example, a change in leadership, a major political shift in thecountry (e.g., end of apartheid), or a report highlighting low learning outcomes. Sequencing popular reforms with those less likely to be supported helped increase leadersaccept less popular reforms. In the case of teacher accountability, starting with collectiveincentives could have helped build an evaluation culture that could have made it easier toimplement individual incentives later on. Such an approach was more likely to guaranteeleaders implement meaningful reforms rather than launch a drastic policy change that may berejected from the outset.The next section reviews literature on some of the technical and political challenges thatgovernments face when introducing education quality reforms and improving learning outcomes.9

2. Literature review: The technical and political challenges of improving learningoutcomesEducation systems and learning outcomes are shaped by a range of technical and politicalcomplexities.8 Achieving systemic change requires aligning these various components towardstudent learning. But even when evidence points to clear interventions that will improve learningoutcomes, it is not always acted upon.Education systems often are constrained by several technical challenges that limit theireffectiveness. First, financing for education systems is often insufficient. Education systems andpolicies may be well designed, but without adequate levels of funding, even the best designedinterventions will not work. Yet most education systems do not yet spend the recommended 6%of GDP on education.9Second, available resources in education systems are often not well used. Teachers may not beprepared to be effective in their work. Many lack enough training and pedagogical skills to deliverthe curriculum. Salaries or other benefits may be low, leaving teachers with little incentive to showup for their own classes. They may take on second jobs outside of teaching, even moonlighting asprivate tutors, sometimes for their own students.10Third, linking inputs to learning outcomes is not straightforward. Many states have proven adeptat building classrooms, increasing access, offering trainings, or introducing new materials toimprove pedagogy. However, the mechanism that links these important investments with students’acquisition of basic skills and proficiencies is poorly understood.11 Even conventional technicalinputs such as textbooks may not bring about the expected improvements to children’s learning.12The challenges of improving quality also face political complexities.13 Scholars have pointed outthat it is reasonably easy for leaders in democratic states to make a case for improving accessbecause these activities are straightforward, visible, and popular.14 But improving educationquality is more challenging than building classrooms, as it is often less visible and can be morecomplex. The evidence base for how to improve learning outcomes is underdeveloped, particularlyin resource-limited settings. Even when promising interventions are identified, results may notcome quickly enough for leaders, who often need to determine the political implications ofinvesting in quality education alongside their voters’ other interests.158Many of the arguments in this section were informed by the work of Hossain et al. (2017), Hickey andHossain (2019), and Bruns, Macdonald, and Schneider (2019).9UNESCO (2014).10Bray and Lykins (2012); CAMPE (2015).11Pritchett (2013).12Ulrich, Kremer, and Moulin (2009).13Grindle (2004); Hanushek and Woessmann (2007); Kingdon et al. (2014); Kosack (2012); Pritchett(2013).14Hossain and Moore (2002); Mani and Mukand (2007); Grindle (2004).15Kosack (2012).10

In short, the incentive for political leaders to introduce reforms to improve education quality maynot be as politically advantageous as we might assume. Quality-focused reforms can entail fightingpowerful teachers’ unions, losing jobs, or shifting budget priorities away from higher education.16Each of these factors can mean losing elections.The elite have formal and informal coalitions, whose priorities and incentives may or may notalign with a commitment to equity and quality. They have other interests, such as maintainingpower and allegiance to the “vital constituencies”17 that are central to keeping them in power orchampioning the needs of special interest groups. Thus, education policies may or may not beintroduced with the expectation that learning outcomes will improve.18Particularly in weaker states, there may be less pressure exerted on governments to improvequality.19 Public forms of accountability cannot be assumed, and demand for education workersmay change. In less developed settings, businesses and industries may have little need for aneducated workforce, placing the burden for improving learning on the state and nongovernmentalorganizations (NGOs).Demand may not be strong from households, either. Parents themselves may lack a formaleducation, so they may not know what to demand or expect from their government when it comesto quality.20 Without demand from businesses or households, the most powerful source of demandto improve quality may have to come from the state itself. This demand must align with globalcommitments concerning sustainable and equitable development for all and with the globalmovement to promote mass education.21The politics of education policies and learning outcomes can be studied at different levels. At thenational level, there is a need to understand the relationship between learning outcomes and thenational political settlement.22 The priorities and incentives for introducing and implementing propoor policies can be studied by looking at the individuals, incentives, and institutions that wouldlead the national government to prioritize quality and equitable education.23At the education sector level, the study of politics can focus on the incentives and forms ofaccountability of local government officials, education officers, and teachers. For example, it canfocus on the effectiveness of decentralization or on the impact of teachers’ unions with respect tolearning outcomes, examination performance, or completion rates.2416Bruns, Macdonald, and Schneider (2019).Kosack (2012).18Kingdon et al. (2014); Kosack (2012); Grindle (2004).19Harding and Stasavage (2013).20Kosack (2012).21Boli, Ramirez, and Meyer (1985).22Political settlement refers to “the balance or distribution of power between contending social groupsand social classes, on which any state is based,” according to Di John and Putzel (2009, p. 4).23Khan (2010); Kingdon et al. (2014); Williams (2017).24McLoughlin (2011); Williams (2019).1711

An investigation into the politics of education quality must also examine the role of informalpolitics.25 Who are the actors and interest groups propelling the trajectory of the sector, and whymight they support quality and equitable education policies? Approaching a study of educationpolitics using this lens can help us understand the distribution of power between different socialand political groups that shape the implementation of education reforms. In short, the study ofpolitics and governance in education can help reveal patterns associated with accountability,technocratic effectiveness, forms of service delivery, and the alignment between national-levelgoals and incentives to improve learning.We now turn to examine findings from the cross-country database to examine episodes and trendsof education reform and learning improvement.25Grindle (2004); Kosack (2012).12

3. Episodes of learning improvement using cross-country dataIn this section we aim to explore the education, economic, and political contexts associated withsuccessful episodes of learning improvement by drawing together the available cross-country data.We first use international and regional assessments to identify country episodes where learningoutcomes changed. Using these “learning episodes” as our main unit of observation, we then builda database of relevant education, economic, and political variables associated with these episodes.We use this database to compare and contrast the characteristics of country episodes of improvingor deteriorating learning outcomes. The analysis in this section does not identify causaldeterminants of changes in learning but instead explores factors correlated with periods ofsustained learning improvements versus periods of decline.3.1. ApproachEpisodes of learning improvements (or declines) were identified using data on reading assessmentsfrom the PISA, LLECE, SACMEQ, and PIRLS assessments. We selected reading because thePISA assessment covers a large number of countries and reading scores are reported on a commonscale over a long period of time.26 Moreover, more data are available for reading in developingcountries than for math and science. For example, in the first round of the SACMEQ testsconducted between 1995 to 1999, students were only tested in reading.27To identify periods of changes in learning, we adopted a standard set of rules across all countriesand assessments (see Box 1 and Annex A). In total, 105 episodes across 76 countries wereidentified.Box 1. Identifying episodes of changes in learningUsing information on learning outcomes, we established a series of learning episodes based on intervals of aminimum of five years, for each country. These episodes identify the main direction of change in learningoutcomes and the period that these changes occurred in. For example, using the Progress in International ReadingLiteracy Study (PIRLs) data, we identified learning changes between 2001 and 2006 and between 2006 and 2011.In the case of the PISA data, we checked the direction of change in learning for six-year periods between 2000and 2006, between 2003 and 2009, and for all other periods between 2000 and 2015.Once these periods were identified, we then sought to establish the direction of change in learning and whetherthese changes were statistically significant. The periods were classified as improving/worsening if the change inlearning was statistically significant. Periods that had statistically insignificant changes in learning were coded as“no change” episodes and were excluded from our database.Since each interval used in the PISA assessments included data from three test years, it is possible that learningoutcomes did not change monotonically. We only included periods where the change between the first and finalyear was statistically significant.Once each period was categorized as an increasing or declining period, we extended the overall period theycovered to identify learning episodes based on the following rules.26Results of reading assessments from all six rounds of the PISA tests have been reported on the samescale. That is not the case for results of math or science assessments; see OECD (2016).27See SACMEQ (2018).13

Let k denote the last year of the interval and j denote the first year of the interval. For an increasing interval, if the score in the assessment year immediately after the end of the interval (year k 1) was greater than thescore at the end of the original period, then the period was extended to include year k 1; and if scores remained unchanged

6 1. Introduction In many developing countries, access to basic education has surged, but learning has not kept pace. Half of the 250 million children who cannot read, write, or do math go to sch