Insight Note

Modeling the Long-Run Learning Impact of the COVID-19 Learning Shock: Actions to (More Than) Mitigate Loss

Published journal version

 

Key points

  • This note uses a calibrated “pedagogical production function” model to estimate the potential long-term losses to children’s learning from the temporary shock of COVID-19 related school closures. It then models possible gains from two mitigation strategies.
  • Without mitigation, children could lose more than a full year’s worth of learning from a three-month school closure because they will be behind the curriculum when they re-enter school and will fall further behind as time goes on.
  • Remediation when children return to school reduces the long-term learning loss by half, but still leaves children more than half a year behind where they would have been with no shock.
  • Remediation combined with long-term reorientation of instruction to align with children’s learning levels fully mitigates the long-term learning loss due to the shock and surpasses the learning in the counterfactual of no shock by more than a full year’s worth of learning.
  • Systems need to begin planning now for remediation programmes, and as they do so they should build programmes and train teachers in ways that can continue to produce benefits beyond the period immediately following reopening.

Authors

Image of Michelle Kaffenberger

Michelle Kaffenberger

RISE Directorate

Blavatnik School of Government, University of Oxford

I. Introduction

The COVID-19 pandemic has forced 1.7 billion children out of school temporarily. While many education systems are attempting varying degrees of remote learning, it is widely accepted that the closures will produce substantial losses in learning (World Bank, 2020; Kuhfeld et al., 2020). However, the real concern is not just that a few months of learning will be lost in the short run, but that these losses will accumulate into large and permanent learning losses as many children fall behind during school closures and never catch up.

A new study suggests that even temporary school closures can result in large medium-term lost learning. Andrabi, Daniels, and Das (2020) analyse the impact of the 2005 Pakistan earthquake on children’s learning four years later by comparing households that were close to the fault line with similar households that were farther away and not affected by the quake. Schools in the affected area were closed for an average of 14 weeks, a little more than 3 months. However, four years later children in the affected areas were not just three months behind, they were the learning equivalent of 1.5 years of schooling behind.1

The direct effect of the school closures alone cannot account for such large deficits in later test scores, suggesting affected children learned less each year after they returned to school because of the short-term interruption (Andrabi, Daniels, and Das, 2020). One possible explanation is that the curriculum and instruction did not adapt to the children’s lower learning levels upon re-entry into school and hence the affected children fell farther and farther behind.

This note uses a calibrated model with a “pedagogical production function” (Kaffenberger and Pritchett, 2020) to estimate the potential long-term losses to children’s learning from the temporary shock of school closures. The model shows that without mitigation, children could lose more than a year’s worth of learning even from a three-month school closure as the short-term losses continue to compound after children return to school. Turning to mitigation strategies, the note examines the long-term effects of two strategies, finding that with some mitigation efforts education systems could come back from the crisis stronger than before.

Leading education experts are already calling for adaptation of instruction when children return following the COVID-19 related closures. Rukmini Banerji, CEO of Pratham, the NGO in India which pioneered the “Teaching at the Right Level” approach, has said education systems should focus on “helping children catch up on basic foundational skills” when children return to school (World Bank Live, 2020). The World Bank has called for education systems to begin planning for large-scale remedial programmes (World Bank, 2020). To get learning back on track, systems will need to assess children’s post-closure learning levels, plan for teaching and prioritising essential foundational skills that may have been missed; and train and empower teachers, including through coaching programmes, to adjust pedagogical practice and content to the learning levels and needs of the students.

How much learning might be lost from the COVID-19 shock in the long run if nothing is done, especially if losses accumulate as the Pakistan study suggests they could? And how much of a difference might mitigation strategies make? I use an existing pedagogical production function model (Kaffenberger and Pritchett, 2020), calibrated to replicate learning trajectories in low- and middle-income countries, to model the possible outcomes. I introduce a learning loss shock for children currently in Grade 3 and model how their learning is affected through Grade 10.2

I find that if learning in Grade 3 is reduced by one-third, roughly the amount of time many children are likely to be out of school, learning levels in Grade 10 (compared to a counterfactual of the same children with no shock) are a full year lower. This is similar to the cumulative learning loss identified by Andrabi, Daniels, and Das (2020). Second, I find that if learning in Grade 3 is reduced by half, reflecting both the time out of school and additional learning regression while away from school (as in the phenomenon of “summer learning loss”)3, then learning in Grade 10 is 1.5 years lower than the counterfactual of no shock.

I then model two remediation approaches.

The first models’ short-term remediation efforts when these children return to Grade 4. It assumes one-third of the Grade 3 curriculum is covered during Grade 4 before moving on to Grade 4 topics. Starting in Grade 5, instruction reverts to the previously established (pre-pandemic) curriculum and instructional levels. This is a “short-term” remediation model. Modelling this with the more conservative assumption of the loss of one-third of Grade 3 learning from school closures, such short-term remediation mitigates about half of the Grade 10 learning deficit, reducing the long-term impact of the shock to one-half of a school year.

Finally, I model an instruction reorientation strategy which combines short-term remediation with long-term reorientation of instruction to children’s learning levels. The steps that education systems will need to take to conduct remedial education, including instituting formative assessments to identify children’s learning levels, training and empowering teachers to conduct such assessments and adapt their instruction and pedagogical practices to students’ levels and needs, and prioritising children’s attainment of essential skills, are well-proven strategies for improving learning outside of a crisis context.4 The final model considers the outcomes if systems both conduct remedial instruction in Grade 4 as described above and reorient instruction and practices to children’s learning levels on a long-term basis. This scenario not only fully mitigates the effect of the shock but increases Grade 10 learning above the counterfactual of no shock by more than a full year’s worth of instruction.

Without the urgent and immediate attention of education systems to the question of how they will handle the learning losses from the temporary closures, the consequences for today’s children of the “short run” measures taken to combat COVID-19 will be long-run and large. Actions to protect children from these losses must be a top priority, even while the crisis persists.

II. Modeling learning

Modeling learning requires a specification of the parameters that drive the learning process. Changes to the parameters then allow modelling of counterfactual scenarios, such as shocks to the learning process or various policy priorities or scenarios. Kaffenberger and Pritchett (2020) develops a pedagogical production function (PPF) which models the learning gained by children at different points in a student distribution in a year of schooling. In this note, I use this PPF to model the COVID-19 learning shock. Here I provide a brief overview of the model, more details can be found in Kaffenberger and Pritchett (2020). The PPF is what, on average, child i with skill level s would learn if they attended grade G. In general terms this is represented as:

Learning process (LP) = LPG(si)

Drawing on the findings of the emerging literature on learning profiles, Kaffenberger and Pritchett (2020) assumes a trapezoidal functional form for the PPF, as in the equation on the left.

Where the learning in grade G of student i of initial skill s is a function of the width w, height h, slope r, and center πG of the trapezoid, as illustrated in Figure 1.

Equation for modelling the learning process

Two distinct features follow from this functional form. First, the PPF has a range of initial skill levels within which children learn and above and below which they do not. If the instructional process is too advanced relative to student skill level (e.g. teaching division to children who cannot recognize numbers) or too rudimentary (e.g. teaching number recognition to students ready for geometry) no new skills are gained. The PPF or instructional process at grade G is centred on a specific skill level, πG, and the width of the PPF, the range of initial child skills over which the instructional process produces learning, is the parameter w. Therefore, a child too far behind will learn nothing from attending grade G (Figure 1).

Second, the trapezoidal shape has a slope parameter, r, so that learning can vary across the initial student distribution. Kaffenberger and Pritchett (2020) assumes an upward sloping trapezoid, with r > 0, so that high performers learn more per year than low performers.3 hmin is the amount learned by the childwith the lowest initial skill level that learns anything at all, and hmax is the amount learned by the child withthe highest initial skill level that learns anything.

Figure 1 illustrates this trapezoidal PPF learning process.

Figure 1: Modelling the learning process: PPF(LP(w,h,r,πG),si)

Illustration that shows how the author modelled the learning process: PPF(LP(w,h,r,πG),si)

A final parameter, pace, p, represents the shift in the PPF from one grade to the next, as the level of instruction shifts to the next grade level. To model learning, we iterate the learning process by applying the PPF to an initial distribution of student skills to produce a new distribution based on the learning acquired in grade G. These dynamics are illustrated in Figure 2. The PPF then shifts to the right according to the pace p to produce the learning for grade G+1.

Figure 2. Initial and end of grade student skill distribution

Graph showing Initial and end of grade student skill distribution

Kaffenberger and Pritchett (2020) calibrate the model to replicate average Grade 10 learning in mathematics in the seven low- and middle-income countries that participated in the PISA for Development (PISA-D) assessment.4 This is just one option for calibration and allows us to model long term learning (through at least Grade 10). The model could be calibrated to other data sources and this will likely be undertaken in future work. For the current modelling, this calibration allows us to model a “typical” learning process in a low- or middle-income country.5

PISA assesses children who are 15 years old and in school and in at least Grade 7. Eligible 15-year-olds are on average in Grade 10. In OECD countries, as a comparison, 89 percent of 15-year-olds are eligible, and PISA is standardized so that the mean score of these participating children is 500 and the standard deviation is 100. Among the PISA-D countries, 43 percent of 15-year-olds were eligible, the average score of participating children was 324, and the standard deviation was 74. In our calibration, we assume dropout is endogenous and that low performers dropout first. More details on the calibration process and dropout assumptions are provided in Kaffenberger and Pritchett (2020). We calibrate the PPF so that the top 43 percent of the Grade 10 distribution roughly replicates the observed PISA-D results, with the combination of parameters that comes closest to replicating the PISA-D results given in Table 1. In Kaffenberger and Pritchett (2020) and in the following modelled scenarios, we use grade attainment data from the World Bank’s EdAttain database, averaged across the seven PISA-D countries, to model dropout after each grade. In this note I assume enrolment and dropout stays constant at pre-pandemic levels, making the learning loss estimates optimistic if some children do not return to school (more details on this in Section III.C.).

Parameter PISA-D calibrated parameters

W (width)

153

hmax

49

hmin

26

r (slope)

0.15

P (pace)

45

N (π1, σ1)

N (0,20)

 

III. Modeling the COVID-19 learning shock and mitigation approaches

I use the calibrated PPF to model five scenarios. The first is the counterfactual of cohort learning with no shock, representing business-as-usual schooling. This serves as the comparison point for learning loss due to the shock – it is the counterfactual of no schooling disruption. Then I model two different learning loss scenarios, followed by two mitigation approaches.

III.A. How much long-term learning may be lost?

Using the calibrated PPF described in Section II, Kaffenberger and Pritchett (2020) estimates average Grade 10 cohort learning among PISA-D countries. While the PISA-D assessment only provides information on the learning of the portion of the cohort that took the assessment, with the parameterised model I can estimate learning trajectories and learning outcomes among the full cohort of in- and out-of-school children. This will serve as the base case counterfactual of learning with no shock.

The calibrated PPF produces average cohort learning at age 15 of 213 and cohort standard deviation of 126, on the PISA scale of OECD average 500 and standard deviation 100. This mean is much lower and the standard deviation much larger than those observed for the PISA-D eligible population, as cohort learning now includes a much longer left tail of low performers (i.e. those who never started or dropped out of school and were therefore ineligible for PISA-D). This distribution of cohort learning is the counterfactual to which the following modelled outcomes are compared.

To model COVID-19 related learning loss, I introduce a shock for the cohort of Grade 3 students and model their learning trajectories and outcomes through Grade 10.6 Today’s Grade 3 students will be in Grade 10 in 2027, three years before the SDG target completion date, making this a relevant cohort for understanding not only implications for long term learning loss but also repercussions for reaching international learning goals. I reduce their Grade 3 learning gains by one-third, the equivalent of about a three-month school closure7 , and, in this initial scenario, assume no remedial efforts are made when children return but that schools return to “business as usual” curriculum and teaching. I also assume, for simplicity, (here and in the subsequent scenarios) no additional school dropout due to the closures so that dropout follows the same trajectory as in the counterfactual. Assuming no changes in dropout makes this a conservative estimate of learning loss, as discussed further in Section III.C.

Figure 3. Modelling long-term lost learning from COVID-19 shock and mitigation strategies for the current Grade 3 cohort: Equivalent years of learning behind/ahead in grade 10 compared to counterfactual of no shock

Chart showing Modelling long-term lost learning from COVID-19 shock and mitigation strategies for the current Grade 3 cohort

Similar to Andrabi, Daniels, and Das (2020), I find that reducing learning by one-third of a school year in Grade 3 reduces later learning by a much larger amount. When this cohort of current Grade 3 students reaches Grade 10, their learning on average is a full year lower than what it would have been had there been no shock (Figure 3).8 While this may at first seem extreme, the mechanism is clear. The lost learning puts children behind the curriculum, and without remediation they cannot keep up. They begin to fall outside the range of the PPF (i.e. outside the range of the curriculum and instruction) and cannot engage with the material. By Grade 10, nearly three-quarters (72 percent) of children who are still in school have fallen outside the range of the PPF and hence are making no learning gains (Figure 4).

Figure 4. Percent of in-school children learning zero, due to falling behind the level of instruction, at each grade level. COVID-19 learning loss shock occurs in Grade 3

Line graph showing the COVID-19 learning loss shock in Grade 3 and then predicts what happens going forward through the grades

Sustainable Development Goal 4 calls for all children to achieve minimum proficiency in reading and mathematics by 2030. One definition for minimum proficiency, established by UNESCO Institute for Statistics (UIS), is achieving a Level 2 on the PISA scale, roughly equivalent to a score of 400 (UNESCO, 2018).9 Because the model is calibrated to the PISA scale, I can estimate the proportion of children who will achieve the SDG in each modelled scenario. The percent of the current Grade 3 cohort that would reach the SDG goal of minimum proficiency in mathematics by Grade 10 drops from 7 percent with no shock to just 3 percent reaching the SDG goal with the shock (Figure 5).

Figure 5. Percent of Grade 10 cohort scoring in each band. Remediation + instruction reorientation produces learning gains far surpassing the counterfactual of no shock On PISA-like scale; 400 roughly corresponds with the SDG goal of “minimum proficiency”

Bar chart showing the percent of Grade 10 students in each band,

I next model another variation of learning loss, reducing Grade 3 gains by one-half, the equivalent of direct learning loss from school closure and additional learning regression during the time out of school. “Summer learning loss” is an established phenomenon in high-income countries. A (very) recent analysis suggests that in the United States, school closures due to COVID-19 could mean children return in the next school with less than 50 percent of the learning gains they would have had in math, and for some grades children could be a full year behind the gains they would have under normal circumstances (Kuhfeld and Tarasawa, 2020). Research on this topic is more sparse for lower income countries, but at least one study indicates that such “grade transition” loss does occur in lower income countries and can be severe (Slade et al., 2017). These suggest that even a 50 percent reduction in the Grade 3 gains may be conservative in terms of the learning loss students experience due to COVID-19 closures.

In this scenario, the Grade 10 learning deficit far surpasses the initial loss of one-half of a year’s learning. In Grade 10, today’s Grade 3 cohort has gained 1.5 years less learning than if the shock had not occurred. The percent of in-school children who have fallen outside the range of the PPF (i.e. behind the level of instruction) and are learning nothing is higher in every grade following the shock, reaching more than 80 percent in Grade 10. Finally, this larger shock further reduces the percent of the cohort who reaches the SDG target for math to just 2 percent. The percent of the cohort with learning levels below 200, considered very low, rises to 73 percent.

III.B. How much difference could remedial efforts make?

There are steps that can be taken to mitigate some or all of these devastating outcomes. It has been widely acknowledged that remedial efforts will be needed when children return to school. A joint framework by UNESCO, UNICEF, the World Bank, and the World Food Programme has called on education systems to implement large scale remediation programmes (UNESCO et al., 2020). The World Bank has said that where full cohorts have missed content, especially in foundational subjects, “plans for teaching essential missed material should be integrated with plans for resuming progress through the curriculum”. How much could such remediation efforts mitigate the long-term effects of the learning shock?

For modelling mitigation, I use the more conservative learning loss assumptions from Section III.A., reducing Grade 3 learning by one-third, and first model remediation when children re-enter school in Grade 4. This is modelled by reducing the curricular pace, p, from Grade 3 to Grade 4 by one-third, representing some of the Grade 3 topics being covered in Grade 4. After Grade 4 the curriculum reverts to the previously established levels (and pace). This is the equivalent of a short-term remediation effort.

Short-term remediation makes up for some of the long-term learning loss. Average Grade 10 cohort learning with remediation is half of a school year higher than without remediation. It does not, however, fully make up for the learning loss of the shock, and cohort learning in Grade 10 is still 0.55 years behind the counterfactual of no shock. In the remediation scenario the percent of the Grade 10 cohort who reach the SDG for mathematics is 4 percent, compared with 3 percent with the shock and no remediation, and 7 percent in the counterfactual with no shock.

Finally, I model a scenario of remediation plus longer term “reorientation” of instruction. This scenario acknowledges the opportunity that education systems have to “build back better,” and particularly to do so building on capabilities they may gain in implementing remediation programmes. Evidence suggests that curricula, and resulting instruction, in many developing (and developed) countries are overambitious, covering many topics with limited time allocated to each. Teachers under pressure to complete the curriculum must rush through the content before students can fully grasp the new knowledge. As a result, many children miss out on foundational and essential skills and fall farther and farther behind (Beatty and Pritchett, 2015; Glewwe, Kremer, and Moulin, 2009). This is represented in the model as children fall outside the range of the PPF, unable to keep up with the pace of instruction, and stop making learning gains.

In Uganda, for example, a study of the national curriculum found that foundational English competencies receive very little emphasis before children are expected to move on to higher order skills (Atuhurra and Alinda, 2017). In 2015 Tanzania reformed its Grade 1 and 2 curricula, which at the time consisted of eight subjects including Vocational Skills and Information and Communication Technology. The reform radically simplified the curriculum, placing 80 percent of instructional time on foundational literacy and numeracy, and preliminary evidence shows large gains in the foundational subjects as a result (Mbiti and Rodriguez-Segura, forthcoming). These studies, and others, suggest that reorienting curriculum to children’s ability levels and ensuring adequate coverage of topics so that children can gain competency can substantially improve learning. The “Teaching at the Right Level” approach, pioneered by Pratham, provides further evidence that such tailoring of instruction to children’s ability levels can have large impacts on children’s learning, and such programmes are now expanding and scaling throughout Africa (Teaching at the Right Level, 2018).

If systems implement effective remediation when schools reopen, many of the building blocks for such adaptations of instruction to meet children where they are would be put in place. For a system to implement remediation efforts effectively, as modelled in the first mitigation scenario, teachers and schools require some ability to conduct formative assessments, to determine children’s learning levels when they return, and to identify the subjects in need of remedial attention. It also requires the ability to adapt instruction to accommodate these needs, tailor content, and adjust curriculum. Doing so will require that teachers receive training and professional development and that they are allowed and empowered to adapt the curriculum they typically are required to fully cover.

If systems embrace the current crisis as an opportunity to “build back better”, and they not only conduct remediation immediately upon return to school, but also carry the capabilities they gain in formative assessment, adaptation of instruction, and ensuring all children master foundational skills into the future, what could be the result for long-term learning? In this final scenario I build on the first mitigation scenario, modelling the shock that reduces Grade 3 learning gains by one-third plus remediation as described above. In addition, I reorient the curriculum for the remaining school years through Grade 10 so that after Grade 4 the pace is “optimised” to children’s learning levels. The “optimised” curricular pace draws from Kaffenberger and Pritchett (2020) which, for the calibrated PPF model, identifies the curricular pace that maximizes Grade 10 learning outcomes. This optimized pace is a pace of 35 on the scale used for the model, a reduction of 22 percent from a pace of 45 which reproduces PISA-D learning outcomes (and is “overambitious” per children’s learning levels). This implies better alignment between instruction and children’s ability and learning levels in each grade. By reducing the pace, fewer children fall behind the level of instruction because sufficient time is spent on content before moving on. With the slower pace, more children stay in the range of the PPF (and continue learning) longer, increasing learning outcomes.

This scenario that combines short-term remediation with long-term reorientation of instruction to children’s learning levels not only fully mitigates the long-term learning loss due to the shock, but also surpasses the learning in the counterfactual of no shock by more than a full year’s worth of learning. Remediation combined with long term reorientation of instruction produces average cohort learning of 259 in Grade 10 on the PISA-like scale, a whopping 2.3 years’ more learning than if the shock had gone unmitigated. This is also 1.3 years’ more learning than the counterfactual of no shock occurring. With remediation and instruction reorientation, 27 percent of the cohort achieves the SDG—nine times more than had the shock gone unmitigated, and even nearly four times more than the counterfactual of no shock.

Large learning gains are achieved in this scenario because the reorientation of instruction enables more children to continue learning for longer. The percentage of in-school Grade 10 children who are learning zero (because they have fallen below the range of instruction) in the remediation plus reorientation scenario is just 16 percent, compared with 72 percent with the unmitigated shock, and 43 percent in the counterfactual of no shock. Because instruction moves at a pace with which children can keep up, they continue learning and gaining new competencies.

Much of the gains come from reducing the number of children scoring between 100 and 200 on the PISA-like scale, and greatly increasing the percent scoring above 400 (Figure 5).

On one hand, it may be surprising that relatively simple efforts to tailor instruction to children’s ability levels could produce such large learning gains. These results, however, are in line with the large impacts achieved by many programmes that have worked to reorient instruction to children’s ability levels. Pratham’s TaRL approaches, for example, have effect sizes ranging from 0.08 to 0.70 standard deviations for relatively short-term programmes (ranging from 10-day remedial camps to a full year of reoriented instruction for foundational subjects) (TaRL, 2018). A computer-aided learning programme that adapted instruction to children’s individual learning levels achieved improvements of 0.29 standard deviations over just a 4.5 month period in India (Muralidharan, Singh, and Ganimian, 2019). The Tusome programme in Kenya achieved impacts of 0.6-1.0 standard deviations in English and Kiswahili learning outcomes after one year through a multifaceted programme that reoriented literacy instruction to ensure all children were learning (Freudenberger and Davis, 2017; Piper et al., 2018; Wilichowski, 2020). It is entirely conceivable that adjusting curriculum pacing and instructional focus to be in line with children’s pace of learning could produce such large long-term effects.

On the other hand, perhaps it is not surprising at all that a multifaceted effort to conduct remediation combined with long-term reorientation of instruction would produce such large gains. Here, it is worth keeping in mind that the large, modelled impacts have incorporated a major shock that, if left unmitigated, would have reduced long term learning by a full school year. An effort that can so fully mitigate a major shock to learning and far surpass the counterfactual learning if there had been no shock deserves attention and consideration by education systems planning for reopening.​​​​​​

Table 2. Summary of modeled outcomes of COVID-19 learning loss shocks and mitigation scenarios
 

Grade 10 average cohort learning (PISA-like scale)

Equivalent years of learning ahead/behind of counterfactual in grade 10

Percent of in-school children in grade 10 learning zero

Percent of grade 10 cohort above PISA 400 (achieving minimum proficiency)

Percent of grade 10 cohort below PISA 200 (very low learners)

Counterfactual – no shock

213 N/A 43% 7% 51%

Shock reduces grade 3 learning by 1/3

175 -1.04 72% 3% 65%

Shock reduces grade 3 learning by 1/2

157 -1.55 82% 2% 73%

Shock reduces grade 3 learning by 1/3 + remediation

193 -0.55 61% 4% 58%

Shock reduces grade 3 learning by 1/3 + remediation + instruction reorientation

259 1.27 16% 27% 36%

 

III.C. Modelled learning loss may be optimistic, making mitigation even more critical

The learning loss modelled in the above scenarios may be optimistic, and actual learning loss could be even worse, making mitigation efforts all the more critical. The modelled learning loss assumes no additional dropout as a result of school closures; it assumes that enrolment and completion rates maintain pre-shock levels once schools reopen. After being out of school for an extended time, however, some (or many) children may not return (UNESCO et al., 2020). During their time out of school, some children may be put to work to help support their household and have to remain in employment once schools reopen. Reductions in children returning to school would further reduce learning outcomes (increasing learning loss) due to the shock.

This model also does not build in macro shocks such as reductions in education spending or losses of parental income. The World Bank forecasts that education budgets in 2020 could fall by as much as 4.2 percent in Sub-Saharan Africa and 6.4 percent in South Asia, though there is much uncertainty in making such forecasts at this stage in the crisis (Al-Samarrai, 2020). The same World Bank report states that in low-income countries households contribute, on average, 29 percent of education funding, and households are being hit hard economically in the crisis. Reduced income will reduce households’ abilities to invest in education. Remittances are also expected to drop significantly, and education is often among the top uses of remittances by receiving households. While education spending is poorly correlated with learning outcomes (World Bank, 2018; Beatty et al., 2018; de Ree et al., 2018) large reductions, if maintained in the long run, could have detrimental effects on outcomes.

IV. Conclusion

The COVID-19 pandemic, which began as a health crisis, has also had tragic economic and educational consequences. The model presented in this note suggests that the long-term repercussions for children’s learning could be devastating, with today’s Grade 3 students losing as much as 1.5 years’ worth of learning (or more) by the time they reach Grade 10 as a consequence of their time out of school. Governments can, however, introduce measures that mitigate some or all of these consequences. The model suggests that effective remediation efforts immediately upon return to school could reduce long-term learning loss for the cohort of Grade 3 students by half. Beyond immediate efforts, there is an opportunity for systems to use the skills gained from implementing large-scale remediation programmes to reorient instruction to better match children’s skill levels in the long run. Such efforts, the model suggests, could not only fully mitigate the consequences of the shock but also surpass learning outcomes compared to the counterfactual of no shock.

All of these mitigation efforts require planning. As systems continue their remote learning programmes, they will also need to begin planning for reopening, putting in place the tools for remedial programmes and, if feasible, beginning to train teachers remotely. As they do so, they should consider how they can build programmes and train teachers in ways that can continue to produce benefits beyond the period immediately following reopening. The present crisis presents an opportunity for systems to build back better.

Footnotes

1 Their test scores were 0.24 standard deviations behind unaffected children and the average annual gain is only .15. The study also finds that, due to government and international relief efforts, affected households were no worse off economically than unaffected households (in terms of wealth, consumption, or infrastructure), so the learning affects are not consequences of economic hardships.

2 The choice of Grade 3 is illustrative, as some learning has already occurred, but enough years remain to model long-term learning loss. The shock could be modelled for children in any grade.

3 This is based on the literature on learning profiles showing that low performers often learn less per year than high performers. It could just as easily be reversed, such as in a remedial environment, and modelled so that low performers learn more and high performers learn less.

4 The PISA-D programme adapted the traditional PISA assessment to lower income countries and included capacity building for participating countries to conduct the assessment. Participating countries included Cambodia, Ecuador, Guatemala, Honduras, Paraguay, Senegal, and Zambia.

5 As a comparison point, for the six PISA-D countries that also have learning poverty measures by the World Bank, the average learning poverty level is 67 percent. Across all low- and middle-income countries with learning poverty measures learning poverty is 53 percent and in low-income countries it is 89 percent, so the PISA-D countries are within reasonable levels for these income levels.

6 The model could be run with a shock to any grade cohort. I choose Grade 3 as some learning has already occurred, and enough schooling remains to model long term consequences of the shock.

7 A similar assumption has been used in other modelling efforts (Cummiskey and Stern, 2020). Given that the majority of low-income countries have no announced distance learning plan, it is not unreasonable to assume that learning will effectively cease while children are out of school (Carvalho and Hares, 2020).

8 The average in-school child in the base case counterfactual learns 36 points on the PISA-like scale per year, and in this shock scenario Grade 10 children are 38 points behind the counterfactual of no shock.

9 The exact scores for achieving Level 2 are 420 for math and 407 for reading.

References

Al-Samarrai, S. (2020) The Impact of the Covid-19 Pandemic on Education Financing. World Bank. http://pubdocs.worldbank.org/en/734541589314089887/Covid-and-Ed-Finance-final.pdf.

Andrabi, T., Daniels, B., and Das, J. (2020) Human Capital Accumulation and Disasters: Evidence from the Pakistan Earthquake of 2005. RISE Working Paper Series. 20/039. https://doi.org/10.35489/BSG-RISE-WP_2020/039.

Atuhurra, J. and Alinda, V. (2017). Basic Education Curriculum Effectiveness Analysis in East Africa: Using the ‘Surveys of Enacted Curriculum’ Framework to Describe Primary Mathematics and English Content in Uganda. MPRA. Paper No. 79017.

Beatty, A. et al. (2018). Indonesia Got Schooled: 15 Years of Rising Enrolment and Flat Learning Profiles. RISE Working Paper Series. 18/026. https://doi.org/10.35489/BSG-RISE-WP_2018/026.

Beatty, A. and Pritchett, L. (2015). Slow Down, You’re Going Too Fast: Matching Curricula to Student Skill Levels. International Journal of Education Development. 40: 276-288

Carvalho, S. and Hares, S. (2020). More from Our Database on School Closures: New Education Policies May Be Increasing Educational Inequality. Center for Global Development Blog. https://www.cgdev.org/blog/more-our-database-school-closures-new-education-policies-may-be-increasing-educational

Cummiskey, C. and Stern, J. (2020). Calculating the Educational Impact of COVID-19 (Part II): Using Data from Successive Grades to Estimate Learning Loss. RTI Shared Blog. https://shared.rti.org/content/calculating-educational-impact-covid-19-part-ii-using-data-successive-grades-estimate.

de Ree, J. et al. (2018). Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia. The Quarterly Journal of Economics. Volume 133, Issue 2. May 2018. Pages 993–1039. https://doi.org/10.1093/qje/qjx040.

Freudenberger, E. and Davis, J. (2017). Tusome External Evaluation Midline Report. Management Systems International for USAID. https://pdf.usaid.gov/pdf_docs/PA00MS6J.pdf.

Glewwe, P., Kremer, M., and Moulin, S. (2009). Many Children Left Behind? Textbooks and Test Scores in Kenya. American Economic Journal: Applied Economics. 1:1, 112–135. https://www.sciencedirect.com/science/article/pii/S0738059314001217.

Kaffenberger, M. and Pritchett, L. (2020). Failing to Plan? Estimating the Impact of Achieving Schooling Goals on Cohort Learning. RISE Working Paper Series. 20/038. https://doi.org/10.35489/BSG-RISE-WP_2020/038.

Kuhfeld, M. et al. (2020) Projecting the Potential Impacts of COVID-19 School Closures on Academic Achievement. EdWorkingPaper. 20-226. Available: https://edworkingpapers.com/sites/default/files/ai20-226-v2.pdf

Kuhfeld, M. & Tarasawa, B. (2020). The COVID-19 Slide: What Summer Learning Loss Can Tell Us about the Potential Impact of School Closures on Student Academic Achievement. NWEA white paper.

Mbiti, I. and Rodriguez-Segura, D. (forthcoming), Evaluating Curriculum Reforms in Developing Countries: Evidence from Tanzania. RISE Working Paper Series.

Muralidharan, K., Singh, A. and Ganimian, A. (2019). Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India. American Economic Review. 109 (4): 1426-60. DOI: 10.1257/aer.20171112.

Piper, B. et al. (2018) Scaling Up Successfully: Lessons from Kenya’s Tusome National Literacy Program. Journal of Educational Change. 19, 293–321 (2018). https://doi.org/10.1007/s10833-018-9325-4.

Slade, T.S. et al. (2017). Is ‘Summer’ Reading Loss Universal? Using Ongoing Literacy Assessment in Malawi to Estimate the Loss from GraDe-transition Breaks. Research in Comparative and International Education.12(4), 461–485. https://doi.org/10.1177/1745499917740657.

Teaching at the Right Level (TaRL). (2018). Evidence. https://www.teachingattherightlevel.org/evidence/.

UNESCO. (2018) Metadata for the Global and Thematic Indicators for the Follow-up and Review of SDG 4 and Education 2030. Available: http://tcg.uis.unesco.org/wp-content/uploads/sites/4/2019/04/sdg4-metadata-global-thematic-indicators-en.pdf.

UNESCO et al. (2020) Framework for Reopening Schools. Accessed: https://unesdoc.unesco.org/ark:/48223/pf0000373348.

Wilichowski, A. et al. (2020). Building Back Better: Accelerating Learning When Schools Reopen, and What Kenya’s Tusome Program Can Teach Us. World Bank Blogs. https://blogs.worldbank.org/education/building-back-better-accelerating-learning-when-schools-reopen-and-what-kenyas-tusome.

World Bank. (2018). World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. doi:10.1596/978-1-4648-1096-1.

World Bank. (2020) The COVID-19 Pandemic: Shocks to Education and Policy Responses.May 2020. Available: https://openknowledge.worldbank.org/bitstream/handle/10986/33696/148198.pdf

World Bank Live. (2020). The Covid-19 Pandemic: Shocks to Education and Policy Responses. Accessed: https://live.worldbank.org/covid-19-shocks-education-policy-responses#discussion.

Citation:

Kaffenberger, M. 2020.  Modeling the Long-Run Learning Impact of the COVID-19 Learning Shock: Actions to (More Than) Mitigate Loss. RISE Insight. https://doi.org/10.35489/BSG-RISE-RI_2020/017