Avoidable failure

School systems are failing, and we don’t know what to do about it — but that doesn’t make it inevitable.

1. Lant Pritchett opened with an anecdote (gasp!) about his visit with students of failing, violent, abusive schools in a Cairo slum, his meeting the next day with the Prime Minister, and how glad he was the PM didn’t ask him how to fix the schools — because the academic literature has no convincing answers.

2. It’s not innate, it’s not just poverty — cross-country learning gaps are big and they emerge due to differential school productivity, Abhijeet Singh argued using comparable longitudinal data from five developing countries from the Young Lives project.

3. On the upside, good schools have both social externalities and long-run impacts on society, as Leonard Wantchekon demonstrated with his research on modern effects of missionary schools under colonialism in Benin.

Beyond RCTs, or bigger, better RCTs?

So what would a research program on education systems look like? Several presenters argued that randomized controlled trials — RCTs — are not restricted to narrow impact evaluations, but can answer big questions about systemic reform.

4. If you give me enough money and a willing government, I’ll randomize whole systems, said Karthik Muralidharan, essentially meaning that if we think school governance at, say, the district level matters more than individual school interventions, then we should randomize at the district level, sample size permitting.

5. Annie Duflo’s RCT of a teaching assistants program run by IPA and the Ghanaian Ministry of Education illustrated another approach to thinking about scalability, i.e., evaluating interventions embedded in dysfunctional government bureaucracies (which we know can produce very different results than small-scale NGO pilots).

6. Or RCTs can use interactions between different policy elements to understand the conditions necessary to make X work — as Karthik and Isaac Mbiti did in Tanzania, showing extra money only raised school performance when coupled with management training (and vice versa).

7. Relatedly, in the Gambia even delivering school grants and management training together had no effect on average, but results hinged on context . The program produced some impact in localities with more educated parents.

Others disagreed, arguing that we need methods to evaluate reforms where n=1.

8. Researching system reform often presents a small-N problem, said Kara Hanson, who illustrated ways of dealing with this in health research.

9. Luis Crouch fleshed out a practical approach to evaluating systemwide reforms and making (non-statistical) causal inference when n=1.

10. Alec Gershberg pushed for n=5, pointing to the potential for comparative case studies across countries.

But enough about methods, what kinds of reforms offer the most promise for dramatically raising student learning?

Public private partnerships: educational success and political failure?

The solution many poor households choose when faced with unaccountable public schools for is to exit:

11. Jishnu Das summarized a decade of data collection from Pakistan on the rise of low-cost private schools and experimental work to highlight and overcome the market failures that prevent them from working better.

Das’s presentation hinted strongly at the potential for voucher programs to combine private efficiency with public finance to ensure greater equity. Other presenters were armed with evidence on precisely such programs.

The results are positive…

12. Vouchers for poor students in Colombia to attend private vocational schools not only helped students, leading to higher graduation rates and higher earnings at age 30, as Michael Kremer showed; they also generated enough extra tax revenue to pay for the program.

13. According to Felipe Barrera-Osorio, building publicly funded private primary schools in Sindh Province, Pakistan, generated big gains in enrollment raised and test scores by a whopping 0.67 standard deviations.

14. Similarly, public funding for private schools in Seoul, South Korea, produced fewer violent incidents, higher college attendance, and better test scores — all for the same cost.

…but the politics don’t always work.

15. “If a Minister of Education is asking for advice on education reform, the first question you need to ask is ‘What kind of unions do you have?’” said Ben Schneider, noting that even the successful Colombia voucher program had been killed.

16. Illustrating these political challenges, James Habyarimana and colleagues found that PPP schools in Uganda attracted a large number of students and produced better test scores, but Prime Minister Museveni still wants to scale back the program because it’s not as politically visible as the higher cost strategy of building new public schools.

Are great teachers born or bought?

If public-private partnerships weren’t controversial enough, there was also an extended discussion of high-stakes testing and teacher incentives. We know that teacher quality matters HUGELY for student outcomes.

17. Remember that experimental study that made headlines showing a good kindergarten teacher in the U.S. is worth about $320,000 a year for his or her students’ future earnings?

18. Norbert Schady and co have done something similar for Ecuador, and found that good teachers are worth even more in Ecuador, relative to local GDP.

19. Sadly, teaching quality is often abysmal: Deon Filmer presented results showing that the average teacher in Mozambique, Uganda, and Tanzania is absent from their class more than half the time, and fewer than 3 percent of teachers in Mozambique, Nigeria, and Togo can competently grade homework based on the curriculum they teach.

Teacher quality is not an innate, immutable trait, though; incentives matter.

20. Chinese school teachers are effectively in a tournament for promotions, according to Albert Park, meaning they work hard to get more wages, but stop trying once their chance for promotion passes by.

21. Derek Neal listed some pitfalls of bad incentive design, and described what the ideal teacher incentive system would look like in theory.

22. Prashant Loyalka has actually tested Neal’s theory in China, and found bigger impacts on learning than with simper pay-for-perofrmance schemes.

Alas, high-stakes testing brings risks.

23. If you’re not careful, using standardized test scores as signals of school quality may encourage schools to manipulate which kids come to school as happened in Chile says Felipe Gonzalez.

24. And using tests to allocate kids to schools can end up being incredibly inefficient, as Kehinde Ajayi showed for Ghana.

25. On the plus side, getting the exam system right can encourage better assessment and better teaching lower down in the system, as Newman Burdett argued based on evidence from Pakistan.

A summary of summaries

The RISE program is just getting ready to launch six years of field research in several developing countries. So what do we take away from this discussion? Methodologically, the answer will probably be “all of the above.” The power of RCTs to provide compelling answers to policy questions is undeniable. But we’re interested in studying big, ambitious reforms that may already be ongoing and not subject to experimental study. We’ll need a diverse toolkit.

Substantively, the papers at the RISE conference suggested big impacts from focusing on systems for motivating teachers to perform and recruiting more high-quality teachers, as well as partnerships with the private sector. But caution is in order. Big open questions remain about the equity implications of market-oriented reforms, not to mention their political sustainability. As Dave Evans noted in the panel discussion, if you read five summaries of the evidence you get five different answers for “what works.” The right answers probably depends on myriad factors we don’t yet understand, from context to state capacity to political resistance to reform — all good topics for RISE research going forward.

Thanks to our CGD colleague Mari Oye for helping to organize the conference (and run the twitter account quoted above) and to our co-organizers, Clare Leaver and Jacobus Cilliers from Oxford University’s Blavatnik School of Government. The RISE program is supported by the UK’s Department for International Development.

This blog first appeared on the CGD website, 2 July 2015