The UN just launched a new Commission chaired by former UK Prime Minister Gordon Brown to better understand how to finance global education. Governments in developing countries already spend a huge amount in absolute terms - about a trillion dollars a year - on education (though this isn’t such a huge amount per student) and are likely to spend more. 

So what can research tell us about the best ways to spend all that money? Is it better textbooks? New technology? Smaller classes? The answer that emerges from a recent RISE-commissioned evidence review by Paul Glewwe and Karthik Muralidharan is, perhaps both disappointingly and unsurprisingly: “it’s complicated.”

Just focusing on more “inputs” pumped into existing systems–new school buildings, smaller classes, or more hours in school – are unlikely to be either consistently successful across countries or produce massive gains.

There is evidence that policy actions like focusing on improving teaching in the classroom and improving the governance and accountability of schools and teachers show promise.  

But what we really learn is the importance of detail of program design, nuance in implementation, and local context. Policies that look the same on paper can have massively different impacts on different kinds of students, depending on what other policies are in place in the specific education system in question. In many cases the differences in the empirically estimated impact on learning of the same policy in different places is larger than the differences in the impact of totally different policies in the same place. “What” is being done is less important than “Who,” “How,” and “Where.”

To give an example from the Glewwe and Muralidharan paper  – four different studies found that distributing textbooks and resources to schools had no impact on learning – but for four completely different reasons (and so with different implications about what might work and hence what should be done differently next time). In one case the books were just locked up in a storage cupboard and never made it to the kids, in one the textbooks were just too hard, so that only the few brightest kids could even read them, in another case they were effective the first year, but the second year parents stopped spending their own money on books, so the overall impact dropped back to zero in the second year, and in another there was only a positive impact where teachers were also being paid based on their performance.

This example highlights six key points made by Glewwe and Muralidharan about how we can do research better.

1. Pay attention to detail. Just like in the textbook example, the precise detail of a policy can make all the difference between whether it works or not. We should be drawing on theory about systems in thinking about these details.
2. Pay attention to differences between students. Many policies work differently for high and low ability students, or for boys and girls. Just looking at average effects will miss this. “Heterogeneity across students is likely to be a first order issue.”
3. Focus more on why policies work or don’t work. We can’t learn anything for the future or a different country if we don’t understand the entire causal pathway of the theory of change. 
4. Focus on interactions or dependencies between policies. There are likely many examples of policies that will only work if another policy is already in place. Another reason we need to think about systems.
5. Better global standards to ease comparisons. On test scores, the rich countries have PISA, but nothing comparable exists that covers all the countries in the world, making comparisons between policies in Pakistan and Malawi difficult.
6. Costs matter.  Poor countries have scarce resources.  Many studies focus exclusively on efficacy and provide nothing about costs, making it very hard to say anything about cost effectiveness. Glewwe and Muralidharan discuss just nine credible studies that have even made an attempt to say something about cost effectiveness.

This is where research like RISE comes in. If the right policies are going to be different for different children, and different in different places, and dependent on a wide range of other policies, it could take forever to test every possible combination. Glewwe and Muralidharan waded through 13,437 studies on what works in education in developing countries, finding just 118 high quality studies that can tell us something credible about causality, so we have a long way to go.

Part of the answer is no doubt being smarter about RCTs – embedding the design in theory and collecting data on steps along the causal chain of the theory of change.

Another part is perhaps drawing on lessons from the new literature on complex adaptive systems and new approaches to building state capability. As well as asking whether we can scientifically and rigorously establish if X causes Y, we should also be asking “what are the conditions in which the actors in a system have the incentives to do their own small-scale experimentation, seek out their own innovations and answers, and then share this new knowledge with colleagues?” How can we motivate people and systems to be driven to strive to be constantly trying new things and trying to improve performance and outcomes for students?

This is a little of what we’re hoping to see from the RISE country research teams, the first three of which are soon to be launched. Watch this space.