Jason Silberstein
Blavatnik School of Government, University of Oxford
Blog
Knowledge is Not Always Power: How a New Study Uses Systems Thinking to Reconcile Why Some Information Interventions Succeed Where Others Fail
Authors
The puzzle: the mixed record of information interventions
The idea that more and better information can empower ordinary people, hold government accountable, and improve public services carries great intuitive appeal. Donors have spent many millions on development programs to provide information to the general public. In the education sector alone, different interventions have tried providing information on school quality (via report cards, rankings, or citizen-led assessments), information on the labor market returns to schooling, and information on the ways that communities can get involved in school governance. Unfortunately, the intuition that information matters hasn’t always been borne out by the evidence. While some meta-analyses of the education literature conclude that providing information helps on average, others find small average effect sizes, and a meta-meta review emphasizes the wide heterogeneity of impact. The entirely unsatisfying conclusion as to whether providing more information “works” seems to be: it depends.
The insight: there are multiple system determinants of useful information
A new study by RISE Nigeria Team PI Leonard Wantchekon and co-author Katrina Kosec points to the way out of this cul-de-sac. To answer the question “Can information improve rural governance and service delivery?” they review 48 studies from developing countries and find the usual heterogeneous effect sizes. However, rather than turning inward and looking for patterns among specific types of interventions, they refocus attention outward toward the system into which these interventions are being introduced. Information is only effective, they propose, in the presence of three necessary conditions: "The information must be relevant to the individual, the individual must have the power to act on it, and the individual must have the incentive to act on it". This is the exactly the sort of system-level thinking that RISE is dedicated to. The RISE systems framework also eschews particular interventions, and instead looks outward toward the key relationships that compose education systems and the different variables that strengthen or undermine each of those relationships.
Let’s examine each of Kosec and Wantchekon’s three conditions in turn. The paper’s first condition is that information must be relevant. Consider, as an example, the role of information in the relationship between a parent and school, where the parent seeks to hold the school accountable for the education offered to their child. For information to improve this relationship, Kosec and Wantchekon argue that it must be relevant in the eyes of the parent. They outline different dimensions of relevance including salience, accuracy, credibility, and specificity. This tracks closely with the RISE message that information must be regular, reliable, and relevant to have any chance of influencing relationships in ways that improve learning.
The paper’s second condition is power. For information to be actionable, the parent in our example must have some accepted authority vis-a-vis the school. The RISE framework refers to this as “delegation,” or the parent’s ability to define the school’s objectives and scope for action.
The paper’s third and final condition is incentives. Relevant information in the hands of a parent is still impotent – and their power is not credible - if they cannot take a decision or action that is “payoff relevant” for the school. The analogy in the RISE framework is the concept of “motivation.” A parent must be able to act on information in ways that impact the motivation of the principal or teachers (i.e. social sanctions, or influence with local government). This also links with another aspect of relationships in the RISE framework, which is control over finance. If parents have a say over school resources - perhaps via a school committee’s sign-off on the school budget, or a parent’s ability to enroll their child in private school and cost the school funding – then it makes it more likely that the school will listen to them.
Table 1: System Determinants for Information to be Useful
| Typology from Kosec and Wantchekon (2020) | RISE Typology from Pritchett (2015) | Example (Parent as principal, school as agent) |
|---|---|---|
| Relevance | Information | Do parents get good information about the quality of education at their child’s school? |
| Power | Delegation | Do parents have the authority to tell the school and teachers what they expect from their child’s education? |
| Incentives | Motivation | Can parents take action in a way that will affect the school and individual teachers’ payoffs? |
The claim at the heart of the paper is that the provision of information will only have an impact when all three conditions – relevance, power, and incentives - are met. This is what the authors poetically dub the “constellation” of conditions, or what RISE calls coherence.
The payoff: how Kosec and Wantchekon’s 3 conditions can help make sense of disparate results
Kosec and Wantchekon then put their proposed conditions to the test. In their review of 48 studies that provide information, including seven focused on education, they find positive impacts only when all three conditions are satisfied. In studies where one or more of their conditions go unfulfilled they find mixed, zero, or negative effects.
To fully appreciate the explanatory power of this pattern, it is useful to zoom in and consider why the exact same kind of information intervention, when introduced into different systems, has wildly divergent impacts. For example, consider two different “report card” interventions – one in Pakistan, the other in Kenya - that provided information to parents about their child’s academic performance. The intervention in Pakistan had big positive impacts: about a 0.1 standard deviation increase in the quality of both private and government schools, lower private school fees, and modest increases in overall enrollment, all of which were sustained 8 years later. A broadly similar intervention in Kenya, in sharp contrast, had no impact whatsoever, not even on intermediate outcomes such as parental actions (i.e. interactions with schools, or helping children with homework). As explored in Table 2 below, the successful intervention in Pakistan was introduced into a coherent system that met all three of the “constellation” of conditions. The intervention in Kenya was introduced into a system that failed to meet both the relevance and power conditions.
Table 2: Applying Systems Thinking to Make Sense of the Disparate Results of the Same Intervention
| System Conditions | |||
| Relevance? | Power? | Incentives? | |
|---|---|---|---|
| Pakistan Andrabi et al. (2017) |
Yes. The report cards included specific information that contextualized each child’s score with her school’s average score and the average score of all other schools in the village. Parents understood the report card results, and correctly updated their opinion of the quality of their child’s school. | Yes. The report cards were introduced into village school systems which operate as closed education markets with high numbers of low-cost private schools. In this competitive system, parents could exert their “voice” within schools, and the number of parent-school interactions greatly increased following the report cards at both government and private schools. While speculative, school responsiveness to parents might have been strengthened by parental “choice”, or the credible threat of exit to another school. | Yes. In private schools, parents could exercise “choice” and enroll their child elsewhere, a direct financial incentive for the private schools. Interestingly, having leverage did not mean parents had to exercise it: learning gains were driven by schools improving the quality of their teachers, not by students changing schools. Parental sway over government schools and teachers is more opaque but might relate to levers of social accountability. |
| Kenya Lieberman et al. (2014) |
No. The information did not change parents’ prior understanding of their children’s academic performance. There was also no relevant benchmark for parents to compare their child’s score to. For example, because a single test of Grade 2 basic skills was done with students of all ages (6-16), older students received “passing” grades even though they would have failed a more grade-appropriate test. | No. Parents reported a low sense of responsibility to improve education, and a large majority said they would not know what actions to take to address problems in their child’s school. | Yes. For example, nearly half of parents were confident in their ability to influence local government decisions. |
Note: The Pakistan study was not one of the 48 studies reviewed by Kosec and Wantchekon (2020), and judgments and reasoning on its fulfillment of the three conditions are the author’s own based on results reported in the paper. The Kenya study was reviewed in Kosec and Wantchekon (2020). The judgments follow theirs, although the reasoning is again the author’s own based on results reported in the paper.
The takeaway: Systems, not just program design and implementation, are a key determinant of what works
A legitimate objection to raise at this point might be that the two interventions compared above were different in many ways, not just in terms of the system they were introduced into. Might their respective success and failure be attributed to those other differences? It’s true that the two interventions differed significantly in terms of program design (i.e. the report cards in Pakistan were given to all parents in a village, whereas in Kenya the assessment results were only delivered to around 12 households per village) and implementation (Pakistan was an experimental RCT overseen by its academic authors, whereas the Kenya intervention was carried out by Uwezo, an East African NGO). These differences assuredly matter, as has been well-demonstrated elsewhere. The lesson here is that the system is also an important determinant of context. Even a perfectly designed and executed program will fail if other system determinants of success are not in place. Rather than thinking about particular interventions in isolation, we need new ways to recognize the patterns in functional education systems. If we can, using frameworks like the one proposed by Kosec and Wantchekon, the result may be nothing less than turning conventional wisdom on its head: more and better information can empower ordinary people only if they independently possess the power to do something with it.
Bibliography:
Andrabi, Tahir, Jishnu Das, and Asim Ijaz Khwaja. 2017. "Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets." American Economic Review, 107 (6): 1535-63.
Andrabi, Tahir, Jishnu Das, and Asim Ijaz Khwaja (2019). “Long(er) Term Impacts of a Report Card Intervention.” RISE 2019 conference presentation accessed at: https://www.youtube.com/watch?v=cvOHvrXxcV0&t=10333
Atuhurra, Julius and Michelle Kaffenberger (2019). System (In)Coherence Seen through a Curriculum Lens: Ugandan Teachers Face Conflicting Demands from Curriculum and Examination Bodies. RISE Blog.
Bold, Tessa, Mwangi Kimenyi, Germano Mwabu, Alice Ng'ang'a and Justin Sandefur (2013). Scaling Up What Works: Experimental Evidence on External Validity in Kenyan Education. Center for Global Development Working Paper 321.
Banerjee, Abhijit V., Rukmini Banerji, Esther Duflo, Rachel Glennerster, and Stuti Khemani. 2010. “Pitfalls of Participatory Programs: Evidence from a Randomized Evaluation in Education in India.” American Economic Journal: Economic Policy 2 (1): 1–30.
Evans, David K.; Popova, Anna. 2016. What Really Works to Improve Learning in Developing Countries? : An Analysis of Divergent Findings in Systematic Reviews. Published by Oxford University Press on behalf of the World Bank.
Jensen, Robert. 2010. “The (Perceived) Returns to Education and the Demand for Schooling.” Quarterly Journal of Economics 125 (2): 515–48.
Kosec, K., & Wantchekon, L. (2020). Can information improve rural governance and service delivery? World Development, 125, 113.
Lieberman, E. S., Posner, D. N., & Tsai, L. L. (2014). Does information lead to more active citizenship? Evidence from an education intervention in rural Kenya. World Development, 60. 69–38.
Murnane, R. J., & Ganimian, A.J. (2014). “Improving Educational Outcomes in Developing Countries: Lessons from Rigorous Evaluations.” Unpublished manuscript.
Pritchett, Lant. 2015. “Creating Education Systems Coherent for Learning Outcomes: Making the Transition from Schooling to Learning”. RISE Working Paper Series. 15/005.
Author bios:
RISE blog posts and podcasts reflect the views of the authors and do not necessarily represent the views of the organisation or our funders.