• How do the central ministries polices improve student learning? What are the factors that contribute to the success of the policies in decentralised Indonesia?
• What reforms do innovative districts focus on to improve learning? How do these reforms work? Does student learning in these districts improve over the life of the RISE project? Do innovations spread across districts and to national policy?

Figure 1 gives a schematic overview of the studies we envision. We will look at both at national and district initiated policies, how they interact with one another, and how they impact learning. For centrally initiated policies, we have selected reforms in each of the key areas of the teacher roadmap.

In particular, we envision the following studies:

Teacher professional development

The low scores on the national teacher competency test (Ujian Kompetensi Guru/ UKG) administered in November 2015 led the Government to implement a nationwide remedial programme (Guru Pembelajar, Learning Teachers) that provides training to teachers who performed poorly on the UKG. The training consists of a combination of online modules and class-based training covering academic and pedagogical content. As part of this training, each teacher will be provided with a customised programme that indicates the modules that should be accomplished to reach competency in the areas where he or she failed. Because the programme was targeted to those who failed the test, i.e., teachers who did not reach the minimum score required, the research intends to compare the group of teachers who narrowly passed the test (those with the lowest passing scores) with teachers who narrowly failed (those with the highest failing scores). The team will undertake this research in the five learning laboratory districts (see the District Innovations section) where detailed data will be collected on teachers and learning over the course of the RISE Programme. The  administrative data will include results from national exams, follow- up teacher competency tests, and training statistics.

Computer-based testing

Research on the use of ‘computer-based testing’ (CBT) aims to understand how and whether computerisation of national exams affects scores; how this, in turn, affects the behaviour of bureaucrats, teachers, parents, and students; and whether student learning changes as the result of using a computer-based format, rather than the hand-written, paper exams that long have been the standard.  The rationale for introducing computerised tests was to curb a culture of cheating on this high-stakes exam. With the computer-based exam, students take the exam online, and each student receives a different set of questions, which greatly reduces the scope of cheating. The introduction of computer-based testing could improve learning outcomes As a consequence of lower scores and fewer passing grades, bureaucrats, head masters and students are likely to face increased pressure to improve national exam results. With less scope to do so though cheating, the hope is that all will work to ensure that students will be better prepared for the exam.

Understanding the system-level ramifications of this change is particularly important considering the potential for this reform to expand, and the potential for increased costs inherent with expanded use of computers for testing a growing number of students. The research team will undertake qualitative analysis of policy adoption using process-tracing and most-similar design to understand how students, teachers and principals have responded to different type of testing. The research team will also examine whether by making cheating more difficult, computerised testing increases effective teaching and student learning. To examine this issue, the team will undertake a Randomised Controlled Trial (RCT) that exploits the phased rollout of computerised testing in 150 schools in 10 districts.

Teacher recruitment

The 2005 Teacher Law gave teachers an opportunity to receive significantly higher salaries. This made the teaching profession more attractive and could, in theory, have led to higher-qualified candidates entering the teaching profession.  Because teacher competence is associated with student learning (Ree Joppe De, 2016), this change could, in turn, have led to improved student learning. The research team will test whether these outcomes indeed occurred. They will investigate whether the prospect of higher salaries led to more and better-qualified students entering teacher training colleges, and to candidates with higher qualifications being hired as teachers.

To understand whether higher-quality teachers are entering the profession as a result of the Teacher Law, the research will analyse the senior secondary national exam and university entrance exam score data over a five-year period, comparing results of entrants to teacher training colleges with those of individuals with similar characteristics from the same cohorts who attended other universities. If the Teacher Law was effective, we would expect to find that exam scores of the teacher training college entrants increased relative to the control group. Researchers will conduct a similar analysis comparing the trends in characteristics of teachers who are hired with a control group. This diverging trend (difference-in-difference) analysis will be complemented with qualitative work to understand how decisions regarding entrance into teacher training colleges and teacher hiring are made. The reason the team wants to better understand hiring decisions is to explore questions over whether civil servant appointments, especially for teaching positions, can be based on nepotism or patronage rather than merit. If this is the case, there is no reason to expect that higher standards or salaries would attract better teacher candidates.

Rewards and incentives

The research team will focus on better understanding the implementation and impact of a novel pilot project, Kinerja dan Akuntabilitas Guru (KIAT Guru), or ‘Improving Teacher Performance and Accountability’. In partnership with the World Bank and the National Team for the Acceleration of Poverty Reduction (TNP2K), the research team will participate in a multi-arm RCT in 200 villages to quantitatively assess the impact of the programme on teacher presence, classroom practice and student learning. KIAT Guru works in remote areas. Communities and schools adopt service agreements, which are subsequently used to discuss education issues in the village and to score teacher performance. In addition, the pilot tests the effects of making the remote area salary allowances dependent on teacher attendance (as recorded by a camera) or the community score cards. The team will analyse the causal mechanism of incentives and teacher quality using principal-agent models and analysis of extrinsic versus intrinsic motivation. Researchers will also undertake a qualitative implementation study in three districts.

District innovations

Complex dynamics underlie the relationship between national policy and district roll-outs of reforms. Some districts are more innovative than others in the implementation of national reforms. Understanding what makes some districts better at rollouts, and gaining insights into how innovation spreads represent crucial aspects of the research agenda. To explore and understand these dynamics, the research team will select approximately five key, innovative districts as ‘learning laboratories’ that feature different resource and geographical conditions. In these key districts, researchers will undertake more-extensive data collection. They will undertake a political economy analysis to investigate how district and national policies interact, and how they eventually affect teachers in schools. At the school level, the team will gather detailed longitudinal student- and teacher-level data, conducting student and teacher assessments, and linking student, teacher, and district policy data to assess the impact of these policies on student learning.

Understanding district-level policies that lead to success or failure across districts

The team seeks to identify features of district policies that are successful in improving learning – and/or features that create barriers to learning. In particular, researchers will aim to understand the roles played by teacher and school management, public spending, teacher incentives, and teacher training – and how these roles may explain the successes and failures of district policies. To explore these issues, they will conduct an analysis of district education policies in 20 most-improved and 20 least-improved districts using media sources, and data from district policy documents and national exam scores since 2005.

Learning whether and how innovations spread

The research team will explore whether and how innovative ideas spread. More specifically, researchers will consider to what extent the district-level innovations in education policy spread to other districts or provinces, and/or were adopted as national policy. This will consist of a qualitative, descriptive study (policy diffusion analysis) that uses qualitative interviews with district leaders and policymakers in learning laboratory districts.

In summary, teacher reforms are an interplay between central and district government policies. The RISE researchers will examine the effect of both centrally and locally initiated policies, investigate how they interact, and, ultimately, analyse how these forces affect student learning. The objective is to understand ‘what works’ in education reform in the decentralised context of Indonesia – and researchers hope to gain insights that may lead to greater understanding to improve education in other settings.

Joppe De, Ree. 2016. How Much Teachers Know and How Much It Matters in Class: Analyzing Three Rounds of Subject-Specific Test Score Data of Indonesian Students and Teachers. Policy Research Working Paper nr 7556, The World Bank.