Students taking an exam

Our team at the RISE Indonesia CRT recently wrote a paper using primary school level mathematics questions from the Indonesian Family Life Survey that showed that performance across Indonesia declined from 2000 to 2014.

Figure 1

Graph showing predicted probability of answering correctly in 2000 and 2014

 

This paper shows results for currently enrolled students but we find similar results by age and for cohorts of 18-24 year olds, a population that recently completed schooling. 

For this analysis we use item response theory (IRT) by predicting the probability that someone answers each item correctly and then take the mean of these probabilities to produce a numeracy score. This score can be interpreted as the predicted probability of answering any of the items correctly. 

Another way of showing this result (without using IRT) is to look at what percent of in-school children correctly answered all questions they would have been expected to have mastered according to the curriculum. In figure 2 below, we see that in 2014, 46 percent of Grade 4 students correctly answered Grade 3-level questions or below; this was 60 percent of 4th graders in 2000. The percent of students correctly answering questions according to the curriculum declined in every grade between 2000 and 2014.

Figure 2

Graph showing percent of students correctly answering all previous grade-level items in 2000 and 2014

We also find that learning has declined even more for the richest 20 percent. Indeed the rich poor gap declined between 2000 and 2014. But the decline for the rich was eight percentage points over 14 years compared with four percentage points for the poor.

Figure 3

Graph showing predicted probability of answering correctly in Grade 1 to 12 for rich and poor in 2000 and 2014

These results are consistent with the TIMSS math results–8th graders in Indonesia participated in TIMSS in 1999, 2003, 2007 and 2011; and the score dropped from 403 to 386 over that timeframe. 

 

One of the first questions we get asked when presenting this paper, as I did at CIES this year is–why?

What happened over this timeframe that might have prompted this change? Given all of the reform underway in Indonesia since the early 2000s, one would have expected an improvement. Here we consider some possible explanations. 

Less spending? No–spending nearly doubled over this timeframe. The government amended the Constitution in 2002 to require that 20 percent of the budget be allocated to education spending (Figure 4). 

Figure 4

         

Percent of government expenditures from 2001 to 2015

 

Teacher deficit? No – the student teacher ratio in Indonesia has been declining since 2005 and in 2017 was 16-1 in primary school, one of the lowest ratios in the region. 

Teacher quality? Unlikely – we know that teachers' education may not explain much variation in teacher effectiveness in developed countries. In Indonesia, teachers with bachelor’s degrees perform slightly better on a series of math, science, and Indonesian test questions than teachers with less education.  Between 2003 and 2016, the share of teachers with a bachelor’s degree rose from 37 to 90 percent. But we know from another study that upgrading teachers' degrees didn’t improve learning. Our point here is that according to crude measures of quality, things improved massively over the study period, so this could not be an explanation for the decline in learning.  

Student composition? No–a common explanation for the decline in learning in many developing countries is that the system is flooded with new learners who have less schooling exposure and thus lower achievement. This is the case in countries with rising enrollments – now that the education system is teaching large proportions of children who have never been to school (and/or whose parents may have never been to school) or children haven’t stayed in school for as long, learning declines, at least temporarily, as these children become familiar with schooling and catch up to peers. 

In Indonesia, six years of schooling became compulsory in 1984 (and nine years of schooling became compulsory in 2003). So at the primary level, the country has seen near universal enrollment since the late 1980s/early 1990s (Figure 5). Indeed, at the junior and senior secondary levels, enrollments rose by around 20 percentage points during our study period. But recall that the test questions we utilize for this analysis are at a Grade 5 level or below – meaning spending more time in school beyond Grade 6 may not benefit students in answering these questions. And if there was a compositional effect at higher grades, we would expect to see differences in the decline in these grades compared to primary–which we don’t.  

Figure 5

Graph showing percent of children enrolled in 2000, 2007, and 2014


Another way to test the composition hypothesis is to look at scores for the entire population (in- and out-of-school children). In figure 6 we show results by age. This shows that learning declined for everyone.  

Figure 6

Graph showing predicted probability of answering correctly by age in 2000 and 2014

Students are repeating grades less? Maybe–we see that between 2000 and 2014 the percent of primary school students repeating a grade overall went down by 38 percent (from 17 percent in 2000 to 11 percent in 2017). The figures below show the percent of children either at the appropriate age for grade (green), behind the appropriate grade for age (red), or ahead of the appropriate grade for age (yellow) in 2000 and 2014. Clearly, fewer children are behind and more children are ahead. Thus, it is possible that students who might have needed more support by repeating a grade would have been able to in 2000 but not in 2014. 

Figure 7a

Age and grade distribution in 2000 of children who are on track, behind, ahead, or out of school


Figure 7b

Age adng rade distribution of children who were on track, behind, ahead, or out of school in 2014

A change in stakes? Maybe–in 2003, Indonesia enacted an education law that included a host of reforms, some to the national examination system. Prior to 2003, a student graduated from 6th or 9th or 12th grade based on yearly grades and national exam results. After 2003, districts took over responsibility for the Grade 6 leaving exam, so the content varied by district. 

For 9th and 12th grade students, national exam score was still a defining factor for graduation. From 2003–2009, the government determined the minimum score in the national exam for student graduation. In 2010, this regulation was loosened, so that the national exam score was only one component in determining student’s final grade. During this period, promotion was based on student’s final grade; this was similar to the system before 2003. Since 2015, which is beyond our study period, the national exam score is no longer a graduation requirement. 

Nevertheless, Grade 6 and 9 exam scores are still important in some areas as they may be used for junior secondary and senior secondary admission. The Ministry of Education and Culture's Center for Educational Assessment still has responsibility for the junior secondary and senior secondary leaving exams.

A change in exposure to math? Maybe–the 1994 curriculum mandated ten hours a week of math instruction for Grades 1-3 and eight hours a week for Grades 4-5. In 2004, the curriculum changed this to five hours per week for Grades 1-6. Moreover, another curriculum change in 2006 required teachers to teach math in Grades 1-3 "thematically," which meant that teachers were to cover all academic subjects related to a theme or topic. This was a big adjustment for teachers who received little training or guidance in implementing this approach and potentially had implications for the amount of material teachers could cover in class.

Decentralization? Has more autonomy had an effect on quality? Very hard to say–in 1999 Indonesia decentralized, giving more financial and political autonomy to the districts. The 2003 Education Law solidified this initiative in education by granting much more autonomy to districts to manage education. Teachers are still hired by the central government and the curriculum is set by the Ministry of Education and Culture but districts distribute teachers, hire/fire contract teachers, allocate funding to schools, and carry out a range of other functions. 

What do you think? Do you know of any research that would help explain (or refute) the decline? We welcome your inputtweet us @SMERUInstitute@dsuryadarma, @mandabeat@BerkhoutEmilie

 

 

 

 

 

Amanda Beatty is a senior researcher at Mathematica Policy Research and has been conducting education research in developing countries for the last fifteen years, and in Indonesia for the last nine. Before joining Mathematica in 2012, she was an education economist with the World Bank in Indonesia, where she led randomized evaluations of nationwide government programs, including a nationwide initiative to upgrade teacher qualifications at the primary and secondary levels; early childhood service expansion to poor communities; and efforts to enhance the effectiveness of school committees and school management through community engagement. From 2014-15, she spent a year working with the Center for Global Development, where she was part of the RISE startup team and helped set the vision and direction for the programme.

Emilie Berkhout is a junior researcher at the Amsterdam Institute for Global Health and Development (AIGHD). She obtained her master’s degree in development economics with distinction at the VU University in 2015. She conducts impact evaluations to assess effects of policy interventions in developing countries. Her main research focuses on the effect of policy reforms in basic education on learning outcomes in Indonesia.

Daniel Suryadarma is the Deputy Team Leader for the Indonesia CRT and is based at SMERU. He conducts applied economics research in the areas of education, poverty and social policy. His work has appeared in peer-reviewed journals, including American Economic Journal: Applied Economics and Journal of Development Economics. His research has informed policy discussions at the Indonesian Vice President’s Office, Indonesian Ministry of Education and Culture, Australian Aid Program, and the World Bank. He is also an honorary lecturer at the Australian National University, a member of the expert roster at the International Initiative for Impact Evaluation (3ie), and an assessor for the Australian Research Council. Previously, he has held positions at the Palladium Group, Center for International Forestry Research, and the Australian National University.

 

RISE blog posts reflect the views of the authors and do not necessarily represent the views of the organisation or our funders.