Our team at the RISE Indonesia CRT recently wrote a paper using primary school level mathematics questions from the Indonesian Family Life Survey that showed that performance across Indonesia declined from 2000 to 2014.

Figure 1

This paper shows results for currently enrolled students but we find similar results by age and for cohorts of 18-24 year olds, a population that recently completed schooling. 

For this analysis we use item response theory (IRT) by predicting the probability that someone answers each item correctly and then take the mean of these probabilities to produce a numeracy score. This score can be interpreted as the predicted probability of answering any of the items correctly. 

Another way of showing this result (without using IRT) is to look at what percent of in-school children correctly answered all questions they would have been expected to have mastered according to the curriculum. In figure 2 below, we see that in 2014, 46 percent of Grade 4 students correctly answered Grade 3-level questions or below; this was 60 percent of 4th graders in 2000. The percent of students correctly answering questions according to the curriculum declined in every grade between 2000 and 2014.

Figure 2

We also find that learning has declined even more for the richest 20 percent. Indeed the rich poor gap declined between 2000 and 2014. But the decline for the rich was eight percentage points over 14 years compared with four percentage points for the poor.

Figure 3

These results are consistent with the TIMSS math results–8th graders in Indonesia participated in TIMSS in 1999, 2003, 2007 and 2011; and the score dropped from 403 to 386 over that timeframe. 

One of the first questions we get asked when presenting this paper, as I did at CIES this year is–why?

What happened over this timeframe that might have prompted this change? Given all of the reform underway in Indonesia since the early 2000s, one would have expected an improvement. Here we consider some possible explanations. 

Less spending? No–spending nearly doubled over this timeframe. The government amended the Constitution in 2002 to require that 20 percent of the budget be allocated to education spending (Figure 4). 

Figure 4
         

Teacher deficit? No – the student teacher ratio in Indonesia has been declining since 2005 and in 2017 was 16-1 in primary school, one of the lowest ratios in the region. 

Teacher quality? Unlikely – we know that teachers' education may not explain much variation in teacher effectiveness in developed countries. In Indonesia, teachers with bachelor’s degrees perform slightly better on a series of math, science, and Indonesian test questions than teachers with less education.  Between 2003 and 2016, the share of teachers with a bachelor’s degree rose from 37 to 90 percent. But we know from another study that upgrading teachers' degrees didn’t improve learning. Our point here is that according to crude measures of quality, things improved massively over the study period, so this could not be an explanation for the decline in learning.  

Student composition? No–a common explanation for the decline in learning in many developing countries is that the system is flooded with new learners who have less schooling exposure and thus lower achievement. This is the case in countries with rising enrollments – now that the education system is teaching large proportions of children who have never been to school (and/or whose parents may have never been to school) or children haven’t stayed in school for as long, learning declines, at least temporarily, as these children become familiar with schooling and catch up to peers. 

In Indonesia, six years of schooling became compulsory in 1984 (and nine years of schooling became compulsory in 2003). So at the primary level, the country has seen near universal enrollment since the late 1980s/early 1990s (Figure 5). Indeed, at the junior and senior secondary levels, enrollments rose by around 20 percentage points during our study period. But recall that the test questions we utilize for this analysis are at a Grade 5 level or below – meaning spending more time in school beyond Grade 6 may not benefit students in answering these questions. And if there was a compositional effect at higher grades, we would expect to see differences in the decline in these grades compared to primary–which we don’t.  

Figure 5

Another way to test the composition hypothesis is to look at scores for the entire population (in- and out-of-school children). In figure 6 we show results by age. This shows that learning declined for everyone.  

Figure 6

Students are repeating grades less? Maybe–we see that between 2000 and 2014 the percent of primary school students repeating a grade overall went down by 38 percent (from 17 percent in 2000 to 11 percent in 2017). The figures below show the percent of children either at the appropriate age for grade (green), behind the appropriate grade for age (red), or ahead of the appropriate grade for age (yellow) in 2000 and 2014. Clearly, fewer children are behind and more children are ahead. Thus, it is possible that students who might have needed more support by repeating a grade would have been able to in 2000 but not in 2014. 

Figure 7a

Figure 7b

A change in stakes? Maybe–in 2003, Indonesia enacted an education law that included a host of reforms, some to the national examination system. Prior to 2003, a student graduated from 6th or 9th or 12th grade based on yearly grades and national exam results. After 2003, districts took over responsibility for the Grade 6 leaving exam, so the content varied by district. 

For 9th and 12th grade students, national exam score was still a defining factor for graduation. From 2003–2009, the government determined the minimum score in the national exam for student graduation. In 2010, this regulation was loosened, so that the national exam score was only one component in determining student’s final grade. During this period, promotion was based on student’s final grade; this was similar to the system before 2003. Since 2015, which is beyond our study period, the national exam score is no longer a graduation requirement. 

Nevertheless, Grade 6 and 9 exam scores are still important in some areas as they may be used for junior secondary and senior secondary admission. The Ministry of Education and Culture's Center for Educational Assessment still has responsibility for the junior secondary and senior secondary leaving exams.

A change in exposure to math? Maybe–the 1994 curriculum mandated ten hours a week of math instruction for Grades 1-3 and eight hours a week for Grades 4-5. In 2004, the curriculum changed this to five hours per week for Grades 1-6. Moreover, another curriculum change in 2006 required teachers to teach math in Grades 1-3 "thematically," which meant that teachers were to cover all academic subjects related to a theme or topic. This was a big adjustment for teachers who received little training or guidance in implementing this approach and potentially had implications for the amount of material teachers could cover in class.

Decentralization? Has more autonomy had an effect on quality? Very hard to say–in 1999 Indonesia decentralized, giving more financial and political autonomy to the districts. The 2003 Education Law solidified this initiative in education by granting much more autonomy to districts to manage education. Teachers are still hired by the central government and the curriculum is set by the Ministry of Education and Culture but districts distribute teachers, hire/fire contract teachers, allocate funding to schools, and carry out a range of other functions. 

What do you think? Do you know of any research that would help explain (or refute) the decline? We welcome your input–tweet us @SMERUInstitute, @dsuryadarma, @mandabeat, @BerkhoutEmilie