Core to the RISE Programme’s systems framework is the need for education systems to be coherent for learning. With the rapid increase in schooling attainment in recent decades, education systems in many developing countries are primarily coherent for schooling—getting more kids in school for more years. As the learning crisis has gained prominence there is increasing recognition that coherence for schooling is not sufficient, and education systems need to be oriented towards learning, in order to achieve education’s promise. This presents a challenge: how to measure (in)coherence in a system and diagnose and improve areas not coherent for learning.

Twaweza recently undertook an effort that helps address this question. Using an innovative approach to analyze Uganda’s national curriculum and primary leaving exam, they quantified the coherence of two (possibly competing) tasks teachers are required to accomplish: completing the curriculum and preparing students for the exam. They find low coherence—or high incoherence—across the two, with the curriculum and exam emphasizing different topics and different depths of knowledge for those topics, leaving teachers with conflicting demands on their limited time. The study also finds suggestive evidence of low coherence between teacher instruction and the curriculum, and high emphasis on rote learning, indicating that teachers may be concentrating teaching on the narrower set of topics on the exam (rather than mastery of the broader content in the curriculum). The researchers conjecture that the combined demand from Uganda’s national exam body and from parents for teachers to prepare children for the exam supersedes the demands from the curriculum body to complete the curriculum. Better alignment between the curriculum and the exam, so that the exam reflects the curriculum and the curriculum provides adequate preparation for the exam, would increase coherence in the system and could lead to a greater emphasis on learning. 

Sources of incoherence

Curriculum and national exams, the first of which articulates what children should learn and the second assessing what children have learned, are often developed through separate processes.  Curriculum reforms in Uganda (and frequently in other parts of Sub-Saharan Africa) almost always enlist high-level political engagement and oversight. This level of oversight is typically missing from the design and conduct of exams (at all levels—school, district or national). Political scrutiny of national exams mostly (only) occurs when performance results are published, with a focus on identifying best and worst performers, whether among learners, teachers, schools, or districts. Throughout the parallel and mostly separate processes it is largely assumed (rather than ensured) that the exams align well with the curriculum standards, presenting an opportunity for incoherence.

The RISE framework for analyzing an education system’s coherence for learning includes four principal-agent relationships, proposing that incoherence across or within these relationships makes achieving learning more difficult. The Twaweza study, delving into the curriculum and exams in Uganda, addresses two main sources of (in)coherence within the framework: The first is within the Compact relationship, involving what policymakers delegate schools to do (A₁ in Table 1) and the information policymakers have to hold schools accountable (A₂). The second is between the Compact and Client Power relationships, involving the (in)coherence in what policymakers delegate schools to do (B₁) versus what parents delegate (demand) schools do for their children (B₂). 

Table 1: Illustrating the incoherence within and across accountability relationships.

How does this apply in the Ugandan context?

(A) Within the Compact relationship, between the “state” (policy makers) and “organizations” (schools), there can be incoherence in what policymakers delegate schools to do, and the information policymakers have to hold schools accountable: 

1. The National Curriculum Development Centre (NCDC, the government’s curriculum body) to schools: Schools are required to implement the curriculum as designed, with a key objective of ensuring children learn the intended content. However, there are no systematic, regular, and reliable means of measuring the actual implementation of the curriculum standards as designed.  For example, no mechanism to ensure schools implement the requirement to use local Language for Instruction in lower primary grades, children get the opportunity to engage with all the intended grade-content before progressing to the next grade, teachers utilize continuous assessment to inform/guide remedial teaching, etc. In other words, the actual implementation of the curriculum by schools and teachers cannot be reliably measured, leaving the NCDC without critical information needed to achieve effective implementation and guide future reforms. 
2. Ugandan National Examinations Board (UNEB, the government’s national examination body) to schools: Schools are required to prepare learners to pass high-stakes examinations whose principal objectives are the selection and sorting of children for further education, depending on their performance outcomes in these exams at the end-of-cycle. This exam, administered after seven years of schooling, is the first standardized performance information collected nationally, meaning little information is available before this point tracking how students are performing. 

Therefore, within the accountability relationship between policymakers and schools, there is potential for incoherence of delegated goals/objectives if the curriculum and assessment bodies do not deliberately harmonize their goals to ensure there are no systemic contradictions. 

(B) Across two delegation relationships, including Compact, between the policymakers and schools, and Client Power, between parents/citizens/students and schools, there can be incoherence in what schools are being delegated to achieve. For parents and students, passing the high-stakes final examinations is a top priority, leading parents to pressure schools and teachers to emphasize exam preparation over curriculum mastery and learning. This is coherent with one part of the Compact relationship, with UNEB delegating schools and teachers to prepare children for the exam. But it could be incoherent with another part of the Compact relationship, with NCDC delegating schools and teachers to complete the curriculum, if the exam and curriculum aren’t aligned. Further complicating these relationships is that citizens are also more able to hold schools accountable for exam preparation than for learning gains because they themselves do not understand what constitutes good quality education in primary grades, furthering their emphasis on the former.

Quantifying Coherence

The situation in Uganda presents the opportunity for incoherence, but how can we know or measure the existence and/or extent of incoherence? Twaweza utilized the methodology called “Surveys of Enacted Curriculum” (SEC) to analyze the extent of (in)coherence between the curriculum standards and the assessments. For SEC, a panel of curriculum experts analyzes the academic content embedded in the national curricula and end-of-cycle assessments and codes and rates them against subject-specific taxonomies. They coded and rated the topics that the curriculum and exams covered, subtopics covered within each topic, target competences within each sub-topic, and the cognitive demand levels required for each topic, subtopic, and competence. It was a massive undertaking, with the team coding four primary-level subjects (math, English, science, and social studies) across a total of 72 topics/subtopics, for all seven grades of the primary school curriculum and the primary leaving exam. To rate the levels of cognitive demand, the panel determined whether the target competences in the curriculum or the items in the exam required the students to (in order of increasing cognitive demand) memorize, perform procedures, demonstrate understanding, conjecture, or solve non-routine problems for each topic/subtopic/competence.

From the results, the team produced powerful visual displays, using heat maps to show the emphasis and cognitive demand of each topic for each curriculum grade and for the exam. Figure 1 shows an aggregated heat map for the math curriculum for Grades 1 through 3, Grades 5 through 7, and for the primary leaving exam. The heat maps in Figure 1 show the 13 math topics on the vertical axes, five levels of cognitive demand on the horizontal axes, and the amount of emphasis for each intersection shown by contour lines on the Z-axes. 

In the visualizations, you can see that in Grade 1 through 3 the math curriculum primarily focuses on number sense and operations, emphasizing the performance of procedures and the students’ ability to demonstrate understanding. In Grades 5 through 7, number sense and operations remain the top priority, but additional emphasis is placed on geometric concepts, basic algebra, and data displays. Through the seven grades, little focus is put on developing the “higher-order” cognitive demands of conjecture and non-routine problems. On the primary leaving exam, most emphasis is narrowly on measurement, including measurement procedures and measurement in non-routine settings, plus some emphasis on basic algebra, geometric concepts, and data displays. Similar to other analyses of high-stakes exams, the primary leaving exam primarily emphasizes low-cognitive-demand capabilities (except for measurement).

Figure 1: Alignment analysis: curriculum standards (lower and upper primary) versus national exams

The team uses the SEC methodology to compute an “alignment index”, describing the alignment of the curriculum and primary leaving exam. The top-level finding is that the aggregate primary curriculum (over the seven years) has an alignment index score with the primary leaving exam of only 0.33 on a zero to 1 scale. They are very poorly aligned.

What this means is that schools, teachers, and head teachers face conflicting demands, and must make a choice regarding what they prioritize or focus their efforts on—passing exams or adhering to implementing the curriculum as designed?¹ In a system where the exam is geared towards selection and sorting, while the curriculum prioritizes learning and content mastery, pressure from policymakers and parents to choose the exam can have unfortunate consequences for learning outcomes. Greater coherence across the two, so that the exam covers the content from the curriculum and the curriculum prepares children for the exam, and therefore the delegation from the curriculum body, exam body, and parents are all well aligned, could place learning more front-and-center in the system.

This blog provides just a snapshot of what could be done with this kind of analysis. This year, for example, Twaweza is training curriculum specialists from the respective curriculum bodies in Tanzania and Uganda on the application of the SEC methodology to improve curriculum alignment aspects these bodies are responsible for in their work.  When applied to teacher instruction, the SEC methodology can also be used for teacher professional development and to facilitate peer-level reflections by teachers on how their instruction aligns with the curriculum standards. Measuring the (in)coherences in what is delegated and enacted in an education system is a crucial step towards aligning the system towards learning and ultimately towards improving learning outcomes, and this methodology provides a great example of a way to do this.

¹The team piloted an effort to address this, analyzing a small sample of teachers’ instruction using the same SEC methodology, and assessing the alignment between instruction and curriculum. This also showed low levels of alignment, as well as higher levels of “memorization” (rote learning) than prescribed by the curriculum, suggesting teachers may de-emphasize the broader curriculum in favor of emphasizing preparation for the exam. For a variety of reasons, including sample size limitations that prevented coding and rating instruction for all seven grades that provide input for the primary leaving exam, they weren’t able to analyze the alignment between instruction and the exam. Conducting this analysis for a larger sample of teachers and to assess alignment across instruction and exam are obviously areas of interest for future inquiry.