As preliminary information, the basic statistics on the three cases (all data refer to most recently available information and are rounded) are:

It is worthwhile to point out that while the gains in the cases noted here are impressive, there is still a lot of ground to cover. Even though Puebla is a state in a rather well-off country, compared to most lower-income and lower-middle-income countries, and even though Spanish is an easy language in which to learn to read, Puebla’s children perform at about 0.6 standard deviations below the OECD, or at about the 20th percentile of the OECD distribution, in spite of recent improvements.³ In Sobral, 5th grade children seem to actually perform at somewhere around the 60th percentile of the better-performing 30 countries on the PIRLS.⁴ Kenyan children, after the intervention shown here, read at about half the fluency expected in the USA.⁵ So, to varying degrees (though much less so in Sobral given that they seem to be at OECD levels in reading in Grade 5—which makes the case study all the more interesting), there is still work to be done.⁶ Still, these results are all much better than their own baselines, and also much better than other localities in their own contexts.

To summarise the matrix in the Annex, these were features of the three efforts, though to varying degrees, and are also supported by the emerging literature from, in particular, early-grade literacy efforts. The case studies show five tips that may benefit other countries or situations. The five tips are described more fully below and can be read from the case studies and the long Annex table. They are:

1. Focus on just a very few achievable indicators, foregrounding foundational learning, for a country emphasising learning.
2. Use data to drive both initial “wake-up” and stock taking but also to support teaching as the process goes along.
3. Emphasis on the teaching by the teachers who are already in place. Use an evidence-based, structured pedagogy that meets children where they are. Support the teachers through coaching or some other form of in-service training. Ensure that related inputs (such as books) are tightly coordinated to exactly match the lesson plans and the assessments.
4. Provide strong motivation through support that works. Teachers are motivated through professional accountability to those who support them, if those who support them know what they are doing and what they suggest actually works. They are also more motivated to the extent that they co-design (Puebla). Provide a foundation of acceptable infrastructure and working conditions. Motivation is also provided by support from the highest political levels. In these cases, the flip side of professional accountability through the coaching and school leadership was trust that what the coaches and school leaders were promoting would actually help children learn better. Thus there is a tight feedback loop between being evidence-based and being able to build relations of accountability that are professional and built on trust.
5. Tight management. Some degree of centralism and prescriptiveness as to pedagogy and other inputs, but after having iterated and adapted to context. Structured pedagogy. Water-tight coordination between the lesson plans for teachers, the assessments, the learning materials (especially books) in terms of sequence over the year and pedagogical approach. Adapted and iterative management was key in all cases, and included items such as iterating the planned lessons taking stock of teacher use, adapting the assessment to make sure it links effectively to formative usage by teachers and coaches and more summative (but low-stakes) reporting up, and so on.

The graphic below summarises.

The graphic makes it clear, from the arrows, that there is a sequencing from the “Why” to the “What”, from the “What” to the “How,” and then also feedback and adaptation, especially from the “How” to the “What.” This creates a virtuous cycle that amplifies the effect over time, maybe a few years, but does not take too long if managed correctly. This was clear from the case studies. This sequencing and iteration are difficult to convey via lists such as the above, and the following, but are key to understanding how the change happened in the three case studies.

The lessons can be provided a bit more at length:

1. Focus. A relentless focus on just a very few doable indicators, avoiding abstraction, loftiness, over-complexity and over-ambition. Indicators that are easy to communicate, and act upon. Rather than more structural reforms such as pay scales, incentives schemes, or decentralisation, there was a sharp focus on the classroom and on teaching and learning reforms. For example, reading as a foundational skill in the early grades, and measured very specifically as in Sobral and Kenya. Puebla started with an Attendance-Completion-Learning (APA in Spanish) model, tracking those three factors. There were conscious choices to not do some things that are “good” because too many small aspects to the intervention meant that fewer schools would implement. Focusing the attention of key change agents on a smaller number of things by dropping the possibilities that would only have a small impact was key. Many primary school reforms or investment programmes do not realise just how relentless and militant the focus needs to be.
2. Data for initial stock-taking, around the focus: learning and improving teaching.
   • “Wake-up value” of an initial simple assessment, often from a party outside the Ministry. These often had to be set to a lower level that the customary assessments as the children could not even reach the base of the customary assessment and were sometimes inspired by relative outsiders (e.g., a la Uwezo in East Africa or ASER in India). However, while outsiders can provide the impetus, the authorities have to be prepared for what might be found and be ready to own the results. Some countries carry out this “wake-up value” strategy at the beginning of a new government, not in the middle.
   • Used for improving teaching, much less for accountability. Measuring and tracking the indicators (including operational or process indicators) in real time and using them for essentially immediate feedback to the teachers and managers. Indicators sometimes measured in real time by the managers themselves. Used to provide better and more targeted coaching, for instance, to set goals for schools, and to report to higher authorities. The distinctive feature was the speed of turnaround and the link between the indicators and the pedagogical model being used.
3. Teaching, with emphasis on the -ing. That is, focus on supporting teachers already teaching, more than on who the teachers are.
   • Evidence-based pedagogy. Reliance on a clear pedagogical model or approach to reading (or other subjects but less so) that was either evidence-based and in line with the scientific evidence or iterative and producing its own evidence. This is key to a professional accountability, since teachers will feel more naturally accountable to professionals who are giving advice that actually works, with their children, in a palpable way (thus the link to simple relatively simple assessment). However, none of the cases started with a complex or more fundamental curricular reform or even serious tinkering.
   • Ensuring other related inputs support the teaching. Teaching and learning materials, structured lesson plans and sequential yearly programming (scope and sequence); devoting curricular space to reading as a skill rather than “language;” assessment that is linked to the teaching; review and improvement of books so they follow the same sequence as the lesson plans and are not out of cognitive reach for the children; and setting of goals. All with a focus on meeting children where they are, instructionally, rather than where one supposes they are.
   • Attendance of both teachers and learners. In the Puebla model this was explicit as one of the three indicators on their three-indicator, focused model.
4. Motivation. The three cases relied much more on support and professional accountability rather than bureaucratic accountability as motivators, though the latter also mattered. They also created an environment that legitimated a focus on, and pivot to, learning. Motivation can also help start the virtuous cycle noted above. Without it, teachers may give up early.
   • Variety of motivations. The approaches saw more of a reliance on professional pride, perhaps through accountability to coaches and supervisors, rather than just monetary rewards or bureaucratic accountability, though the latter were also used in some of the cases. Smart communication of goals and progress with public recognition of success, including (in some cases) allocation of resources and public recognition to persons and teams who made sustainable progress.
   • Overall support and right preconditions. This varied in degree, but in some models a focus on attendance, nutrition, and a modicum of decent infrastructure were also used. Support sometimes targeted to those struggling was used, including the selection/prioritisation of human and pedagogical resources for those in need. The fact that the suggestions made by coaches and supervisors to teachers were evidence-based, and visibly worked to improve learning outcomes, was to a significant degree what created a respect for coaches and supervisors, which in turn underpinned professional accountability. Resource dumps were not the answer. But having basic conditions right, such as 1:1 books: pupil ratio, at least a modicum of comfort and sanitation in the school for both teachers and students, school lunches, all can help, probably causally but, if nothing else, by making a pivot to learning feel more legitimate and as a matter of rights.
   • Motivation at the political or high policy level. Motivation from the highest levels of government (even though more local government in some cases) to shake things up was key. This motivation or pre-disposition pre-dated the wake-up value of an initial learning assessment but was abetted by that wake-up value.
5. Tight, clever management around the instructional core. A central point in the system where the various threads of the intervention (coaching, lesson plans, books, assessment) could be orchestrated, using some creative combination of hierarchy and participation.
   • A bureaucracy that was ready to react to opportunity. Luck in finding the right advisors or hearing of a reform done elsewhere played a role in all three cases. Luck cannot be replicated, but preparedness to use the luck can be. And supplying examples of successful cases can also be replicated. In all three cases luck played into bureaucracies that were well prepared to use that luck, were receptive and eager for change, worked with a non-education political context ready for change, and above all recognised the value of the external input that worked with the bureaucracy to kick-start the process.
   • Some prescriptiveness and centralism. Relatively prescriptive approaches (e.g., to structured pedagogy), without being too heavy-handed and in some cases providing some choice, as long as some clear structure was used, and results were produced. Selection of key educational materials was done centrally, based on clearly stated learning goals and consistent with the lesson plans or scope and sequence leaving room for feedback, innovation, and collaborative work from the classrooms. 
   • An iterative, adaptive approach. Once honed to a sharp edge the approaches were deployed centrally, but  process did not start pre-designed (though they did start with sufficient design to reflect the evidence base). They all adapted a great deal, for example, in fine-tuning the degree of centralism or prescriptiveness. Not necessarily to make the approach more or less prescriptive or structured, but to make sure that the type of prescriptiveness actually helps the teachers and students. Kenya went through six adaptations of some aspects of the approach. Some experts in at least one of the cases did note that had they had a more centralist approach to start with, from which to compromise as needed, they would have saved much time. 
   • Integration and coordination among the various “inputs”. Tight coordination between the various inputs (book content, lesson plan content, assessment, coaching for the teachers) to the schools. Ensuring tight synergy among these was key. Approaches to these “inputs” were all aligned with other and had the same sequence over the year. 
   • Some initial support by an external “agency”. An important role for external agents who in all cases were motivators and networkers, brought in scientific approaches, and ensured an initial ethos of integration among the inputs as per the point immediately above. However, those external agents were always “used” by a supportive bureaucracy and political leadership eager for improvement and who could provide passion and leadership. Over time the system absorbed the ethos and there has been less and less reliance on the external agents.
   • Readiness to struggle. In at least two of the cases, and in others in the literature, there were groups that resisted and those that actively fought back, so while luck was part of it, fighting the political fights was also part of it (and having factions that were supportive enough to enlist the fight against the opposing forces). Demonstrating that even less expensive books can be very effective might not be welcomed by the publishing industry, for example, and they may be powerful with a Ministry of Education.

The three governments provide a case-oriented narrative that adds some nuance and chronology to the matrix above. Each case starts with a graphical summary of why the case is and should be of interest.

Sobral, Ceará, Brazil

Sobral is a municipality in the state of Ceará, with a population of just over 200,000. Sobral ranks number one out of approximately 5,700 municipalities in Brazil in the national basic education ranking, up from 1,366th place in 2005 as shown in Figure 1 (next page). This rise has come despite high poverty levels, a five-year drought in the region, and a national recession. Sobral’s scores are about 80 percent higher than would be expected given their education expenditure.

The graphic above provides a static picture. The empty white dots are Sobral in the context of Ceara (solid black dots) and Brazil (grey dots).

The rapid transformation of Sobral in comparison with the rest of Brazil is noted in Figure 2 (for Grade 5). Sobral is on the right, Brazil as a whole on the left. Both Sobral and Brazil were at the same level in 2005. Since then, Sobral has improved much faster, to about 285 on a 350-point scale, as opposed to 210 for the rest of Brazil (the minimum expected level is 200, in a range of 200 to 275).

The transformation of Sobral’s education system began in 1997 with the election of Cid Gomes as Mayor. Coming from an influential local family, Mayor Gomes first instituted reforms to increase access, improve infrastructure, and mitigate over-age and dropout rates. When these reforms did not have the intended effect on student learning outcomes, Mayor Gomes introduced an additional set of reforms in his second term aimed at literacy, the professionalisation of teachers and staff, decentralisation of non-pedagogic activities, and school autonomy.

The core goals were already set out in 2001, but the specific focus on “alphabetisation at the right age” and the interventions that make the case interesting, and led to the improvements, did not start until 2004. In 2001 the following goals were set:

• Literacy for all 6- and 7-year-olds enrolled in 1st or 2nd grades.
• Remediating literacy of all students from 2nd to 6th grade who could not read.
• Streamlining of student flow by reducing grade failure.
• Reducing dropout rate to less than 5 percent.
• Reaching universalisation of childcare services of good quality.
• Restructuring the education system for ISCED 2.
• Providing literacy instruction for all young people and adults who were still illiterate.

These reforms started in 2001, but tended to emphasise the student flow issues, in reality. What changed in 2006 was that learning was not improving as markedly as was desired. This led to the introduction of a set of policies that could be summarised as (borrowing liberally and paraphrasing from interviews with Ilona Becskehazy, the analyst and presenter of the Sobral case, at https://tinyurl.com/u7dmoew):

1. Absolutely clear, numerical, precise teaching objectives (such as of reading fluency with accuracy and prosody).
2. A phonics-based method with clear sequence of lessons and smaller, step by step objectives.
3. Scripted lessons based on the method.
4. A data-based, numerical monitoring system that allows quick feedback for schools, teachers, and municipal management, that includes assessments linked to the teaching methods with 100 percent of children assessed twice per year.
5. Unified teaching materials (at least the basic, since teachers can, together with their coordinators, complement the didactic sequences of the textbooks and those that are prepared centrally).
6. Ongoing professional development based on those materials, in sequence.
7. Teacher motivation, including good communication, distribution of learning materials, substantial bonuses, awards, and public recognition events for high performing teachers and schools. Results-based rewards.

All these factors were integrated into a coherent whole. In addition, the basic principles derived from the “effective schools” literature were applied. These are well known and widely available so are not summarised here.

More recent interventions include fuller and better specification of the curriculum and a focus on reading comprehension, for example, in expository writing, by asking students to a) explain the specific theme and main idea of a passage and explain why that is the case, b) summarise the arguments and connections established between them to support main the thesis, as well as the flaws in the argument, through excerpts that prove them; c) the conclusion of the author of the text on the topic through excerpts that prove it; d) the hypothesis exposed by the author in a specific way; e) the counter-arguments presented by the author in a specific way. (From Ilona Becskehazy at https://tinyurl.com/qsbo72c.)

From a political point of view, it is essential to note the prominence of the Gomes family. Two brothers were both mayors of Sobral and/or governors or congressmen with some focus on education reform for at least 20 years. One brother was Chief of Staff for the other at least once. It is rare to have such championship for an education cause. Countries and development agencies may wish to explore how such championship could be induced.

School autonomy is important in Brazil, so to strike a balance, the terms are negotiated and agreed with the school level, and then the processes are fully monitored and controlled by the central team. In some sense, autonomy is more about operational matters than pedagogical matters, as the pedagogical changes were centralist. Since 2016, Sobral has had a detailed curriculum for language and mathematics. There are annual teaching plans that are detailed down to a weekly basis at the central level, with school representatives participating in their development. Assessment is tightly linked to the lesson plans, as are the learning materials.

Teaching materials for the early years (pre-primary and 1st grade) are provided by an external vendor and are very prescriptive and detailed, including model lessons. Textbooks mainly come from the federal books programme and are complemented by locally produced, structured activities, but the schools (not the teachers) have some room to add materials and activities. Reading books are chosen and acquired centrally.

Though many aspects of the approach are prescriptive, the development was gradual and fairly organic, and came to be accepted, so in the end it may not “feel” all that prescriptive. There is some choice, for example, but the choices may not contain that much difference; they may conform to the same approach. At the same time, the political leadership was able to bring actors along so that the approach does not “feel” that centralist or prescriptive.

It is important to note that many of the reforms have the force of legislation. As elements succeed, they are formalized onto legislation or regulation.

As a last point, in Sobral, the schools in question are no-frills spaces but have all the basic elements for good pedagogy: infrastructure, books, teaching materials, absenteeism control, good and universal free meals in public schools (mandatory in all of Brazil, but better-implemented in Sobral), and playing areas. Having these features is not common in Brazilian schools.

After showing success in Sobral, and with the Gomes family elected to the state government, the Sobral approach is being scaled to all of Ceará.

Kenya

Kenya has a population of almost 50 million and ranks 25th out of 54 African countries in GDP per capita. Kenya’s 2007 SACMEQ scores were much higher scores than other countries in the SACMEQ region, controlling for both expenditure and income per capita (Figure 3 below). Kenya performed about two standard deviations better than the expected value based on expenditure and GDP per capita.

More recently, the Tusome experience in Kenya has produced remarkable results even by mid-line, as noted in Figure 4.

The dark blue color represents the percentage of pupils in either Grade 2 (right two columns) or Grade 1 (first two) reaching mastery at baseline and midline. The red color represents the percentage of students unable to read at all. To those familiar with learning impacts, it will be obvious that these are remarkable changes in just a year or two (but based on much prior refinement in PRIMR).

Since its independence in 1963, Kenya has had six commissions and several taskforces to reform its education system. Between roughly 1995 and 2010-2015, the emphasis of these reforms was on accountability, pay-for-performance, and measurement for accountability. Since 2010-2015, there has been a shifted focus to pedagogy, paying attention to the foundational grades of the education system, and having better books and lesson plans. There were additional reforms that were not necessarily part of these waves, such as reducing the role of double-shifting, moving to full-day schooling, and pro-poor funding.

In 2009-2012, several surveys of student learning, including the Uwezo citizen-led assessments and the baseline of literacy and numeracy through the USAID-funded Primary Math and Reading (PRIMR) pilot reading initiative, found that an alarming proportion of students could not read and write. Given Kenya’s traditional concern with learning, these results were a bit of a wake-up call. Interventions through PRIMR led by the Ministry of Education and RTI resulted in relatively large increases in reading levels, which garnered attention within the ministry and among teachers and motivated key actors to try this approach at full national scale. Beginning in 2014, early pilots were scaled to all public primary schools and 1500 low-cost private schools through Tusome, meaning “let’s read” in Kiswahili.

The Kenya case study is significant in that it is one of the few cases in the literature where the impact at scale-up (Tusome) has been equal to, or better than, the impact of the pilot, and also very quick. However, one should note that that impact would almost certainly not have been possible without the PRIMR pilot.

Since Tusome was to a large extent based on the features and success of PRIMR, it is worthwhile recalling the key lessons and components of PRIMR (borrowing liberally and literally from PRIMR’s final evaluation at https://tinyurl.com/wdzv9jk [and especially circa p. 18]):

• Developing an experimental research design that included rigorous baseline, midterm, and endline assessments administered to randomly assigned treatment groups.
• Designing a clear, specific, year-long scope and sequence of curriculum-based content in Kiswahili, English, and mathematics.
• Preparing teachers’ guides for Kiswahili, English, and mathematics for Classes 1 and 2.
• Specific, scripted lessons, though degree and type of scripting evolved. Training teachers and head teachers to implement PRIMR lessons and use teachers’ guides.
• Supporting regular supervision and monitoring of teachers by coaches and supervisors/tutors.
• Providing literacy and mathematics books for pupils at a 1:1 ratio, matched and closely integrated with teachers’ guides.
• Revising learning and teaching materials regularly, based on teachers’ feedback, to make them more relevant and user-friendly. This resulted in significant improvements and refinements, specifically of the scripted lessons.
• Training teachers to employ continuous assessment methods.
• Using EGRA and EGMA results to revise and update programme materials.
• Carrying out policy studies to inform the MoEST on issues related to education quality and the policy reforms revisions needed to improve student outcomes.
• Actual training and classroom support to be done by existing government officers and research to be undertaken to understand whether and how these officers would be able to accommodate the PRIMR activities in their daily work.

Specific implementation features included:

• Inexpensive books: The project team made several decisions that would ensure savings on book purchases. For example, all elements of the literacy program—which for English and Kiswahili include phonics activities, illustrations, and decodable stories—were embedded in the books. The books had attractive illustrations, which were initially in black and white, but later were produced in color. The books utilized in the 2013 academic year cost US$0.75 each. Prices at the national scale were somewhat lower than that, even when including delivery to each school in the country.
• Basic instructional aids: Tusome provided simple instructional aids, including an A3-sized pocket chart and a set of letter flashcards.
• Self-contained teachers’ guides: The main resource for teachers was the teachers’ guide. The final version of the Tusome teachers’ guides included the entire program in one teachers’ guide and one textbook for learners.
• Modest initial teacher training: Tusome decided to invest proportionally more resources in follow-up and observation than in traditional training, so the entire standard training program was only six days for the two subjects.
• Focused observations: Much of Tusome ’s attention and energy were spent in supporting coaches to visit schools and observe classrooms. Project funds reimbursed coaches’ travel based on the proportion of teachers observed per month to ensure that coaches had an incentive to provide equal support to distant or remote schools. For much of the lifespan of Tusome, coaches were observing upwards of 20,000 classrooms per month and assessing more than 60,000 children monthly.
• Setting expectations: “Reading outcomes for Class 1, Class 2, and Class 3 pupils improved,” according to measured numerical targets, was the ultimate objective, with two intermediate objectives: 1) improved supervision support and delivery of reading instruction to target pupils, and 2) improved government capacity in target directorates to sustainably improve reading outcomes.
• Monitoring and holding schools accountable for meeting those expectations.
• Intervening to support the students and schools that are struggling to meet expectations.
• Holding the system accountable for delivering that support.

Tusome clarified expectations for implementation of the curriculum and outcomes using national benchmarks for Kiswahili and English, and these expectations were communicated down to the school level. The initiative used functional, simple accountability, and feedback mechanisms to track performance against the benchmarks. These included regular assessments using the Early Grade Reading Assessment (EGRA). The data collected through the feedback mechanisms were used to inform instructional support at the county level. The effect sizes seen during the PRIMR pilot have been sustained, and in many cases strengthened, in the scale-up of Tusome.

The mid-term evaluation of Tusome (available here: https://tinyurl.com/y3h7wcgz) concludes that the main features responsible for success and suggestions for ongoing and further improvement (borrowing literally and liberally from it, especially pp. 29, 53) were:

• Access to reading materials at home and in school.
• Access to a reading teacher’s guide.
• Practice reading aloud and silently at school.
• Supervisor/tutor/coach observations and suggestions (coaching) and recording results. Focus on class activities and actions that research in Kenya and elsewhere (including PRIMR) has shown work.
• Accountability of coaches and supervisors.
• Frequency of lesson plans being reviewed.
• Full-day school shifts (in comparison to half-day).
• Classroom libraries.
• Pupils of the correct age range (5–9 years old).
• Female teacher or head teacher (obviously of limited replicability).
• Fidelity of implementation.
• Large scale use of data.
• Use of benchmarks as targets.

Puebla, Mexico

Puebla is the fifth most populous state in Mexico (6.2 million inhabitants; 1.8 million students K-12), and also ranked fifth highest in poverty levels out of 32 states. The state contains 14,000 schools (K-12). It has several native languages in addition to Spanish. In 2015, Puebla became the top state in Mexico at both the lower and upper secondary levels in the national Plan Nacional para la Evaluación de los Aprendizajes (PLANEA) assessment. Since then, Puebla has retained the 1st position in mathematics at the lower secondary levels in 2016, 2017, and 2019, and at the upper secondary level in 2016 and 2017. In language (Spanish), Puebla has ranked between 2nd and 3rd since 2015. When compared internationally, Puebla is one of the fastest improving education sectors participating in PISA. Between 2003 and 2012, the average performance of 15-year-olds increased by the equivalent of almost one full school year.⁷

This summary takes many points from the presenter’s (Bernardo Naranjo) inputs, including presentations, background writing, and subsequent information exchanges. Because the Puebla case was more iterative, consultative, and complex than the two others highlighted here, it also benefitted from other sources more so than the other cases.⁸

Two graphics explain why the Puebla case is so compelling, and what was achieved in such a short time. The first one shows Puebla to be a major outlier in terms of expected achievement conditional on income. The fact that the regression line is not very strong is immaterial. The point is that Puebla is very much on the edge of the scatter.

Figure 6 below shows academic results over time.

In 2011, a new administration was about to take office in Puebla for the 2011-2017 period. The state Secretary of Public Education (SEPP), a former senator not born in Puebla, immediately started a cultural change. Against a tradition of hiring local people only, he invited people from all over the country to work at the local Ministry.

The Secretary invited several external advisors to help him and requested help from the OECD to write policy recommendations for Puebla (this was the first time the OECD worked at the local level with a Mexican state). Most external advisors had to leave in 2013, when a very large budget cut also made it necessary to reduce staffing in the Ministry. The Puebla intervention survived after its results became known, and it helped that the advisors to that process were based on Puebla.

Initially at the upper secondary level, and by the end of 2013 in the entire K – 12 system, SEPP and “Proyecto Educativo”, a local group of advisors, developed a programme to improve the essential objectives of the education system. The Asistencia, Permanencia, Aprendizaje (APA) (Attendance, Completion, and Learning) model was developed with the following targets, and only three targets:

• Asistencia (attendance). Each person 3 – 17 years old attends school (about 250,000 out of 1.8M in the state did not attend in 2015)
• Permanencia (persistence and completion). Every student K-12 graduates from upper secondary school (currently, only about 61 percent of primary school new students will)
• Aprendizaje (learning). All students learn Spanish and mathematics above the minimum acceptable level in the national assessment (one third in Spanish and two thirds in mathematics do not achieve this goal in grade 12)

The SEPP and Proyecto Educativo referred to recommendations by international organizations and national specialists, but the strategy was entirely designed and implemented with local citizens. The model was adapted from an intervention in Ontario, Canada, in the early 2000’s that sought to improve graduation rates and basic learning in English and mathematics.⁹

The strategies developed to implement the APA model included:

• Targeting schools, in a number small enough to guarantee a significant impact (between 100 and 200 per level). Criteria: lowest-performing schools, with 100+ students, willing to participate. Schools that were too small were not included as they required a lot of effort relative to impact on the total. Initially, 100 schools were chosen, but at some point there were 600 schools receiving special support.
• Articulate/integrate efforts, to make the most of available resources. Articulate across institutions, education levels, resources, and ‘technical’ interventions (measurement, learning materials, teacher capacity development etc.) Approach became increasingly coherent, within itself, over time.
• Careful implementation and follow-up to make sure everyone understands their role and identifies mistakes and opportunities to improve.

One of the first actions in 2011 was to revive the CEPPEMS (State Commission for Upper Secondary Education), a group that formally integrated all institutions at the upper secondary level. The group met weekly, and technical subgroups formed to work on specific issues (teacher training, information systems, evaluation, dropout prevention, supply-demand analysis). Targeted schools at the upper secondary level kept using the same curriculum and textbooks as the rest of the schools, but received help (teacher training, institutional support) from other schools.

CEPPEMS decided on general strategies, and each subsystem was able to translate them into specific actions, but they all shared their experiences in the group. This allowed structure and creativity at the same time.

Interventions in the primary schools included:

• A summer course for 2nd graders to ensure basic literacy and numeracy. This allowed one to also detect and fix issues in the focus and materials design.
• A chosen method for beginning literacy, materials designed and distributed to the targeted schools.
• A new assessment for Grade 2 (mid-year) so schools have time to correct, and ongoing assessment in Grade 5.

The early-grades primary interventions were coordinated and designed to work together with preschools. Integrating the approach to these two levels was key. At the pre-school level, important steps included:

• Workbooks with prewriting and math skills, designed specifically as a prequel to the early-grades primary grades, by the same experts.
• Pre-school and early primary grade teachers to work as a team. In fact, in Puebla there was a recognition that even problems in secondary school arose in primary and problems in primary arose in pre-primary (or lack thereof) so there was a big emphasis put on “articulating” the whole system.
• Campaign to increase pre-school attendance. Attendance ratio increased five times faster than the national average.

A certain style of intervention was common to all levels. In training and teacher capacity building, it included:

• Emphasis on quality, not quantity. Training groups were limited to 25 teachers or less.
• Content designed by specialists centrally in order to address specific needs and opportunities.
• Facilitators recruited locally among the best teachers/principals/supervisors, and they received preparation to do so. External facilitators required a high profile.
• Courses were mostly face to face. Online resources were used as a complement only.
• Coordination with teachers and co-design with teachers, but working in a collective manner so that a unitary model emerges, rather than just letting each individual teacher figure things out.
• Getting teacher union leadership interested in pedagogical factors.

And in terms of systems management:

• Supervisors. Strategy to train all supervisors (950) on leadership skills and the APA Model. The best supervisors  chosen as facilitators for ongoing teacher training (see above on “Facilitators”).
• The Academia Poblana de Supervisión (Puebla Supervision Academy) was created to give identity to facilitators. Positions in it were honorary. The aim was to recover the status and prestige of a somewhat forgotten role in the system, that of supervisors, as the key link between schools and the system.
• Servicio de Asistencia Técnica a la Escuela de Puebla (SATEP) (System of Technical Assistance to the Puebla School), to coordinate all training efforts statewide (and training to trainers). It included a renewed role of supervisors.
• Information distribution through the webpage (escuelapoblana.org) to have a common base for all schools and officials
• Reporte APA, a customized report identifying by name to every at-risk student in each school of the state, and critical areas in the last assessment results in mathematics and Spanish. Sent every two months through the existing systems.
• Unidad de Promoción del Derecho a la Educación (UNIPRODE) (Unit for Promotion of the Right to Education) created to coordinate statewide efforts to ensure enrollment and attendance, as the A and A in the APA model.

The following are lessons learned from the Puebla effort:

1. Improvement is always possible, even with the same resources. Improvement is about people and systems, not about money.
2. School systems require to set up conditions to discuss academic issues, articulate efforts, and follow up on critical processes. Academic Boards in Puebla were effective means as they had clear objectives, a good structure, and strong leadership.
3. Targeting (on academic results) is essential: equity is a fair, efficient, and fast way to increase quality in the whole system. One can also target on poverty but for other purposes and programs. Learning improvement is efficiently and equitably targeted through learning results.
4. Let people at each level of the structure make some decisions. Think about teachers as professionals, not just operators.
5. Identity and pride can be more powerful than cash among teachers when looking for additional effort from them. If they are convinced, they are likely to participate.
6. Quality control mechanisms must include visits to schools by decision-makers to be effective. Visits are not to evaluate those schools, but to evaluate the design and implementation processes.
7. Massive provision of information in many cases down to the child level is vital. Also, the information has to be tied to action items and has to have rapid feedback loops.
8. Make the feedback cycle as short as one can to monitor closely and generate faster gains. Remember that one has to deliver in a very clear manner before the top sector leadership that welcomed the change is replaced, as top-level leadership changes often.
9. Keep legal arrangements as flexible as possible in order to react in a timelier manner.
10. Sustainability increases when permanent personnel (supervisors, principals) and external institutions get involved. Private institutions, international organizations, and NGOs must be considered.
11. To be transferred (level, region, country), strategies and actions need to be adapted to people and circumstances. It is not about recipes and resources. Systems need to be adapted to local realities, and people may have different priors and motivation in circumstances beyond those of the first success.
12. Give the most credit to local authorities and teachers. Governments need political rewards, and teachers deserve recognition. Unless they decide otherwise, low profile work from external participants is more efficient.

The annex includes a table that captures some of the essential elements of the four cases. The table is based on the PowerPoint presentations made by three of the country experts at the December 2019 workshop but also on prior and ensuing discussions regarding each case. We do not present the Chile case here as it covers a complex set of policies developed over a few decades, in a context of already-high performance relative to the rest of Latin America, which has also improved over recent decades. The Chile case also did not describe a pedagogical focus on foundational literacy and numeracy, so a comparison of the Chile case with the other three cases was not possible. Following the summary table are narratives that provide the context and story for each case, after providing data on why the case is worthy of study. In this table, the cases of Sobral and Puebla relate to fairly specific interventions that had a relatively clear beginning and specific development, and the Kenya case relates to a particular intervention with a clear starting point but in a context of historically good performance relative to peers.  The table is organised in terms of the "what" (describing the approach or intervention), "why" (why that specific approach and what caused the government to embark upon it), and "how" (how the reforms were accomplished). Each of these three big themes has a set of sub-themes. The case of Puebla had different approaches for various levels; for this note we emphasize the primary or generic (all grades) interventions. 

Click here to open the annex.

1 http://data.uis.unesco.org/

2 http://uis.unesco.org/en/news/6-out-10-children-and-adolescents-are-not-learning-minimum-reading-and-math

3 Approximate, informal but more than back-of-the envelope data only using ENLACE to link to PISA.

4 Even more approximate, informal, but more than back-of-the envelope data using SAEBE data to link to PISA and PIRS.

5 Fluency is used in many efforts as a goal or benchmark that is reasonably easy to explain and measure with reliability. Most experts would agree that the real goal should be comprehension. Some would argue that a good alternative proxy is accuracy of reading. (It is what is being used in UNICEF’s MICS learning module.)

6 We understand that Sobral’s relative performance at the secondary school level is much less noteworthy, which is understandable given that that is not where the efforts were directed.

7 OECD. (2016). Improving School Leadership and Evaluation in Mexico: A State-Level Perspective from Puebla. Paris: OECD. https://www.oecd.org/education/school/Improving-School-Leadership-and-Evaluation-in-Mexico-A-State-level-Perspective-from-Puebla.pdf

8 A working paper by Rafael de Hoyos and Benardo Naranjo on the case, forthcoming in Xaber, an upcoming civil society portal with data and evidence on education in Mexico; personal communication for now. Another useful one is also a study by UNESCO/Mexico titled “Marco de la Política Educativa en el estado de Puebla.” An informative interview with the Secretary of Education for Puebla at the time of the reforms, Patricia Vasquez, can be found at: https://www.inee.edu.mx/origen-recorrido-y-logros-del-modelo-educativo-poblano-asistencia-permanencia-y-aprendizaje/.

9 Some features of the Ontario model generalise what is noted for all three cases in this note; others do not. In common are: a limited number of goals (just three in Ontario, also three in Puebla: hence high focus), a focus on pedagogy rather than general management, province-wide assessment that was relatively low-stakes but thorough, using assessment to assess the system and approach, and help the teachers, rather than for accountability in a narrow sense, assessments that were criterion-reference (see benchmarks used by some of the efforts described in this note), and a province-wide achievement data set. Other features, perhaps more appropriate for a system that was already achieving at levels much higher than those found in Mexico, Brazil, or Kenya, included that the approach was much more teacher-led even in a technical sense, and worked less on teaching technique than on teacher reflection. See https://internationalednews.com/2015/10/28/learning-from-successful-education-reforms-in-ontario/ for a short summary.