Janeli Kotzé compared the effectiveness of virtual, tablet-based coaching and on-site coaching for South African primary school teachers. Charlotte Jones discussed a study of the delivery of teacher professional development through communities of practice implemented at scale in Kenya and Rwanda. Andy de Barros examined the efficacy of teacher training and instructional materials for a primary school mathematics programme in Karnataka, India. Todd Pugatch presented results from an evaluation of the impact of an extensive teacher training programme on the delivery of a compulsory entrepreneurship course in Rwandan secondary schools across a range of outcomes.

Due to the time constraints of the event, our panellists were unable to answer all of the questions submitted by our attendees. Fortunately, the panel has enthusiastically tried to answer the remaining questions. Click on one of the questions below to learn more about our panellists’ research.

1. Can the panellists speak about the role of teacher motivation to meaningfully engage in continuing professional development, and apply what they learn? Go to Question 1 ↓
2. I was wondering if in any of the analyses we disaggregate teachers according to seniority or years in the profession? Or more broadly what characteristics seem to have a bearing on whether these teachers are more or less likely to take up reform-oriented professional development? Go to Question 2 ↓
3. To what extent were teachers themselves involved in designing your programme or intervention? Go to Question 3 ↓
4. Our discussion here of existing norms affecting changes in teacher practice is limited to ‘external factors’, such as classroom conditions or system-level factors like an exam-oriented system. What role do you think factors ‘intrinsic’ to the teacher, such as pre-existing beliefs about teaching and learning for instance, play in not only bringing about a change in practice but also ensuring that it persists? I know the panellists talked briefly about teacher emotion… Go to Question 4 ↓
5. Apart from in-service training interventions, in your experiences around the world, what are some things that work in teachers' initial training (bachelor's programs, teacher training programs, entry requirements or teacher assessments etc.)? Go to Question 5 ↓
6. Do you have a hypothesis for why the program only had an impact on girls? Are they farther behind on math? Given all the data collected to unpack the black box, what do you hypothesize is driving the effect? Go to Question 6 ↓
7. Teachers teaching and learning from fellow teachers has always proved beneficial, but did you relate your study with learning achievement? Go to Question 7 ↓
8. My (biased?) reading of the evidence is that when the community of practice lead is in the school, but is rather insistent in, and is well trained in, "direct and structured instruction," it can work. But if it is just feel-good leadership, it does not. Go to Question 8 ↓
9. Were there any teachers from low cost private schools or private schools in general who took part in the coaching programme? Go to Question 9 ↓
10. Might there be some benefit in developing coaching the coach interventions to enable face-to face professional development, where a coach has virtual support? Go to Question 10 ↓
11. The lack of trust due to years of disciplinary approaches are difficult to shift. What were the mechanisms that were used to achieve this behaviour change re emotional readiness? is your approach scalable and transferable to other contexts and what is the cost? Go to Question 11 ↓
12. To what extent might decreased income be accounted for by the fact that the program increased university enrolment (and therefore for at least some students delayed wage earning)? Go to Question 12 ↓
13. It was mentioned that the implementation of Skill Labs was low in the control schools. Was it higher in the treatment schools? If so, what factors helped in improving the rate of implementation? Go to Question 13 ↓
14. Entrepreneurship requires a lot more than technical/practical skills. The non-cognitive dimensions are also needed. Could teaching for attitude change and value orientation (soft skills) not have been added? Go to Question 14 ↓

1. Can the panellists speak about the role of teacher motivation to meaningfully engage in continuing professional development, and apply what they learn?

   • Charlotte Jones: Wider international evidence (e.g. Darling-Hammond’s work) shows that when teachers learn together (for example in communities of practice) this can also enhance their professional motivation. We will be measuring changes in confidence and self-efficacy and our hypothesis is that this will increase as a result of working in the communities of practice.
   • Janeli Kotzé: We realised that teacher motivation played a critical role in the virtual coaching relationship. Since the on-site coaches visit the teachers in the classroom, they had the opportunity to motivate teachers and therefore often acted as an ‘external motivator’. The virtual coach was dependent on the teacher opting in to being coached. That is, the teacher needed to answer the phone calls, submit videos/ photos to the virtual coach, or participate in the WhatsApp group conversations. Teachers who were not motivated could easily ‘avoid’ the support from the virtual coach.
   • Todd Pugatch: The program we study gave no incentives to participate in continuing professional development, nor require any punishment for not participating. Nonetheless, we find takeup of 80 percent or more. This suggests high demand from teachers for continuing professional development. Top ↑

2. I was wondering if in any of the analyses we disaggregate teachers according to seniority or years in the profession? Or more broadly what characteristics seem to have a bearing on whether these teachers are more or less likely to take up reform-oriented professional development?

   • Charlotte Jones: Yes, we are recording years in the profession, as well as other teacher characteristics such as age/gender.
   • Janeli Kotzé: The average age of the teachers in our intervention was 49 years old, with 80 percent of the teachers being older than 45 and 25 percent of the teachers being older than 55. We tried to see if we can find any correlation between programme uptake and age, especially in terms of the technology component, but could not find any relationship.
   • Todd Pugatch: We don’t find any differential treatment effects on student outcomes based on teacher experience. Top ↑

3. To what extent were teachers themselves involved in designing your programme or intervention?

   • Andy de Barros: In our case, the intervention had been iterated on in government schools, together with government teachers, since 2011. You may find additional background information on the program’s inception and design here.
   • Charlotte Jones: Good question. They were not. However, the programme delivery teams were deeply involved, including operational leads who work with coaches and teachers on the ground, who fed in key issues and challenges that this research could help to shine a light on.
   • Janeli Kotzé: Our programme builds on a series of similar programmes since 2010 and has implicitly taken the inputs of teachers on board as we went along. In terms of the lesson plan design, we also have a reference group with the district officials to provide inputs. However, we did not explicitly involve teachers in the design.
   • Todd Pugatch: The Rwandan government partnered with the NGO Educate! to design the intervention. Both drew on their prior experience with entrepreneurship teachers—the Rwandan government based on 7 previous years requiring entrepreneurship as a secondary school subject, and Educate! based on its youth entrepreneurship education program in Uganda. Top ↑

4. Our discussion here of existing norms affecting changes in teacher practice is limited to ‘external factors’, such as classroom conditions or system-level factors like an exam-oriented system. What role do you think factors ‘intrinsic’ to the teacher, such as pre-existing beliefs about teaching and learning for instance, play in not only bringing about a change in practice but also ensuring that it persists? I know the panellists talked briefly about teacher emotion…

   • Charlotte Jones: Our qualitative work is looking at this, and we will be investigating shifts in mindset such as an increasing sense of collective responsibility, a sense of professional agency, etc.
   • Janeli Kotzé: We realised that teachers’ emotional readiness to adopt new practices played a key role. Coaches first needed to build a trusting relationship, so that teachers will be willing to take their advice and support on-board. Secondly, we realised that teachers will only persist with the new practices once they saw a change in their learners’ performance and were convinced that the new methods worked. In a previous study, we found the effects of on-site coaching to persist much more than traditional teacher training sessions.
   • Todd Pugatch: Our study finds an increase in student business participation in response to the program, but this effect is dampened when a student has a qualified (certified) teacher. This suggests qualified teachers were less willing or able to embrace the program’s focus on encouraging entrepreneurship among students. We hope to perform additional analysis on these or other “intrinsic” factors among teachers. Top ↑

5. Apart from in-service training interventions, in your experiences around the world, what are some things that work in teachers' initial training (bachelor's programs, teacher training programs, entry requirements or teacher assessments etc.)?

   Todd Pugatch: Our study doesn’t address this topic directly. But in other work I have done, pre-service teacher training has little relationship to student performance. Some ways to improve pre-service training include: “First, emulate successful practices used by in-service training programs. Second, reposition certification as one point of continuous professional development, not merely a one-time event. Finally, install clear and transparent incentives for career advancement based on performance, not merely formal qualifications and experience. Such reforms would help to support and retain the most talented among the present teaching corps, as well as to attract bright and ambitious graduates to the profession.” Top ↑

6. Do you have a hypothesis for why the program only had an impact on girls? Are they farther behind on math? Given all the data collected to unpack the black box, what do you hypothesize is driving the effect?

   Andy de Barros: We don’t entirely know, at least not yet, but we can already rule out a couple of hypotheses. Girls were indeed slightly farther behind in math, but we don’t find differential impacts by baseline ability. I just took a brief look at our classroom observation data—it also doesn’t look like we find differences in the extent to which teachers “treated students of all genders with equal regard” (as measured by TEACH). This will be more exploratory, but we can still look into differential effects on student attitudes. Top ↑

7. Teachers teaching and learning from fellow teachers has always proved beneficial, but did you relate your study with learning achievement?

   Charlotte Jones: In phase 1 of the Kenya programme (3 years 2014-2017), Education Development Trust saw gains of 0.52 SD on girls’ learning outcomes, validated by Coffey International’s final evaluation impact report (p.33 table 11, page 40 table 14). The budget envelope for this study will allow us to track changes in instructional quality only unfortunately. However, the link between instructional quality and student outcomes is well established. The two programmes do obviously track changes in learning outcomes as part of their monitoring requirements (see above). Top ↑

8. My (biased?) reading of the evidence is that when the community of practice lead is in the school, but is rather insistent in, and is well trained in, "direct and structured instruction," it can work. But if it is just feel-good leadership, it does not.

   Charlotte Jones: This is one of the key takeaways of the paper. Too much school leadership development is generic. What are we training school leaders for? The Teachers Learning Together study has tried to get quite forensic on this question, and the leadership and facilitation attributes that actually lead to professional growth. We had no time to discuss in the panel, but we are looking inside the black box of the community of practice and have invested in a tool which looks at community of practice features and attributes, including leadership inputs. Top ↑

9. Were there any teachers from low cost private schools or private schools in general who took part in the coaching programme?

   Charlotte Jones: Yes, in Kenya a significant proportion of the schools in the urban slum areas are low cost private schools. The study finds some interesting differences between community of practices between types of schools and this will be further analysed in the endline analysis. Top ↑

10. Might there be some benefit in developing coaching the coach interventions to enable face-to face professional development, where a coach has virtual support?

    Janeli Kotzé: There could be a benefit in supporting coaches virtually, but your up-front training of the coach will be very important. For instance, we employed coaches who have a couple of years of experience in both teaching and coaching early grade reading. We still spend a week each term in training them on the content, reading methodologies and coaching support methods. Top ↑

11. The lack of trust due to years of disciplinary approaches are difficult to shift. What were the mechanisms that were used to achieve this behaviour change re emotional readiness? is your approach scalable and transferable to other contexts and what is the cost?

    Janeli Kotzé: Firstly, our coaches were employed by an external organisation and not the government. That already meant that there was an additional layer of confidentiality between the coach and the teacher, since the coach will not report on the teacher’s performance. At this point, we are not sure how critical this aspect is to the trust relationship. Secondly, we spent a lot of effort to train the coaches on the soft-skills, and even ‘scripted’ this slightly. For instance, coaches had to always start their feedback with a compliment, and they had to ensure that the majority of their feedback was encouraging. We are currently in the process of developing a coaching course and the intention is to train district officials on these coaching methods. We will see whether this will change their approach to supporting teachers. Top ↑

12. To what extent might decreased income be accounted for by the fact that the program increased university enrolment (and therefore for at least some students delayed wage earning)?

    Todd Pugatch: We’re looking into this, as the results on income and university enrolment remain preliminary. Top ↑

13. It was mentioned that the implementation of Skill Labs was low in the control schools. Was it higher in the treatment schools? If so, what factors helped in improving the rate of implementation?

    Todd Pugatch: Yes, Skills Lab implementation was considerably higher (52 percentage points) in treatment schools. We attribute this to the emphasis placed on Skills Labs in the training and outreach program. Teachers and head teachers got the message that Skills Labs were essential to curricular implementation. Top ↑

14. Entrepreneurship requires a lot more than technical/practical skills. The non-cognitive dimensions are also needed. Could teaching for attitude change and value orientation (soft skills) not have been added?

    Todd Pugatch: Soft skills were indeed included in the curriculum and our evaluation, though I didn’t focus on these in the presentation. We find no treatment effect on a range of dimensions, though we do find increases in grit among treated students in preliminary results of our follow-up study. Top ↑

For more information regarding the event, including links to the papers that were presented, please visit the event page on the RISE website.

• Katie Cooper