For nearly two decades, we have been hearing about the potential of technology to disrupt education. Yet, after two decades and countless laptops, smartboards, and other educational gadgets distributed across schools in developed and developing countries, ed-tech does not seem to have delivered on its promise.

In the developing world, several studies have found that providing schools and students with laptops does not improve their learning and it may actually get in the way. Using technology to allow students to review the material from class has only had small positive effects.

Why are we still waiting for technology to deliver on its promise to radically improve learning? An efficacy trial of a “blended learning” program in India that we recently published in the American Economic Review suggests that we may not have been taking full advantage of the things that technology does best, and that when we do, its effects on learning are much larger.

Program, evaluation, and results

We randomly assigned 619 students, mostly in Grades 6 to 9, in Delhi, India to either receive free access to a computer-assisted learning software in math and language and small-group instruction during the study or once the study had concluded. The software, called “Mindspark”, had been developed by Educational Initiatives, one of the leading assessment firms in India, over a ten-year period of iterative design and field testing. It draws on an item bank of over 45,000 questions developed by pedagogical experts and 400,000 users who interact with these questions more than 1 million times a day. Students assigned to receive this program attended stand-alone centers before or after school six days a week for 90 minutes (45 minutes of software and 45 minutes of small group instruction). Each week, they received two days of math instruction, two days of Hindi (the local language), one day of English, and one day devoted to a subject of their choice. The study lasted for about half a school year (4.5 months).

The Mindspark software leverages four comparative advantages of technology to improve learning: its capacity to standardize the delivery of high-quality content, adjust the level and pace of activities to each student’s preparation level, identify common misconceptions and redirect students to activities that remedy them, and its reliance on activities that require the continuous engagement of students to move forward. Importantly, the blended learning program also had an instructor in the room while children interacted with the software to provide them with technical support and maximize their time “on task.”

After less than five months, students assigned to attend the program performed .37 standard deviations better in math and .23 standard deviations better in Hindi—i.e., 2 and 2.4 times the progress made by their control counterparts over the same period, respectively (Figure 1). The average treatment student only attended 58% of the after-school sessions. Our estimates indicate that attending the program for 90 days (80% of the sessions) would have increased achievement by .6 standard deviations in math and by .39 standard deviations in language. To our knowledge, this is one of the largest effects of an educational intervention evaluated experimentally in a low- or middle-income country—and it was achieved in just half a school year. This is particularly remarkable, given how little evidence on how to improve learning in middle schools in developing countries.

The centrality of “teaching at the right level”

How can the effects be so large? In part, the answer lies in a crucial fact about our context: the wide heterogeneity in student preparation. In many South Asia and Sub-Saharan African countries, recent expansions in access to schooling have incorporated children and youths who are among the first in their families to enroll in school. These children lag far behind curricular expectations. In our study, we found that the average sixth grader was about 2.5 grade levels behind their expected performance in math in grade 6 and 4.5 grade levels behind in Grade 9 (Figure 2).

There is growing evidence that these children are typically poorly served by the school system. High-stakes exams at the end of middle and secondary school encourage teachers to teach to the top of the ability distribution, which ends up discouraging new generations of learners and pushing them out of school. A differentiated instruction approach known as “teaching at the right level” has made some headway in addressing this problem at the primary school level, but it has proven to be challenging to encourage teachers to implement this approach during regular school hours. Further, implementing differentiated instruction gets much more challenging at higher grades where the content is more complex and the variation in the classroom is higher.

The blended learning program that we evaluated benefits students at all levels of preparation, but it produces the greatest relative improvement for students with low levels of preparation, who learned close to nothing despite being enrolled in school (Figure 3). This is partly because it presents less well-prepared students with easier problems and progressively graduates them to more difficult activities. In fact, on every session, the software is able to do what no human—no matter how talented at teaching—can achieve: to simultaneously present each student with an activity of different difficulty (often spanning several grade levels at the same time) and to adjust the level of subsequent activities based on that student’s pace.

From efficacy trial to scale-up

Our study illustrates what we see as the real potential of technology in education: to integrate technology and pedagogy, to allow for personalization of the level and pace of the material, and to maximize time “on task.” It serves as a “proof of concept” that we can leverage the comparative advantages of technology to improve learning. Yet, it raises the question of whether this software would work at scale within the public-school system.

Currently, two of us are involved in evaluating a version of the Mindspark software at scale, within the public-school system of the Indian state of Rajasthan. With three years’ exposure to the program, starting early in primary schools, we hope that we can prevent the vast learning deficit from opening up in the first place. In an ideal scenario, all students would reach the standards of foundational literacy and numeracy. We will not know how close (or not) we get to this goal until the results come in next year. But, in the meantime, these first results already offer hope—not all uses of technology in education are promising, but when technology solves an important constraint in the classroom, big improvements are possible!

This post originally featured on the Education Next website on 22 April 2019 and has been cross-posted with permission.