Whether better test scores can increase college attendance among poor students in low- and middle-income countries remains an open question. Using data from five long-running panels in Ethiopia, India, Pakistan, Peru and Vietnam, we show that (a) at age 22 there are substantial gaps in years of schooling by socioeconomic status but, (b) conditioning on test scores at the end of primary school eliminates only between 15--50% of these gaps. An exclusive focus on test score improvements in primary schools or earlier will not equalize access to post-secondary education for the poor in these five countries.