More findings about school vouchers and test scores, and they are still negative
More findings about school vouchers and test scores, and they are still negative
Vouchers to pay for students to attend private schools continue to command public attention. The current administration has proposed vouchers in its budget, and more than half of states are operating or have proposed voucher programs.
Four recent rigorous studies—in the District of Columbia, Louisiana, Indiana, and Ohio—used different research designs and reached the same result: on average, students that use vouchers to attend private schools do less well on tests than similar students that do not attend private schools. The Louisiana and Indiana studies offer some hints that negative effects may diminish over time. Whether effects ever will become positive is unclear.
Test scores are not the only education outcome and some observers have downplayed them, citing older evidence that voucher programs increase high school graduation and college-going. We lack evidence that the current generation of voucher programs will yield these longer-term outcomes. We also lack evidence of how public schools and private schools differ in their instructional and teaching strategies that would explain negative effects on test scores. Both questions should be high on the research agenda.
Vouchers to pay for students to attend private schools continue to command public attention. The current administration has proposed vouchers in its budget, and more than half of states are operating or have proposed voucher programs.
Dynarski wrote in this forum last year about recent studies that had shown negative effects of vouchers on test scores in Louisiana and Indiana. Since that time, new studies of vouchers in DC and Ohio have been released, and the Louisiana and Indiana studies released findings from an additional year.
The four different studies use four different designs but arrive at the same result: on average, students that use vouchers to attend private schools do less well on tests than similar students that do not attend private schools. With voucher programs expanding rapidly and with each of the four studies measuring effects of vouchers differently, it’s worth unpacking each study a bit to see what they say and do not say about effects of vouchers.
Table 1, in the appendix (please see attached PDF), compares features of the four studies, including the populations served by the programs, sample sizes of the studies, the test that studies used as their outcome, and how the studies measured impacts on those tests.
Figure 1, below, shows measures of test-score impacts, starting with the four studies at the top, and then effects on test scores from previous studies, roughly in reverse historical order. The point in the middle of the bars for each study is the estimate of the score effect, which is negative in both subjects in all four studies. The bars are confidence intervals for the estimated effects—when confidence intervals include zero, the effect is not statistically significant. Below the blue divider, we show effects for eight prior studies of vouchers. These show some positive effects for both subjects, though most are not statistically significant.
Figure 1. Findings from four current studies of vouchers and eight previous studies
The question is why the pattern of recent studies differs from previous studies. As Dynarski had written previously, public schools and private schools have been under different accountability pressures for the last 15 years or so, which might explain some of the findings. Recognizing that researchers often call for more research, we think that call is merited here. It is rare for policy initiatives to be expanding in the face of evidence that those initiatives may have negative effects on key outcomes.
THE DISTRICT OF COLUMBIA OPPORTUNITY SCHOLARSHIP PROGRAM
This study is a classic ‘field experiment’ consistent with the authorizing legislation that called for the program to be studied using the ‘strongest appropriate design.’ Students selected to receive a voucher could attend private schools that agreed to accept the voucher as payment, which was more than half of all private schools in the District. Students and families had no obligation to use the voucher and, after a year, the study reported that about 30 percent of students in fact had not used their vouchers. This is a useful reminder that being offered a voucher expands options for parents but does not by itself require parents to do anything.
The study administered the Terra Nova test at the time students applied for vouchers (generally spring or early summer), and again about a year later. It also collected other data about students and families such as demographic characteristics, parent education, length of time at current residence, and parent ratings of the child’s current school. These characteristics were used in statistical models to adjust for whatever differences remained between students who were offered and not offered vouchers.
The findings showed that after one year, students who had been offered a voucher scored lower on the math part of the test, and the amount by which they were lower was statistically significant (the difference could not be explained by random variation). Students also scored lower on the reading test but the amount by which they were lower was not statistically significant.
The study considered three possible explanations for the negative results. One was that students not offered vouchers went on to attend high-performing public schools (either traditional or charter schools). This did not explain much, however—students not offered vouchers attended public schools that had achievement levels that were average for the District. A second possible explanation was that students did less well on tests because they were adjusting to new schools. This explanation also did not hold up, in part because more than half of students not offered a voucher also switched schools, either because they had to (such as students who were moving from an elementary school to a middle school) or because they wanted to.
The third explanation was that private schools provided less instruction in reading and math. Data from a survey of principals that the study administered found that instructional time was lower by about an hour a week in both subjects, about twelve minutes a day. The District was not unusual in this regard—the difference in instructional time between private and public schools was about the same as the National Center for Education Statistics reported from a national survey of principals. But it’s at least plausible that students in private schools may have scored lower because they received less instruction in reading and math.
THE LOUISIANA SCHOLARSHIP PROGRAM
The Louisiana Scholarship Program (LSP) began in 2012. It is a statewide program and almost 10,000 students applied in its first year, making it considerably larger than the DC program, which averaged about 600 eligible applicants a year during the three years when students were enrolled in the study sample. Private schools that elected to participate by accepting vouchers as payment also had to administer the Louisiana state assessment to voucher-receiving students and were graded by the state using the same A-F scheme the state used for its public schools. Private schools whose voucher-receiving students scored poorly and received low grades from the state could be removed from the program.
The study of the LSP is an experiment but it is more complex than the one in DC. The lottery at the heart of the LSP experiment was conducted only when schools did not have enough available spaces at a grade level for the number of students that wanted to attend that school and grade level. A school may have had enough spaces for the number of applying fourth-graders, for example, but not enough spaces for the number of applying third-graders. That would have triggered a third-grade lottery at the school. The combination of applicant priorities, preferences parents expressed for schools, and available spaces resulted in a complex structure with 150 different lotteries, which required a complex analytic approach to measure voucher effects that is described in study reports.
The study estimated that students using vouchers had lower math scores on the Louisiana state assessment—in fact, scores were quite a lot lower. The study presented results for two samples, one that was restricted to students who had baseline scores because they had previously participated in the state tests in public school before they applied for a voucher, and another that included the full sample of students that had a test score three years later regardless of whether they had a baseline score. Scores were negative and statistically significant for the full sample, but less negative and not statistically significant for the sample that was restricted to students with baseline scores. Experiments do not have to use baseline data to estimate effects because simple differences of outcomes at follow-up are effects of the program. And using larger samples can yield more precise estimates. It depends on whether the sample is sufficiently larger to offset not having baseline test scores. In this case, our preference is for the results from the full sample, but the results from both samples point in the same direction.
Media reporting of the findings pointed to the larger negative effects in the first year and smaller negative effects in the third year as good news. This is an odd conclusion. There are different arguments for vouchers, such as that they would give parents more choice, reduce the role of government in education, enable parents to transmit values and religion to their children, and deliver cost-effective education. But certainly one of the arguments for vouchers is to enable students to thrive academically in private schools. If this is the case, there should have been no catching up to do in the first place, beyond whatever adjustments students need to make when they change schools. And it’s noteworthy that Louisiana students have not yet caught up after three years.
