On negative effects of vouchers
Executive summary
Recent research on statewide voucher programs in Louisiana and Indiana has found that public school students that received vouchers to attend private schools subsequently scored lower on reading and math tests compared to similar students that remained in public schools. The magnitudes of the negative impacts were large. These studies used rigorous research designs that allow for strong causal conclusions. And they showed that the results were not explained by the particular tests that were used or the possibility that students receiving vouchers transferred out of above-average public schools.
Another explanation is that our historical understanding of the superior performance of private schools is no longer accurate. Since the nineties, public schools have been under heavy pressure to improve test scores. Private schools were exempt from these accountability requirements. A recent study showed that public schools closed the score gap with private schools. That study did not look specifically at Louisiana and Indiana, but trends in scores on the National Assessment of Educational Progress for public school students in those states are similar to national trends.
In education as in medicine, ‘first, do no harm’ is a powerful guiding principle. A case to use taxpayer funds to send children of low-income parents to private schools is based on an expectation that the outcome will be positive. These recent findings point in the other direction. More needs to be known about long-term outcomes from these recently implemented voucher programs to make the case that they are a good investment of public funds. As well, we need to know if private schools would up their game in a scenario in which their performance with voucher students is reported publicly and subject to both regulatory and market accountability.
Recent research on statewide voucher programs in Louisiana and Indiana has found that public school students that received vouchers to attend private schools scored lower compared to similar students who did not attend private schools.[i] This is the kind of research finding that generates a reaction of ‘wait, what?’ Negative effects are rare in education research.
The magnitudes of the negative impacts were large, too. In Louisiana, a public school student who was average in math (at the 50th percentile) and began attending a private school using a voucher declined to the 34th percentile after one year. If that student was in third, fourth, or fifth grade, the decline was steeper, to the 26th percentile. Reading declined, too: a student at the 50th percentile in reading declined to about the 46th percentile. In Indiana, a student who had entered a private school with a math score at the 50th percentile declined to the 44th percentile after one year.
Voucher programs emerged in the nineties in Milwaukee, and later in New York City, Dayton, Ohio, and Washington, DC.[ii] The Milwaukee, Dayton, and New York programs were supported by state and local funds or philanthropies, and the DC program was supported by Federal funds. In each location, students applied to receive a voucher, which would be paid to the private school they chose to attend.
Since then, vouchers and voucher-like mechanisms for supporting private school choice with public funds (tax-credit scholarships, education savings accounts, and individual tax-credits) have exploded at the state level.[iii]
In publicly funded voucher programs, many of the private schools that are recipients are religious organizations. This generates thorny legal issues in the 38 states that have so-called Blaine amendments to their constitutions (which prohibit direct government aid to educational institutions that have a religious affiliation and were originally intended to target Catholic schools in states with large immigrant populations).
Legal issues aside, the present rationale for vouchers—perhaps the rationale—is to help low-income students attend private schools because they will learn more. And if parents look at data from the National Assessment of Educational Progress (NAEP), they would see that, historically, private school students regularly score higher than public school students by 15 to 20 points in reading and math, which is a big difference (on average, NAEP scores rise about 10 points for each grade level). So, a parent might surmise, if their child attends private school, their child’s scores will be higher too. But some part of that score difference arises because higher-performing students select into private schools. Parents do not know howtheir child would score in private school.
And, in fact, studies of vouchers have reported mixed results on scores. Scores improved for some students in some places, and scores did not improve for other students in other places.[iv] The evidence showed higher reading and math scores for black students in New York City, and higher reading scores in the third follow-up year of the DC program. The evidence showed no differences in scores for Milwaukee students in reading, for Hispanic students in New York City in reading and math, and for DC students in math. What little research there has been on other education outcomes reported higher rates of high school graduation in New York City and DC, and higher rates of college attendance in New York City.
None of these earlier studies reported significant negative effects on test scores, which adds to interest in what might be happening in Louisiana and Indiana that could explain negative effects. Is it something about how the research studies were designed, how the programs were structured, the quality of the public schools against which private schools were being compared?
The research designs are unlikely to explain the findings. The Louisiana study used a random assignment experimental design, which is a high research standard for measuring effects. Louisiana private schools that had more applicants than available seats operated lotteries to choose applicants randomly. Students who won the lottery constituted the treatment group, whereas those that lost were the control group. Random assignment experiments are central in medical research and are increasingly used in education. For example, studies of the effects of charter schools have relied on experiments created by school lotteries.[v] And two different teams of researchers studying the Louisiana data reached similar conclusions.
In Indiana, effects were measured by comparing time trends of test scores for students who initially attended public schools and then received vouchers to time trends for students who continued to attend public schools, a ‘fixed-effects’ estimator. This is a lower research standard for measuring effects than experimental designs, but sample sizes were quite large (more than 3,000 voucher students and 500,000 public school students) and the findings are consistent with findings from Louisiana.
There was an oddity in the structure of testing in Louisiana. Tests used there were mandated by the state to be the ones used for the Louisiana Education Assessment Program (LEAP) carried out in public schools. Students that used their voucher to attend private schools had to take LEAP tests, though other students in those schools did not. Because LEAP tests are part of the state’s accountability structure for its public schools, the tests are ‘high stakes’ for those schools but low stakes for private schools. Perhaps students in the voucher group did worse than those who remained in public schools because the public schools taught to the test, whereas the private schools did not.
To address this hypothesis, the Mills and Wolf study exploited the fact that some LEAP tests had more content related to Louisiana state standards, and those tests might favor public school students. The study reported that private school voucher students did worse than public school counterparts regardless of the degree of alignment of the tests with the state standards guiding instruction in public schools.
Maybe students that applied for vouchers were attending excellent public schools rather than mediocre or bad ones, so the private schools that received voucher students were up against the cream of Louisiana public schools. The Abdulkdiroglu et al. study rejected this hypothesis with the finding that Louisiana public schools attended by students applying for a voucher were below average in the state. On the recent NAEP, Louisiana ranked 43rd among states in reading scores of its fourth graders and 49th among states in math scores of its eighth graders. To test the effectiveness of the instruction of voucher students in private schools by comparing the students’ learning outcomes with those of similar students in below-average public schools in Louisiana is not a high bar.
Another possibility that these studies did not investigate is that our historical understanding of the superior performance of private schools is no longer accurate. Since the nineties, public schools have been under heavy pressure to improve test scores. No Child Left Behind and conditional waivers from it under the Obama administration are visible manifestations of this pressure, with various kinds of sanctions for not meeting targets for score growth. Private schools were exempt from these accountability requirements.
A recent study by Wong et al. used NAEP data to compare public schools to private schools from the nineties to 2011.[vi] The study reported that private schools started with higher scores in the nineties—this is the difference between private school students and public school students mentioned at the outset, above. By 2011, the gap had closed significantly, especially in math. The authors concluded that No Child Left Behind improved scores. But a broader conclusion is relevant here. Public schools improved relative to private schools, for whatever reason.
The figure showing national private and public school scores on fourth-grade math indicates some of what the authors found. In 2000, scores of private school students are well above scores of public school students, a gap of 14 points. By 2013, the gap had closed to 5 points.
The same contrast of public and private schools cannot be done for Louisiana or Indiana per se. Their samples of private schools are too small to meet reporting standards of the National Center of Education Statistics. But Louisiana and Indiana public schools show trends on NAEP similar to national trends. The second figure uses the same national trends as the previous figure (the dashed lines) and adds score trends for Louisiana and Indiana. General improvements in the trend that were evident nationally also are evident in Louisiana and Indiana. In fact, by 2013, Indiana’s public school fourth graders score above the national private school average.
Improvements in scores also are evident for the low-income population of students most likely to apply for vouchers. The next figure compares fourth-grade math scores for all public school students in Louisiana and students eligible for a free or reduced-price school lunch, On negative effects of vouchers | Brookings Institution:
