Some Dissection of the Latest Credo Study on Charter Schools
The recent CREDO (Center for Research on Educational Outcomes) report comparing charter schools and traditional public schools has been described by charter proponents as deciding proof of charter superiority. Blogger Derek Black, with analysis from Bruce Baker, says not so fast.
Education Law Prof Blog: Are Charter Schools Finally Outperforming Traditional Public Schools?
Derek Black March 25, 2015
Probably not, but the news stories surround the most recent charter school study by the Center for Research on Educational Outcomes (CREDO) would have the public believe so. CREDO’s studies have been a center point in the debate over the efficacy of charter schools since 2009. Charter school advocates used the 2009 study to demonstrate that some charters (17% to be precise) were outperforming traditional public schools. Those advocates ignored the 37% that were under-performing in comparison to traditional public schools. Charter school skeptics hammered that point and backed it up with subsequent studies.
CREDO’s second report in 2013 was more equivocal than the first and moved in a direction to the liking of charter schools. Rather than focusing on raw performance, it sought to identify educational improvement, finding that charter schools in general were showing more growth than traditional public schools. Some argued that larger growth was potentially easier because charters were starting from a lower baseline. The changed frame of analysis also elicited criticism from both sides regarding the methodology of the study.
CREDO is now out with its 2015 report, and its equivocation is all but gone. The study finds that “urban charter schools in the aggregate provide significantly higher levels of annual growth in both math and reading compared to their TPS peers. Specifically, students enrolled in urban charter schools experience 0.055 standard deviations (s.d.’s) greater growth in math and 0.039 s.d.’s greater growth in reading per year than their matched peers in TPS. These results translate to urban charter students receiving the equivalent of roughly 40 days of additional learning per year in math and 28 additional days of learning per year in reading.”
This finding was met with applause by education reformers, charter school advocates, and the business community. It was met with keen interest by the media. It has been met largely with silence from those formerly critiquing charters (or they have been unable to capture headlines). Does this study and the silent reaction to it mean that charter schools have finally matured and are demonstrating superiority over traditional public schools? Is the debate, in effect, nearing resolution? Not so fast, says Bruce Baker. We still must compare apples to apples, and it is not clear that CREDO has done that.
Those seeking to demonstrate charter superiority have almost always compared apples to oranges. If the student demographics of charters differ from traditional public schools, raw achievement scores between the two cannot be accurately compared. Responding to this problem, newer studies, including CREDO’s, have attempted to account for differing student demographics.
But CREDO’s new study may have done both too much and too little in this regard. CREDO’s new study narrows the field further than every before, largely in the attempt to triangulate some area of advantage for charters. The new study does not compare charters and traditional public schools on the whole, but only urban charters to urban traditional public schools. That comparison is probably correct, but, of course, those are not the only charter schools in operation. Thus, at best, the study suggests that under certain circumstances, charters outperform traditional public schools.
Bruce Baker, however, says the new study still presents a distorted picture in regard to student demographics, even when narrowed to urban schools. The variables the study uses to “match” an urban public school to a charter for comparison “are especially problematic.” It is inaccurate to treat charters’ “poor kids” as equivalent to traditional public schools’ “poor kids.” And it is, likewise, inaccurate to assume that charters’ special education kids are the same as traditional public schools’ kids. In fact, there is a lot of variation within those two categories, and charters may very well have the most advantaged students within those otherwise narrow groups. Baker further explains:
Newark data are particularly revealing of these problems. Charters undersubscribe the poorest students and oversubscribe the less poor, but CREDO treats those kids as matched anyway…
Charters undersubscribe high need special education kids and oversubscribe mild learning disabled (as a share) but CREDO treats those kids as matched.
This creates a severe bias in favor of charters in Newark and in many other cities with similar sorting patters and high average poverty rates.
This perhaps provides partial explanation for why CREDO tends to find stronger charter effects in poor urban centers than, say in suburbs, where their matching measures – at least for income status – would potentially be more useful.
The point is that the virtual record comparison asserts that these kids are otherwise similar, and thus the gains are somehow attributable to “charter” schooling as a treatment. This assertion is deeply flawed at two levels. First, the as noted above the variables they are choosing for matching are nearly useless. They don’t necessarily identify similar kids at all. Nearly all kids fall below the income threshold they are using and thus they might label as “matched” (likely do in fact) a kid in deep poverty/homelessness, etc. in a district school with a kid marginally below the reduced lunch cut point in a charter. They might also label as “matched” a mild specific learning disability kid in a charter (since that’s all they have for disability) with a far more severely disabled kid in a district school (where district schools have disproportionate shares of those kids now because charters have siphoned some of the less needy spec ed kids).
The second level problem here is that the CREDO study doesn’t then account separately for who these kids attend school with – the peer effect. It conflates that effect with “school” effect, by omission.
Deep stuff. It is probably deeper than the average reporter cares to consider, which might explain some of the silence. But these distinctions are crucial in understanding the new CREDO Larry Miller's Blog: Educate All Students! | Educate All Students:
