It's not the science that is junk, it's the measures
So I recently had occasion to read a whole bunch of studies on charter schools and one type I read was about their effectiveness. I read the CREDO studies and I read critiques of the CREDO studies and I read meta-analyses and I read smaller studies.
Anyway, I want to go back to something I used to say and that I have heard others who are similarly skeptical of Big Ed Reform, and that is the notion of "junk science." A lot of us have called VAM and have called other studies of educational effectiveness "junk science." I know I did, indignantly. But you know what? I didn't really know what I was saying. (This is one reason I went back to get my PhD, so I would have more understanding of these kinds of things.)
And I was reading all of these studies on the effectiveness of charter schools, I remembered reading this post by Matt DiCarlo on the Shanker Blog from over 3 years ago. I remembered that reading it gave me pause about calling what I did "junk science" and I ceased doing so, but even so, I couldn't fully relate to what he was saying:
Now, I personally am not opposed to using these estimates in evaluations and other personnel policies, but I certainly understand opponents’ skepticism. For one thing, there are some states and districts in which design and implementation has been somewhat careless, and, in these situations, I very much share the skepticism. Moreover, the common argument that evaluations, in order to be "meaningful," must consist of value-added measures in a heavily-weighted role (e.g., 45-50 percent) is, in my view, unsupportable.
All that said, calling value-added “junk science” completely obscures the important issues. The real questions here are less about the merits of the models per se than how they're being used.
If value-added is “junk science” regardless of how it's employed, then a fairly large chunk of social scientific research is “junk science." If that’s your opinion, then okay – you’re entitled to it – but it’s not All Things Education: It's not the science that is junk, it's the measures:
