The dark side of education research: widespread bias
Johns Hopkins study finds that insider research shows 70 percent more benefits to students than independent research
Critics have attacked Big Pharma for widespread biases in studies of new and potentially profitable drugs. Now, scholars are detecting the same type of biases in the education product industry — even in a federally curated collection of research that’s supposed to be of the highest quality. And that may be leaving teachers and school administrators in the dark about the full story of classroom programs and interventions they are considering buying.
An analysis of 30 years of educational research by scholars at Johns Hopkins University found that when a maker of an educational intervention conducted its own research or paid someone to do the research, the results commonly showed greater benefits for students than when the research was independent. On average, the developer research showed benefits — usually improvements in test scores — that were 70 percent greater than what independent studies found.
“I think there are some cases of fraud, but I wouldn’t say it’s fraud across the board,” said Rebecca Wolf, an assistant professor in the Center for Research and Reform in Education at Johns Hopkins University and lead author of the draft study. “Developers are proud of their products. They believe in them. They’ve worked hard in developing these products. They want a study that puts the best face forward.”
Biased research matters because current federal law encourages schools to buy products that are backed by science. In order to tap into federal school improvement funds, for example, low-achieving schools with disadvantaged children are required to select programs that have been rigorously tested and show positive effects.
The study, “Do Developer-Commissioned Evaluations Inflate Effect Sizes?” was presented at a March 2019 conference session of the Society for Research on Educational Effectiveness (SREE) in Washington, D.C. The paper is a working paper, meaning it has not yet been published in a peer-reviewed journal and may still be revised.
Wolf and three of her colleagues analyzed roughly 170 studies in reading and math dating as far back as 1984 that are part of the What Works Clearinghouse. That’s an archive of research that the U.S. Department of Education launched in 2002 to help educators decide which educational products to buy. It is by no means a complete or an exhaustive collection of educational research but a group of high quality studies curated by experts. The studies track test score gains and compare students who got the intervention with those who didn’t.
More than half, or 96, of the studies were conducted by independent researchers while 73 of them had some sort of insider connection with creating or selling the product. Wolf labeled the research a CONTINUE READING: The dark side of education research: widespread bias
