New Research Study: Controlling for Student Background Variables Matters
An article about the “Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables” was recently published in the peer-reviewed Journal of Research on Educational Effectiveness. While this article is not open-access, here is a link to the article released by Mathematica prior, which nearly mirrors the final published article.
In this study, researchers Matthew Johnson, Stephen Lipscomb, and Brian Gill, all of whom are associated with Mathematica, examined the sensitivity and precision of various VAMs, the extent to which their estimates vary given whether modelers include student- and peer-level background characteristics as control variables, and the extent to which their estimates vary given whether modelers include 1+ years of students’ prior achievement scores, also as control variables. They did this while examining state data, as also compared to what they called District X – a district within the state with three-times more African-American students, two-times more students receiving free or reduced-price lunch, and generally more special educations students than the state average. While the state data included more students, the district data included additional control variables, supporting researchers’ analyses.
Here are the highlights, with thanks to lead author Matthew Johnson for edits and clarifications.
- Different VAMs produced similar results overall, almost regardless of specifications. “[T]eacher estimates are highly correlated across model specifications. The correlations [they] observe[d] in the state and district data range[d] from 0.90 to 0.99 relative to [their] baseline specification.”
This has been something that has been evidenced in the literature prior, although many critics argue that this is because all VAMs (and growth models) are using the same, fallible, large-scale standardized test data; hence, this might be less surprising if even the most sophisticated models are processing “garbage in” and “garbage out.”
- “Yet even correlation coefficients above 0.9 do not preclude substantial amounts of potential misclassification of teachers across performance categories.” The New Research Study: Controlling for Student Background Variables Matters | VAMboozled!: