ACT’s Dan Wright on a Simple VAM Experiment
A VAMboozled! folllower, Dan Wright, who is also a statistician at ACT (the nonprofit company famously known for developing the college-entrance ACT test), wrote me an email a few weeks ago about his informed and statistical take on VAMs. I invited him to write a post for you all here. His response, with my appreciation, follows:
“I am part cognitive scientist, part statistician, and I work for ACT, Inc. About a year ago I was asked whether value-added models (VAMs) provide good estimates of teacher effectiveness. I found lots of papers that examined if different statistical models gave similar answers to each other and lots showing certain groups tended to have lower scores, but all of these papers seemed to bypass the question I was interested in: do VAMs accurately estimate teacher effectiveness? I began looking at this and Audrey invited me to write a guest blog post describing what I found.
The difficulty is that in the real world we don’t really know how effective a teacher, that is in “objective” terms, so that we do not have much with which to compare our VAM estimates. Hence, a popular alternative/way for statisticians to decide if an estimation method is good or bad is to simulate data using particular values, and then see if the method produces estimates that are similar to the values expected.
The trick is how to create the simulation data to do this. It is necessary to have some model for how the data arise. Hence, I will use a simple model to demonstrate the ACT’s Dan Wright on a Simple VAM Experiment |:
