Grading Teachers by the Test
In 2004, the Chinese government decided there were too many accidental deaths. China’s safety record, it decreed, should be brought in line with those of other middle-income countries. The State Council set a target: a decline in accidental deaths of 2.5 percent per year.
Provincial authorities kicked into gear. Eventually, 20 out of a total of 31 provinces adopted “no safety, no promotion” policies, hitching bureaucrats’ fate to whether they met the death ceiling. The results rolled in: by 2012 recorded accidental deaths had almost halved.
It wasn’t, however, all about increased safety. For instance, officials could reduce traffic deaths by keeping victims of severe accidents alive for eight days. They counted as accidental deaths only if the victims died within seven.
In a study of China’s declining deadly accidents, Raymond Fisman of Columbia University and Yongxiang Wang of the University of Southern California concluded that “manipulation played a dominant role.” Bureaucrats — no surprise — cheated.
This is hardly unusual. It is certainly not exclusive to China. These days, in fact, it has acquired particular importance in the debate over how to improve American education.
The question is, what will happen when teachers are systematically rewarded, or punished, based to some extent on standardized tests? If we really want our children to learn more, the design of any system must be carefully thought through, to avoid sending incentives astray.
“When you put a lot of weight on one measure, people will try to do well on that measure,” Jonah Rockoff of Columbia said. “Some things they do will be good, in line with the objectives. Others will amount to cheating or gaming the system.”
The phenomenon is best known as Goodhart’s Law, after the British economist Charles Goodhart. Luis Garicano at the London School of Economics calls it the Heisenberg Principle of incentive design, after the defining uncertainty of quantum physics: A performance metric is only useful as a performance metric as long as it isn’t used as a performance metric.
It shows up all over the place. Some hospitals in the United States, for example, will often do whatever it takes to keep patients alive at least 31 days after an operation, to beat Medicare’s 30-day survival yardstick. Last year, Chicago magazine uncovered how the Chicago Police Departmentachieved declining crime rates, simply by reclassifying incidents as
“We don’t know how big a deal this is,” said Jesse Rothstein, a professor at the University of California, Berkeley, who has criticized evaluation metrics based on test scores. “It is one of the main concerns.”
American education has embarked upon a nationwide experiment in incentive design. Prodded by the Education Department, most states have set up evaluation systems for teachers built on the gains of their students on standardized tests, alongside more traditional criteria like evaluations from principals.
Fourteen more states are expected to have fully developed systems this academic year, according to the National Council on Teacher Quality — an advocacy group that supports rigorous assessments. All but six states are expected to have one by the 2016-17 school year.
The assessments are backed by sophisticated research. An important study by Professor Rockoff and two Harvard professors, Raj Chetty and John Friedman, found that teachers who improved students’ scores, termed high value-added teachers, raised the students’ chances of going to college as well as their salaries later in life.
But teachers — and parents — are up in arms. In New York, the teachers’ union strongly opposes Gov. Andrew Cuomo’s proposal to increase the weight of test scores to 50 percent of a teacher’s evaluation. The governor is being hammered over the issue in opinion polls.
“People who claim to be market-based reformers want to sell the theory noncriminal.
