The Thrill Of Success, The Agony Of Measurement
The recent release of the latest New York State testing results created a little public relations coup for the controversial Success Academies charter chain, which operates over 20 schools in New York City, and is seeking to expand.
Shortly after the release of the data, the New York Post published a laudatory article noting that seven of the Success Academies had overall proficiency rates that were among the highest in the state, and arguing that the schools “live up to their name.” The Daily News followed up by publishing an op-ed that compares the Success Academies’ combined 94 percent math proficiency rate to the overall city rate of 35 percent, and uses that to argue that the chain should be allowed to expand because its students “aced the test” (this is not really what high proficiency rates mean, but fair enough).
On the one hand, this is great news, and a wonderfully impressive showing by these students. On the other, decidedly less sensational hand, it’s also another example of the use of absolute performance indicators (e.g., proficiency rates) as measures of school rather than student performance, despite the fact that they are not particularly useful for the former purpose since, among other reasons, they do not account for where students start out upon entry to the school. I personally don’t care whether Success Academy gets good or bad press. I do, however, believe that how one gauges effectiveness, test-based or otherwise, is important, even if one reaches the same conclusion using different measures.
It’s not possible, using available data, to check the testing performance for Success Academies’ students upon initial entry (which is kindergarten for all the schools currently operating). This is because the state begins testing in third grade. We can, however, try to get a rough idea (or, more accurately, illustrate the futility of trying) by taking a quick look at the latest (2014) math results for third graders attending Success Academies (at least those campuses for which testing data are available). The distributions are presented in the table below.
As you can see, the Success Academy third grade math proficiency rates range from 85 to 100 percent, with most of them falling above 90 percent. Moreover, huge proportions of these third graders — between about 50 percent and almost 90 percent — score above the cutoff for level 4 (“advanced”). Citywide, the third grade rates in 2014 were 14.8 percent (level 4) and 38.6 percent (level 3 + level 4). The comparisons are similar in previous years.
Now, it may be the case that Success Academy students enter kindergarten performing at very low levels, and that the school performs remarkable feats for its students before they reach third grade, bringing virtually all of them up to proficiency level, and a very large group to the “advanced” level 4, over that 3-4 year time period. Or, perhaps, Shanker Blog » The Thrill Of Success, The Agony Of Measurement: