Proficiency Rates And Achievement Gaps
The change in New York State tests, as well as their results, has inevitably resulted in a lot of discussion of how achievement gaps have changed over the past decade or so (and what they look like using the new tests). In many cases, the gaps, and trends in the gaps, are being presented in terms of proficiency rates.
I’d like to make one quick point, which is applicable both in New York and beyond: In general, it is not a good idea to present average student performance trends in terms of proficiency rates, rather than average scores, but it is an even worse idea to use proficiency rates to measure changes in achievement gaps.
Put simply, proficiency rates have a legitimate role to play in summarizing testing data, but the rates are very sensitive to the selection of cut score, and they provide a very limited, often distorted portrayal of student performance, particularly when viewed over time. There are many ways to illustrate this distortion, but among the more vivid is the fact, which we’ve shown in previous posts, that average scores and proficiency rates often move in different directions. In other words, at the school-level, it is frequently the case that the performance of the typical student — i.e., the average score — increases while the proficiency rate decreases, or vice-versa.
Unfortunately, the situation is even worse when looking achievement gaps. To illustrate this in a simple manner, let’s take a very quick look at NAEP data (4th grade math), broken down by state, between 2009 and 2011.
The table below summarizes “score versus rate trends” for three measures. In the first column are the trends in rates and gaps for students eligible for free/reduced-price lunch (FRL), a rough proxy for poverty. The second column is students who are not eligible. And, finally, the third column is the trend in score/rate gaps between these two groups. The numbers in the table represent the number of states.
So, for example, among FRL-eligible students, there were 43 states in which the average NAEP score and the proficiency rate moved in the same direction between 2009 and 2011 – that is, either both increased or both
