Why the UK Department for Education is wrong on promoting OECD Pisa
Why PISA ranks are founded on a methodological thought disorder
Dr Hugh Morrison
(The Queen’s University of Belfast [retired])
(drhmorrison@gmail.com)
When psychometricians claimed to be able to measure, they used the term ‘measurement’ not just for political reasons but also for commercial ones. … Those who support scientific research economically, socially and politically have a manifest interest in knowing that the scientists they support work to advance science, not subvert it. And those whose lives are affected by the application of what are claimed to be ‘scientific findings’ also have an interest in knowing that these ‘findings’ have been seriously investigated and are supported by evidence. (Michell, 2000, p. 660)
This essay is a response to the claim by the Department of Education that: “The OECD is at the forefront of the academic debate regarding item response theory [and] the OECD is using what is acknowledged as the best available methodology [for international comparison studies].”
Item Response Theory plays a pivotal role in the methodology of the PISA international league table. This essay refutes the claim that item response theory is a settled, well-reasoned approach to educational measurement. It may well be settled amongst quantitative psychologists, but I doubt if there is a natural scientist on the planet who would accept that one can measure mental attributes in a manner which is independent of the measuring instrument (a central claim of item response theory). It will be argued below that psychology’s approach to the twin notions of “quantity” and “measurement” has been controversial (and entirely erroneous) since its earliest days. It will be claimed that the item response methodolology, in effect, misuses the two fundamental concepts of quantity and measurement by re-defining them for its own purposes. In fact, the case will be made that PISA ranks are founded on a “methodological thought disorder” (Michell, 1997).
Given the concerns of such a distinguished statistician as Professor David Spiegelhalter, the Department of Education’s continued endorsement of PISA is difficult to understand. This essay extends the critique of PISA and item response theory beyond the concerns of Spiegelhalter to the very data from which the statistics are generated. Frederick Lord (1980, p. 227-228), the father of modern psychological measurement, warned psychologists that when applied to the individual test-taker, item response theory produces “absurd” and “paradoxical” results. Given that Lord is one of the architects of item response theory, it is surprising that this admission provoked little or no debate among quantitative psychologists. Are politicians and the general public aware that item response theory breaks down when applied to the individual?
In order to protect the item response model from damaging criticism, Lord proposed what physicists call a “hidden variables” ensemble model when interpreting the role probability plays in item response theory. As a consequence item response models are deterministic and draw on Newtonian measurement principles. “Ability” is construed as a measurement-independent “state” of the individual which is the source of the responses made to test items (Borsboom, Mellenbergh, & van Heerden, 2003). Furthermore, item response theory is incapable of taking account of the fact that the psychologistparticipates in what he or she observe. Richardson (1999) writes: “[W]e find that the IQ-testing movement is not merely describing properties of people: rather, the IQ test has largely created them” (p. 40). The participative nature Why the UK Department for Education is wrong on promoting OECD Pisa | Pace N.Ireland Education Weblog: