Computerized Grading: Purloining the Analysis, the Most Fundamental Exposition of Humanity
Les Perelman, a researcher at the Massachusetts Institute of Technology, has been doing some interesting experiments to test the capacity of computerized grading systems to accurately judge the quality of written work. In a recent interview with the Canadian Broadcasting Company's Carol Off, he shared what he has learned.
As our government agencies and various reform efforts seek to shift high stakes testing away from multiple choice questions, there is growing interest in computer programs that can read and score student essays. But questions persist, given the limitations of the algorithms these programs use. So Mr. Perelman has done an experiment. He created something he calls the Basic Automatic BS Essay Language Generator, BABEL for short. During his interview with Carol Off, Perelman fed his machine a topic she suggested, "Fair Elections Act."
Here is what the BABEL machine provided in response:
Fun fair for adherents and presumably will never be altruistic in the extent to which we purloin the analysis. Fair is the most fundamental postulate of humankind. Whiner to act in the study of semiotics in addition to the search for reality. Act is intrepidly and clandestinely axiomatic by most of the scenarios. As I have learned in my semiotics class, act is the most fundamental exposition of humanity.
Mr. Perelman then submits this essay for grading. The result, a score of 5.4 out of 6, placing this essay in the 90th percentile.
Perelman explains his purpose:
I did this as an experiment to show that what these computers are grading does not have anything to do with human communication. If you think about writing or any kind of human communication as the transfer of thoughts from one mind to another mind, then if the machine takes something that anyone would say is complete incoherent nonsense, and scores it highly, and we know that it's not, then we know that it's not grading human communication.
Two years ago there was some excitement about computer-scored essays, when a demonstration showed that computers could yield results that aligned reasonably well with the scores given by Computerized Grading: Purloining the Analysis, the Most Fundamental Exposition of Humanity - Living in Dialogue - Education Week Teacher:
