The 7 steps to making a standardized test
Inside the New Hampshire not-for-profit where assessments like the MCAS are born.
Who designs the standardized tests that spread a blanket of silence and stress across schools in Massachusetts and other states each spring?
A) A phalanx of drill sergeants who would ban recess if they could
B) A coven of witches who feed on the sweat and tears of small children
C) Dozens of PhDs and former teachers who spend their days in cubicles in Dover, New Hampshire.
The answer is C. The Massachusetts tests for grades 3 through 8 — used to rate schools, partially evaluate teachers, and help decide if students move to the next grade — are created by the employees of the not-for-profit Measured Progress, an enthusiastic bunch who care deeply about improving education through testing. Measured Progress has worked with the Massachusetts Department of Elementary and Secondary Education to make and score the MCAS for all but four of the 19 times it has been given. The company also works on large-scale assessments for 16 other states and is one of two companies in the running to design the MCAS 2.0.
Here’s how they do their work:
The list of standards — the skills and knowledge states decide their students should have at each grade level — is the test maker’s bible. Examples from the Massachusetts standards, which incorporate the Common Core, include: “Determine a theme of a story, drama, or poem from details in the text; summarize the text” or “Use the four operations with whole numbers to solve problems.” The state and Measured Progress agree on how many questions to devote to each standard and the ratio of multiple-choice questions to open-ended ones.
Next, it’s time to start writing “items,” as they’re called in the industry. Each year, MCAS tests included some recycled items, but a large number will have been field-tested in previous years and used for scoring for the first time. A good item is clear and concise and will address the standard it’s supposed to be assessing. Multiple-choice questions can have only one right answer. Reading passages need to be drawn from actual fiction or nonfiction. In math, word problems need to be realistic. “You can’t have someone going on a hike where he’s hiking 20 miles per hour,” says Raymond Reese, a math content development specialist.
After an internal review, the state assembles six to 12 educators from across Massachusetts for each grade (and sometimes subject area) that’s being tested to go through items with them, one by one. If teachers don’t like an item, they can either suggest revisions or vote to reject it (fewer than 10 percent are discarded at this stage). Meanwhile, separate committees of state educators vet the items for bias. “They’re really looking for things that might affect performance on items other than the student’s ability,” says Stuart Kahl, Measured Progress’s founding principal. A writing prompt to The 7 steps to making a standardized test - The Boston Globe:
