Introduction. 

In June, 2002, the New York Times reported that New York Regents Exams altered literature passages on its tests according to "sensitivity guidelines."[1]  Some of the changes were ludicrous, some merely stupid, and some would lead testtakers to the wrong answer.  The state promised it would never do it again, but as the Times' Michael Winerip found, the censorship and bowdlerization is still going on.  How the Department got caught and other aspects of the affair were detailed in "The Twelfth Bracey Report on the Condition of Public Education," Phi Delta Kappan, October, 2002.  More important for this essay is a sentence in Winerip's article that draws attention to another problem with tests--passing or failing can turn on a single item.

---------------------

A statement buried in the 15^th paragraph of Michael Winerip's "How New York Exams Rewrite Literature (A Sequel)[2]" deserves to be brought center stage.  Winerip wrote "In the world of make-or-break exams, one question scored incorrectly can make all the difference in a student's future."  To people unfamiliar with the technology of testing, Winerip's words probably look like hyperbole.  They are not.

We who have worked in the field of testing have known this for some time, but the capriciousness of high-stakes testing entered public discourse only because of Martin Swaden, a Minnesota parent. 

Minnesota requires students to pass a test in order to obtain a high school diploma.  Swaden's daughter failed.  Swaden reasoned that the best way to help her next time round was to look at the test and his daughter's answer sheet to see what kinds of questions were giving her trouble.  The state denied his repeated requests to see them, yielding only when he threatened to sue (he's a lawyer, giving the threat some credibility). 

Matching his daughter's answer sheet against the key containing the officially right answers, Swaden discovered that the testing company, NCS Pearson, mis-scored six questions, enough to put his daughter over the top.  She was hardly alone: 47,000 students got lower scores than they deserved, 8000 students failed when they had should have passed and 525 seniors who had actually passed were denied the right to walk across the stage and pick up their diplomas at graduation ceremonies. 

These 8000 falsely failed students did not suffer mere temporary humiliation.  Some gave up jobs to attend summer school they didn't need.  Others forked out for tutors.  They suffered slights from classmates.  Some changed plans for the future and some dropped out of school.  They also filed a class-action suit against NCS.

NCS claimed that the error was a one-time affair and couldn't happen again.  It found a scapegoat and fired him.  It hired the current or former state directors of testing from Virginia, Washington and Texas to testify in its behalf (parents, do not expect state officials to take the lead in ferreting out errors).  The testing director from Virginia said that such errors as NCS made were likely.

The judge, though, rejected NCS' one-time-mistake arguments, concluded that NCS had a long history of shoddy quality control and had failed to hire enough employees to cure the problem.  He permitted the plaintiffs to sue for punitive damages. NCS settled out of court.  Some students received as much as $16,000, but most got less than $2,000.  The total award amounted to over $8,000,000.

Test companies fail the quality control test far too often.  CTB-McGraw Hill errors forced thousands of New York City students to attend summer school when they didn't need to.  Nevada officials caught a similar error in 2002.  Also in 2002, Georgia jettisoned the entire state's results on Harcourt's SAT-9--the company mis-scored all 340,000 tests.  CTB-McGraw Hill's sloppiness so offended Arizona that the state sought another contractor (but given the limited number of testing companies with the capacity to handle large state contracts, had to accept another with similar risks for errors.  Basically there are only four companies: Harcourt Educational Measurement, CTB-McGraw Hill, NCS, and Riverside.  Educational Testing Service, once a major player in the K-12 testing arena has decided to become on again).

All of these errors were made before the Elementary and Secondary Education Act of 2001, the so-called No Child Left Behind Act, had any impact.  That law, requiring all children be tested in grades three through eight every year in reading and math (with science to be added in two years) will vastly increase the amount of testing in this country (and some think that it is no accident because the Bushes and the McGraws who founded CTB-McGraw Hill have been vacation-together families for 75 years).  The same, short-staffed, error-prone contractors will develop and score these tests.

We can thus safely predict the test companies will make many more errors.  We cannot be confident that anyone will detect and report them.  Some kids will falsely flunk and some institutions will falsely bear the label--and the consequences--of being a "failing school."  The states and the testing companies, fearing to look bad more than they fear doing wrong by children, will have no incentive to look for the mistakes.

Indeed, one wonders how many other errors already lie undetected in computer files because parents from other states have not pressed to see how the tests match their children's answer sheets.  They should.  No one else will.  Swaden won without litigation in Minnesota.  A Florida parent had to sue to see her daughter's answer and.  Florida Governor Jeb Bush fought the suit claiming that letting parents see their children's answer sheets would cost the state too much money.  He lost.

Most parents likely don't realize how narrow and arbitrary the difference between pass and fail is. When Virginia changed its test equating procedure for one test, the new procedure required one less item correct to pass than the old procedure called for (a peculiar outcome the state has not accounted for).  That one-item difference changed 5,625 flunks into passes (the children were informed months later that they were no longer failures).  Virginia also lowered the passing score on several of its tests in 2002, but did not apply the new standards retroactively.  As a consequence, some 50,000 kids who flunked in 2000 and 2001 would have passed had they been lucky enough to take the test in 2002.

The testing industry is perhaps the largest unregulated business in the nation. There is no FDA of testing to stamp the tests for quality or make sure that the tests won't poison anyone.  Legislators, governors, and boards demand accountability from teachers and school administrators, even though they only have control over their "products" for a few hours a day, half the year.  Test companies have total control over their products.  One wonders when they will be held accountable.  Minnesota and Florida have made starts, but only after determined parents forced the issue. Parents in all other states should take heed.