Education Disinformation Detection and Reporting Agency
-- a Gerald Bracey Report on the Condition of Education
SIMPSON'S PARADOX MASKS THE GOOD NEWS ABOUT AMERICAN SCHOOLS AND UNDOES STEPHAN THERNSTROM, WINTHROP PROFESSOR OF HISTORY AT HARVARD UNIVERSITY
By golly, I was right. Back in 1991 I published an article, "Why Can't They Be Like We Were?" debunking misguided nostalgia about some non-existent Golden Age of American public education from which the situation had deteriorated. I presented data showing that schools were doing at least as well as ever. The editors who published the analysis asked that it be an annual publication and the more I looked into it, the more I became convinced that the schools were doing better than ever.
The gloom merchants first tried to ignore the data I presented. When that tack failed, they argued that it's not good enough to be as good as ever--our schools were turning out kids who could not compete in the international marketplace. With the economy roaring for almost a decade and the then-ballyhooed Asian economies in the tank, this argument is now seldom heard.
Then the critics moved on to admit that the schools were "a little better" but to argue that the increased demands from business and industry meant that they had to be a lot better. They proferred no evidence to prove this contention.
Even today, some critics are capable of actually misreading data in order to demean the schools. In the fall, 1998 issue of The Concord Review, Stephan Thernstrom, Winthrop professor of history at Harvard, and co-author, with his wife, Abigail, of America in Black and White, uttered this lament: "It is hard not to feel discouraged about the state of our elementary and secondary schools these days…The performance of our very best students has been particularly depressing. In every subject, the scores required to make it into the top decile and top 5 percent [on the National Assessment of Educational Progress] are lower than they were in the early 1970's. We have displayed a commendable concern for improving the educational performance of students from disadvantaged backgrounds in recent years, but at the cost of dumbing down the curriculum and making it less challenging for the academically gifted."
The good professor should cheer up: he is wrong on all almost counts.
For almost two decades (longer in reading) the National Assessment of Educational Progress (NAEP) has tested knowledge and skills in a variety of topics at periodical international using nationally representative samples of students. It is not linked to any specific curriculum so no one can really prepare for it. It is not a high stakes test so no one even tries.
There are 8 NAEP then-and-now comparisons that bear on Thernstrom's allegation, the 90th and 95th percentiles in reading, writing, science and mathematics, only five are "down". The other three are "up." The quotation marks signify that virtually any objective observer would say that there has been no change. The differences between the early scores and the most recent are tiny, often less than one point on a 500-point scale.
But percentiles for the nation mask changes for different ethnicities. To look at scores in the 1970's and the most recent scores, from 1996, is to look at two different Americas. Minorities now make up a much larger proportion of the population than 25 years ago.
Why is this important? Because the scores of blacks and Hispanics are lower than those of whites (Asians make up such a small proportion of the country that only in the most recent NAEP assessments has the sample of Asians been large enough to report separately). To add more and more scores from groups that score low is to depress the overall average even if all groups are scoring higher.
An imaginary example will make the situation concrete and clear:
Time 1 Time 2
Consider that the 500's at Time 1 are the scores of white students and that the 400's are the scores of minority students. Together, the average is 490. Now imagine that the 510's at Time 2 are the scores for white students and the 430's are the scores for minority students. Whites and minorities have both gained and minorities' scores have gained substantially more than whites. Yet the overall average is lower, 486. That's because minorities constituted only 10% of the total at Time 1 and 30% of the total at Time 2. Increasing the proportion of improving-but-still-low scores depresses the average.
This outcome--falling averages as a result of proportional changes in the Subgroups is known in statistics as Simpson's Paradox and it is Simpson's Paradox that really undoes professor Thernstrom.
The NAEP trends for reading, math, and science are shown in the table for whites, blacks and Hispanics. Results are presented for the average score (50th percentile), the low scores (5th percentile) and high scores (95th percentile).
If professor Thernstrom were right, gains should be seen for the low scoring students and they do appear--for all subjects, for all ages, and for all ethnicities. Scores are up across the board with the lone exception of the 5th percentile reading score for 13-year-old Hispanics which is the same at 174.
Assuming that disadvantaged students constitute a disproportionate share of the low scorers, professor Thernstrom's contention for disadvantaged students holds. The low scorers have improved significantly. What about at the top?
There is no evidence whatsoever for Thernstrom's allegation of "dumbing down". Of the 9 comparisons for 17-year-olds (three subjects by three ethnicities), the math 95th percentile for white students is the same and the reading score fell all of one point (on a 500 point scale). The other seven are up. Given that it is harder to raise the ceiling than the floor, these results border on the remarkable. Given the decline in the family and community institutions outside of school, these results are remarkable.
I invite the reader to pore over these tables. I invite the critics to shut up.
© 1999 Gerald Bracey|
Last updated June 27, 1999
Web Services by |