EDDRA


Education Disinformation Detection and Reporting Agency


-- a Gerald Bracey Report on the Condition of Education

Index of
EDDRA
Articles


Related
Article:
The Unseen
Progress of
American
Schools

SIMPSON'S PARADOX MASKS THE GOOD NEWS ABOUT AMERICAN SCHOOLS AND UNDOES STEPHAN THERNSTROM, WINTHROP PROFESSOR OF HISTORY AT HARVARD UNIVERSITY

By golly, I was right. Back in 1991 I published an article, "Why Can't They Be Like We Were?" debunking misguided nostalgia about some non-existent Golden Age of American public education from which the situation had deteriorated. I presented data showing that schools were doing at least as well as ever. The editors who published the analysis asked that it be an annual publication and the more I looked into it, the more I became convinced that the schools were doing better than ever.

The gloom merchants first tried to ignore the data I presented. When that tack failed, they argued that it's not good enough to be as good as ever--our schools were turning out kids who could not compete in the international marketplace. With the economy roaring for almost a decade and the then-ballyhooed Asian economies in the tank, this argument is now seldom heard.

Then the critics moved on to admit that the schools were "a little better" but to argue that the increased demands from business and industry meant that they had to be a lot better. They proferred no evidence to prove this contention.

Even today, some critics are capable of actually misreading data in order to demean the schools. In the fall, 1998 issue of The Concord Review, Stephan Thernstrom, Winthrop professor of history at Harvard, and co-author, with his wife, Abigail, of America in Black and White, uttered this lament: "It is hard not to feel discouraged about the state of our elementary and secondary schools these days…The performance of our very best students has been particularly depressing. In every subject, the scores required to make it into the top decile and top 5 percent [on the National Assessment of Educational Progress] are lower than they were in the early 1970's. We have displayed a commendable concern for improving the educational performance of students from disadvantaged backgrounds in recent years, but at the cost of dumbing down the curriculum and making it less challenging for the academically gifted."

The good professor should cheer up: he is wrong on all almost counts.

For almost two decades (longer in reading) the National Assessment of Educational Progress (NAEP) has tested knowledge and skills in a variety of topics at periodical international using nationally representative samples of students. It is not linked to any specific curriculum so no one can really prepare for it. It is not a high stakes test so no one even tries.

There are 8 NAEP then-and-now comparisons that bear on Thernstrom's allegation, the 90th and 95th percentiles in reading, writing, science and mathematics, only five are "down". The other three are "up." The quotation marks signify that virtually any objective observer would say that there has been no change. The differences between the early scores and the most recent are tiny, often less than one point on a 500-point scale.

But percentiles for the nation mask changes for different ethnicities. To look at scores in the 1970's and the most recent scores, from 1996, is to look at two different Americas. Minorities now make up a much larger proportion of the population than 25 years ago.

Why is this important? Because the scores of blacks and Hispanics are lower than those of whites (Asians make up such a small proportion of the country that only in the most recent NAEP assessments has the sample of Asians been large enough to report separately). To add more and more scores from groups that score low is to depress the overall average even if all groups are scoring higher.

An imaginary example will make the situation concrete and clear:

Time 1 Time 2

  1. 510
  1. 510
  1. 510
  1. 510
  1. 510
  1. 510
  1. 510
  1. 430
  1. 430
  1. 430
  1. 486

Consider that the 500's at Time 1 are the scores of white students and that the 400's are the scores of minority students. Together, the average is 490. Now imagine that the 510's at Time 2 are the scores for white students and the 430's are the scores for minority students. Whites and minorities have both gained and minorities' scores have gained substantially more than whites. Yet the overall average is lower, 486. That's because minorities constituted only 10% of the total at Time 1 and 30% of the total at Time 2. Increasing the proportion of improving-but-still-low scores depresses the average.

This outcome--falling averages as a result of proportional changes in the Subgroups is known in statistics as Simpson's Paradox and it is Simpson's Paradox that really undoes professor Thernstrom.

The NAEP trends for reading, math, and science are shown in the table for whites, blacks and Hispanics. Results are presented for the average score (50th percentile), the low scores (5th percentile) and high scores (95th percentile).

If professor Thernstrom were right, gains should be seen for the low scoring students and they do appear--for all subjects, for all ages, and for all ethnicities. Scores are up across the board with the lone exception of the 5th percentile reading score for 13-year-old Hispanics which is the same at 174.

Assuming that disadvantaged students constitute a disproportionate share of the low scorers, professor Thernstrom's contention for disadvantaged students holds. The low scorers have improved significantly. What about at the top?

There is no evidence whatsoever for Thernstrom's allegation of "dumbing down". Of the 9 comparisons for 17-year-olds (three subjects by three ethnicities), the math 95th percentile for white students is the same and the reading score fell all of one point (on a 500 point scale). The other seven are up. Given that it is harder to raise the ceiling than the floor, these results border on the remarkable. Given the decline in the family and community institutions outside of school, these results are remarkable.

I invite the reader to pore over these tables. I invite the critics to shut up.

READING
9-YEAR-OLDS
PERCENTILEWHITEBLACKHISPANIC
RANK 197119961971199619711996
5th146154107123120125
50th215222171190184197
95th277279232255240257
13-YEAR-OLDS
5th205205163176174174
50th262269223237234242
95th313324277292289300
17-YEAR-OLDS
5th219225165200184198
50th293296239265252264
95th358358310330320329

MATH

9-YEAR-OLDS
PERCENTILEWHITEBLACKHISPANIC
RANK 197119961971199619711996
5th166183134158144164
50th225238193213204215
95th278289249260260266
13-YEAR-OLDS
5th211233170204180204
50th272281229253237256
95th330328288300296305
17-YEAR-OLDS
5th252266217241224243
50th307312268286275293
95th358358321333332341

SCIENCE

9-YEAR-OLDS
PERCENTILEWHITEBLACKHISPANIC
RANK 197119961971199619711996
5th163173107139125142
50th230240174202191208
95th295303244264261268
13-YEAR-OLDS
5th191209144168147174
50th256267207225213231
95th321321275285282283
17-YEAR-OLDS
5th231237172192194196
50th298309240259262270
95th365371310327331340


© 1999 Gerald Bracey
Last updated June 27, 1999
Web Services by