TIMSS, RHYMES WITH DIMS, AS IN WITTED

TIMSS, RHYMES WITH DIMS, AS IN WITTED

(This Essay to Appear in the May, Phi Delta Kappan)

Gerald W. Bracey

As in the movie "The Official Story", the official TIMSS story is not the real story. Indeed, for TIMSS, the official story is not even the story contained in the official report. The simplistic and misleading statements of TIMSS study director, William Schmidt, NCES Commissioner, Pat Forgione, and Secretary of Education, Richard Riley, seriously distort what the TIMSS official report actually says. I am happy that William Jefferson Clinton made it as President because his TIMSS-related comments on the impact of poverty on learning reveal a logic deficit that would prevent him from becoming a sound educational researcher. Clinton said poverty couldn't be the cause of our low standing because our 4th graders did so well. Evidently he is not familiar with cumulative effects.

The U. S. Department of Education states that the math and science literacy tests measure "the knowledge of mathematics and science needed to function effectively in society as adults." For this, the Department earns my "Unmitigated And Overweening Horsehockey Award." Readers, call the Department and ask them first to define "function effectively in society" and then to tell you how they know that the TIMSS math and science tests measure this ability. The published sample items strike me as quite reasonable things to teach students, but to then say this stuff is needed "to function effectively in society" is utter nonsense (it recalls the similar nonsense of the 70's over minimum competency testing and the debate then over what skills are "essential" to function in society).

I know, I know, skeptics will say that I've gotten locked into this position as the nation's leading "defender" of public schools and am now trying to put a good spin on what everyone else agrees are "devastating results" (Schmidt's and Forgione's phrase) and "unacceptable" (Clinton). Well, then, check out the facts and decide for yourself (if only reporters had). My take is that the physics and advanced mathematics scores are beyond redemption; no statistical manipulation can make them wholly credible. They are condemned to lie in a muck of uncertainty. If the math and science literacy scores were accurately calculated factoring in appropriate variables, the United States would be about average. Not a cause for celebration, surely, but not the disaster so far painted, either.

NCES' Forgione said that the graduation gap between the United States and other nations had been closed meaning that 12th graders in various countries are now comparable. This is not true. The official TIMSS report says "there remains considerable variation among countries in completion rates" (p. 17). What the TIMSS table on page 22 actually shows is that there remains considerable variation among countries in secondary school enrollments. When one considers that for most nations this is the proportion of students aged 12-17 who are enrolled, the variation is surely much larger for graduation rates.

Here are a few of the other statistical knee-slappers: The TIMSS "kids" in Iceland are the same age as American college seniors. That's right folks, the Icelandic students who were compared to our 12th graders averaged 21.2 years. Those in Germany, Norway, Italy, Austria, Sweden and Switzerland were merely closer in age to our college sophomores. For this we paid $31 million? (Think that spending $31 million, while other nations ponied up a pittance, has anything to do with our making such a big deal about the study?)

Norwegian kids topped the rest of the world in physics. But the question is: Why didn't they do better? They'd been studying physics for three years. Ditto the second ranked Swedish. The performance of these two nations with three years of physics study puts to rest a notion likely held in many quarters: Had any of the Asian nations, whose students had headed the list at the 8th-grade level, shown up for the 12th-grade testing, American students would have looked even worse. Given the substantial distance between Norway, Sweden and the rest of the world, it is unlikely that any other nation would have come close, unless they, too, had multi-year physics programs.

Nobody saw fit, apparently, to mention at the official-story press conference that the American TIMSS cadre contained our students taking pre-calculus. Is this important? Well, those students scored fully 100 points lower than those who had actually taken calculus, no doubt because 25% of the questions presumed that the testtaker had absorbed a calculus course. American students with calculus actually under their belts scored one item less correct than the international average of all 21 nations.

And how's this for a statistical fluke: On the island nation of Cyprus, students ranked 18th of 26 nations in TIMSS grade 4 math, 37th of 41 nations in grade 8, and 20th of 21 nations in grade 12 in math literacy. The best Cypriot students did not score as high as some countries' average students. The average math literacy score for the upper 25% of students in the 21 nations combined was 585. America's best 25% scored 559 and the Cypriots a mere 501. Yet the Cypriot students outscored us in "advanced mathematics" at the 12th grade level. Is something screwy here or what? My guess is that Cyprus has a dual education system that coddles an elite and largely ignores the rest. Either that or they cheated.

In fact, the official TIMSS report says that students in Cypriot lycea can pursue five courses of study, one of which is math and science. Only students in the math-science program were tested. Cyprus also has technical schools where students pursue vocational studies or math and science and from the latter head to universities. The math and science students in technical schools were tested, the vocational students were not. These two groups still counted for a smaller proportion of the student body than the Americans who were tested.

The U.S.-Cyprus difference in the structure of schools is just one of many cultural-difference problems afflicting these studies that cannot be solved by taking ever more precise samples. Because of these, international comparisons on test scores can never be very meaningful (what can be meaningful is to look at what other countries do in their schools and decide if it makes any culturally-relevant sense for us to do it, too).

A more noteworthy example: fully 55% of American students tested for math-science literacy reported themselves working at a job more than 15 hours a week. Research studies have found, for American students, jobs and school work enjoy a curvilinear relationship. Kids who work up to 15 hours a week do better in school than those who don't work and those who work longer hours. The TIMSS results corroborate this. American students who reported that they worked 10 hours a week actually scored above the international average. Those who worked more than 25 hours a week were 60 points lower (in almost all other countries very few students work at all). [NOTE: The Bureau of Labor Statistics has provided figures indicating that the proportion of seniors working more than 15 hours a week is only about 20%. At this writing I have not determined whether the TIMSS figure is an error or whether the TIMSS American sample is unrepresentative of seniors]

Who knows what other cultural and social factors affect these data? Such factors often come to light only by accident. For instance, in the First International Adult Literacy Survey (FIALS), the United States did pretty well overall. Only Sweden had a larger proportion performing at the highest two levels. But our immigrants in the study had severe reading limitations, no doubt because of having first learned a language other than English. On the other hand, FIALS revealed that Canada has a higher proportion of immigrants who read at the highest two levels (28%), than nativeborns at those levels (20%). Turns out that Canada has an aggressive policy of recruiting skilled, highly educated immigrants to come live in the land of the maple leaf. Thus these scores that at first glance appear to reflect school quality are in fact strongly influenced by immigration policy.

Finally, the official story about TIMSS has not reported that most countries did not meet the TIMSS criteria for participation. To have their results counted, countries were supposed to have a combined student-school participation rate of at least 75% and a sample of students representative of at least 90% of all students eligible to participate. Only 9 countries met the first criterion and only 14 met the second, and only four met both. Some countries excluded 22% of eligible students (Cyprus), or 30% (Italy) or even 43% (Russian Federation). Reporting the results for all 21 countries is thus comparing the incomparable, something that is strictly forbidden under normal rules of research.

I say again as I have said so often in the past, that there are serious problems in American schools and that our poor rural and urban schools need a Marshall Plan. But we should attend to problems that actually exist. Those that were purportedly revealed in the official story of TIMSS, do not. Forgione admits only that TIMSS has "rough edges." I say rotten at the core.

The official TIMSS story is an exercise in political rhetoric and comes very close to being a hoax perpetrated on the nation. The only thing it has done is give a great deal of aid and comfort to the enemies of public education. "Replace Public Schools Rather Than Fix Them" was the headline on Cal Thomas' syndicated column. For this the Department of Education and TIMSS staff have a great deal to answer for.

Gerald W. Bracey is an independent researcher and writer living in Alexandria, Virginia. He is author of the annual "Bracey Report on the Condition of Public Education" and Setting the Record Straight: Responses to Misconceptions About Public Education in the United States, and of Put to the Test: An Educator's and Consumer's Guide to Standardized Testing.