At the recent education "summit" IBM CEO and summit convener, Lou Gerstner, had this to say about the Third International Mathematics and Science Study (TIMSS):

The press, which appears to have no critical thinking powers when it comes to the "official story" in education, (another failure of American education?) dutifully and slavishly reported this crap. Let's take a look.

In brief, the situation is not so great at fourth grade, no so bad at 8^th grade and certainly not so grim at 12^th grade.

American fourth graders finished 3^rd among 26 nations in science, and 10^th among 26 in math. Not bad.

American 8^th graders did rank 28^th among 41 nations in math, but their score was only two percentage points off the international average: the countries are tightly bunched and small differences in scores make huge differences in ranks.

We were 16^th of 41 in science, above average, but the score difference was the same as in math. American 8^th graders got 58% correct and the international average was 56%. The nations were even more tightly bunched in science with some 24 countries being within plus or minus 6% percentage points of the American score. I can't imagine that such small differences on a paper-and-pencil test have any practical ramifications.

I think there really is slippage from 4^th grade to 8^th in math. We treat the middle school years as the culmination of elementary school and do a lot of review. Other nations are more likely to treat it as the beginning of secondary school and infuse math, for instance, with lots of algebra and geometry.

We need to review math in the middle years because we of how we approach math in the elementary grades. TIMSS found that American textbooks are about three times as thick as those in other nations. American teachers try to do it all with the consequence that coverage is often brief and shallow and has to be done over the next year.

This really is a situation where less would be more: fewer topics, more time on each.

My hunch is that the change in science is a matter of other countries not emphasizing science as much in the elementary grades and then catching up. My reasoning here comes from the fact that French students did very well in math and they did very well in another international study of reading. But they did poorly in science. I can't believe there's something about French brains that can't accommodate science so I conclude that they're probably not teaching a lot of it before high school.

There really is no "twelfth grade" study. There is a "Final Year" study. The TIMSS people called it that because after 8^th grades, the education systems in the various countries are really no longer comparable. In other nations, curricula differentiate: some kids head for college prep, some for technical schools, some for vocational schools (in some countries, these are over half of all students). Some get concentrated programs in math and science or science and technology. Unfortunately the way the U.S. Department of Education released the results, everyone--and I mean everyone--in the meeting got the impression that our 12^th graders went up against their 12^th graders and got stomped and that our best 12^th graders went up against their best 12^th graders and got stomped (see the series of media quotes that open my September, 1998 article on the subject in Phi Delta Kappan). That is not what happened at all but it is a cliché that has entered the popular culture pretty much as Gerstner told it.

The Final Year data are really meaningless for a variety of reasons:

1. TIMSS established criteria for permissible exclusion rates and participation rates. If you let the countries choose who gets tested, most countries will choose only those students who will make them look good. Only 5 of the 24 nations met these criteria for valid data. Russia tested only in Russian speaking schools. Italy lopped off whole provinces, etc. Why was such invalid data even released?, you ask. Good question. No answers except maybe that we spent $51 million American taxpayer dollars on the study.
2. Because of differentiated programs and their length, students in other countries were on average more than a year older than American students. Many were 19, some were 20 and one country (Iceland) tested kids the same age as American college seniors.
3. Who knows what cultural factors might be producing differences. One that made a huge difference was working. In most countries, you're either a worker or a student, not both. But 55% of American seniors said they worked more than 21 hours a week. Research clearly shows in this country that kids who work up to 20 hours a week do better in school than those who don't work or those who work more than 20 hours.

   This shows up in TIMSS: The 28% of American students who said they worked 21-35 hours a week are below average. The 27% who said they worked more than 35 hours a week are off the chart. Those who work less scored slightly above the international average.

4. In the advanced math test, we tested students in calculus and pre-calculus courses. But 23% of the items in that test presume that the student has taken calculus. American pre-calculus students scored fully 100 points lower than those with calculus who were at the international average. When I asked the TIMSS people why they tested pre-calculus kids they said that the just wanted to see how they'd do. That's fine, but don't lump these students in with ones that have actually studied what the test tests.
5. Swedish and Norwegian kids topped the world in physics and why not? They'd had three years of it. It would be legitimate to ask if American high schools should offer multi-year programs in science. It is not legitimate to compare students with three years of study against students with one.
6. I don't know if other countries have a senior slump, but we tested our 12^th graders in May. We don't generally test kids in 12^th grade except with consequential tests like the SAT because they blow the tests off. One can only imagine with what diligence American kids approached the TIMSS tests. Motivation, in and of itself can produce enormous differences in test performance of the same kids (one day I'll tell you a story about this from my years as Director of Research, Evaluation and Testing for the State of Virginia).
7. The enrollment rates in the final year are probably vastly different in different countries. A few countries have now surpassed the US in high school graduation rates, but most have not. I used the word "probably" earlier because we don't have good figures. Then-commissioner of education statistics, Pat Forgione, said the countries were "roughly comparable", but there was over a 20% variation and what the hell does "roughly comparable" mean in a supposedly scientific study? Those roughly comparable figures were for ages 12-17, but the kids tested were mostly 18 and over. In most countries, it seems likely that low-achieving kids have left and taken their low test scores with them.
8. There are peculiar things in the data: Cyprus, for instance, was 19^th of 26 in math at the 4^th grade level, 37^th of 41 at grade 8 and 20^th of 21 in the Final Year study. Yet, on the advanced math test, they were 6^th and within the advanced math arena, they were #1 in the world in calculus. How did a country that got really did get progressively worse at each level do so well in advanced math? I don't know. Their kids were not older than American students, although they did test a far smaller proportion. And they tested only students who had studied virtually nothing but math and science for the last three years, studys in the final year of the Lyceum's math and science program. I also haven't ruled out that they cheated.

Finally, consider reading. When the most recent international comparison in reading came out, the ideologues at the Department of Education (Lamar Alexander, Diane Ravitch) tried quite successfully to ignore it. Unlike an earlier math and science study which they had touted with a large press conference, the reading study received absolutely no attention except from the paper in Binghamton, NY (the person who directed the US part of the study taught at SUNY Binghamton). No mention of it has ever been made in the Washington Post or New York Times.

Even the industry newspaper, Education Week, did not find out about the study until two months after publication--and then only by accident. Ed Week then-reporter Bob Rothman received a copy from a friend living in Germany. Ed Week, of course, ran the story on page 1. USA Today was the only media outlet to play off the Ed Week article. Their own page 1 story came complete with a quote from Francie Alexander, then a deputy assistant secretary, dismissing the study. I found out later that Ravitch had instructed her staff at OERI to "disappear" the study--they were instructed to analyze it for flaws. They produced about six inches worth of reports, but carefully let the study stay whole until the Bushmen had left town.

Four years later, Dick Riley re-released the results. I saw the story in the Houston Chronice, but the story carried the by-line of Josh Greenberg, with the Washington bureau of the LA Times. When I got back to DC, I called and asked "Josh, how come you guys were interested in data that is four years old?" He said, "When Riley called us over, we were really suspicious, but then we called around and found that no one knew about this story so it was still news." By that criterion, it still is.

Here's the news: Among kids from 27 countries at one age and 31 at another, American students were second*.

*Technically, the older kids were 8^th, but at both ages only one country, Finland had a significantly higher score. And the actual score of the 14-year-olds was as close to first place as the score of the second place 10-year-olds.