In a recent Washington Post, Jay Mathews reports the findings of Paul Peterson and Company on the Washington Scholarship Fund, an organization that provides scholarships to Washington, DC public school students so that they might attend private schools. Ironically, the story carries a large picture not of Paul Peterson, the study's director, but of me, over the caption "Education consultant Gerald W. Bracey said the study should have emphasized the benefits of small class sizes found in private schools." It's true.

The two basic findings about achievement are these:

1. After 1 year, voucher students in grades 2-5 outscored their public school peers by 6 percentile ranks on the ITBS math and 2 ranks in reading. The math score difference is statistically significant, the reading score difference is not.
2. After 1 year, voucher students in grade 6 to 8 outscored their public school peers by 2 percentile ranks on the ITBS math but TRAILED them by 8 percentile ranks in reading. The math score difference is not statistically significant, but the reading score difference is. Voucher students also expressed a variety of negative statements about their new schools, more about this later.

Several questions come quickly to mind:

1. Why does Peterson always launch his findings via a news story rather than at a professional conference? I confess I suspect his motives are not benign. I don't know if the mores of political science differ from those in education and psychology, but I can never remember any story being released prior to an AERA or APA meeting or even at any other conference. The Peterson results will be released at a conference next month. I suspect Peterson vents his "findings" initially into the popular press so that the results will "stick" in the popular culture even if or when they are rejected by scholars. His "study" of the results from Milwaukee--not replicated by anyone--we released as an AP wirestory. The same day, he had an op-ed essay in the Wall Street Journal. This just happened to be the same day as then presidential candidate, Robert Dole, was addressing the Republican National Convention and calling for vouchers. Within a couple of days, Bill Bennett and Lamar Alexander had both hurled themselves onto national TV proclaiming the wonders of vouchers. "How can we deny children choice," said Alexander, now that we KNOW it works?" World famous pedagogue, Rush Limbaugh, sent the good news over the airwaves.

Peterson is also the guy who described himself and voucher advocates as "a small band of Jedi attackers, using their intellectual powers to fight the unified might of Death Star forces led by Darth Vader, whose intellectual capacity has been corrupted by the urge for complete hegemony." Doesn't sound like much of a truth seeker to me. On NPR, one hears a lot of segments that start "In Today's edition of the Journal of the American Medical Association (Nature, NEJM, etc)…" Scholars obtain publicity once their studies are published. Peterson and his colleagues seek the publicity, not the truth.

2. Are the voucher students and the public school students equivalent? Maybe. The vouchers were offered through a lottery conducted by the evaluation team. The voucher applicants were tested in the Spring of 1998 during eligibility verification sessions with their parents/caretakers. Those who were offered scholarships did not differ significantly on demographic variables or test scores from those who were not offered scholarships. The samples are large enough that even small differences would have been significant.

However, only 53% of those offered scholarships exercised their option and used them. This makes one pause: If DC public school children and their parents are desperately seeking alternatives, why did such a small percentage opt in? Some parents decided that they could not afford the difference between the school's tuition and the value of the voucher, but all parents had gone through the application process so one presumes they have some idea of what they will have to cough up to meet the schools' tuition requirements.

So…We can't know about the comparability of those who actually used their scholarships and those who did not. These students could be different both academically and behaviorally. While scholarships were awarded by lottery, admission to schools was not: the students applied to the schools, but the schools exercised discretion over who was admitted. This could produce differences in the groups.

Moreover, of those who were not offered a scholarship, the control group in the study, only 68% attended a regular DC public school.

3. Should we care much about what the results look like after one year? Probably not. Certainly, the study should have been conducted to make sure that horrible things were not happening to the students in the experimental group. Publishing the results of, such a short-term study, on the other hand, is another matter. The "Hawthorne Effect" applies: the newness of a change itself produces other changes.

In addition, the most obvious source of impact from the change comes from the teachers and administrators at the receiving private schools. One can imagine that such teachers might well give the scholarship children special attention. And the administration could be counted on to encourage those attentions: Catholic schools are in economically dire straits all over the nation, and an infusion of money via the scholarship programs would have to be welcomed initially and sought after later.

Haven't Peterson and his colleagues ever heard of a longitudinal study wherein the results are not reported for years? Indeed, one can, should question the wisdom of publishing the data at all, and especially popularizing them in the mass media. Such publication means that the results of the first year study are now part of the experiment and could easily affect the course and outcomes of the experiment. This really seems professionally irresponsible. It really looks like the only reason for the publication is the publication--get the story out there no matter how unreliable the data might be in the long term.

4. If the results are real, are they produced by some aspect of "choice?" It's a guessing game here, but I would guess probably not. The major reason is a datum published, but not commented on by the report, namely that public schools were 50% larger than the private schools (438 students vs. 217) and that public school classes had more students in them, 22 on average, than did the private schools, 18. The difference between 22 and 18 is itself sufficient to make substantial differences between two otherwise equivalent groups.

   The failure to comment on the difference is class size, given the other research on class size research, is another indication that the researchers are not really a disinterested, objective group of scientists.

5. If the results are real, the probably have less educational significance than choice advocates would like to contend. At the earlier grades, the gains for math were significant, the gains for reading were not. This was true in Milwaukee as well. In another study, I noticed that in most cities, the mathematics scores on tests in the elementary grades are higher than those for reading. Why? My guess would be that in the elementary grades, test scores in math are susceptible to drill-and-practice effects. To score well on a reading test, though, you have to…read. I have seen some evidence that the advantage of mathematics fades at the later grades as the tests cover more difficult material that cannot be learned by rote.

As noted, the reading scores for older voucher students were 8 percentile ranks below those of public school students. In addition, there were large differences in student responses to "I like my school a lot", "students are proud to attend my school," and "I would give my school the grade "A", all favoring public schools. The difference on the last question was especially large. Only 8% of the voucher students awarded their private school and "A", while 48% of public school students gave out "A's." In their discussion, the researchers cite Maria Montessori as claiming that early adolescence is not a good time to move kids around in school.

Parental satisfaction with the private schools was high and there are no doubt some good reasons for this. There is probably some good old Festingerian "cognitive dissonance," too. Festinger would contend that the voucher parents, having to fork out a considerable sum of their own money would say to themselves, "I'm paying a lot for this so it has to be good."

In the reality arena, more public school parents than voucher parents reported tardiness, fighting, truancy, cheating and destruction of property as serious problems at school. We might also expect that mothers who are spending a lot of their not-very-disposable income on tuition are likely to try and make it work (over 75% of the students in both groups lived with mother only; voucher parents were somewhat more educated and had slightly high incomes, $17,774 vs. $15,781).

This research does have one substantial advantage over other studies: because the evaluation was designed in advance, plans could be made to insure that there are data for most parents and students. Participation in data collection was a condition of being part of the program. The spring eligibility verification meetings allowed testing of the children. They also permitted not only the collection of questionnaires from the caretakers, but, since filling these out took less time than the tests, the convocation of focus groups to talk about various issues.

Other studies of voucher programs have reported high attrition rates. Eighty percent of the voucher moms say they plan to send their children to the same school next year. It will be interesting to see what the data actually reveal. Too bad the researchers didn't wait until such data were actually available rather than engaging in their rush to judgment.