There is a shocking amount of ignorance in the Williams community about the effect of SAT prep classes (Kaplan, Princeton Review and so on). Many Ephs seem to believe (incorrectly) that taking a prep class has a significant effect on one’s scores and that, since rich families can more easily afford such classes, there is a bias in the Williams admissions process. Consider EphBlog’s own president, Derek Catsam ’93.

Asserting that SAT scores reflect intelligence is stupid. All things being equal, it would be stupid. But in a world where SAT prep classes have proliferated, and in a world in which the affluent have dozens of more options for elite test prep classes and one-on-one tutoring that will improve student scores well beyond any gap between them and these allegedly lesser poor students, to argue that SAT sores are a reflection of intelligence represents an exponential leap in stupidity.

Here (pdf) is a thorough survey of the academic literature. Summary:

The existing academic research base indicates that, on average, test preparation efforts yield a positive but small effect on standardized admission test scores. Contrary to the claims made by many test preparation providers of large increases of 100 points or more on the SAT, research suggests that average gains are more in the neighborhood of 30 points. Although extensive, the academic research base does have limitations. Most notably, few published studies have been conducted on students taking admission tests since 2000. Only two studies have been published on the effects for ACT scores, and no studies have been published since the 2005 change to the SAT, which added the Writing section among other changes. In addition, many previous studies were conducted on small samples or had other methodological flaws. Additional large-scale studies of test preparation—including both the ACT and SAT and examining a variety of test preparation methods—will be important to understanding more about the relative value of different types of test preparation. However, even with these caveats in mind, students and families would be wise to consider whether the cost of a given test preparation option is worth what is likely to be a small gain in test scores.

And note that the organization behind this report, National Association for College Admission Counseling, is, if anything, probably (?) biased against finding this result, i.e. more likely to believe if the efficacy of coaching. Note also that the 30 point effect is for math and critical reading combined.

I also suspect that this is an overestimate — read the whole report for details — because it (mostly) compares paid SAT prep with doing nothing when the fair comparison, when considering the plight of poor students, is look at X hours spend in prep class versus X hours spent studying on one’s own, reviewing old tests and so on. In that context, the overall effect is, from the point of view of Williams admissions, indistinguishable from zero.

In other words, even if the federal government gave every 17 year old in the country a $5,000 voucher for use in SAT prep classes/tutoring, the distribution of SAT scores would be largely unaffected.

More details:

Nonetheless, over the past 10 years evidence has emerged from three large-scale evaluations of coaching that point to a consensus position about its average effects on admission exams. This consensus is as follows:
• Coaching has a positive effect on SAT performance, but the magnitude of the effect is small.
• The effect of coaching is larger on the math section of the exam (10–20 points) than it is for the critical reading section (5–10 points).
• There is mixed evidence with respect to the effect of coaching on ACT performance. Only two studies have been conducted. The most recent evidence indicates that only private tutoring has a small effect of .4 points on the math section of the exam.

Read the whole report.

Print  •  Email