A question that often comes up at EphBlog, and which is related to our seminar, is: How accurately can Williams, or any other elite school, predict college academic performance? I had a back-and-forth e-mail exchange with Jen Doleac ’03 on the topic. See below for details. The punchline is that the academic rating (AR) used by places like Williams (details here) works moderately well in predicting academic performance. The difference between an AR 1 and AR 4 is 0.4 in GPA on average. Given the tight distribution of Williams grades, this is a large difference and probably underestimates the true effect since it does not account for easier courses/majors selected by weaker students.

My initial question to Jen.

The question arose on the blog as to whether or not things like SAT scores and high school grades really predict performance at Williams. I think that they do (at least as well as any measure) and that a graph which showed something like average GPA for each academic rank would make the point clearly.

Am I right? Is there some page in your thesis that addresses this? Could I quote you on the topic? This sort of stuff is covered in places like The Shape of the River, but I would love some Williams specific testimony.

Jen replied:

I don’t have a graph or table showing precisely what you’re looking for, but the regressions on page 23 are close. Each 100 point increase on the SAT correlates with an additional .18-.22 points on the freshman GPA, or .13-.19 points on the cumulative GPA for all 4 years (this is consistent with other studies, which show SATs best predict freshman academic performance, but become less relevant over time). An improvement of 1 point on the AR, similarly, correlates with an improvement of .16 points on the freshman GPA, and .14 points on the cumulative GPA.

However, the total predictive power of the Academic Rating is better than that of SAT scores, because it factors in such things as HS GPA, teacher recommendations, and essays, and — most importantly — judges all of those achievements in the context of opportunities available (i.e., socioeconomic status). Money buys good schools and SAT prep courses, and you have to take that into account when considering someone’s potential. Colleges have been refining this “context” part for many years, so the AR works pretty well these days (though my results do suggest it may still overweight test scores). It is hands-down the best predictor we have of future academic achievement; even so, it explains less than 40% of variation. (In my opinion, GPA isn’t a great measure of achievement, anyway, since students often self-select into courses they’ll do well in. Think the stereotypical “rocks for jocks” type of classes, in addition to simply being able to avoid all math or English courses if they’re not your cup of tea.)

As a sidenote, since I seem to recall this coming up on the site as well, the reason the NAR doesn’t work as well as the AR, in my opinion, is that it doesn’t consider SES/opportunities available when rating extracurricular accomplishments. In my analysis, the predictive power of the NAR frequently depends on the student’s income level. So it’s not that rating non-academic achievements (and predicting future ones) is a futile endeavor, we just haven’t refined our rating system yet. This is becoming more of an issue as extracurricular accomplishments are increasingly being used to break “ties” between the hundreds/thousands of academically-stellar applicants. Not considering the context in which these achievements were earned unfairly benefits high-income applicants.

Did that answer your question? Feel free to quote the above, if it’s helpful. I’m flattered people are interested!

Note that AR means academic rank/rating and NAR means something like everything-else-but-athletics rank. (I think that the initials stand for non-academic rating.) Jen’s thesis demonstrates that the NAR is largely useless. Although it would be nice to be able to predict which students will be campus leaders, serve in student government, be selected for JAs, lead campus organizations and so on, the Admissions Department can’t do it. Note that this isn’t a knock on Admissions! They are smart people doing the best job that they can, the best job that probably can be done. You just can’t predict those things.

I sought clarification on one issue.

Great stuff! I will quote this on the blog soon.

One question, though. I was pretty sure that Williams, like most places, assigned AR in a “straight” fashion, without adjustment for SES and the like but with adjustment for high school opportunities. So, for example, the Socio-ec tag is *not* checked when deciding if someone is an AR 1 or 2. But, after assigning an AR “fairly”, the admissions office will tend to favor one AR 2 over another AR 2 if one is, for example, first generation college.

There is, I think, an adjustment in AR for high school opportunities. So, to be an AR 1 from an high school that only offers two APs, you need to take those AP classes. But someone at Exeter who only takes two APs would never be an AR 1 because there are so many AP opportunities offered.

Or is my understanding off?

Jen confirmed my understanding.

You’re exactly right. I’m using SES and opportunities available interchangeably here, which is admittedly lazy but I think largely consistent with reality, since the two are very highly correlated. When I say SES, I’m not refering to the Socio-ec tag, which isn’t considered until after the AR is assigned.

I hope that this helps. I think that these are important facts to understand in the context of the debate about “access” to elite schools like Williams for poor students. As I have ranted on in the past in the context of Anthony Marx’s crusade at Amherst, every time you accept a “poor” AR 4 applicant in place of an “rich” AR 1 applicant, you are decreasing the average academic achievement at the College, not just the average SATs of the incoming class.

Whether or not that trade-off is worth making is one of our main topics this month. The reality that there is a trade-off is undeniable.

Print  •  Email