Currently browsing posts filed under "Doleac ‘03"

Follow this category via RSS

What Pinker Gets Wrong About Harvard (and Williams) Admissions

Steven Pinker’s essay in The New Republic (hat tip Razib Kahn) provides a false description of admissions at places like Harvard and Williams.

At the admissions end, it’s common knowledge that Harvard selects at most 10 percent (some say 5 percent) of its students on the basis of academic merit. … The rest are selected “holistically,” based also on participation in athletics, the arts, charity, activism, travel, and, we inferred (Not in front of the children!), race, donations, and legacy status (since anything can be hidden behind the holistic fig leaf).

This is not true. Summary: More than 80% of admissions at Harvard (and other elite schools like Williams) is determined by academic merit, measured by past success in high school (high grades in the most rigorous classes with the best teacher recommendations and top standardized test scores), all of which best predicts academic success in college.*

First, leave aside athletics for the moment; the preferences there are real and large.

Second, consider the raw data in terms of 25th and 75th percentile SAT scores. (I have taken the latest available data and simply added the Math and Critical Reading scores together.)

Harvard:  1390 -- 1590
Williams: 1330 -- 1540
Cornell:  1320 -- 1520 

A difference of 50 or 60 points may seem small, but this is (back-of-the-envelope) 1/4 to 1/3 a standard deviation.** If we were talking about height, it would be as if the average student at Harvard were an inch or so taller than the average student at Williams or Cornell. There is no way, in a large population, to get this sort of difference unless the selection procedure has a major focus on SAT scores (or their correlates). In particular, there is no way that the top 25% (!) of the Harvard class has almost perfect SAT scores if only 10% (much less 5%!) is selected on the “basis of academic merit.” It is mathematically impossible.

Third, there are no meaningful preferences given for “the arts, charity, activism, travel” and other non-academic, non-sport reasons. Why?

  • Harvard is not that different from Williams and, as Professor of Music David Kechley explained 11 (!) years ago, there is no meaningful preference given for musical talent.
  • There is no need to give preference for things like music and art because academically strong students are often talented in music and art. Go meet some!
  • There is no reason to give preference for music/arts because schools don’t compete with each other on that basis. Imagine that the quality of the arts and music was twice as good at Williams as at Harvard. Would anyone notice? No! No one goes to enough events at both Williams and Harvard to make that judgment. (This is one aspect by which athletics is different.)
  • Even if you wanted to give preference to those students who would go on to be heavily involved in things like, say, student government and charitable work, there is no way for the admissions department to predict which students will do so, as Jen Doleac ’03 demonstrated in her thesis.
  • Harvard does not have the time or money to meaningfully evaluate the artistic ability of applicants. With 14,000 applicants, the logistics are impossible. As books like The Gatekeepers and A is for Admission make clear, admissions officers make some notes about non-academic talents, but these attributes play a de minimus role in the process.
  • “Travel?” Harvard prefers students who have done a grand tour of Europe? Give me a break! The biggest thing that teenage travel correlates with is family income, and Harvard gets plenty of rich kids already. Might Pinker be able to point to Harvard students who traveled a lot? Sure! But he could also find plenty of blond Harvard students. That fact doesn’t mean that the Admissions Office selects by hair color.

Now, every once in a while does something like music help? Sure! If the orchestra conductor calls up the admissions office and begs for some decent drummers, he may get helped out. But, overwhelming, even those drummers will have amazing academic credentials.

Fourth, even affirmative action does not change the basic story because black (and Hispanic) applicants are accepted under the same criteria as white/Asian students. The same process of looking at high school grades, course schedule, teacher comments and standardized test scores applies to everyone. Whatever it is that Harvard is looking for in white/Asian students, it is looking for the exact same thing in black/Hispanic students. Harvard just sets the bar lower for the latter. Being poor is probably an advantage. Being a non-US citizen is probably a disadvantage. But, whatever bucket you are competing in, the key criteria is academic success.

Fifth, legacy is a red herring. Do the math! There are 1,600 Harvard students in the class of 2018. There were around (I think) 1,600 Harvard students in each class in the 80’s. I can’t find good data on fecundity, but, judging from Williams, elite students from the 80’s go on to, at least, achieve replacement levels of fertility. So, there are 1,600 or so legacy students born in 1995/1996 who would love to come to Harvard (or at least be accepted by Harvard) for the class of 2018. But Harvard only enrolls about 200 of them!*** You think the other 1,400 go to Stanford? Ha! It is easy for Harvard, like Williams, to ensure that enrolled legacy students are academically equivalent to non-legacy students because the legacy pool is so strong. Turns out that Harvard parents tend to have academically talented children. Who knew?

Sixth, even in the case of athletics, academics matter because the admissions department insists. See here for some details. But, to the extent that Pinker has a point, he is correct that athletics plays an important part. And so does major wealth. But even if we combine the athletes and the donors, we are still talking about less than 20% of the class.

Big picture, Pinker’s description of Harvard admissions is fundamentally flawed because the vast majority of it (80%?) is, in fact, driven by “academic merit.” Unless you are a recruited varsity athlete or a billionaire’s child, you got in because your classes/grades/scores were better than the other applicants (at least within your race and/or socioeconomic class and/or nationality).

And this is easy to see if you follow the admissions process at your local high school, assuming it is the sort of school that sends lots of students to elite schools. On average, the high school students who get into Harvard have done better — higher grades in tougher classes with better teaching recommendations and standardized test scores — than the students who get into Williams, and then the same down the academic pecking order.

Steven Pinker is a voice of reason in many of the debates surrounding higher education. It is too bad that he is so misleading about Harvard admissions in this essay.

* Of course, it is not clear what scale Pinker is using for his 5% or what scale we should be using for our 80%. The main clarification that applies to the 80% is that, although the academic evaluation system is the same across categories of students, students are mostly competing against peers in their own racial, citizenship, and socio-economic bucket. If you are, say, rich and black, then Harvard admits use on the basis of academic merit in comparison with other rich/black applicants.

My preferred scale is to imagine that the Harvard admissions system is blinded to everything non-academic. All they see is your high school transcript and standardized test scores. Even in this scenario, more than 50% of the students in Harvard today would still have been accepted. Athletics and affirmative action do have a meaningful impact on admissions, but most of what is going on is still Pinker’s “academic merit.”

** Yes, I realize that this is a rough estimate. The standard deviation of individual SAT tests is around 100. I can’t find good estimates of the standard deviation of combined scores. If the scores from the two tests were uncorrelated, then the combined standard deviation would be around 141. But the positive correlation means that this is a lower bound. And, of course, we are talking about the far right tail of the distribution, where all sorts of weird stuff might happen. The larger point stands: it is impossible for Harvard’s combined SAT scores to be 50+ points higher than Williams/Cornell, year after year, without significant focus on SAT scores by the Admissions Department.

*** See our legacy admissions category for various calculations with regard to Williams. I doubt that things are much different at Harvard or any other elite school. Why would they be?


Predicting Academic Performance

A question that often comes up at EphBlog, and which is related to our seminar, is: How accurately can Williams, or any other elite school, predict college academic performance? I had a back-and-forth e-mail exchange with Jen Doleac ’03 on the topic. See below for details. The punchline is that the academic rating (AR) used by places like Williams (details here) works moderately well in predicting academic performance. The difference between an AR 1 and AR 4 is 0.4 in GPA on average. Given the tight distribution of Williams grades, this is a large difference and probably underestimates the true effect since it does not account for easier courses/majors selected by weaker students.

My initial question to Jen.

The question arose on the blog as to whether or not things like SAT scores and high school grades really predict performance at Williams. I think that they do (at least as well as any measure) and that a graph which showed something like average GPA for each academic rank would make the point clearly.

Am I right? Is there some page in your thesis that addresses this? Could I quote you on the topic? This sort of stuff is covered in places like The Shape of the River, but I would love some Williams specific testimony.

Jen replied:

I don’t have a graph or table showing precisely what you’re looking for, but the regressions on page 23 are close. Each 100 point increase on the SAT correlates with an additional .18-.22 points on the freshman GPA, or .13-.19 points on the cumulative GPA for all 4 years (this is consistent with other studies, which show SATs best predict freshman academic performance, but become less relevant over time). An improvement of 1 point on the AR, similarly, correlates with an improvement of .16 points on the freshman GPA, and .14 points on the cumulative GPA.

However, the total predictive power of the Academic Rating is better than that of SAT scores, because it factors in such things as HS GPA, teacher recommendations, and essays, and — most importantly — judges all of those achievements in the context of opportunities available (i.e., socioeconomic status). Money buys good schools and SAT prep courses, and you have to take that into account when considering someone’s potential. Colleges have been refining this “context” part for many years, so the AR works pretty well these days (though my results do suggest it may still overweight test scores). It is hands-down the best predictor we have of future academic achievement; even so, it explains less than 40% of variation. (In my opinion, GPA isn’t a great measure of achievement, anyway, since students often self-select into courses they’ll do well in. Think the stereotypical “rocks for jocks” type of classes, in addition to simply being able to avoid all math or English courses if they’re not your cup of tea.)

As a sidenote, since I seem to recall this coming up on the site as well, the reason the NAR doesn’t work as well as the AR, in my opinion, is that it doesn’t consider SES/opportunities available when rating extracurricular accomplishments. In my analysis, the predictive power of the NAR frequently depends on the student’s income level. So it’s not that rating non-academic achievements (and predicting future ones) is a futile endeavor, we just haven’t refined our rating system yet. This is becoming more of an issue as extracurricular accomplishments are increasingly being used to break “ties” between the hundreds/thousands of academically-stellar applicants. Not considering the context in which these achievements were earned unfairly benefits high-income applicants.

Did that answer your question? Feel free to quote the above, if it’s helpful. I’m flattered people are interested!

Note that AR means academic rank/rating and NAR means something like everything-else-but-athletics rank. (I think that the initials stand for non-academic rating.) Jen’s thesis demonstrates that the NAR is largely useless. Although it would be nice to be able to predict which students will be campus leaders, serve in student government, be selected for JAs, lead campus organizations and so on, the Admissions Department can’t do it. Note that this isn’t a knock on Admissions! They are smart people doing the best job that they can, the best job that probably can be done. You just can’t predict those things.

I sought clarification on one issue.

Great stuff! I will quote this on the blog soon.

One question, though. I was pretty sure that Williams, like most places, assigned AR in a “straight” fashion, without adjustment for SES and the like but with adjustment for high school opportunities. So, for example, the Socio-ec tag is *not* checked when deciding if someone is an AR 1 or 2. But, after assigning an AR “fairly”, the admissions office will tend to favor one AR 2 over another AR 2 if one is, for example, first generation college.

There is, I think, an adjustment in AR for high school opportunities. So, to be an AR 1 from an high school that only offers two APs, you need to take those AP classes. But someone at Exeter who only takes two APs would never be an AR 1 because there are so many AP opportunities offered.

Or is my understanding off?

Jen confirmed my understanding.

You’re exactly right. I’m using SES and opportunities available interchangeably here, which is admittedly lazy but I think largely consistent with reality, since the two are very highly correlated. When I say SES, I’m not refering to the Socio-ec tag, which isn’t considered until after the AR is assigned.

I hope that this helps. I think that these are important facts to understand in the context of the debate about “access” to elite schools like Williams for poor students. As I have ranted on in the past in the context of Anthony Marx’s crusade at Amherst, every time you accept a “poor” AR 4 applicant in place of an “rich” AR 1 applicant, you are decreasing the average academic achievement at the College, not just the average SATs of the incoming class.

Whether or not that trade-off is worth making is one of our main topics this month. The reality that there is a trade-off is undeniable.


Doleac ’03 Thesis on Academic Achievement

One of EphBlog’s continuing missions is to bring the excellent work done by Williams undergraduates to a broader audience. For now, the focus continues to be on senior theses. Although I hope to convince the College to post all those prior theses related to specifically to Williams, we will do what we can in the meantime. Consider:

Admission criteria as predictors of achievement : a case study of Williams College, by Jennifer Doleac ’03.

Special thanks to Jennifer for providing me with a copy of her thesis. There is a lot of great stuff here, which I hope to be able to comment on at some point. Her thesis lacks a abstract, which is too bad for those looking to get a quick overview. (The Economics Department ought to require all theses to have abstracts, a handy tool for readers and a useful method for forcing students to focus on the specific point that they are trying to make.)

Below are selections from Doleac’s conclusion (pages 96 to 99):

Read more


Currently browsing posts filed under "Doleac ‘03"

Follow this category via RSS