I reckon the readers of Ephblog would like a market based solution to college rankings rather than US News & World Report’s rankings. Why base ranking on arbitrary criteria when you can examine what consumers actually select?

A group of economists (headlined by Harvard’s Caroline Hoxby, whose work is top notch) decided to test out the idea (in a paper that can be found here). They selected 510 high schools that typically send many students to competitive colleges. Guidance counsellors at these schools then randomly selected students to participate in the study. A host of questions were asked, but the most important issue was which schools were they accepted by and which school did they decide to attend?

The schools were then ranked using an algorithm for ranking chess players developed by a Hungarian mathematician named Elo (Jeff Sagarin uses the algorithm in his USA Today rankings of sports teams). Schools that accepted the same student were compared head to head. The school that was selected by the student was deemed the “winner” and the schools that were turned down by the student were deemed the “losers.” In this way, Hoxby and crew were able to come up with rankings (I loaded up the top 50 Download file).

Williams places 18th overall.
Among liberal arts schools, it placed 4th behind Amherst; Wellesley; and Swarthmore.

A couple of quick thoughts:

1) The survey of students is obviously incomplete. Hoxby and crew had limited resources and put together a very interesting dataset with what they had (and the difficulty in execution should not be under-estimated);

2) Self-selection is an issue for the design. Students pick to which schools they apply. Specialty schools (like Caltech and MIT) might receive a boost as a result of this selection process. It isn’t obvious to me where else bias from self-selection might creep into the estimates.

3) Selection on the part of schools is also a slight problem. Harvard accepts a small percentage of people. Presumably a large number of people who applied to Harvard would have accepted had they been accepted. Harvard cheats itself out of “wins” against other schools — not that it needs much help.

4) Oddly enough, Williams did best in Region #4, which contained Minnesota, Nebraska, Missouri, and Kansas. I’d like to think that my promotion back home was the cause, but I haven’t done any, so I guess that can’t be it.

5) In general, big schools were more of a draw than small schools. Given the prestige that comes with brand name schools, it should not be surprising that a boutique school like Williams is snubbed in favor of the Ivies.

6) I was a little surprised to see that my new home institution, Notre Dame, ranked more highly than Williams. Go Irish! Notre Dame would beat Williams at football, too.

7) I wonder if this ranking system sets up perverse incentives for admissions offices. US News and World Report definitely provides incentive to keep yields high, but this ranking system sets up LARGE incentives to not lose out to other schools. In order to game the system, admissions should not admit people who look destined to land at MIT or Caltech.

8) Contrariwise, this ranking punishes early admission because you don’t go head to head with other schools. Williams would like your most enthusiastic applicants to get into other schools and then pick Williams over Princeton and te other Ivies.

9) Markov Chain Monte Carlo is cool. I need to get better at WinBugs so I can do stuff like this.

Print  •  Email