A regular reader sent us (pdf) these details behind this year’s US News rankings. Let’s spend five days discussing them. Today is Day 1.

1) The most important data point here is the huge gap between Williams and Amherst. (Thanks to our regular reader for pointing this out.) Recall the methodology:

To arrive at a school’s rank, U.S. News first calculated the weighted sum of its standardized scores. The final scores were rescaled so that the top school in each category received a value of 100, and the other schools’ weighted scores were calculated as a proportion of that top score. Final scores were rounded to the nearest whole number and ranked in descending order.

Exercise for the reader: Assume that by “standardized,” US News means that they take the mean and subtract the standard deviation, leading to sub-scores that are N(0, 1). How much does Williams have to be leading the other schools in various categories for it to have a 5 point lead in the overall ranking based on 100?

2) Note how well Williams does in the Peer Assessment and High School Guidance Counselor rankings. Note the circularity that this can generate. Williams has been ranked #1 by US News for 14 years. What sort of high school guidance counselors are likely to fill out a random questionnaire from US News? The sorts that care about the US News rankings. What sorts of schools are they likely to rank high? Schools that they have read about before in US News! Williams could, in truth, become a horrible school tomorrow and, for years, these counselors would rank it highly.

3) For giggles, not this part of the methodology:

To reduce the impact of strategic voting by respondents, U.S. News eliminated the two highest and two lowest scores each school received before calculating the average score.

Watt said that Clemson officials, in filling out the reputational survey form for presidents, rate “all programs other than Clemson below average,” to make the university look better. “And I’m confident my president is not the only one who does that,” Watt said.

If such strategic voting is widespread, it is not clear if eliminating just four outlier scores will be enough to fight it.