Circular Ratings

New research raises additional questions about the "reputational" survey that is worth 25 percent (more than any other factor) on the U.S. News & World Report rankings of colleges.

What the research found is that the reputational scores don't correlate with changes in factors such as resources or graduation rates, but correlate with the previous year's rankings. In other words, the way you get a good reputational score -- and in turn a good ranking -- is to already have a good ranking.

This finding is potentially significant because the reputational survey is one of the most criticized parts of the U.S. News rankings, and frustrated college officials have long said that they think the survey largely reflects old reputations and in effect rewards colleges for having once been thought of as good, not for actually being particularly good (or improving) in a given year. The new research -- published in the American Journal of Education (University of Chicago Press) -- appears to back up that complaint. (An abstract and ordering information may be found here.)

In the study, two scholars evaluated changes in reputational scores of colleges and then looked for correlations between those changes and other factors that U.S. News declares are important and recalculates each year: graduation and retention, faculty resources, selectivity and financial resources. These factors are of course also controversial with many educators, who say that they reward colleges for being wealthy and rejecting many students. But the theory behind the study was that if these are key measures of quality in the magazine's view, institutions that change in these categories should also experience reputational changes over time. But they didn't -- while the correlation that was clear was reputation with the previous year's rankings. In other words, rankings beget rankings.

The study is by Michael N. Bastedo, an associate professor of education at the Center for the Study of Higher and Postsecondary Education at the University of Michigan, and Nicholas A. Bowman, a postdoctoral research associate in the Center for Social Concerns at the University of Notre Dame.

In an interview, Bastedo said that the results raise serious questions about the validity of the reputational survey. He noted that consumers of rankings tend to mistrust rankings that are "only the aggregation of a bunch of factors" and believe that there are intangible factors that may be best reflected in some measure of reputation. But he said that this idea is based on the belief that reputation isn't just a repetition of everything else in a rankings formula but is actually something new. The correlations the study found, he said, suggest that reputational rankings won't change, even as the quality of colleges does change.

"The problem is that rankings and reputation are becoming the same thing," he said. "The way reputations are being done now is harder and harder to justify.... You want reputation to be a perceptual indicator of something that's not the rankings you just produced."

Robert Morse, who leads the college rankings at U.S. News, said that it was true that reputational scores are "relatively stable from one year to another," but he said that this is no surprise. "Schools themselves say they change slowly, not rapidly."

Morse said that the magazine's research suggests that colleges with the highest reputational scores also have the highest or best data in academic categories, such as graduation and retention rates and admissions data. "U.S. News believes that the peer assessment scores are measuring something valuable and help provide highly useful information about the relative merits of schools."

Bastedo noted in the interview that the problem with reputation primarily measuring past rankings and not anything independent may not be unique to U.S. News rankings. He said that another study (also done with Bowman and currently under review at a journal) looks at an international rankings system of universities and compares reputational responses before and after the rankings were publicized. Once the rankings are known, Bastedo said, reputation follows the rankings, in what he called "an anchoring effect."