Potemkin Rankings

As the U.S. News & World Report rankings have faced something of a rebellion from colleges in the last year, the magazine's editors have increasingly framed their work as part of the accountability movement. You can see that line of defense in the blog of Robert Morse, who oversees the rankings, in a recent post in which he noted that the magazine and Education Secretary Margaret Spellings share common goals of keeping students informed about colleges' quality and cost. Morse noted with apparent pride that Spellings had recently cited the magazine's popularity as a sign of public hunger for information about quality and costs.

Accountability is of course quite the buzz word these days. But what if the rankings have the opposite effect on key factors -- what if they punish colleges for being, for example, less expensive than the institutions they want to emulate?

That's a question raised by Tori Haring-Smith, the president of Washington & Jefferson College, a liberal arts institution south of Pittsburgh. She is not a believer in the rankings -- and has signed the stronger of the two letters issued by presidents to oppose them -- joining the Education Conservancy's letter in which she pledged not to participate in the "reputational" survey of presidents in which they judge other institutions, or to use the rankings in promotional materials. While Haring-Smith said she has never taken the rankings that seriously, she said that they do seem to matter to certain groups of people, among them alumni and parents of prospective students. Generally, she said, "the further people are away from the college," the more they seem to use the rankings.

So after getting some questions about the college's rankings, in light of progress at the college on key measures covered by the rankings, Haring-Smith dug up a lot of numbers about Washington & Jefferson, which she shared with Inside Higher Ed. She argues that they show that even if you improve in key areas U.S. News values (like selectivity), you can fall in the rankings -- unless you start to spend a lot more money. The issue of institutional wealth is not a new one in the rankings debate. But it is typically used in regard to the institutions at the very top -- which almost always are among those with the largest endowments in the United States and those that have longstanding reputations for excellence.

The argument Haring-Smith makes is different and may challenge the claims of the magazine that it's just giving parents and students what they want. Do they really want an institution that charges more, she asks? If you serve more students with roughly the same endowment, is that a bad thing? Do they really want to judge an institution by how it used to be, not how it is now? (Not surprisingly, Morse of U.S. News has a different take on the numbers -- more on that later.)

So why does Haring-Smith think the rankings don't work? Washington & Jefferson has done a number of things one would think would be rewarded. Over the last four years, applications increased to 7,450 from 4,480, as part of a plan to increase enrollment by 50 percent, to 1,500 students. The quality of applicants went up -- 30 points higher on average on the SAT, and the percentage of students in the top 25 percent of their high school class went to 79 percent, from 58 percent. The college has been able to grow without changing its 12:1 student-faculty ratio, and the percentage of classes with fewer than 20 students increased to 69 percent from 61 percent. The college has built 16 buildings, added academic programs, and so forth. Twenty-eight percent of students are the first generation in their families to go to college.

All of those stats would seem to play into the U.S. News view of the world, which rewards selectivity over, say, training the greatest number of school teachers. And, in fact, in some categories of its rankings, Washington & Jefferson went up -- to 120 from 135 on selectivity, reflecting its acceptance rate dropping to 36 percent from 51 percent.

"Peer assessment" -- the much-criticized category where college presidents just express their feelings about other institutions on a five-point scale -- didn't change at all while the college was undergoing these changes. It was 2.8 at the beginning, 2.8 at the end, and 2.8 every year in between.

What went down dramatically was the college's results for "financial resources." During the years in question, Washington & Jefferson's endowment increased, but it ended up just over $100 million -- a fraction of what top-ranked liberal arts colleges have and not proportionately greater on a per-student basis than before the college increased enrollment by half. Tuition has gone up to, topping $29,000, but considerably less than the Pennsylvania private colleges in the $34,000-$36,000 group. As a result, Washington & Jefferson's ranking on "financial resources" fell to 110th place among liberal arts colleges from 78 in the last five years. (The college's overall rank fell to 106 from 91.)

Had the college raised tuition more, it might have benefited in two ways: more spending on educational resources that count for the magazine and -- if money gets spent on merit aid -- buying more high-SAT students to further raise competitiveness and SAT averages.

"We made moves that made us financially more efficient," Haring-Smith said. "In this era of concerns about rising college costs, you would think that would be a good thing, provided you don't hurt your programs. But it seems to me that the philosophy is: 'The more money you have, the better program you have.' This encourages people to spend money and raise tuition. They reward you if your tuition goes up."

Likewise, she said, the fact that the reputation ranking was static amid all the change at the college shows how stagnant most presidents are in evaluating other colleges. Haring-Smith said there's no reason to think that presidents of liberal arts colleges nationally would know about her institution's changes (especially because the college, unlike others, doesn't send out brochures boasting of them to fellow presidents before rankings season). They are busy managing their own institutions, not reading up on her enrollment changes. But as a result, she said, presidents are making a guess, based on what they have heard years ago.

Morse, of U.S. News, has a different analysis, although on a few points, he acknowledged the validity of the criticism.

First, he noted that the impact of increasing tuition more than the college did was hypothetical and might have had other impacts. So he said it wasn't fair to say that colleges are rewarded for being more expensive. But he said it was correct that if the college spent more on certain educational items and recruited more competitive students, its rankings might well have been higher. He noted that in one category -- graduation rates -- the college went down over the period in question, to 68 percent from 73 percent. And while the college still "overperforms" based on what the magazine projects its graduation rate should be, the overperformance isn't as much as it used to be.

On the question of presidents grading other institutions on a 1-5 scale, he said: "I think the reputation can be a weakness -- it's not a fast-moving indicator. Maybe it's lagging. It certainly is stable and it may not reflect changes to a school -- either that has quickly deteriorated or transformed itself in a positive way."

So why have the largest part of the formula, on a survey that boasts new results every year, be something that doesn't reflect the changes going on from year to year? "The reputation does serve its role. It's a rating by a school's peers," he said.

Finally, Morse noted, some of the ground Washington & Jefferson lost was because different colleges ended up in the liberal arts category this year, including the military academies at West Point and Annapolis.

Of course that reinforces another point Haring-Smith made: The unique circumstances of any college mean that real comparisons to others (even in a group with similar qualities like liberal arts colleges) can quickly get silly. One college is undertaking a major enrollment boom, another isn't; one college is wealthy, another isn't; one college is a military academy and one isn't.

"One of the things rankings do," she said, "is to compare not even apples and oranges, but apples and cucumbers. I find that problematic."