Refusing to Rank

The ever controversial rankings from U.S. News & World Report arrive today -- amid some signs that the protests against them may be gaining ground. The magazine's editors acknowledged Thursday that the proportion of presidents participating in the "reputational survey" -- in which they rank similar institutions -- had fallen to 51 percent this year, from 58 percent a year ago. Only a few years earlier, participation was as high as 67 percent.

While participation in the reputational survey -- which receives more weight in the rankings than any other factor -- is falling, U.S. News took a step to respond to a common criticism of the rankings, which is that they favor colleges that educate wealthy, well prepared students. For the first time, U.S. News included calculations based in part on the percentage of students who are Pell Grant recipients. But the formula is being used in such a way that the impact of the Pell figures appears to be small.

In addition, the magazine for the first time singled out competitive colleges that do not use the SAT or ACT for unfavorable treatment:Institutions that don't look at such information (there is only one such competitive college right now, Sarah Lawrence College; most of those that have ended SAT requirements still will look at scores applicants voluntarily submit) will now be bumped out of the regular rankings to an "unranked" category. So Sarah Lawrence, which was No. 45 among liberal arts colleges last year, between Centre and Rhodes Colleges, is now in the rankings purgatory for colleges that can't be classified because they didn't turn in their forms or are too small or don't have enough data.

The arrival of the new rankings from U.S. News (and many others) every August always prompts a flurry of boastful press releases from colleges that are happy with their placement -- and complaints from those who have gripes over methodology. The rankings have long been criticized by educators (including some of the same people who have issued press releases when their colleges do well) and have long been popular with prospective college students and their families. This year's rankings arrive, however, at a time that the griping has turned into something of a movement, with dozens of presidents having taken a public pledge not to fill out the reputational survey or to use the rankings in promotional material.

Brian Kelly, the top editor at U.S. News, played down the drop in participation. "We don't believe there was any significant effect," he said. "We don't know exactly why" there was a drop, Kelly said, but he added that many presidents reported being particularly busy when the surveys were due, which was around the time of the Virginia Tech killings.

Robert Morse, who heads the rankings project at the magazine, said that while "it's certainly a drop," he felt confident that enough people were still participating to make the survey valid. He declined, however, to say what level was necessary to sustain the survey's reliability. He referred to a spokeswoman questions about whether the dip was even greater in some sectors of higher education (the presidents are asked only to rank similar institutions, so liberal arts presidents are ranking liberal arts colleges, and so forth). But the spokeswoman said it was the magazine's policy not to release such data.

The reputational survey has been a target of rankings critics because they view it as particularly unscientific, vulnerable to manipulation (some colleges send out brochures to "remind" presidents of how wonderful their institutions are), and susceptible to being out of date (many presidents say they worry that they are going on reputations that may be decades old). The survey -- dubbed the "beauty pageant" by many presidents -- is important in the rankings, accounting for 25 percent of the total score for the major institutional rankings, more than any other factor.

Even the 51 percent participation rate may overstate how many presidents are ranking colleges. U.S. News tells presidents to skip institutions about which they don't know much, and in the past the magazine has found that presidents on average rank 56 percent of the institutions in their sector.

Lloyd Thacker, founder of the Education Conservancy, a group dedicated to reforming college admissions, which is coordinating efforts to take on the rankings, said he was pleased to hear that more presidents aren't participating. "I think it's a hopeful sign that college presidents are beginning to behave according to their educational sensibilities," Thacker said. He predicted that as more presidents see that it's possible to move away from rankings, more will follow.

The overwhelming majority of colleges -- even many where presidents have vowed not to fill out the reputational survey -- do provide U.S. News with the data that the magazine uses. Only a few colleges over the years -- most notably Reed College -- have adopted a non-cooperation policy, forcing U.S. News to gather data from other sources (such as college Web sites).

On Thursday, Columbia College Chicago announced that it would be joining such full-boycott institutions. The college released a letter from its president, Warrick L. Carter, in which he said: "The ratings -- with their emphasis upon selectivity and exclusion based upon standardized test scores -- do not promote the kind of diversity that fosters creative expression and intellectual engagement. Neither do the ratings measure the inventiveness or ingenuity necessary for future creative achievement."

SAT Scores and Fairness

The emphasis on SAT or ACT scores in the rankings, noted by Carter, has long been controversial -- and U.S. News faced a particular challenge this year from Sarah Lawrence College. Most colleges that have ended SAT requirements have continued to collect scores from applicants who wish to submit them, and so have had average figures to submit to the magazine. In fact, many admissions officers at institutions that have ended SAT requirements believe that they are helped in the rankings because their SAT averages tend to rise (those with high scores submit and those with low scores don't) and their application total goes up. Sarah Lawrence appears to be the only competitive college that not only ended the SAT requirement, but said that it wouldn't accept scores submitted.

Sarah Lawrence's then-president, Michele Tolela Myers, charged in an op-ed in The Washington Post in March that U.S. News responded to her college's policy by saying that the magazine would just assume that the average SAT would have been one standard deviation (about 200 points) below the average of Sarah’s Lawrence’s peers. “In other words, in the absence of real data, they will make up a number,” wrote Myers. U.S. News officials said at the time that no final decision had been made, but the Myers article galvanized many rankings critics.

In the end, U.S. News did not make any assumptions about Sarah Lawrence, but simply moved it to the "unranked" colleges. Kelly, the editor, said that after weighing all the options, that was the best approach because "we realized that they were not providing one of the crucial industry standards, which is the SAT scores. If you don't have that information, it's not fair to the other schools that are producing that information."

Kelly has repeatedly said that the magazine does not try to dictate colleges' educational decisions. Asked if he wasn't punishing Sarah Lawrence (which lost a much coveted place in the top 50 liberal arts colleges), or taking a stand on the debate over whether the SAT has validity, Kelly said No. "This is a hugely significant factor that they do not have," he said. "In the case of the SAT, it is such an accepted industry standard. We are not creating this measure. We are reflecting what is the industry standard."

Thacker, of the Education Conservancy, said that fairness wasn't the real motivating factor for the magazine's decision. "It just bothers me that they think they are value neutral," he said. "Their value is to sell magazines to Americans who like lists. We all know that the way the SAT is used is way out of proportion to its educational relevance," he said, and U.S. News is adopting a policy that will discourage other colleges from taking a principled stand similar to that of Sarah Lawrence.

U.S. News "is not in a position to assess the educational relevance" of Sarah Lawrence's testing policy, Thacker said.

Sarah Lawrence released a statement saying that all of its evidence shows that the students who have been admitted since the college stopped using admissions tests have been as strong as the previous classes of students. Sarah Lawrence has said that it plans to post more information about itself on its Web site and to participate in new projects being created by college organizations to provide more information to prospective students.

As for being "unranked," the college said that "other than not fitting neatly into U.S. News’ template, Sarah Lawrence has little in common with most of the other schools in that category."

Adding a Pell Grant Factor

In a move that has been requested by some educators for years (although in a more significant way), U.S. News this year is for the first time including consideration of what percentage of undergraduates are Pell Grant recipients. Pell data are considered by many to be a good proxy for low-income students. Many factors in the rankings, such as faculty resources, favor wealthier institutions, and many college officials have said that the rankings in effect discourage colleges from enrolling low-income students. Because U.S. News rewards colleges with high graduation rates and highly selective admissions, the argument goes, colleges that admit many low-income students (who are likely on average to have lower SAT scores and graduation rates) are set back by the magazine.

Pell Grants are being added to a very small category of the rankings, graduation rate performance, which is worth 5 percent of an institution's total score. Graduation rate performance seeks to reward colleges that do better than expected on graduation rates. A regression analysis is run in which a college's "expected" graduation rate is determined based on such factors as students' high school rank and standardized test scores, college spending per student and public/private status. Generally, colleges that admit only well prepared students and that are wealthy are expected to have higher graduation rates. So institutions that exceed their "expected" rate get extra points. What's new this year is that the enrollment of Pell Grant students is part of the regression analysis, so a college with many such students would have a slightly lower expected rate, and thus would do slightly better overall.

Kelly said that the magazine was concerned by the idea that it had created a disincentive to enroll low-income students and that "we're trying to factor that out." He added, however, that "the philosophy of this is not to create an incentive to bring in low-income kids, but not to create a disincentive."

In the short term, he said, the change "may not have a significant effect," but he said that the magazine would track the impact and might change the relative weight assigned in the future. He said that it was important "to proceed cautiously," and that "we're not willy-nilly going to throw new metrics in there." Morse said that the impact would probably be greatest for public flagship universities, which tend to enroll far more Pell students than do elite privates.

A look at the rankings suggests that the impact has been minimal at the top levels. The usual suspects occupy the top slots and they generally aren't the institutions with top performance on enrolling low-income students. Princeton University, which is at the bottom of the Ivy League in percentage of undergraduates with Pell Grants (7 percent), leads the "national research university" category, as it did last year. Ivies with double Princeton's percentage of Pell students -- Columbia University (15 percent) and Cornell University (14 percent) -- didn't budge from their positions in the rankings (No. 9 and No. 12, respectively) as a result of a factor that should have helped them.

Among the liberal arts colleges to which U.S. News gives its highest marks, Smith College leads the nation in Pell students, with 26 percent. But it is in a three-way tie for No. 17 on the rankings of liberal arts college. Perpetual Nos. 1 and 2 Amherst and Williams Colleges have half of the percentage of Pell enrollments.

The colleges that come out on top of U.S. News generally are among the wealthiest in the nation, which is no surprise given the way the formula values resources and raw totals. The Pell Grant data influences in some small way a category worth 5 percent, while raw data about graduation and retention rates are worth 20 percent.

Kelly said that there is nothing wrong with having factors that favor wealthy institutions. "Wealth matters," he said. "In American society, wealth is a crucial factor. Richer schools have more," he said. Kelly stressed that by "more," he meant educational resources, not "two extra salad bars in the cafeteria."

But Thacker of the Education Conservancy said that while it was a good thing to include Pell Grants, the fact that it was such a small factor was worth noting. Thacker also said that the rankings' continued emphasis on selectivity meant that colleges that recruit and admit many low-income, minority students -- many of whom thrive despite being admitted with test scores that are, on average, lower than those of wealthy, white applicants -- will end up behind in the rankings.

The entire rankings structure, Thacker said, encourages colleges to focus on themselves, not "their public interest mission," which would be served by recruiting such students.

It's no surprise, Thacker said, that the addition of Pell Grants didn't lead to any meaningful change in the top of the rankings. U.S. News doesn't want to see a college that does a great job of educating low-income students topple Harvard, Yale and Princeton, he said. "They've got a lot at stake in keeping the rankings the way they are. They are not going to do anything that is going to interrupt to pecking order."