'U.S. News' Makes Modest Tweak in Methodology

U.S. News & World Report is today releasing its rankings of colleges and universities -- largely unchanged at the top.

And the magazine is also releasing very modest changes in the methodology that favor colleges where most students submitted standardized test scores -- even though the magazine said in June 2020: "U.S. News believes now is the right time to end the use of standardized tests in admissions decisions as a requirement for inclusion in the rankings."

Others that rank colleges have also faced a challenge of doing rankings amid a pandemic. Forbes changed its methodology such that a state university was the winner, for the first time ever. And The Princeton Review abandoned (temporarily) rankings based on surveys of students.

U.S. News

U.S. News is the giant of college rankings in the United States, despite widespread criticism of its methodology, with critics saying it favors wealth and criticizing its reliance on a survey of college administrators in its methodology. And U.S. News has also been criticized for its use of standardized test scores in its formula.

The overall rankings categories in the U.S. News rankings are the same this year as last. Many have been waiting to see how U.S. News would change its consideration of standardized testing, given that a majority of colleges for the first time were test optional or test blind in admissions. (U.S. News rankings are based on the prior year's statistics, so last year's rankings were based on the year prior to the pandemic.)

Standardized tests count for 5 percent of the total score, the same as before. But U.S. News did change its methodology, slightly.

The magazine said: "A change for the 2022 edition -- if the combined percentage of the fall 2020 entering class submitting test scores was less than 50 percent of all new entrants, its combined SAT/ACT percentile distribution value used in the rankings was discounted by 15 percent. In previous editions, the threshold was 75 percent of new entrants. The change was made to reflect the growth of test-optional policies through the 2019 calendar year and the fact that the coronavirus impacted the fall 2020 admission process at many schools."

An important note on that change is that it probably protected the most competitive colleges. The Common Application issued a report last week that said 43 percent of students submitted test scores this year, but "more selective member institutions" received more applications with test scores.

U.S. News also released its policy for test-blind colleges, which won't even look at SAT or ACT scores. While this group of colleges is small, it included the University of California system. "U.S. News again ranks 'test blind' schools, for which data on SAT and ACT scores were not available, by assigning them a rankings value equal to the lowest test score in their rankings. These schools differ from ones with test-optional or test-flexible admissions for which SAT and ACT scores were available and were always rank eligible."

There wasn't much of an impact on the top colleges in the rankings. The top college in the county is unchanged as are the top five (although there is some movement among the top five). The top five liberal arts colleges are the same as they were last year. Among public universities, there was a change in the top five: The number five university is different (it was tied for sixth last year).

The reaction to U.S. News making only minor changes in its formula from critics of the use of test scores was anger.

Angel B. Pérez, chief executive officer of the National Association for College Admission Counseling, noted that NACAC has asked U.S. News four times (over two decades) to remove test scores from its methodology, "given that the scores are input variables, and as such do not contribute to an understanding of how effective the institution is at educating its students."

He said the magazine's practice of changing the test scores of institutions (when they are test optional or test blind) "is an added distortion to an already distorted and flawed methodology."

Stephen Burd, a senior writer and editor with the education policy program at New America, a liberal research organization, noted that New America (along with NACAC and other groups), in July asked U.S. News to stop using the test scores.

He said via email of the new rankings: "We don't particularly care how the methodology affects an individual college's ranking. The key thing for us is how the rankings affect college behavior. Some colleges appear to be making their decision about whether or not to remain test optional or test blind based on how U.S. News treats these schools in the rankings. The fact that U.S. News, a private commercial entity, has that much influence is incredibly disturbing."

And Robert Schaeffer, executive director of the FairTest: National Center for Fair and Open Testing, said U.S. News's explanation of the changes "is misleading if not dishonest."

He said that the headline on the section of the news release reads: "Full credit for schools reporting fewer SAT/ACT takers."

"That statement is not true," Schaeffer said. "Full credit is granted only if a college or university reports ACT/SAT results from at least half of its enrollees. Otherwise, the school is punished by an unjustified 15 percent penalty in the test score component of its ratings."

He added: "This is further evidence that U.S. News rankings are bogus. Much of the underlying data is false or out of date; the rankings formula is arbitrary and changes from year-to-year, and the overall exercise still over-emphasizes input variables related to family affluence rather than measures of educational quality and equity. Garbage in, garbage out."

However, Robert Morse, the longstanding rankings czar at U.S. News, said: "These tests are used in the rankings because they measure in a standardized way schools' ability to attract students who can handle rigorous coursework. For the 2020 entering class, which was prior to many admission policy changes caused by COVID, standardized tests were required for admission at a large percent of U.S. schools and served as key criteria for whether they were admitted."

He also denied that U.S. News is doing anything harmful to test-optional colleges. "The methodology does not punish test-optional schools," he said. "In general, schools that reported on fewer test takers for the 2022 rankings edition than the 2021 edition were not positively or negatively impacted as a result."

Forbes

Forbes also changed its methodology this year and, as a result of the change, for the first time named a state university -- the University of California, Berkeley -- as the country's top college.

In the past, the honor has gone to a private college that is highly competitive in admissions or (one year) a service academy.

Forbes was quite proud of its selection, writing: "Forbes has ranked a public school, the University of California at Berkeley, in the No. 1 spot. Harvard, unseated from first place, is a distant No. 7. There are five other publics in the top 25 including three U.C.'s, the University of Michigan and the University of Florida."

The methodology change was developed with Michael Itzkowitz, a higher education consultant who developed the Education Department's College Scorecard in the Obama administration.

With the changes, Forbes rankings were based on:

• Alumni salaries: 20 percent.
• Student debt: 15 percent.
• Return on investment, using a formula created by Third Way: 15 percent.
• Graduation rate; starting with the federal rate (which measures only first-time, full-time students), adding in part-time and transfer students, with a bonus for colleges that enroll and graduate lots of students eligible for Pell Grants for needy students: 15 percent.
• Forbes's "American Leaders List." The list includes alumni who have a Forbes 30 Under 30, Forbes 400, Richest Self-Made Women and Most Powerful Women lists. They also included individuals in public service, including members of the Presidential cabinet, Supreme Court, Congress and sitting governors. Finally, the list included winners of the MacArthur Fellowship, Nobel Prize, Breakthrough Prize, Lasker Prize, Fields Prize, Academy Awards, Oscars, Tony's, NAACP Awards, Guggenheim Fellowship, major sport all-stars, Presidential Medals and Pulitzer Prizes: 15 percent.
• Retention rates, the percentage of students who choose to stay after their freshman year: 10 percent.
• Academic success, defined as the number of graduates of each college who have gone on to win Fulbright, Truman, Goldwater, Rhodes, Gates and Cambridge scholarships over the last four years and data from the federal government's National Center for Science and Engineering Statistics on the total number of undergraduates who went on to earn Ph.D.s over the last three years: 10 percent.

The main changes from the previous year were adding the return on investment and scrapping a student satisfaction category. Those changes boosted not only Berkeley, but the public universities that were added to the top 25 colleges.

The Princeton Review

Each year when The Princeton Review releases its rankings of colleges, it captures the most attention with its ranking of the top "party school." The ranking comes from student surveys on the popularity of fraternities and sororities and alcohol and drugs on their campus and the number of hours the students report they study each day (outside class time). This year, there were many cases where campuses were closed (except for online education), and conducting regular surveys was impossible. So The Princeton Review didn't attempt to organize a ranking of party schools. But Robert Franek, editor in chief, said the lists should return next year.

The Princeton Review did include rankings for "great" things on campus: career services, classroom experience, college library, etc. But instead of relying on one year's survey, it lists all the colleges that have made each list many times.