Great Marketing Tool – but That’s About All

Now in their fifth year, U.S News & World Report's annual rankings of online programs get wildly mixed reviews -- favorable assessments from institutions that have ranked programs, but poor marks from most higher education experts.

On the one side, colleges say the U.S. News online rankings are a great marketing tool. “It has raised our visibility in important ways, nationally and locally,” said John Bartle, dean of the College of Public Affairs and Community Service at the University of Nebraska at Omaha. “Our [graduate] criminal justice program is ranked No. 2. We use that in our publicity and on our website. We don’t have a big ad budget, so this helps us.”

Bartle also said Omaha made changes to its undergraduate online programs to better meet the U.S. News criteria, and that "our rankings did go up significantly." But, he added: "It led to good [for students] because we improved the quality of the ... degrees."

Critics, however, say the rankings offer learners little guidance on finding the online program that’s right for them. “[Students] value the rankings way too highly,” said Russ Poulin, director of policy and analysis at the WICHE Cooperative for Educational Technologies (WCET). “Should [the rankings] have an impact? No -- but they do.”

Karen L. Pedersen, chief knowledge officer of the Online Learning Consortium, said students focus on rankings because the information available online programs can be confusing or overwhelming, especially for learners who lack experience with higher education. "Institutions haven't done a good job helping students make decisions," she said. 

More Than 1,300 Ranked

U.S. News's 2017 best online programs rankings, announced in January, include 1,338 bachelor's and graduate degree programs. The total number of rankings of these rapidly expanding distance education programs has almost doubled since the program’s inception in 2012.

According to the Integrated Postsecondary Education Data System (IPEDS), an arm of the U.S. Department of Education, in 2015, 2,354 institutions had at least one online degree program; the total number of programs was 26,844.

"We developed the Best Online Programs rankings to provide students with the tools to identify programs that offer top faculty, positive student engagement and generous support services -- all of which are key to productive online learning," Anita Narayan, managing editor of education at U.S. News, said in a statement.

Over the past five years, U.S. News has tweaked the rankings criteria, which include peer reputation, admissions selectivity, faculty credentials and training, student engagement, and student services and technology. Peer reputation makes up 10 to 25 percent of a degree program’s score. (U.S. News said that the peer reputation percentage varies by category because the institutions’ responses vary.)

Karen Pollack, assistant vice provost for online and blended programs at Penn State World Campus, said the criteria U.S. News employs to rank online programs allow Penn State to "shine" in the categories it's well regarded for, such as faculty credentials or peer reputation, and recognize the institution's investments in student services and technology.

Serves a Purpose

According to U.S. News, the annual rankings help bring attention to little-known programs at regionally accredited public and private institutions as well as at for-profit universities.

“U.S. News exposes students to the breadth of distance learning offerings available, and signals that these programs can have utility,” said Robert Morse, chief data strategist at U.S. News. “Our [adult] audience … has grown by 200 percent in the past five years. There is an interest in learning more about … online degrees.”

Institutions agree that the U.S. News rankings boost awareness.

“There are thousands of institutions that offer online education,” said Pollack of Penn State, which has had online programs listed since 2013. “The rankings can help prospective students sort through the many colleges and universities that offer degree programs online.”

People “are looking for clear and concise information about academic programs and a way to normalize the information across institutions,” said Darlene Smith, executive vice president and provost of the University of Baltimore, whose M.B.A. program is ranked. “They often use multiple resources, one of which is U.S. News. It’s part of the due diligence that I believe is necessary to make an informed decision.”

Peer Reputation: Weighted Too Heavily

Nevertheless, some experts and administrators say the addition of the peer reputation factor a few years has been detrimental, especially for lesser-known institutions. (The reputation elements are the most controversial parts of U.S. News and other college rankings.)

Poulin provides this example: “Suppose that you are a community college in Boston that is not terribly prestigious. Since you are in Boston, there are several institutions within 50 miles, and many of them will complete the reputation score on your institution and you will be ranked.

“Now suppose that you are a community college in Wyoming that has an excellent distance education program…[but] because you are so remote, you don’t receive the necessary reputation scores to be ranked,” he added. “So you are in that twilight zone at the bottom of the list. That’s unfair and is out of the institution’s control.”

Smith of the University of Baltimore, whose online M.B.A. program has been listed since 2012, agreed that “[i]t’s clear that reputation is heavily weighted -- sometimes to the detriment of other more qualitative categories."  She added this can lead to “PR wars” as institutions aggressively promote their brands to impact the rankings.

“Smaller institutions like the University of Baltimore [struggle] to match the reputation-enhancing campaigns of our larger peers,” Smith said. “The great teaching, research and academics that go on [here] can become a bit lost in this process.”

Deb Adair, executive director of Quality Matters, said peer opinion has value, “but there is no level setting with that. The big brands will have an advantage over the small regional schools.”

Nevertheless, Poulin said he credits U.S. News for listening to institutions’ thoughts and concerns about the overall ranking criteria. “I participated in one [session], and they were very eager to hear about the concerns of those in the room,” he said.

Poulin added that he thinks it would be helpful to let students develop their own personalized lists by weighting the criteria that are most important to them.

For the 2017 rankings, U.S. News said it had made minor adjustments to how some factors were calculated based on its internal review and feedback from colleges.

Meaningful Impact?

Besides promotional value, college officials say that the U.S. News listings can help institutions identify best practices, reflect on and enhance their offerings and develop more effective performance metrics to drive student success.

 “We look at our level of commitment and investment … in the categories that U.S. News evaluates, and we ask ‘What can we do better?’ Pollack said.

“For our undergraduate degrees, we did make changes in response to the criteria,” said Bartle, of the University of Nebraska, Omaha. “It led to some good because we improved the quality of our bachelor degrees.”

However, others are dubious.

"Are students well served [by the rankings] -- no,” Adair said. “I don’t believe this is the way to do it. There is a void. People are attracted to at a glance.”

Adair added that U.S. News could make the rankings more meaningful by providing more contextual information about the results and by expanding the data collected. She also said that companies that help students figure out the best online programs for their needs could be more beneficial to learners than the ratings.

“The reputation factor drives me crazy,"Poulin said. It’s hard to imagine that those who complete the survey have more than superficial knowledge of more than a few other colleges. So the ‘reputation’ score is based upon what they know about the other college, which can be tainted by factors such as advertising or having a good basketball team or my cousin’s son went there.”

Smith said that increased emphasis on unique offerings and other factors could provide a more realistic picture of the ranked institutions – and “that could benefit the consumer."

In the end, the rankings are just one resource learners can use when selecting an online program, said Jill Buban, senior director of research and innovation at the Online Learning Consortium.

“It’s important to remind students to take any type of ranking into consideration with their own research on an institution,” she said.

That research, Buban added, could include an extensive review of the institution’s website, its accreditation standing, a conversation with institutional representatives, and, perhaps more important at the graduate level, a conversation with a department-specific adviser or faculty member.